The UK Independence Party has gained its first elected MP, with Douglas Carswell taking the seat of Clacton by 12,404 votes.
Mr Carswell, who defected from the Conservatives, knocked his old party - which enjoyed a 12,068 majority at the 2010 election - into second place.
UKIP leader Nigel Farage said he had "shaken up British politics".
In the night's other by-election, Labour held on to Heywood and Middleton but UKIP slashed its majority to 617.
Mr Farage told BBC Breakfast UKIP could hold the balance of power after next year's general election.
He said UKIP's second place in Heywood and Middleton was "even more significant" than its win in Clacton, saying the party was now the main opposition to Labour in northern cities.
"Something big is happening here. People want change, they have had enough of career politicians in three parties."
Mr Farage said he expected more Conservative MPs to join UKIP following the defection of Mr Carswell and Mark Reckless, who has triggered a by-election in Rochester and Strood, Kent.
"I think it would be very surprising if more people did not come across", he said, saying he had also spoken to Labour MPs "frustrated that they are not able to change things in British politics".
Conservative Party Chairman Grant Shapps told BBC Breakfast the result in Clacton was a "wake-up call" saying that while UKIP were a threat to all the Westminster parties, they "cost Conservatives seats", so put Labour leader Ed Miliband "one step closer to Downing Street".
Prime Minister David Cameron said: "If you see a big UKIP vote you end up with Ed Miliband as prime minister, Ed Balls as chancellor, Labour in power."
BBC political editor Nick Robinson said the Conservative strategy of saying a UKIP vote helped Labour was "the only way they think they can get those voters back".
The UKIP vote could still "slump back" at the general election, or it could "carry on breaking every prediction and really make a breakthrough", he added.
Speaking in Heywood, where he congratulated winning candidate Liz McInnes, Mr Miliband said UKIP voters did not think political parties listened to them or that the country represented them.
He said Labour had changed and realised it was "not prejudiced" to worry about immigration, but said he did not think UKIP could "represent the interests of working people".
The next election would be a fight against "disillusionment and despair", he predicted.
Speaking on a walkabout with Mr Farage in Clacton on Friday morning, Mr Carswell, who gained 21,113 votes - 60% of the total - said he did not want to be UKIP leader. and said his old party had "smeared" Mr Reckless since his defection.
In his acceptance speech he told UKIP supporters there was "nothing that we cannot achieve".
Addressing the people of his constituency, he said: "I resigned from parliament to face this election because I answer first, foremost and last to you. You are my boss. I will not let you down.
"To my new party I offer these thoughts: humility when we win, modesty when we are proved right. If we speak with passion, let it always be tempered by compassion."
He also addressed the party's prospects on a wider front, saying "we must be a party for all Britain and all Britons: first and second generation as much as every other".
"Our strength must lie in our breadth. If we stay true to that there is nothing that we cannot achieve. Nothing we cannot achieve in Essex and East Anglia, in England and the whole country beyond."
Mr Carswell condemned the other Westminster parties for operating "cosy cartel politics".
Michael Dugher, Labour's vice-chairman, said his party would continue to "expose UKIP for what they are".
"We will take them on in our areas", he said.
Mr Dugher said UKIP had inflicted a "humiliating defeat for David Cameron" in Clacton.
But Labour MP and former Pensions Minister Frank Field MP said: "If last night's vote heralds the start of UKIP's serious assault into Labour's neglected core vote, all bets are off for safer, let alone marginal seats at the next election."
Analysis by Chris Mason, BBC political correspondent
If Clacton provided the headline, Heywood and Middleton provided the surprise: UKIP nearly won there too. Little wonder party leader Nigel Farage was on the champagne in the pub afterwards.
The faces of Tory activists in Clacton said it all. Imagine how you'd feel if you had to listen to the victory speech of a former colleague who walked out on you and then gave you a walloping to boot.
Mr Carswell is the first MP to be elected under their banner although it did have one MP for two years after 2008 after another Tory MP, Bob Spink, defected to them.
Conservative candidate Giles Watling, one of the stars of the old BBC sitcom Bread, said: "I intend to stand here next May at the general election when this country will face a clear choice of who they want to be the leader - David Cameron or Ed Miliband."
Labour 'in trouble'
It had also looked at one point that UKIP could have been celebrating a double success with the result in the Greater Manchester constituency of Heywood and Middleton - held comfortably by Labour since its creation in 1983 - deemed close enough to warrant a "bundle check" of votes.
But in the end, Labour's Liz McInnes got 11,633 votes - 41% - with UKIP's John Bickley gaining 11,016 - 39%.
Labour's newest MP, Ms McInnes, told the BBC: "I'm going to work damn hard in this constituency. People here will have an MP who is going to fight for their interests."
But Mr Bickley said: "Labour are in big trouble. This should have been a safe seat. They have thrown everything at it and they have only just scraped home."
'Very different election'
Elections expert Professor John Curtice of Strathclyde University said: "We have to accept that the battle in England for May 2015 is not going to be a battle between simply three parties; it's going to be a battle between at least four parties.
"And bear in mind that also, according to some of the opinion polls, the Greens are running at 5% or 6% as well. We are looking at a very different kind of electoral competition in England from anything we have previously experienced in post-war elections."
The Heywood and Middleton contest was prompted by the death of long-serving Labour MP Jim Dobbin last month. His majority at the last general election was 5,971.
Clacton by-election: result in full
Douglas Carswell (UKIP) 21,113 (59.75%)
Giles Watling (Con) 8,709 (24.64%)
Tim Young (Lab) 3,957 (11.20%)
Chris Southall (Green) 688 (1.95%)
Andy Graham (LD) 483 (1.37%)
Bruce Sizer (Ind) 205 (0.58%)
Howling Laud Hope (Loony) 127 (0.36%)
Charlotte Rose (Ind) 56 (0.16%)
Turnout was 51%
Heywood and Middleton by-election: result in full
Liz McInnes (Lab) 11,633 (40.86%)
John Bickley (UKIP) 11,016 (38.69%)
Iain Gartside (Con) 3,496 (12.28%)
Anthony Smith (LD) 1,457 (5.12%)
Abi Jackson (Green) 870 (3.06%)
Turnout was 36%