Broadcasters have said plans for three TV general election debates in April will go ahead, despite David Cameron saying he will take part in only one.
It means Mr Cameron - who has rejected a head-to-head debate with Ed Miliband - could be "empty chaired".
Mr Miliband said it was "judgement day" for the prime minister, who was "a weak leader running from his record".
Downing Street said it would not back down on its "final offer" of one seven-way debate in March.
Mr Cameron's communications chief Craig Oliver said it was "disappointing" that the broadcasters were not prepared to take up the prime minister's offer.
"I am ready to discuss at your convenience the logistics of making the debate we have suggested happen," he added in a written response to the broadcasters' decision.
Downing Street has blamed the broadcasters for "chaotic" negotiations, which have gone on for months.
Mr Miliband said: "I think the broadcasters are absolutely right to reserve the right to empty chair the prime minister.
"These debates don't belong to him; they don't belong to me. They belong to the British people.
"He should stop ducking and weaving. He should stop running scared. He should agree to these debates."
Lib Dem leader Nick Clegg urged David Cameron to drop his objections, tweeting: "You haven't got your own way so accept it and take part."
Analysis by BBC Political Correspondent Ben Wright
This is a defiant move by the broadcasters.
They could have chosen to blink and accept the "final offer" made by Number 10 on Wednesday and end the negotiation saga.
Instead, the BBC, ITV, Channel 4 and Sky have said they will continue with three debates in April as planned.
They have made it clear all the debates will happen regardless of whether invited parties turn up or not - and today described the head-to-head debate, with Mr Cameron and and Miliband, as "vital".
But proceeding without Mr Cameron would be a massive step for the broadcasters to take.
And to complicate matters further the one-on-one contest would have to comply with the broadcasters' own rules on fairness and balance - perhaps the basis of a challenge from Number 10.
Downing Street is adamant that it is the broadcasters who have badly mishandled the whole process and that it is "disappointing" that they have not accepted the prime minster's final offer.
But Mr Miliband said it was "make your mind up time" for Mr Cameron - and the ball is back in Downing Street's court.
UKIP leader Nigel Farage said: "I am pleased that the broadcasters have stood firm at last, but it would have been far better had they stuck with their original proposal which included fewer parties.
"Nonetheless we accept the challenge."
SNP leader Nicola Sturgeon said: "A Tory prime minister simply cannot be allowed to dictate terms to everyone else taking part."
Green leader Natalie Bennett said Mr Cameron's refusal to take part in the debates as planned was further damaging the reputation of politics and the proposed format was "very fair".
Plaid Cymru leader Leanne Wood said: "I'm delighted that the broadcasters are holding firm. It would have been wrong for one individual to dictate and change the terms of these debates."
But Peter Robinson, leader of the DUP and First Minister of Northern Ireland, said the debate negotiations had been the "greatest electoral shambles of all time".
"Just who do the broadcasters think they are, that they can set down a diktat?" he told Sky News.
Mr Cameron has raised concerns about the exclusion of the DUP from the broadcasters' plans but the BBC Trust has rejected calls for it to reconsider their participation.
Sky, the BBC, Channel 4 and ITV have invited the leaders of the Conservatives, Labour, Lib Dems, UKIP, the SNP, the Green Party of England and Wales and Plaid Cymru to take part in two debates lasting two hours each during the election campaign.
Planned debate schedule
April 2: Seven-way debate featuring David Cameron, Ed Miliband, Nick Clegg, Nigel Farage, Nicola Sturgeon, Natalie Bennett and Leanne Wood (ITV broadcast)
April 16: Seven-way debate featuring David Cameron, Ed Miliband, Nick Clegg, Nigel Farage, Nicola Sturgeon, Natalie Bennett and Leanne Wood (BBC broadcast)
April 30: Head-to-head debate featuring David Cameron and Ed Miliband (Channel 4 and Sky News broadcast)
As it stands, the broadcasters have said the debates will go ahead as originally planned on 2 and 16 April, while a planned head-to-head encounter between the Conservative and Labour leaders will also continue on 30 April.
In a joint letter to Craig Oliver, the four broadcasters said they believed their proposal was more suitable than the one debate proposed by the prime minister.
"We have given your proposal serious consideration but we don't think it achieves the goal of providing our viewers with election debates that can properly explore a reasonably full range of issues," they said.
They added: "We very much hope that all invited leaders will participate in the broadcast debates. However, in the end all we can do - as impartial public service broadcasters - is to provide a fair forum for debates to take place.
"It will always remain the decision of individual leaders whether or not to take part."
The broadcasters said they would be prepared to meet Mr Cameron or his representatives for talks on the way forward.