David Miliband says he believes the Labour Party is at its weakest point in half a century.
The ex-foreign secretary spoke after the Conservatives won the Copeland seat to record the first by-election gain by a sitting government in 35 years.
Jeremy Corbyn has said he will stay on as Labour leader, after his party won a separate by-election in Stoke Central.
Deputy leader Tom Watson said it was "not the time" to change leader but the party had "to do better".
In an interview with the Times, Mr Miliband declined to rule out a return to UK politics.
Mr Miliband, who lost out on the Labour leadership in 2010 after being defeated by his younger brother Ed, told the newspaper he was "deeply concerned" about the party's future.
"I don't think this is just a repeat of the 1980s," he said. "We have to really understand the historic nature of the challenge that we have to face."
He now works as the head of the International Rescue Committee in New York, helping with aid for humanitarian crises around the world.
But when asked if he would come back to British politics, he said: "I honestly don't know what I am going to do. It's hard to see - but what's the point of saying never?"
BBC political correspondent Chris Mason said some in Labour saw Mr Miliband as the best leader they never had, but it was not the first time he had been "a public doom-monger" about his party under Mr Corbyn.
Mr Miliband's criticism comes as others urged Mr Corbyn not to "pass the buck" after the loss of the seat in Copeland, Cumbria, which Labour had held for 80 years.
Trudy Harrison got 44.3% of the vote, increasing the Conservatives' vote share by more than 8%. She overturned a Labour majority of more than 2,564 to take the seat by 2,147 votes - a swing of more than 6%.
At the Scottish Labour Party conference in Perth, Mr Watson said "we need to take a long, hard look at ourselves" after losing Copeland.
Referring to criticism of Mr Corbyn, he added: "This is not the time for a leadership election, that issue was settled last year - but we have to do better."
PM Theresa May said the "astounding" victory in Copeland showed her government was "working for everyone".
The leader of trade union Unison, Dave Prentis, told the Guardian that the loss was "disastrous" for Labour.
"The blame for these results does not lie solely with Jeremy Corbyn, but he must take responsibility for what happens next," he said.
"Nurses, teaching assistants, care workers and ordinary people everywhere need a Labour government. Jeremy has to show he understands how to turn things around and deliver just that."
Gerard Coyne, who is standing to become general secretary of the Unite union, said there was "no doubt" Copeland was a "meltdown" for Labour.
Labour must "face up" to the "disconnect" between Westminster and ordinary Labour voters", he said, but stopped short of calling on Mr Corbyn to step down, telling the BBC's Today programme the leadership was a matter for Labour MPs and the party itself.
'Solutions' not leadership
Shadow Brexit secretary Keir Starmer told the Guardian he was concerned the party was not "acknowledging how bad a result it was".
But shadow foreign secretary Emily Thornberry defended the party, telling the BBC: "What makes a difference to people's lives is having the sort of government that will address the concerns of people, that has some solutions."
The opposition's challenge "cannot be laid at the door of one individual", she said.
Mr Corbyn said Labour's "message was not enough to win through in Copeland" but hailed victory in Stoke as a "decisive rejection of UKIP's politics of division and dishonesty".
Labour's share of the vote for newly elected MP Gareth Snell was 37% - slightly down on the 39.3% it got in 2015.
UKIP got 24.7% of the vote and the Conservatives, who came a close third, 24.4% - both slightly higher than their 2015 vote share.
Labour had held both Stoke and Copeland since their creation but was forced to defend them when two former frontbenchers, Tristram Hunt and Jamie Reed, resigned as MPs.
The by-election results mean the government's majority is now 12 - the same as it was immediately after the general election, as the Conservative's new Copeland seat makes up for the one they lost to the Lib Dems in the Richmond Park by-election.
With Sinn Fein's four MPs not taking their seats, Mrs May's working majority is 16.