Shadow chancellor John McDonnell has urged four Labour MPs who backed Theresa May in key Brexit votes to stop "propping up" the government.
Pro-Brexit MPs Kate Hoey, John Mann, Frank Field and Graham Stringer defied Labour's whip in the votes last week.
The government won one by six, avoiding a potential confidence vote that might have triggered a general election.
Mr McDonnell told the BBC he was trying to persuade the four they were "wrong" to support the government.
He said he spoke to them in the Commons chamber before the vote on an amendment to the trade bill, which would have allowed the UK to join a customs union if it does not agree a free trade deal with the EU.
The amendment was tabled by pro-EU Conservatives but Labour ordered its MPs to support it.
"I said 'look this government is reeling now and the longer this goes the more damage it's going to do to our country - leave aside party, this is damage to the country'," Mr McDonnell told BBC 5 Live's Anna Foster.
He said their response was that "this government's going to go in the autumn anyway". He told them that was "a short-term view".
"I am going to keep on talking to them to convince them that the more they prop up this government, the more damage this government will do and we might as well get to the situation where we enable Labour to take over negotiations," he added.
"Or, failing that, let's have a general election and let the people decide."
He said he respected their view but added: "I just say to them, as respectfully as I possibly can, you are making the wrong decision here, come alongside."
He said the Labour Party was making preparations "to go into government" including a Queen's Speech and a Budget, in case there is another election.
Kelvin Hopkins, who is currently sitting as an independent after being suspended by the Labour Party, also backed the government in one of the key Brexit votes.
The four Labour rebels have come under fire on social media from supporters of Jeremy Corbyn for "propping up" the government.
But they say they are standing up for the millions of Labour supporters who voted to leave the EU.
After the Commons vote, Mr Field told his local newspaper, the Liverpool Echo, he did not think it could have brought about the fall of the government.
"I voted knowing that two-thirds of Labour constituencies in the country voted to leave," he said.
"One reason for that was that people in Westminster and Whitehall and in elite groups don't wish to listen to them.
"I wanted to make sure that those two-thirds of Labour constituencies had a voice."
Ms Hoey, who is facing a campaign by Corbyn-backing Momentum in her Vauxhall constituency to expel her from the party, has been similarly defiant.
"I have had the same views on the EU since being elected and they knew these views as they were always in my election addresses," she told the Huffington Post.
"I voted against the Maastricht and Lisbon treaties and for a referendum and always in the same lobby as the now leader."