Ex-Chancellor Alistair Darling has warned of a potential "nightmare" on England's existing railways if the multibillion-pound HS2 line is built.
He was in the cabinet when the high speed rail scheme was approved in principle but has now changed his mind.
A rise in its budget from £32bn to £42.6bn would drain cash from other lines, he said.
But Transport Secretary Patrick McLoughlin hit back by saying not building HS2 would be a nightmare.
He said "massive growth" in passengers and freight meant the line was a necessity.
Labour's front bench has backed the scheme, which ministers say is vital and will create billions of pounds of benefits.
Its first phase would see 225mph trains running on a new railway line to be built between London and the West Midlands by 2026. A second phase would see the line extended further north with branches to Leeds and Manchester by 2033.
Mr Darling, also a former transport secretary, was chancellor in the previous Labour government when the first stage of HS2 was approved in principle in 2010.
He told the BBC that, at the time, he thought the case for HS2 was "just about stateable". But he had changed his mind because of the revised costs - which he said meant future governments would not have the money to spend on other lines.
"My experience as transport secretary is that if you do not spend money on upgrading, improving the track, improving the trains, then the thing will simply, eventually start falling apart, as it did by the mid-1990s."
He said the West Coast Mainline had been ignored for 30 years and became "an absolute disaster" which had to be rebuilt at a cost of £8bn.
"That is an example of the sort of money you need to be spending. The East Coast Mainline, which was last done up seriously probably about 25 years ago, it will need money spent on [it]."
'No blank cheque'
He added: "My fear is, if you build this visionary project that you will have a nightmare on the rest of the network because you don't have the money to spend on it."
Some Conservative MPs whose constituencies would be affected have also raised concerns and another former Labour cabinet minister, Lord Mandelson, has warned it could be "an expensive mistake".
Asked if Labour would continue to support the project, shadow chancellor Ed Balls told the BBC: "We have consistently supported plans for a north-south rail link but it's got to work, it's got to be value for money.
"There will be no blank cheque from a Labour Treasury."
Labour leader Ed Miliband says he is a "supporter" of the HS2 rail project but it should be scrutinised for "value for money".
He said having a high-speed rail link is "part of being a modern country" and will help boost growth in areas outside of London.
And reiterating the party's support for the scheme, shadow transport secretary Maria Eagle said it was the only way to address the "capacity issue" for passengers and freight on the railways.
Reports in the Financial Times suggested that Treasury officials were preparing for an overspend of £31bn on the project, including account inflation and VAT over 20 years.
But Transport Secretary Patrick McLoughlin said the budget of £42.6bn had not changed and included contingency money.
He told the BBC that the government would be investing heavily in rail over the next five years, with £37bn to electrify 880 miles of railway, the "northern hub" programme to upgrade rail links between northern cities and infrastructure investment in Reading and Birmingham stations.
"There's no doubt in the government's mind as to the importance of our railway network.
"The simple fact is, Mr Darling says that it would be a nightmare if we do HS2. It would also be a nightmare if we didn't do HS2 because what we have seen is a massive growth in [passengers and freight on] our railways over the past 20 years."
"I want to see freight continue to increase on our railways, I want to see more people using our railways but there's a big capacity problem."
He said no new railway had been built north of London for more than 120 years and Britain needed better links between its cities and more capacity on the railways for freight.
"No major infrastructure project gets under way without huge controversy. When we built HS1 [from St Pancras in London to the Channel Tunnel] people were complaining about that and saying that it shouldn't go ahead... that it would wreck the countryside. Actually if you go and look at what's happened now, people aren't saying that at all."