Delayed repairs to the Palace of Westminster cost taxpayers £2m a week, a government spending watchdog says.
Politicians had been scheduled to leave the building for at least six years while building work is completed at an estimated cost of £4bn.
But in May the body overseeing the work said the move should be reviewed due to the financial pressures of coronavirus.
The Public Accounts Committee said the "few decisions" that have been taken so far did not need reviewing.
The committee's Labour chair Meg Hillier also warned that "excessive political interference" in the process "may muddy the waters" and make delivery of the project more difficult.
Earlier this year, the prime minister suggested the House of Lords could be moved to York on a permanent basis.
This idea was later rejected. but the government said it would continue to push for a move out of London, even though the final decision will be down to MPs and peers.
Ms Hillier said: "Parliament is literally falling apart around the thousands of people who work there and the million or so who, in better times, visit every year. It poses a very real risk to health and safety in its current state,"
"After nearly 20 years of discussion and costs to the taxpayer of just maintaining Parliament now rising by £2m a week, what we don't need is for the authorities to keep reopening and reviewing what few decisions have been taken.
"The cost of the project will be high but doing nothing is not an option and is certainly not a cost-free option - without action we are just ratcheting up the bill to the taxpayer."
'Get to work'
The Labour MP also suggested the coronavirus pandemic meant Parliament was quieter than usual, providing an opportunity for building work to go ahead.
"It's time for those responsible to get creative and get to work," she said.
Responding to the committee's report Sarah Johnson, CEO of the Houses of Parliament Restoration and Renewal Sponsor Body, said the review was aimed at securing value for money and providing "certainty about the way forward".
"The work to save our Parliament buildings for the nation is essential and urgent, and the Palace of Westminster continues to be at a high risk of catastrophic damage, be that a major fire, flood or falling masonry," she added.
The Palace of Westminster, which houses the Houses of Commons and Lords, has long been in need of repair and in 2018 MPs voted for refurbishment work to go ahead.
Under current plans, parliamentarians will move into temporary accommodation while the major repairs are carried out.
However, the Sponsor Body - a panel of politicians and building experts set up to oversee restoration - said it would review the project, looking at ways of cutting costs, including alternatives to decanting Parliament to a different building.
Renovation could happen around politicians as they continue working in the building, although the process could take longer and cost more.
In its report, the Public Accounts Committee warned that "excessive political interference may muddy the waters of the Sponsor Body's work and has the potential to make delivery of the programme more difficult".
It advised the body to publish details of "what it considers are the main risks to building political consensus" and how it plans to mitigate those risks.
And it suggested the Sponsor Body launch an "engagement strategy" to ensure the public can feed into decisions.