David Cameron has insisted the government is sticking to planned housing benefit cuts after Labour claimed the policy was in disarray.
The PM said: "These are difficult changes but I think it is right."
Labour leader Ed Miliband said the policy was "a complete shambles" after reports welfare secretary Iain Duncan Smith was rethinking it.
Some Lib Dem and Tory MPs are calling for the planned cap on housing benefit, of £400 a week, to be dropped.
But Mr Cameron, speaking at prime minister's questions in the Commons, said it was not fair for working people to see their taxes used to fund homes "they couldn't even dream of".
'Out of touch'
And he stressed that the government was "going forward with all the proposals we put in the Spending Review and in the budget".
Mr Miliband accused Mr Cameron of being "out of touch" and warned that thousands of people would be forced out of their homes as a result of the changes, which start to come into effect from April next year.
He told the prime minister: "You are about to make 500,000 people redundant. Your policy on housing benefit is a complete shambles. In London councils are saying 82,000 people will lose their homes. How many people do you think will lose their homes as a result of this policy?"
The coalition has announced plans that would limit housing benefit at about £400 a week for a four-bedroom home, and cut payouts by 10% when people have been on jobseeker's allowance for more than a year.
Mr Cameron said: "The point everyone in this House has got to consider: are we happy to go on paying housing benefit of £30,000, £40,000, £50,000?
"Our constituents working hard to give benefits so people can live in homes they couldn't even dream of? I don't think that's fair."
The Treasury believes the changes will trim £2bn from the £20bn housing annual housing benefit bill, which grew dramatically under Labour.
The Department of Communities and Local Government is to grant £10m from its homelessness budget to local councils' funds, to ease the consequences of the change.
The money will go to councils' "discretionary funds" - a pot of money they can use for special cases. It is not expected to be aimed at London alone, the BBC understands.
For example, if a family's rent was more than £400 but one of their children attended a local special school, the council could - at their discretion - allow the family to stay in that accommodation and keep paying a higher rate.
But ministers accept that many thousands of people will have to move house if the changes go ahead.
Chancellor George Osborne had already allocated £60m in his June budget, a tripling of the cash for "discretionary funds" to help the changes.
According to government figures, 21,000 people will be affected by new caps on the amount families can claim for five, four, three, two and one-bedroomed properties across the UK including 17,000 in London, the majority of whom are out of work.
But 775,000 claimants could be affected by changes to the way local housing benefit levels are calculated.
At the moment, the Local Housing Allowance is based on the median rent for an area, but from next year it will be based on the 30th percentile of local rents, which the government says will drive down housing costs. Claimants would lose an average of £9 a week.
'Hardship and distress'
Lib Dem backbenchers have threatened to join forces with Labour in a bid to block the changes in Parliament.
Shadow work and pensions secretary Douglas Alexander urged them to rebel against the government, saying: "Ed Miliband told David Cameron it was time to listen - now I'm telling Lib Dem MPs it's time to talk and work together to force the government to think again."
One potential rebel, Torbay MP Adrian Sanders, said he was seeking a meeting with Iain Duncan Smith and was confident of gaining concessions from the work and pensions secretary.
He said the changes were not just unfair on those living in cities, as London MPs have claimed, but also on people like his constituents who were in receipt of Local Housing Allowance.
"The whole thing is completely and utterly unrealistic and it is going to cause hardship and distress," Mr Sanders told the BBC News website.
And Conservative Mayor of London Boris Johnson said special dispensation was needed for people living in the capital.
"It is vital that we mitigate the impact of that housing benefit cap in London in a place where rents are so high, where the cost of living is so high and where some families will simply need to live in a certain area just because their kids go to school nearby or they are near their place of work," he told Channel 4 News.
Last week's Spending Review also included a cut of 50% on the amount spent on new social housing - but the government hopes to make up the shortfall by allowing housing associations to charge close to the full market rate for rent.
The coalition claims the extra private sector investment this will attract will deliver 150,000 new social homes by the end of its five-year term.