The minister for disabled people has defended planned changes to support payments aimed at saving £1.3bn, amid growing calls for a U-turn.
Justin Tomlinson told BBC News there would still be an increase in the number of people claiming Personal Independence Payments (PIP).
And more money would be directed at the people who "needed it most".
Labour and some Tory MPs are threatening to derail the change, spelt out in Wednesday's Budget.
Labour leader Jeremy Corbyn said Mr Osborne had "declared war on the disabled" and Tory MP Andrew Percy said he had gone "too far" and the changes were "just not acceptable".
"This hits exactly the wrong people, potentially. I think it sends out the wrong message about the priorities of the government," Mr Percy told BBC Radio 4's The World at One.
He said the changes were more about helping Mr Osborne meet his self-imposed cap on overall welfare spending than reforming the benefits system.
Mr Tomlinson told BBC political editor Laura Kuenssberg 640,000 people would be affected by the changes in 2020-21 - but it did "not necessarily mean they will lose their benefit".
"They might lose one of the points in the assessment but a significant chunk of that 640,000 will continue to receive the benefit," he said.
He said it was a "future measure", which would only affect current claimants when they came up for reassessment.
"We will still see an increase in the numbers of people using PIP and the amount of money paid out," he added.
He said the government was committed to improving the system and that more people were receiving the payments at the highest end of the scale.
The Institute for Fiscal Studies said 370,000 people would lose out under the planned changes, with an average loss of £3,500 a year.
PIP claimants can get between £21.80 and £139.75 per week and are assessed using a points system to determine what level of help they receive.
The money is meant to help people cope with the extra cost of living with a disability or long-term health problem and are used to fund everything from mobility cars to adapted baths and showers.
The weight given to the use of aids and appliances in two of the 10 daily living activities - dressing and managing toilet needs - will be reduced from January.
'Happy to listen'
It follows an independent review, commissioned by the Department for Work and Pensions, which said a "significant number of people" were likely to be getting the benefit despite having minimal-to-no ongoing daily living extra costs.
But disability campaigners say the changes will make it harder for some disabled people to qualify for the benefit and prevent people hit by other benefit cuts from living independently.
Asked on BBC Breakfast about whether cuts to PIP were harming the most vulnerable in society, Mr Osborne said: "We have got to make sure we have a system that works.
"I'm always happy to listen to proposals about how improve on that but we have got to control our disability budget and make sure help goes to the people who need it most."
Labour's Shadow Work and Pensions Secretary Owen Smith said: "There is no political, economic or moral justification for George Osborne's decision to cut support for disabled people while increasing tax benefits for the wealthiest, symbolising the unfairness of his Budget.
"I do not believe that those who voted Conservative at the last election did so to see a further £4 billion stripped away from people who may require assistance when washing or dressing."
He said Labour would resist any attempt by the government to push through the changes in secondary legislation, without a full debate in Parliament.
Analysis by Ross Hawkins
Why George Osborne has a problem with disability benefits.
1. Introducing the new Personal Independence Payment for disabled people has saved much less money than expected. There was "little evidence" for the original optimistic savings assessment says the Office for Budget Responsibility, which has repeatedly reduced its forecast.
2. More people are getting PIP than the government expected, and they are being paid more. Average awards were expected to be £86 a week. In fact they will be £100.
3, The government can boast it is spending more on disability benefits. The bill rises by about £1bn over the next five years. Had it not been for the most recent cuts it would have been yet higher - by an extra £1bn. Despite the overall bill rising individuals will see cuts to their payments.
4. The problem could get worse. Even after the latest cuts there is "considerable uncertainty" about the figures on future savings says the OBR.
5. It's a big political problem. If Mr Osborne U-turns he will have to find more than a billion pounds of savings elsewhere, and doubts about his ability to reach a surplus will intensify.
One lifelong Conservative voter, Graeme Ellis, said he had quit the party over the cuts - and made his views known on the official website of the Conservative Disability Group, on whose executive he has served.
"This website is temporarily closed owing to Disability Cuts," a message read after Mr Osborne confirmed the cuts.
"The owner of the hosting package, Graeme Ellis, has resigned over disability cuts from the group and will no longer develop or host this site."
The message was later amended to stress that no other member of the group was involved in the action and they had not known about it in advance.
Mr Ellis, a former NHS worker who has diabetes and uses a wheelchair, said Mr Osborne was "destroying lives".
"I've been a Conservative voter since I could vote. But as a lifelong Conservative I could no longer agree with what the Government's doing," he told the Mirror.
A Conservative Party spokesman said: "The Conservative Disability Group has not deactivated its website.
"The owner of the domain, who is no longer a member of the group, has deactivated it without any instruction to do so."