Disability benefits should go to "really disabled people" not those "taking pills at home, who suffer from anxiety", a key Theresa May aide says.
No 10 policy unit head George Freeman said personal independence payments (PIP) reforms were needed to roll back the "bizarre" decisions of tribunals.
Ministers say the changes will save £3.7bn but leave a "strong safety net".
But disability charity Scope criticised Mr Freeman's "crude" distinction between physical and mental health.
And Labour said the comments were "an insult to disabled people".
Responding to criticism online to his interview on BBC 5 live's Pienaar's Politics, Mr Freeman later tweeted that he had suffered from anxiety and depression in childhood, adding: "I don't need any lectures on the damage anxiety does."
The government is proposing changes to PIPs, which replaced the Disability Living Allowance (DLA), after two tribunal rulings at the end of 2016 which it said would have added £3.7bn to the benefits bill by 2023.
The benefit is intended to help people cope with the extra costs of living with ill health or disability and are made according to the points a person scores in an assessment of their needs.
In his BBC interview Mr Freeman said: "These tweaks are actually about rolling back some bizarre decisions by tribunals that now mean benefits are being given to people who are taking pills at home, who suffer from anxiety," he said.
"We want to make sure we get the money to the really disabled people who need it."
The Conservative MP for Mid Norfolk added that he and the prime minister "totally" understood anxiety. "We've set out in the mental health strategy how seriously we take it," he added.
After the interview, shadow chancellor John McDonnell tweeted: "This is an insult to disabled people. (George Freeman) should apologise immediately or Theresa May should make him."
And Scope chief executive Mark Atkinson said: "It is unhelpful to make crude distinctions between those with physical impairments and mental health issues because the kind of impairment someone has is not a good indicator of the costs they will face.
"Many disabled people will now be anxiously waiting to hear as to whether or not these tighter rules will affect their current PIP award.
"The government must offer clarity and reassurance that these new measures will not negatively affect the financial support that disabled people receive now or in the future, and that they stand by their commitment to making no further changes to disability benefits in this Parliament."
Disabilities minister Penny Mordaunt said she was reforming the PIP payments to "restore the original aim of the benefit" to make sure support was given to the most needy.
Mr McDonnell said he was "furious" about the proposed changes to PIPS, and said Labour would pressurise the government to reverse them in next month's Budget.
"Next week the Tories will make out that the economy and the public finances are doing better, however, they are planning to go ahead with a £3.7bn cut to the disabled," he said.
The cuts would mean many people with severe disabilities "are going to be trapped in their homes", he added.
The Liberal Democrats said the government was using court losses "as an excuse to severely restrict disability benefits".
A DWP spokeswoman said the government was "committed to ensuring our welfare system is a strong safety net for those who need it".