The Chancellor has been criticised by a disabled charity for saying high levels of disabled people in the workforce may have had an impact on productivity.
Speaking to the Treasury Committee, Philip Hammond said the UK should be "extremely proud of high levels of participation by disabled people".
But he said that may have had an impact on the UK's overall productivity.
Disabled charity Scope called for an apology, saying the comments were "unacceptable and derogatory".
Last month the government announced plans to get a million more disabled people into work within a decade.
Anna Bird, director of policy and research at Scope, said Mr Hammond's comments "fundamentally undermine the government's policy and the ambition set out by the prime minister just a week ago".
She called on him urgently to withdraw the remarks.
Office for National Statistics figures show that disabled people remain twice as likely to be unemployed as their able-bodied peers.
Labour MP Marsha de Cordova, shadow minister for disabled people, tweeted : "As a disabled person I am shocked and appalled that Philip Hammond is trying to blame me and other disabled people for the Tories' economic failure.
"He should apologise immediately for this disgraceful comment."
She said the disability productivity gap had been "stuck at 30 odd percent" since 2010 and she called for a massive investment strategy and education programmes.
Sophie Morgan, a disability rights campaigner and presenter, expressed her anger on Twitter.
She was joined by inclusion expert Mik Scarlet, who responded to Mr Hammond by documenting his journey to work.