Former Foreign Secretary Boris Johnson has been accused of Islamophobia after saying Muslim women wearing burkas "look like letter boxes".
He said he was against bans on face-covering veils in public places, in his Telegraph column, but that it was "ridiculous" people chose to wear them.
The Muslim Council of Britain accused him of "pandering to the far right".
Labour MP Jess Phillips said she would report Mr Johnson to the Equality and Human Rights Commission.
The Muslim Council of Britain said the comments were "particularly regrettable in this current climate, where Islamophobia and anti-Muslim hatred is becoming worryingly pervasive".
The group said that the government had shown "little action" to tackle anti-Muslim hate and repeated its call for an inquiry into Islamophobia within the Conservative Party.
The chairman of the Conservative Muslim Forum, Mohammed Amin, said the article was "anti-Muslim" and would "whip up hatred of women who wear the niqab and burqa".
"Boris Johnson is a master of the English language - he must understand exactly what effect his language will have. I find it deplorable he chose to write such an article," he said.
The imam of Finsbury Park mosque, Mohammed Mahmoud, has, meanwhile, accused the government of failing to show "meaningful engagement" with the Muslim community.
In an article for the Evening Standard, Mr Mahmoud, who was praised for his response to last year's terror attack on his mosque, criticised the government's "lacklustre" efforts to fight Islamophobia.
Conservative peer Baroness Warsi, the first Muslim woman to sit in a British cabinet, accused Mr Johnson of indulging in "dog whistle" politics.
“Tories are in denial on Islamophobia- says the hero praised around the world for his brave & compassionate response to the murderous attack in #FinsburyPark” @standardnews— Sayeeda Warsi (@SayeedaWarsi) August 6, 2018
Meanwhile @Conservatives its business as usual with #Boris #Burkha dog whistle 😔https://t.co/KXFl3GtgEo
Mr Johnson - who last month quit the government in protest at Prime Minister Theresa May's Brexit policy - was commenting on the introduction of a burka ban in Denmark.
He said he felt "fully entitled" to expect women to remove face coverings when talking to him at his MP surgery - and schools and universities should be able to take the same approach if a student "turns up... looking like a bank robber".
"If you tell me that the burka is oppressive, then I am with you," he said.
"If you say that it is weird and bullying to expect women to cover their faces, then I totally agree - and I would add that I can find no scriptural authority for the practice in the Koran.
"I would go further and say that it is absolutely ridiculous that people should choose to go around looking like letter boxes."
He said businesses and government agencies should be able to "enforce a dress code" that allowed them to see customers' faces.
But he said: "Such restrictions are not quite the same as telling a free-born adult woman what she may or may not wear, in a public place, when she is simply minding her own business."
He said a total ban on face-covering veils would give a boost to radicals who said there was a "clash of civilisations" between Islam and the West and could lead to "a general crackdown on any public symbols of religious affiliation".
Labour MPs accused of him of stoking Islamophobia to boost his ambitions of becoming the next Conservative leader.
Labour MP David Lammy said: "Muslim women are having their burkas pulled off by thugs in our streets and Boris Johnson's response is to mock them for 'looking like letter boxes'.
"Our pound-shop Donald Trump is fanning the flames of Islamophobia to propel his grubby electoral ambitions."
Labour MP Stella Creasy said the former foreign secretary was in danger of "going full Morrissey" - a reference to the former Smiths singer whose comments on immigration have sparked controversy.
Her advice to Mr Johnson was that "being a buffoon, not being a racist and misogynist, [is] more British".
Lib Dem foreign affairs spokesman Christine Jardine said Mr Johnson had "proven himself to be a xenophobe".
"This desperate pitch to stay in the news completely disregards the women who are facing daily discrimination on our streets," she added.
Mrs May's official spokesman said: "The long-standing government position on this is clear, that we do not support a ban on the wearing of the veil in public.
"Such a prescriptive approach would be not in keeping with British values of religious tolerance and gender equality."