Former foreign secretary Boris Johnson is facing growing criticism over his remark that Muslim women wearing the burka "look like letter boxes".
Dominic Grieve, the ex-attorney general, said he would quit the party if Mr Johnson became leader.
Ex-Tory chairwoman Baroness Warsi said Mr Johnson's remarks could trigger a rise in hate crime.
Senior Tories have urged him to apologise but he has not done so, and has stood by his comments.
In a Daily Telegraph article, he said full-face veils should not be banned, but it was "absolutely ridiculous" women chose to "go around looking like letter boxes". He also compared them to looking like "bank robbers".
A source close to the former London mayor has said: "We must not fall into the trap of shutting down the debate on difficult issues.
"We have to call it out. If we fail to speak up for liberal values then we are simply yielding ground to reactionaries and extremists."
But, speaking on BBC Radio 4's The World at One, Mr Grieve - a former Remain campaigner who has previously clashed with Mr Johnson over Brexit - said his behaviour was "very embarrassing".
Mr Grieve said he would "without the slightest doubt", quit the Tories if Mr Johnson became leader, "because I don't regard him as a fit and proper person to lead a political party".
The leader of the Scottish Conservatives, Ruth Davidson, has also backed calls for an apology over what she called "gratuitously offensive" comments.
Speaking at an event in North Berwick, East Lothian, she said: "This wasn't an off-the-cuff slip, he wrote a column, he knew exactly what he was doing and I think it crossed from being provocative and starting a debate and actually it became rude and gratuitous.
"It's also not been shown through history that when men make sweeping statements about what women should or shouldn't wear that it goes well for them.
"I think he should apologise for them."
Earlier Culture Secretary Jeremy Wright said there was no reason not to have a "robust conversation" about the subject, but added: "We're not talking to our friends in the pub. We are public figures and we have an additional obligation to be careful."
A former Tory chairman, Lord Pickles, said Mr Johnson, who was foreign secretary until resigning last month over Brexit, risked "closing down" the debate with his "illiberal language".
Supporters of Mr Johnson say the row is politically-motivated and that other senior Tories have made similar remarks without such criticism.
Outspoken Johnson carries risk
By Jonathan Blake, BBC political correspondent
By saying what he did, in the way he said it, Boris Johnson was guaranteed to provoke a reaction.
The question of what Muslim women choose to wear in public is a sensitive one, around which most politicians tread very carefully.
But Boris Johnson is not most politicians. And by using language you rarely hear in the normal course of political debate, the former foreign secretary has attracted attention and criticism.
If it was his intention to remind everyone that Boris is still Boris, not afraid to say what he thinks, despite being out of government and away on holiday, it was not without risk.
Figures show a rise in Islamophobic attacks and the Conservative party faces criticism for not addressing anti-Muslim prejudice in its own ranks.
In this context, many feel he has picked the wrong time to tackle the wrong issue.
But as ever, being outspoken is at once appealing to Boris Johnson's supporters and distasteful to his detractors.
Mr Johnson, who is the current MP for Uxbridge and South Ruislip, has long been seen as a potential candidate for the leadership of the Conservative Party.
He fronted the successful Leave campaign in the 2016 EU referendum, and resigned as foreign secretary last month in protest at Theresa May's Brexit plans.
Writing in the Guardian, Baroness Warsi said Mr Johnson's remarks were "indefensible" and "have no place in the modern Conservative Party".
She said, although he was setting out a liberal position on the burka, he was doing it in an "alt-right" way, and using Muslim women as "political fodder… to stake out a leadership bid".
"Johnson's words… send out a message that Muslim women are fair game," she wrote.
But mother-of-seven Tahira Noor, who has been wearing a burka for 20 years, said it was "100% my choice" and Mr Johnson's comments showed a "lack of knowledge".
She told BBC Radio 5 Live: "In today's day and age, the majority of the women who wear the burka are born and brought up in this country, are educated in this country, they've been to colleges, universities, and have understood why they want to do what they're doing.
"They're under no oppression, they're not doing it because their husbands want them to or their fathers want them to."
Ms Noor has four daughters and none of them wears a burka, adding: "It's not a must, it's not an obligation."
Sahar Al-Faifi, from Cardiff, who wears the niqab, told BBC News: "I absolutely have no problem with anyone disagreeing with my choice to wear it and criticising it.
"But when you demonise and dehumanise a minority, who happen to be women of colour, I have a problem with this.
"It's not for the state, it's not for the government and it's definitely not for middle-aged white men to tell me what to wear and what not to."
Tory MP for Mansfield, Ben Bradley, said it was not the language he would have used, but added: "Ultimately he is a journalist... and journalists want their pieces to be read.
"They want people to talk about what they have written [and] I would say that is probably the motivation."
Mr Johnson's former adviser Munira Mirza said Mr Johnson's views on the burka had been "entirely consistent" and other Conservative politicians had expressed the same view, without being called Islamophobic.
In 2013 former Justice Secretary Ken Clarke - who also opposed a ban on the public wearing of burkas - said they should not be worn while giving evidence in court. He referred to burkas as a "peculiar costume" and a "kind of bag".
Ms Mirza added: "The reality is there is a political fight here. People who frankly couldn't care less about the issues that Muslim women face are piling into Boris because Boris said it."
Conservative MP Conor Burns also defended Mr Johnson, accusing his political colleagues of "bandwagon jumping".
The member for Bournemouth West tweeted those who criticised the former foreign secretary "clearly didn't understand" his comments.
What is Islamophobia?
By BBC home affairs correspondent Dominic Casciani
Last year, Zaynab Hussein, a mother from Leicester, was left fighting for her life after a man she didn't know smashed into her with his car - and then ran over her again. She was attacked because she was a Muslim.
Tell Mama, the national organisation that collects anti-Muslim attack statistics, says that the majority of street victims of such abuse and violence are women, for the same reason that Mrs Hussein was singled out: some Muslim women are easily identifiable by their mode of dress - and therefore easy to target.
Seven years ago Baroness Warsi said prejudice against Muslims had passed the "dinner table test".
And Mr Johnson's critics regard his "letter box" and "bank robber" comments as part of the problem the peer defined: normalising prejudice and dehumanising women, rather than calmly debating the complexities of the veil in an open society.
Since Baroness Warsi's warning, there has been the launch of a cross-departmental working group to tackle anti-Muslim hate.
But it has been criticised as toothless, not least because the government can't agree a definition for Islamophobia.
What Boris Johnson said
In his Daily Telegraph column, Mr Johnson - who last month quit the government in protest at Prime Minister Theresa May's Brexit policy - was commenting on the introduction of a burka ban in Denmark.
He said he felt "fully entitled" to expect women to remove face coverings when talking to him at his MP surgery - and schools and universities should be able to take the same approach if a student "turns up... looking like a bank robber".
"If you tell me that the burka is oppressive, then I am with you," he said.
"If you say that it is weird and bullying to expect women to cover their faces, then I totally agree - and I would add that I can find no scriptural authority for the practice in the Koran.
"I would go further and say that it is absolutely ridiculous that people should choose to go around looking like letter boxes."
He said businesses and government agencies should be able to "enforce a dress code" that allowed them to see customers' faces.
But he said: "Such restrictions are not quite the same as telling a free-born adult woman what she may or may not wear, in a public place, when she is simply minding her own business."
He said a total ban on face-covering veils would give a boost to radicals who said there was a "clash of civilisations" between Islam and the West, and could lead to "a general crackdown on any public symbols of religious affiliation".