LGBT+ campaigners have urged the government to "make good" the prime minister's promise to ban "conversion therapy" in England and Wales.
It comes as three advisers quit the government's LGBT+ advisory panel amid concern it is being too slow in bringing in legislation.
Ministers have promised to "work at pace" to end conversion therapy.
But Labour leader Sir Keir Starmer said the government had a "blind spot" on the issue that the PM had to "address".
Conversion therapy involves treatments ranging from psychotherapy to religious teaching and discussion.
Campaigners want a "meaningful" law against the practice, which aims to change sexual orientation or gender identity.
All forms of conversion therapy are already prohibited in Switzerland and parts of Australia and Boris Johnson pledged last summer to ban what he called an "abhorrent" practice in England and Wales.
But the government has not yet published the findings of a consultation on the issue which began in 2018.
In a debate in Parliament on Monday, equalities minister Kemi Badenoch said conversion therapy had "no place in a civilised society", but added that "robust" laws were already in place to deal with the most troubling examples, involving violence and sexual assault.
And the government did not want to stop those who "seek spiritual counselling as they explore their sexual orientation", she said.
But LGBT+ groups and campaigners, including Stonewall, the LGBT Foundation and Trans Media Watch, say a ban on all forms of conversion therapy, not just the most distressing types, is needed.
In a letter to Ms Badenoch, they express "deep concern" over her comments, arguing they appeared to "imply that you have been meeting with perpetrators of conversion therapy and those who want the practice to continue".
The letter continues: "This is very worrying and gives the impression that the government is more interested in appeasing the extreme fringes of religious people, despite the calls for a ban from senior religious leaders from across a wide range of faiths."
Two advisers quit the government's LGBT advisory panel on Wednesday, and a third resigned on Thursday.
An awkward issue for Boris Johnson
There have been rumblings for a while about a growing unease amongst some panel members which are now playing out in a very public way.
The advisory group was first set up under Theresa May but hasn't met for some time - I'm told at least six months.
And now we've heard claims that, even when it did meet, panellists didn't necessarily feel listened to.
If the group does appear increasingly defunct (and we'll have to see what happens) it may be seen, by some, as symbolic of a fading commitment to LGBT+ equality under this administration.
Ministers would reject that and, during a debate on Monday, Kemi Badenoch said the government has a "proud record" of championing equal opportunity.
But it was her response overall that seems to have sparked these resignations.
Among concerns was her refusal to commit to a legislative "ban" of so-called conversion therapy, promising instead only to "end" the practice.
The equalities minister described the legal landscape as "complex". But it's an awkward one for Boris Johnson who did, last summer, use the word "ban" during a TV interview.
Jayne Ozanne, who resigned first, accused ministers of creating a "hostile environment" for LGBT+ people.
She said there had been "a lack of engagement", telling ITV News that both Ms Badenoch and minister for women and equalities Liz Truss were known among the LGBT+ community as "the ministers for inequality".
Another of those who quit the panel, James Morton, told BBC Radio 4's Today programme: "For the past 12 months I've been increasingly concerned about the direction of travel on trans rights."
He added that it felt "really scary" being a trans person "in the UK at the moment".
And Ellen Murray said she had resigned "due to the government's persistent and worsening hostility towards our community".
"From conversion therapy to trans healthcare, to the shameful treatment of LGBT refugees, the government has acted in appalling faith," she added.
Labour Leader Sir Keir Starmer said the government "has a blind" spot in the issue.
"The PM needs to address what is obviously a problem in his government and a pattern of behaviour he needs to address," he said at his party's local election campaign launch.
Asked if he would ban conversion therapy if he were prime minister, Sir Keir said: "yes, absolutely."
Business Secretary Kwasi Kwarteng said the government "bows to nobody in terms of our pushing for LBTQ issues, LGBT rights and also rights of minorities".
On Wednesday, a government spokesman said: "We have repeatedly made clear that we will take action to end conversion therapy and we are working to bring forward plans to do so shortly."
The LGBT Advisory Council was set up in 2019, with the term of current panel members set to end on 31 March.