Victims say they are "saddened, but not surprised" by the resignation of the most senior lawyer working for the independent inquiry into historical child sexual abuse.
The Shirley Oaks Survivors Association described the decision to quit by Ben Emmerson QC as "devastating".
It said some survivors were losing faith, believing the investigation's large remit was designed to fail.
Prime Minister Theresa May has said she still has confidence in the inquiry.
It was set up to examine whether public bodies, including the police, in England and Wales had failed in their duty to protect children from sexual abuse, and to examine claims of abuse involving "well-known people".
Mr Emmerson had been suspended on Wednesday due to "concerns about his leadership", and in his resignation letter he said he was no longer the "right person" for the role, but denied he had to step down over a difference of opinion with current chairwoman Prof Alexis Jay.
His resignation on Thursday evening came hours after it emerged that his colleague Elizabeth Prochaska - the inquiry's second most senior lawyer - had stepped down on 15 September.
A spokesman for the Shirley Oaks Survivors Association, a group representing 600 people who allege they were abused in south London children's homes, said: "We are devastated for all those victims who were hoping to finally get justice, but I cannot say it is surprising."
He described the inquiry's remit - investigating 13 separate strands ranging from the Roman Catholic Church to "the internet" - as "everlasting" and said some survivors believed the inquiry's framework was "default by design".
Gabrielle Shaw, chief executive of the National Association for People Abused in Childhood, said Mr Emmerson's resignation was "disappointing" as it was likely to reduce public confidence.
But she said suggestions that the inquiry was in crisis were "wrong" and rejected calls for its scope to be reduced, saying an "overarching umbrella" was needed to bring the inquiry together.
"It's not about scaling back and reducing the terms of the reference - it should be about better operational delivery of the terms", she said.
She added that she thought time "will be the healer" for confidence in the inquiry.
A spokesman for the Minister and Clergy Sexual Abuse Survivors group said the resignation of Mr Emmerson was "concerning".
He added: "It's the latest in a long line of problems and I think it's important the inquiry has somebody of the experience and calibre of Ben Emmerson at its heart, because whether you are a fan of Alexis Jay or not she's essentially a senior social worker - she does not have the legal expertise or experience to make snap judgements in a public inquiry."
A spokesman for the White Flowers Alba group, which helps survivors of abuse by the clergy in Scotland, said Mr Emmerson had been the "glue" that kept the inquiry together.
He said members were "absolutely devastated" by Mr Emmerson's resignation and now felt the inquiry was in "meltdown".
"Ben Emmerson was the one constant. This [the inquiry] isn't rocket science but we seem to keep getting it wrong. That's weird to me."
Abuse survivor Ian McFayden tweeted: "Honestly this is truly a sad day for victims & survivors of abuse, when we lose one of the world's leading civil rights lawyers @BenEmmerson1."
Barrister Michael Mansfield QC told BBC Radio 4's Today programme the inquiry had been "chaotic from the very beginning" and "appalling" appointments had been made without consulting survivors.
He said the inquiry needed to be split up into "very obvious parts" and said he was "very willing to consider" calls for him to replace Mr Emmerson.
But Labour MP Sarah Champion, whose Rotherham constituency was the subject of a sexual abuse scandal, said there was merit in keeping the inquiry "overarching".
"We ought to be more supportive", she told the BBC.
"The reason that we want to be whole... is we need to know in this country is the establishment predisposed to cover up child abuse? I think if you separate the inquiry we're going to lose a lot."
By Tom Symonds, BBC home affairs correspondent
Morale among victims of child abuse who have been waiting for this inquiry to make progress is at rock bottom. It is not hard to see why.
The inquiry now has no counsel, the senior lawyer who will question witnesses.
Ben Emmerson's replacement must be found and prepared for hearings, which are due to start early next year.
Another challenge is the need to reshape the inquiry's working methods to make its huge task more achievable.
Several of the inquiry's critics are considering court challenges to aspects of its work.
Meanwhile, newspaper articles and blogs question the existence of the inquiry on a daily basis.
But the prime minister - who started this in the first place - insists it has to go ahead, its wide-ranging remit intact, because children were failed in the past. And lessons must be learnt.
In his resignation letter to Prof Jay, Mr Emmerson said he was "sad" to leave but that he had had doubts about whether he was the right person for the inquiry.
He wrote that "it has become clear to me that I am not the person to take this review forward on your behalf.
"It is now time for someone else to take the helm with a different leadership of the counsel team.
"There is no truth in suggestions that I have resigned due to a difference of opinion with you about the next steps for the inquiry."
Abuse inquiry: How we got here
- 7 July 2014 - The government announces an independent inquiry into the way public bodies investigated and handled child sex abuse claims. Baroness Butler-Sloss is chosen as chair
- 14 July - Baroness Butler-Sloss stands down, saying she is "not the right person" for the job after questions are raised about her late brother, Sir Michael Havers, being attorney general in the 1980s
- 31 October - Lord Mayor of London Fiona Woolf steps down amid calls for her resignation after she admits having five dinners with Lord Brittan from 2008-12
- 4 August 2016 - New Zealand high court judge Dame Lowell Goddard writes to Home Secretary Amber Rudd to resign from her post
- Professor Alexis Jay, who led the Rotherham abuse inquiry, appointed new chairwoman
Mr Emmerson, who represented the widow of Alexander Litvinenko at the inquiry into the Russian dissident's death in London from radiation poisoning, is a deputy High Court judge.
He is also a visiting professor of human rights law at Oxford University and a leading international lawyer.
Prof Jay said Mr Emmerson would "continue to be available to the inquiry whilst his replacement is recruited and brought up to speed".
It had been suggested Mrs Prochaska's departure was unconnected to Mr Emmerson's resignation. However, BBC Newsnight understands there were "serious problems" in the working relationship between Mrs Prochaska and Mr Emmerson.
The two resignations are the latest blows to an inquiry that has already been beset by problems - Prof Jay is the fourth person to be appointed to lead the inquiry after three chairwomen stood down.
Prime Minister Theresa May said the "really important" hearings would still go ahead as planned.
Preliminary hearings began at the Royal Courts of Justice in London in March.
Speaking on Thursday, Mrs May said the inquiry's original terms of reference "were the right ones".
However, Labour MP Chuka Umunna, said: "This is a dysfunctional inquiry and we need urgent reassurance by the chair that she is getting a grip of the situation."
Strands of the inquiry
The inquiry has launched 13 investigations into a broad range of institutions. They are named as follows:
- Accountability and Reparations
- Cambridge House, Knowl View and Rochdale
- Children in custodial institutions
- Children outside the UK
- Child sexual exploitation by organised networks
- Lambeth Council
- Lord Janner
- Nottinghamshire Councils
- Residential schools
- The Anglican Church
- The internet
- The Roman Catholic Church