English Defence League leader and founder Tommy Robinson has left the group, saying he has concerns over the "dangers of far-right extremism".
The EDL organises protests across the UK against "radical Islam".
Mr Robinson said it was still his aim to "counter Islamist ideology", although "not with violence but with better, democratic ideas".
Several senior figures have quit, but an EDL spokesman said the group "will not die because they're walking away".
He added that supporters would spend the next few weeks discussing how to proceed.
'No longer productive'
The EDL, formed in 2009, has organised marches and demonstrations in several cities across the UK, which have seen sometimes violent confrontations with anti-fascism campaigners.
Mr Robinson's co-leader, Kevin Carroll, has also opted to leave.
Their decision follows discussions with the Quilliam group, which describes itself as a "counter-extremism think tank".
Mr Robinson said: "I have been considering this move for a long time because I recognise that, though street demonstrations have brought us to this point, they are no longer productive.
"I acknowledge the dangers of far-right extremism and the ongoing need to counter Islamist ideology not with violence but with better, democratic ideas."
He explained his motives for leaving, telling BBC Radio 5 live's Nicky Campbell: "When some moron lifts up his top and he's got the picture of a mosque saying 'boom' and it's all over the national newspapers, it's me, it's when I pick up my kids from school the parents are looking at me, judging me on that.
"And that's not what I've stood for and my decision to do this is to be true to what I stand for. And whilst I want to lead the revolution against Islamist ideology, I don't want to lead the revolution against Muslims.
"I believe that the revolution needs to come from within the Islamic community and they need to stand up. And I believe this is a step forward not a step back."
Mr Robinson said his decision would be a "massive problem" for him.
He added: "Do I feel English Defence League members are going to plant bombs and target my family? No I don't. [But] I feel there will be a backlash."
Mr Robinson also said he wanted to become a more "credible voice" through working with Quilliam: "I'm going to be more hated than I was yesterday. That's for sure. If I saw this as a way out [of public life]... I would have just quit the English Defence League."
In a press conference, he later added: "I apologise for the fact that what I've represented and said has not resonated individually with Muslims.
"From day one in the English Defence League, we've wanted to embrace everyone: all colours and all creeds."
Later, in an interview with the BBC's Newsnight programme, he reiterated his beliefs that there should be better regulation of mosques in the UK to curb the influence of "preachers of hate" and "an end to Islamic immigration until the problem is solved".
Quilliam said it hoped Mr Robinson would "invest his energy and commitment in countering extremism of all kinds, supporting the efforts to bring along his former followers and encouraging his critique of Islamism as well as his concern with far-right extremism".
Chairman Maajid Nawaz said: "As well as being a very positive change for the United Kingdom, this is a very proud moment for Quilliam. This represents not a change but a continuation for us, as challenging extremism of all kinds forms the basis of our work.
"We have been able to show that Britain stands together against extremism regardless of political views and hope to continue supporting Tommy and Kevin in their journey to counter Islamism and neo-Nazi extremism."
Mr Robinson and Mr Carroll are due to face trial later this month, charged with obstructing police officers near the scene of the killing of soldier Lee Rigby.
The men allegedly tried to defy a ban on marching to the scene in Woolwich, south-east London, via a major mosque, on 29 June.
Mohammed Shafiq, chief executive of the Muslim youth organisation the Ramadhan Foundation, said: "I met Tommy Robinson last week and during that meeting he indicated that he was leaving the EDL because he couldn't control the extremist group, impact on his family and wider legal cases he faces.
"At no stage did he reject his previous disgusting attacks on Islam and Muslims or apologise to the British people for the millions wasted policing their protests.
"I cannot take Tommy Robinson or Kevin Carroll's announcement seriously until they reject their fascist views on Islam and Muslims and would caution other organisations celebrating this announcement as a massive personal achievement."
And Professor Matthew Goodwin of Nottingham, who has studied the EDL, said: "I would treat the announcement with extreme caution given Stephen Lennon's (aka Tommy Robinson's) recent comments against Islam in general on Twitter, and already signs that a new movement that is not geared around demonstrations may be formed.
"In hindsight, this may all owe more to the quest for publicity than to a genuine conversion to democratic ideas"