The populist, anti-immigrant Alternative for Germany (AfD) has rejected a bid by its co-leader to move the party closer to the mainstream.
Frauke Petry had wanted the AfD to seek coalition with other parties and reject extremist views but delegates in Cologne rejected her approach.
She has announced she will not contest general elections this year and the party must choose a replacement.
Outside the conference protesters clashed with police.
Up to 15,000 people demonstrated against the AfD's meeting, with some 4,000 officers deployed to stop them entering the hotel where is was taking place.
Two were injured in clashes, one while escorting party members. There were several reports of arrests.
Many shops were closed for the day and police helicopters were heard overhead, with some suggesting this was to monitor the protests.
About 100 people tried to break through a police line on the opposite side of the Rhine river from the conference venue, Die Welt newspaper reports (in German).
Outside the hotel, up to 60 people blocked access using a chain of bicycles. Police broke this up, leading to skirmishes with protesters.
It is not the first time an AfD conference has attracted demonstrations. When the party met last year in Stuttgart, hundreds of protesters were detained.
'Here to stay'
AfD co-leader and public face Frauke Petry announced earlier this week that she would not be the party's chief candidate in the September general election.
At the conference Ms Petry urged the party to reject infighting and unite behind the leadership, to prove the AfD was "here to stay".
But those attending refused even to hold a debate on her arguments that the party should become more mainstream and seek to govern in coalition rather than staying in opposition.
Fractious scenes: BBC's Jenny Hill in Cologne
Thousands of armed police officers were deployed to hold protesters back from the city centre hotel where Germany's most controversial party was holding its pre-election conference. Inside, the atmosphere was no less fractious.
The AfD, which is notorious for its fierce anti-Islam rhetoric, is slipping in the polls. The anti-immigrant platform which won it support during the refugee crisis is no longer enough to satisfy the electorate.
The party is on course to win seats in the general election but its chances of significant political success depend now on whether its members can agree on a candidate to stand against Angela Merkel and on the party's future direction.
While this weekend is officially a party conference, in truth it will be a weekend of crisis talks.
The party pushed Mrs Merkel's governing conservatives into third place in a regional election last year and hopes to build on its success by getting its first seats in the federal parliament.
Founded in 2013, it rose on a wave of opposition to Mrs Merkel's 2015 open-door policy to refugees.
However, all the mainstream parties have said they will refuse to form a coalition with the AfD.
Opinion polls suggest a sharp recent drop in the party's popularity - from 15% late last year to between 7% and 11% now.
It is beset by infighting between Ms Petry's pragmatic wing and a wing that is further to the right.
The party's reputation has been dented by a local leader's comments in January that the Holocaust memorial in Berlin was a "memorial of shame".
The party has voted to expel him and he will not be allowed to attend the conference.