German Chancellor Angela Merkel has been re-elected for a fourth term while nationalists have made a historic surge in federal elections.
Her conservative CDU/CSU bloc has seen its worst result in almost 70 years but will remain the largest in parliament.
Its current coalition partner, the social democratic SPD, says it will go into opposition after historic losses.
The nationalist AfD has won its first seats and is set to be the third party, a result that sparked some protests.
Dozens of demonstrators gathered outside the right-wing, anti-Islam party's headquarters in Berlin on Sunday night, some with placards saying "Refugees are welcome".
Protests were also held in several other cities, including Frankfurt and Cologne.
What does the result mean for Mrs Merkel?
While her alliance has remained the largest party, it is the worst result for the alliance between the Christian Democrat (CDU) and the Christian Social Union (CSU) since 1949, when national elections were held in Germany for the first time after World War Two.
Addressing supporters, Mrs Merkel, who has been in the job for 12 years, said she had hoped for a "better result".
She added that she would listen to the "concerns, worries and anxieties" of voters of the Alternative for Germany (AfD) in order to win them back.
Mrs Merkel also said her government would have to deal with economic and security issues as well as addressing the root causes of migration - one of the main reasons behind the AfD's result.
"Today we can say that we now have a mandate to assume responsibility and we're going to assume this responsibility calmly, talking with our partners of course."
A chaotic day for German politics: Jenny Hill, BBC News, Berlin
It has been a long and bruising election campaign. Angela Merkel may have won the election but it does not feel like much of a victory.
This election will go down in the history books for two reasons. Mrs Merkel may have won a fourth term but it is her worst-ever general election result. And right-wing nationalists are now part of the German establishment.
The result is a verdict, perhaps, on Mrs Merkel's decision to open Germany's doors to one million refugees.
What is the political norm in many other European countries was considered unthinkable in post-war Germany. Not any more.
What are her coalition options?
The Social Democrats (SPD) also had their worst election result since 1949. The party's loss of support while junior partner in government saw leader Martin Schulz declare the end of the "grand coalition" with Mrs Merkel's alliance, to cheers and applause.
"It's a difficult and bitter day for social democrats in Germany," Mr Schulz told supporters. "We haven't reached our objective."
He also vowed to prevent the AfD from being the main opposition party.
With the possibility of an alliance with the SPD rejected, Mrs Merkel's options are narrow, and the process of forming a new coalition could take months.
Six parties will be in the German parliament for the first time since the 1950s.
The most likely scenario is of a "Jamaica" coalition, so-called because of the colours of Jamaica's flag. It includes the black CDU/CSU, the yellow, business-friendly Free Democrats (FDP) - who are returning to parliament after a four-year hiatus - and the Greens.
It is not a marriage made in heaven, as the Greens want to phase out 20 coal-fired power plants and the FDP disagree, but it is the only formation that would guarantee enough seats in the new Bundestag, German broadcaster ZDF says.
All parties have rejected working with the AfD.
What about the AfD?
The party is expected to take 94 seats in the 709-seat federal parliament after capitalising on a backlash against Mrs Merkel's policy towards migrants and refugees, many of them from war-torn, mainly Muslim countries like Syria.
Alternative for Germany was founded in 2013 as an anti-euro party but later turned its focus to immigration and Islam.
It called for a ban on minarets and declared Islam incompatible with German culture. Several of its candidates have been linked to far-right remarks.
Those hardline positions helped it to win seats in 13 of Germany's 16 state parliaments in the last few years.
Prominent AfD figure Frauke Petry said on Twitter (in German) that Germany had experienced an incomparable "political earthquake". The party's performance was better than forecast in opinion polls.
It performed particularly well in what was formerly East Germany, taking 21.5% of the vote as the second most popular party.
Beatrix von Storch, one of the party's leaders, told the BBC that the result would change the political system in Germany, giving "a voice" to the people she said were not represented in the last parliament.
"We'll start debates on migration, we'll start debates on Islam, we'll start debates on ever closer union."
AfD calls for stricter asylum rules to curb abuse of the system, including vetting of claims in countries of origin that are deemed "safe".