The moderate Democratic Left party in Greece says it will not join pro-bailout parties in a coalition without the more radical far-left Syriza.
Syriza is refusing to attend coalition talks because it will not back any government which supports austerity measures demanded by the EU and IMF.
The Greek president has summoned the four main parties for last-ditch talks in an effort to avoid new elections.
Major European markets fell sharply because of the uncertainty.
By 13:00 GMT, shares on the Athens Stock Exchange had dropped by 3.5%. The London FTSE-100 index was down 1.7%, the Frankfurt Dax fell by 1.95% and in Paris, the Cac-40 saw losses of 2.27%.
The euro stood at $1.2861 - its lowest level since 23 January.
'No unity government'
A majority of Greek voters in last week's election backed parties opposed to the austerity plans demanded by the EU and the International Monetary Fund (IMF) in return for two bailouts.
"No unity government can emerge," Fotis Kouvelis, head of the Democratic Left party, told Greek television.
"A government without Syriza would not have the necessary popular and parliamentary backing."
EU finance ministers are due to meet in Brussels to discuss the Greek crisis later on Monday.
The fear over holding new elections is that parties that oppose austerity measures might do well again in new polls, says the BBC's Gavin Hewitt in Brussels.
And with no sign Europe's leaders are prepared to renegotiate the deal, Greece could end up leaving the eurozone.
For the first time, some central bankers have spoken openly about the consequences of a Greek exit from the single currency, our correspondent adds.
President Papoulias has invited four parties, including Syriza, to further talks with the aim of forming an emergency government.
Neither New Democracy nor Pasok has so far been able to form a new coalition.
They both agreed to swingeing cuts in return for the last EU/IMF bailout, but suffered at last week's polls.
Syriza, which came second, insists any new government must cancel austerity measures agreed in return for EU-IMF loans worth 130bn euros ($170bn; £105bn).
Leading European figures, including European Commission head Jose Manuel Barroso, have warned that Greece must respect the terms of the bailout deal if it wants to remain in the euro.
On Monday, Germany's finance minister Wolfgang Schaeuble said the EU could not amend agreed rules on tighter fiscal discipline despite voters' rejection of austerity policies in several countries, including Greece.
Officials are weighing up the fallout of a potential Greek withdrawal from the euro and how that would be managed, say analysts.