Brexit negotiations have hit a "real problem" over the issue of the Irish border, government sources have warned.
Hopes of a breakthrough were raised when the Brexit secretary made an unscheduled trip to Brussels on Sunday.
But talks faltered over the need for a back-up plan - known as the backstop - to avoid a hard border.
Theresa May, who is briefing MPs, has insisted any backstop arrangement must apply to the UK as a whole so not to create a border in the Irish Sea.
Ahead of a key summit of European leaders on Wednesday, the EU is believed to be seeking further reassurances that its internal market will be protected and that additional customs and regulatory checks can be carried out without the return of physical infrastructure on the border.
Sunday's talks had broken down after the EU had insisted on a second backstop arrangement - just involving Northern Ireland - if the UK's version wasn't ready in time, Downing Street sources indicated.
Sources say EU proposed 'backstop to the backstop' solution is "a real, real problem."— norman smith (@BBCNormanS) October 15, 2018
The Democratic Unionist Party has vowed to oppose any new checks on goods passing between Great Britain and Northern Ireland. And the party's Brexit spokesman has said the prospect of a no-deal Brexit is "probably inevitable".
Ireland's Foreign Minister, Simon Coveney, suggested a deal would not be done at this week's summit, which starts on Wednesday.
Mr Coveney said he was "frustrated and disappointed" that it was going to take "more time than people had hoped".
Scotland's First Minister Nicola Sturgeon, meanwhile, has set out her alternative Brexit plans, saying it is "time to compromise".
What is the Irish 'backstop'?
The UK is leaving the EU in March 2019, along with its single market and customs union, which allow for friction-free trade between members.
After Brexit, it will have a land border with the EU between Northern Ireland and the Republic.
Both the UK and the EU want to avoid a "hard border" - physical checks or infrastructure between Northern Ireland and Ireland - but cannot agree how.
So, the backstop is a position of last resort - to protect an open border on the island of Ireland in the event that the UK leaves the EU without having agreed a solution as part of trade negotiations.
The two sides do not agree on what this safety net should look like, however. The EU has suggested Northern Ireland stays aligned with its trade rules so new border checks are not needed.
But Mrs May has said this would undermine the integrity of the UK by creating a new border in the Irish Sea.
She has suggested the UK as a whole could remain aligned with the EU customs union for a limited time after 2020, when the planned transition period ends.
But the EU says a backstop would not work if it is time-limited.
Some Tory Brexiteers say the backstop is not necessary at all because technological solutions can avoid a hard border.
Domestic pressure on May
This week's summit comes as domestic political pressure on Mrs May increases amid threats of potential cabinet resignations.
Some Brexiteers are unhappy at the idea of the UK staying aligned to EU rules without a time limit being specified.
At the weekend, former Brexit Secretary David Davis urged ministers to "exert their collective authority" and rebel against the plans.
Brexiteer MPs trying to persuade Cabinet ministers to force PM to ditch Chequers at tomorrow's meeting - serious warnings flying around about what happens if she doesn't - 'I'm not sure how we stop chaos if she doesn't budge', says one— Laura Kuenssberg (@bbclaurak) October 15, 2018
But International Development Secretary Penny Mordaunt said she wasn't going to resign over the prime minister's Brexit plan.
"Everyone needs to calm down. We're entering the final stages of these negotiations and we're all behind the PM trying to get the best result," she added.
Arriving at an EU meeting in Luxembourg, Foreign Secretary Jeremy Hunt said the Brexit talks had entered a "difficult period", adding: "Whether a deal is done this week or not who knows?"
He insisted it was possible to do a deal and "with goodwill on all sides we can get there".
But his predecessor, Boris Johnson, said the backstop plan should be scrapped altogether, saying the EU was trying to change the UK's constitutional arrangements and "treating us with naked contempt".
Sinn Fein leader Mary Lou McDonald told BBC Radio 4's Today programme: "It would be a chronic miscalculation and an utter disgrace if the policy of the British government was to be set by the most extreme elements of the Brexiteers and the needs and desires and the idiosyncrasies of the Democratic Unionist Party."
Labour, meanwhile, has called on the government to publish its plan for the backstop.
Shadow Brexit secretary Sir Keir Starmer said any proposal needed full scrutiny from MPs before an agreement could be struck with the rest of the EU at the Brussels summit.
A Number 10 source said the prime minister had made sure Parliament was regularly updated on the talks.