The EU's chief negotiator Michel Barnier says there has not been enough progress to move to the next stage of Brexit talks as the UK wants.
He said there was "new momentum" in the process but there was still "deadlock" over how much the UK pays when it leaves, which he called "disturbing".
Brexit Secretary David Davis said he still hoped for the go-ahead for trade talks when EU leaders meet next week.
The pair were speaking after the fifth round of Brexit talks in Brussels.
Mr Barnier said: "I am not able in the current circumstances to propose next week to the European Council that we should start discussions on the future relationship."
The UK's Brexit Secretary David Davis urged EU leaders at the summit, on 19 and 20 October, to give Mr Barnier a mandate to start trade talks and to "build on the spirit of co-operation we now have".
He said there had been progress on the area of citizens' rights that had moved the two sides "even closer to a deal".
The EU chief negotiator told reporters at the joint press conference he hoped for "decisive progress" by the time of the December summit of the European Council.
He said Theresa May's announcement that Britain would honour financial commitments entered into as an EU member was "important".
But he said there had been no negotiations on the issue this week because the UK was not ready to spell out what it would pay.
"On this question we have reached a state of deadlock which is very disturbing for thousands of project promoters in Europe and it's disturbing also for taxpayers."
Not doomed yet
Analysis by BBC Political Editor Laura Kuenssberg
Not even Brexit's biggest cheerleader could claim the discussions in Brussels have been going well. And there are visible frustrations on both sides.
But before claiming this morning's drama means the whole thing is doomed there are a few things worth remembering.
At the very start of this whole process, the hope was that in October, the EU would agree to move on to the next phase of the talks, to talk about our future relationship. But for months it has been clear that the chances of that were essentially zero.
It is not, therefore, a surprise to hear Mr Barnier saying right now, he doesn't feel able to press the button on phase 2, however much he enjoyed the drama of saying so today.
Second, behind the scenes, although it has been slow, there has been some progress in the talks but officials in some areas have reached the end of the line until their political masters give them permission to move on.
The so-called divorce bill covers things like the pensions of former EU staff in the UK, the cost of relocating EU agencies based in the UK and outstanding commitments to EU programmes. The UK has said it will meet its legal requirements and there has been speculation the bill could be anywhere between £50bn and £100bn, spread over a number of years.
BBC Europe Correspondent Kevin Connolly said the UK sees its total financial commitment "as its best negotiating card to be played somewhere near the end of the talks - the EU wants that card to be shown now at a point which is still relatively early in a two-year game".
The UK has also offered to keep paying into the EU budget during a proposed two-year transition period.
The EU had two other issues on which it would not make any "concessions", said Mr Barnier - citizens' rights and the Northern Ireland border.
On the status of the border, Mr Barnier said negotiations had "advanced" during this week's discussions.
But he said there was "more work to do in order to build a full picture of the challenges to North-South co-operation resulting from the UK - and therefore Northern Ireland - leaving the EU legal framework".
Asked about speculation that the UK could exit the EU in March 2019 without a trade deal, Mr Barnier said the EU was ready for "any eventualities" but added: "No deal will be a very bad deal."
Mr Davis said: "It's not what we seek, we want to see a good deal, but we are planning for everything."
Both men said progress had been made on citizens' rights, with Mr Davis saying there would be an agreement "soon" to ensure EU nationals in the UK would be able to enforce their rights through the UK courts.
He said EU citizens would still have to register with the UK authorities but the process would be streamlined to make it as simple and cheap as possible.
According to Mr Davis, the remaining sticking points include:
- The right to bring in future family members
- The right to "export a range of benefits"
- To "continue to enjoy the recognition of professional qualifications"
- To vote in local elections
- To "leave for a prolonged period and yet continue to enjoy a right to remain or permanent right of residence on return"
Labour leader Jeremy Corbyn said: "I think it's quite shocking. We're now 15 months on since the referendum and the government seems to have reached deadlock at every stage."
He said "falling out" of the EU without a trade deal would threaten "a lot of jobs all across Britain".
Labour is calling for "emergency" talks between Mr Davis and the EU early next week, to try to break the deadlock ahead of the EU summit.
Earlier this week, European Council President Donald Tusk suggested that the green light to begin talks about a post-Brexit trade deal would not come until December at the earliest.
Meanwhile, draft conclusions for next week's summit of EU leaders - which could yet change - call for internal work to begin on possible transitional arrangements and trade talks with the UK.
That would mean they could move ahead with negotiations on a future relationship, if "sufficient progress has been achieved" in talks.
But the draft conclusions seen by the BBC, if adopted, suggest EU leaders are not yet ready to begin talks with the UK about a post-Brexit transition deal.
Last month Prime Minister Theresa May used a speech in Florence to set out proposals for a two-year transition period after the UK leaves the EU in March 2019, in a bid to ease the deadlock.