Up to 80 Tory MPs are prepared to vote against Theresa May's Chequers plan, a former Brexit minister has warned.
Steve Baker told the BBC the PM's blueprint for future relations with the EU had alienated Leavers and Remainers.
He warned of a "catastrophic split" if his party was not able to unite around a different, more flexible arrangement.
No 10 said its plan was the "only one on the table" able to deliver on the will of British people while avoiding a hard border on the island of Ireland.
The cabinet backed the so-called Chequers plan when it met at the PM's country residence in July.
The agreement, which led to the resignations of then-Brexit Secretary David Davis and the former Foreign Secretary Boris Johnson, sets out a blueprint for the future relationship with the EU once the UK leaves in March 2019.
Mr Baker is the former chairman of the Eurosceptic group the European Research Group (ERG), which plans to publish its own proposals for Brexit ahead of the Conservative Party conference.
He told the Press Association that the lack of support for Mrs May's plan, which would involve the UK accepting a common rule book for trade in goods, among Tory MPs left her with a "massive problem" in the run-up to next month's Conservative conference.
If the government tried to push through a Brexit deal through the Commons, on the back of Labour votes, he said the Tory Party "would suffer the catastrophic split which thus far we have managed to avoid".
"It is absolutely no pleasure whatsoever to me to acknowledge that, but I look at the mood of colleagues and the mood of the Conservative Party in the country and I am gravely concerned for the future of our party."
Mr Baker told the BBC that opposition to Chequers was so widespread that some colleagues who backed remaining in the EU would prefer the UK to leave without a deal than sign up to the current proposal.
Despite his criticism, he insisted he was not seeking a change in leadership and he would back the PM if she changed tack and backed a free trade deal modelled on Canada's arrangements with the EU.
"We want to change the policy and we would be delighted to unite behind Theresa May with a different policy.
"The Conservative Party must come together to deliver Brexit behind a policy and the policy we are arguing for is the one the EU has offered us in March."
'Suicide vest' row
Former leader Iain Duncan Smith said the PM's plan was "unacceptable" to the majority of Tory MPs and was "essentially dead".
Asked when Brexiteers outside of the government would come up with a detailed alternative, he said he and others would be putting forward a "whole set of new ideas" in the coming weeks but it was up to ministers to mould them into a plan that would be approved by Parliament.
The latest interventions follow a week of criticism of the Chequers deal from MPs, notably former foreign secretary Boris Johnson.
He caused controversy over the weekend with a column in the Mail on Sunday, calling the PM's plan a "suicide vest" around the UK, with the detonator in Brussels' hands.
A number of senior Tories criticised the language he used, with one minister saying it marked a "disgusting moment" in politics but other colleagues of Mr Johnson said critics were "terrified of his popular appeal".
In his weekly column in Monday's Daily Telegraph, Mr Johnson wrote that the Conservative government should cut taxes to help "striving" families and give the UK a post-Brexit boost.
But Justice Secretary David Gauke said there was "no vacancy" in Downing Street and ex-education secretary Nicky Morgan, who backs a Norway-style association agreement with the EU, said getting a Brexit deal through the Commons would potentially depend on the support of Labour MPs.
"Surely the public want politicians to work together on this matter of national interest," she told BBC Radio 4's Today.
"That is why I hope the government is going to appeal to MPs on all sides to do the right thing for the country."
When will a deal be done?
The UK is due to leave the EU on 29 March 2019.
A deal on the UK's withdrawal agreement was expected to be agreed in time for an EU summit in October, along with a statement from both sides on the future relationship.
However, a number of senior figures have said this deadline may slide into November.
What is the latest from the EU?
The EU's negotiator Michel Barnier has said he is "strongly opposed" to aspects of the Chequers plan.
But with the clock ticking to next month's summit, the Financial Times said EU leaders now seem prepared to begin discussions on the most contentious points of Mrs May's plan.
The paper reported they would give Mr Barnier the green light to begin talks at a meeting in Salzburg later this month.
But, at the same time, The Republic of Ireland's EU Commissioner Phil Hogan signalled that further UK compromises would be needed and "factionalism" within the Conservative Party could scupper any chance of an agreement.
"If the UK attitude is Chequers and only Chequers, there will be no agreement before March next year on the future trade relationship," he said in a speech in the town of New Ross.