David Davis, who has been leading UK negotiations to leave the EU, has quit his role as Brexit Secretary
He told the BBC that he was no longer the best person to deliver the PM's Brexit plan - agreed by the cabinet on Friday - as he did not "believe" in it.
He said the "career-ending" decision was a personal one but he felt the UK was "giving away too much and too easily" to the EU in the negotiations.
Mrs May said she did not agree but thanked him for his work.
The resignation is a blow to Mrs May as she seeks to win over Eurosceptic MPs to her proposed Brexit vision, which would form the basis of the UK's position in on-going talks with the EU.
Dominic Raab, who campaigned for Leave during the UK's 2016 EU referendum, has been promoted from housing minister to take over from Mr Davis.
The UK is due to leave the European Union on 29 March 2019, but the two sides have yet to agree how trade will work between the UK and the EU afterwards.
There have been differences within the Conservative Party over how far the UK should prioritise the economy by compromising on issues such as leaving the remit of the European Court of Justice and ending free movement of people.
Mrs May's Conservative Party only has a majority in Parliament with the support in key votes of the 10 MPs from Northern Ireland's Democratic Unionist Party, so any split raises questions about whether her plan could survive a Commons vote - and has also led to renewed questions about whether she will face a challenge to her position.
In a sign of how delicately positioned the numbers are on Brexit strategy it has emerged that the government has taken the unusual step of arranging a briefing for opposition Labour MPs on the detail of the Brexit plan agreed on Friday.
In his resignation letter, Mr Davis told Mrs May that "the current trend of policy and tactics" was making it "look less and less likely" that the UK would leave the customs union and single market.
He said he was "unpersuaded" that the government's negotiating approach "will not just lead to further demands for concessions" from Brussels.
Mr Davis, who was appointed Brexit Secretary in 2016, said: "The general direction of policy will leave us in at best a weak negotiating position, and possibly an inescapable one."
In her reply, Mrs May said: "I do not agree with your characterisation of the policy we agreed at cabinet on Friday."
She said she was "sorry" he was leaving but would "like to thank you warmly for everything you have done... to shape our departure from the EU".
Mr Davis told BBC Radio 4's Today programme that he had objected to Theresa May's plan at the Chequers meeting, telling cabinet colleagues at the outset that he was "the odd man out".
He said it was "not tenable" for him to stay in post and try to persuade Tory MPs to back the policy when he did not think it was "workable".
"The best person to do this is someone who really believes in it, not me."
He said he feared the EU would seek to further water down the UK's plans and his resignation would make it easier for the UK to resist attempts to extract further concessions.
Mr Davis told BBC political editor Laura Kuenssberg that he had been compromising for two years and that the latest plan was "a compromise too far".
But Mr Davis insisted he continued to back Theresa May, saying that if he "wanted to bring her down", the time would have been after she failed to win last year's general election outright.
A leadership contest now would be "the wrong thing to do", adding: "I won't throw my hat into the ring."
Eurosceptic MP Steve Baker has also resigned. He played a leading role in the Brexit campaign in the run up to the 2016 referendum. He was promoted to the Department for Exiting the EU as a parliamentary under-secretary in June last year.
Conservative MP Peter Bone hailed Mr Davis's resignation as a "principled and brave decision", adding: "The PM's proposals for a Brexit in name only are not acceptable."
Labour Party chairman Ian Lavery said: "This is absolute chaos and Theresa May has no authority left."
David Davis resigning at such a crucial time shows @Theresa_May has no authority left and is incapable of delivering Brexit.— Jeremy Corbyn (@jeremycorbyn) July 8, 2018
With her Government in chaos, if she clings on, it's clear she's more interested in hanging on for her own sake than serving the people of our country.
'No choice but resignation'
By BBC political editor Laura Kuenssberg
After many months of rumours that he would pull the plug, David Davis has actually quit as Brexit Secretary.
His unhappiness in government has been no secret for some time, but after the prime minister's Chequers agreement with cabinet ministers to pursue closer ties with the EU than he desired, he found his position untenable.
After a visit to Downing Street on Sunday he concluded that he had no choice but to walk.
The move, while not completely surprising, throws doubt on to how secure the government's Brexit strategy is.
Mrs May is due to address MPs on Monday afternoon and is expected to tell MPs that the strategy agreed by the cabinet at Chequers on Friday is the "right Brexit" for Britain.
Brexiteer MP Jacob Rees-Mogg said it would be "very difficult" for Mrs May's plans to win the backing of MPs without Mr Davis.
He told BBC 5 Live: "These proposals will have to come to the House of Commons in legislation and the question is 'will they command support from Conservative MPs?'
"And I think without David Davis there, without his imprimatur, it will be very difficult for them to get the support of Conservative MPs and therefore the prime minister would be well advised to reconsider them."
BBC political editor Laura Kuenssberg said she understands Mr Davis was "furious" after a meeting at No 10 earlier on Sunday and "concluded he could not stay in post".
The resignation came as people awaited the verdicts of senior figures from the Brexit side of the 2016 referendum. There has yet been no on the record comment from Foreign Secretary Boris Johnson, while Environment Secretary Michael Gove told the BBC on Sunday he was urging Tory MPs to support Mrs May.
One of the leading pro-Remain Conservative MPs Anna Soubry did not refer directly to Mr Davis's resignation, but tweeted it was "not the time for egos, grandstanding and blind ideology".
The director-general of the CBI, Carolyn Fairbairn, said the resignation was "a blow", adding that business had welcomed the agreement of ministers on Friday.
Meanwhile, some Remain-supporting politicians said the resignation was evidence of the need for a second referendum.
Lord Adonis, a prominent backer of a second vote, tweeted: "People's Vote to put Brexit out of its misery a big step closer after DD's resignation. Now the Brexiteers holding Mrs May hostage are falling out, there isn't a majority for any withdrawal treaty in Parliament."
The Liberal Democrats called on people to sign a petition for a vote on the proposed deal, adding: "The resignation of David Davis is yet more evidence of the chaos of this Tory Brexit. You deserve the final say".
Nigel Farage congratulated Mr Davis for quitting and called for Mrs May to be replaced as prime minister, accusing her of being "duplicitous" and claiming her response "shows she is controlled by the civil service".