Glenn Tamplin has put Billericay Town up for sale after suffering "personal abuse" which he says has "started to affect his health and family".
Tamplin was spoken to by police after Saturday's defeat at Woking in National League South following a complaint from a fan that he had been taking cocaine.
"For me it has now crossed the line," the 46-year-old said.
The multi-millionaire has invested heavily since taking over the Essex club in December 2016.
"It is with deep regret that I have no choice but to cut my ties with Billericay Town Football Club," Tamplin said in a statement on the club website.
"The club is now up for sale with immediate effect.
He said "the final nail in the coffin" was after Saturday's 2-1 defeat to Woking, "where I was informed at the ground, after the match, that the police wanted to talk to me regarding a complaint from a fan that that I had been using cocaine.
"After speaking to the police they were satisfied that the complaint was unfounded, although angry and upset I felt that was the end of the matter.
"However, 20 minutes after I had left I was pulled over by the police. Bizarrely I was once again questioned regarding the complaint.
"This led to my two young children who were with me becoming absolutely terrified and in floods of tears in the back of my car."
The club, currently second in National League South, later uploaded an audio recording on their Twitter account of the second conversation Tamplin had with police after leaving Woking's ground.
Tamplin's statement continued: "Again, after assisting the police in every way possible I was again allowed to continue my journey home."
Billericay have attracted plenty of headlines since Tamplin bought the club, with former Premier League players Paul Konchesky, Jermaine Pennant and Jamie O'Hara all having had spells at New Lodge - and O'Hara is still a member of their first-team squad.
Tamplin had a spell as manager last season which saw the players sing R Kelly's 'The World's Greatest' in the changing room and him leave a game at Hendon 20 minutes early to attend a business meeting.
He vowed to get the club into the English Football League in five years and The Blues won promotion last season as Isthmian Premier League champions.
But they are currently managerless after Harry Wheeler was sacked as boss last month - reportedly by text message five minutes before the end of their defeat at Chippenham.
"My last actions will be to find suitable investors that will be able to run the Billericay Town and also to try my hardest to reinstate Harry Wheeler as manager of this great club."