Dover chairman Jim Parmenter has threatened to withdraw his investment if attendances do not improve.
Only 588 people were at Crabble on Tuesday as the Whites lost 2-1 to Southport in the National League
"I am going to walk away from trying to get this club to the Football League unless the people of Dover get behind the club," he told BBC Radio Kent.
"There is no point in me investing hundreds of thousands of pounds if that is not what the people want."
Dover's match was the third-best attended game in Kent on Tuesday, with Bromley against Welling drawing in 1,110 and 854 watching Ebbsfleet's match against Havant & Waterlooville in National League South.
Defeat saw the Whiles slip to ninth in the table, five points off the play-offs, although Chris Kinnear's side have games in hand on all of the teams above them.
The club is on the verge of making ground improvements which would make Crabble fulfil Football League requirements, but Parmenter says he will think twice before starting building works.
"I've all but confirmation of a £400,000 grant to rebuild a stand, with league-standard dressing rooms," he said.
"It involves personal expenditure of close to half a million pounds. When the grant is finalised - and it should be finalised in the next week - we have got six months to start building.
"If we don't start building we won't be able to get promotion. I am not going to do it unless the people want it.
"I will walk away. I have had enough."
Parmenter says he has invested close to £3m in the club over the last 11 years, during which time Dover have risen from Division One of the Ryman League to the top flight of the non-league pyramid.
Dover average 921 for home league games this season, with their highest attendance of 1,537 coming against Bromley on 28 December.
"We have been challenging at the top of the table all season and have had 10 years of success, and everybody is completely blasé about it," he added.
"People are just too used to success. They would rather stay indoors and watch European football.
"We should be getting crowds of 1,000 plus where we are and what we are doing.
"Everyone in the area needs to make up their mind - do they want a club that is ready to go in the league, or do they want us to go back down to the Ryman League and play local football?
"If that is what people want, fine. I am ready to say 'I give in'."