Alfreton Town chairman Wayne Bradley fears the club will struggle to grow because of a lack of supporters.
The Conference Premier high-flyers would be able to play in the Football League despite concerns about the suitability of their Impact Arena home.
But while the ground has a capacity of more than 4,000, Bradley said their average league attendance of just 771 remains a cause for concern.
"Unless we can double our home support we will struggle to progress," he said.
Talking to BBC Radio Derby, Bradley continued: "Support levels are definitely a concern - at the moment we are progressing almost despite ourselves.
"It's too easy for me to put blame at the supporters' door, but it's not the supporters that I'm complaining about - it's the stay-away individuals, it's the community at large.
"I do accept we trade in an area of football population, there is Derby County, Nottingham Forest and Chesterfield doing well, then there is Mansfield just on our doorstep in Nottinghamshire."
Alfreton, who have been deducted three points for fielding an ineligible player against Kidderminster in December, are just one place and one point outside the play-off places with five matches remaining.
Bradley said the club's ability to accommodate away support had been a major point of contention as they endeavoured to prove they meet the Football League's standards for promotion.
With little home support, however, Alfreton open more than half of their ground to visiting fans.
"When we showed them our current ground lay-out they quickly recognised that 50% is given over to away support," said Bradley.
"It is unique, it is a concern and it is a factor I consistently bang the drum about."
If promoted, Alfreton are capable of increasing the capacity of their ground to 5,200 with Bradley confirming they have three years of redevelopment works already planned out.
"It's fair to say that if we are promoted, within three years the current stadium would be unrecognisable," he said.