Expensive tickets, officious stewarding and inconvenient kick-off times are turning off fans, according to the Football Supporters' Federation (FSF).
More than 4,000 supporters replied to the FSF's 2012 National Fans' Survey.
And 92.2% of them said ticket prices were too high.
"There's a huge amount of wealth swilling among Premier League clubs, and there's no excuse to keep charging the prices they often do," said Malcolm Clarke, the FSF chairman.
Four out of 10 respondents said they were now attending fewer games because of high ticket prices.
That is an increase on the last survey in 2009, when 25% said they were watching less live football than before.
Clarke said: "It's about time that match-going fans were rewarded with lower prices rather than having to fork out as much as £50 or £60 to watch their team at some grounds.
"But it's not just the top-flight where fans face high prices - those in the Championship and beyond can face similar hurdles in following their teams.
"We're not asking the football industry to do something out of the goodness of its heart. Self-interest should play a part as those huge TV deals will disappear if stadiums begin to look empty.
"The game is playing Russian roulette when it comes to attracting younger fans, who might start voting with their feet to become 'pub only' fans, or even turn their backs on the game altogether."
In response, a Premier League spokesman told the BBC that grounds have been at least 90% full for 15 seasons in a row, with the last four seasons seeing 92% capacity.
"So fans are clearly enjoying the football on offer," the spokesman said.
"Pricing is a matter for individual clubs, but there are a variety of ticket offers made during the season and all clubs offer concessions to young people."
The survey also found that nearly six out of 10 supporters said they had seen an illegal feed of a game - from an overseas satellite provider - in a pub, and more than half admitted to watching football via an unauthorised "stream" on the internet.
On Thursday, the Premier League announced it had shut down 30,000 illegal streams last season, or an average of 75 per game.
Another rising statistic is the weight of support for the return of standing areas at football grounds, with 54.4% of the survey's respondents saying they prefer to stand during games, and 91.1% saying fans should be given the choice to sit or stand.
All-seater stadia have been compulsory in the Premier League since the start of the 1994-95 season following a recommendation by the inquiry into the Hillsborough disaster.
The measure was later extended to all levels of the professional game, although exemptions are allowed for smaller grounds further down the league pyramid.
There have been campaigns to revoke this legislation but the authorities have so far resisted calls for the reintroduction of standing areas at football grounds.
"Recent research looking at crowd diversity shows that more women, families and fans from different ethnic backgrounds are attending than ever before, and that reflects the welcoming nature of our grounds," the Premier League spokesman added.
The FSF is a supporters' organisation that claims more than 200,000 members from clubs at all levels of the game in England and Wales. The 2012 survey was promoted online and 4,074 replies were gathered in a four-week period after the end of last season.