Football would "collapse" if the private text messages of everyone in the game were examined, says Kick It Out chairman Lord Ouseley.
Speaking to BBC Radio 5 live's Sportsweek, Lord Ouseley claimed racism and sexism were endemic in the sport.
"All the people in power in football have to come to clean and admit that there is a problem," he said.
Former Cardiff manager Malky Mackay is being investigated by the FA after admitting sending offensive texts.
But Ouseley says focusing solely on Mackay would be a mistake because the issue went much deeper.
"Mackay is probably thinking 'hey I'm unlucky' because if the texts of every other football manager and senior administration were examined there wouldn't be many people in work," he said.
"The game would collapse."
Former Football Association chairman Lord Triesman told 5 live he thought Mackay could be facing a year-long ban from the sport.
But he wondered whether the FA had the jurisdiction to act after declining to take action against Premier League chief Peter Scudamore over private sexist emails recently.
"The case is very strong now and the FA is duty-bound to intervene," said Lord Triesman.
"If it turns out to be three texts, a ban might be a season. If more, it may be much longer."
Meanwhile, Sport Wales chair Professor Laura McAllister thinks there is a "legitimisation of really bad behaviour" in football.
She said: "There's always been a soft underbelly of pretty hideous behaviour around the game and that contrasts very, very strongly with a lot of other sports. It is depressing."