Football administrators need to speed up the process of change or they could be 'forced' to do so, according to Kick It Out chairman Lord Ouseley.
In a report to celebrate 20 years of the campaigning group, Ouseley says more willingness needs to be shown to tackle discrimination in the game.
He also claims the top of the football pyramid is still "a bastion of white male domination".
"Progress, while notable and helpful, remains painfully slow," he said.
Kick It Out, which is jointly funded by the Football Association and Premier League, works across all levels of the game on all forms of discrimination, including racism and sexism.
Ouseley said: "Sustainable transformation can only be achieved if those people with the power, influence and resources - who can therefore make it happen - willingly do so.
"The alternative is for them to be made to do so forcefully. That alternative is neither a practical nor desirable option and would only be viable through legislation or conflict.
"But, there has never been any gain without pain in the struggle against racism, discrimination and exclusion."
Lord Ouseley, who founded Kick It Out in 1993, also claimed:
- Professional football clubs have only a "passive interest" in equality and diversity outcomes.
- More often than not there is "little or no interest whatsoever" in boardrooms or senior management teams for any high-profile equality and diversity activity.
- Many leaders in football hold the view there is no longer a race or discrimination problem and believe it is "time for Kick It Out to pack its bags and move on".
Ouseley's remarks come as Premier League chief executive Richard Scudamore is due to attend a meeting on Monday of the committee that will decide his fate after he sent sexist emails.
Kick It Out revealed that, so far in the 2013-14 season, it has received 281 complaints covering all levels of the game and social media, with 67% relating to racism and 20% to anti-Semitism.
The 79 complaints it has received about in-stadium incidents is up 443% from the same point last season.