Kick It Out chairman Lord Ouseley claims the Football Association needs more help to tackle issues of inequality in the game.
The FA last week announced that former Cardiff City boss Malky Mackay and head of recruitment Iain Moody will face no action over allegations they shared racist and sexist texts.
But Ouseley says such issues should not just fall on the FA to sort out.
"Clarity is needed about who should initiate action," he added.
The FA conducted an 11-month investigation into the allegations against Mackay and Moody before ruling that the communications "were sent with a legitimate expectation of privacy".
Ouseley, who has headed up the anti-discriminatory body Kick It Out since its inception in the 1990s, claimed that the FA's privacy argument could allow prejudicial behaviour to continue unchecked in some areas.
And he called on other bodies to play their part in investigating any alleged incidents.
He added: "It should not only be the FA. In most cases the employer is the football club and surely it is there where the action should be initiated.
"The reality is that, repeatedly, all the other bodies in football, such as the leagues, the clubs, the Professional Footballers' Association (PFA) and the League Managers Association (LMA) and others shy away from making decisions, which are theirs, as employers and as integrity standard bearers, relying instead on The FA to bail them out."