The Football Association has been accused of "institutional racism" for not referring alleged comments by referee Mark Clattenburg to the police.
Peter Herbert, chairman of the Society of Black Lawyers, believes the FA has not done enough to investigate a complaint against Clattenburg.
"We think the FA is institutionally racist," Herbert told BBC Radio 5 live.
But FA chairman David Bernstein responded by calling Herbert's comments "ill-informed and unhelpful".
The referee denies using "inappropriate language" towards two Chelsea players.
The Metropolitan Police has into a complaint, made by the Society of Black Lawyers, that Clattenburg used a racially offensive term during Chelsea's 3-2 defeat at home to Manchester United on 28 October.
An FA investigation into that matter is ongoing, but Herbert labelled as "derisory" the punishments handed out by the governing body to Chelsea captain John Terry and Liverpool striker Luis Suarez for other incidents.
Terry was banned for four games and fined £220,000 last month for racially abusing QPR defender Anton Ferdinand during a Premier League game at Loftus Road in October 2011.
Suarez received an eight-match suspension for repeated racial abuse directed towards Manchester United defender Patrice Evra during a league game played in the same month.
Herbert said: "What we're saying is that the FA has no hate-crime strategy. Now that, for anybody in the industry of dealing with criminal justice, means they've got no systematic way of dealing with hate-crime in sport.
"To give you an example, the penalties on John Terry and Luis Suarez were derisory.
"And then you have 104 incidents where players are sent off for racial abuse and not one of them is reported to the police, while you have 120,000 teams playing every weekend up and down the country.
"We think the FA is institutionally racist, has not got a handle on this issue, and its refusal to report this matter to the police looks like it is one of those football cover-ups, which is well-versed."
But FA chairman Bernstein dismissed Herbert's criticisms.
He said: "These ill-informed and unhelpful remarks are at odds with the progressive and responsible approach being followed by the game with the support of Government to deal with these serious issues.
"The FA will continue working to strengthen processes to eradicate all forms of discrimination in football."
In an earlier interview with BBC Radio 5 live, Herbert had accused both the FA and Chelsea of a "cover-up" for not involving the police.
He said: "It sounds remarkably like the football industry wanted to have this issue swept under the carpet."
Chelsea made a on 29 October. A day later, the into the allegation, part of which was that Clattenburg used racial language.
Meanwhile, the Metropolitan Police began its own probe based on a written complaint by Herbert - not Chelsea or the FA.
It is understood Chelsea did not give evidence to the police because they felt the FA was the appropriate body to deal with the matter.
On Tuesday, the police investigation was discontinued because "no victims have come forward" and "without a victim and/or any evidence that any offence has been committed, the matter cannot currently be investigated".
The FA confirmed its investigation would proceed, but Herbert is furious that neither the governing body nor Chelsea have taken the matter to the police.
Herbert said the society would take up the FA's handling of the allegations with sports minister Hugh Robertson, who responded: "They are welcome to approach me but I cannot instruct Chelsea to make criminal complaints; that is an argument he needs to have with Chelsea.
"We had the Downing Street summit on racism in football earlier in the year and the FA are coming back to us before the end of the year and as soon as we have that response we will assess what to do next."