Chelsea's complaint against Mark Clattenburg could hinge on the evidence of the assistant referees, who would have heard every exchange, according to ex-referee Graham Poll.
The Blues claim he used "inappropriate language", including racial words, in Sunday's defeat by Manchester United.
Poll told the BBC: "A referee's microphone is on open. Everything he says is heard by two assistants.
"So if Mark said something, the assistants would have heard it."
He added: "If a comment of a racial nature was made, I think it should be reported and I think assistant referees will report it because there's no place for it."
Chelsea filed a complaint to the Premier League match delegate after the 3-2 defeat at Stamford Bridge, claiming "inappropriate language" was used towards two of their players.
BBC Sport understands part of the allegation is that Clattenburg used racial language.
It is also understood midfielder John Mikel Obi is involved in the complaint.
The Football Association launched an investigation on Monday after Chelsea made a formal complaint against Clattenburg.
Referees' union Prospect has given its "full support" to Clattenburg, adding in a statement: "It is now important that the allegations are fully investigated through the proper process as quickly as possible."
All four officials - the referee, his two assistants and the fourth official - wear microphones and ear pieces that allow them to hear what each other is saying throughout the match.
That means assistant referees Michael McDonough and Simon Long and fourth official Michael Jones should have heard conversations between Clattenburg and the players.
The conversations are not recorded but Poll, who refereed 1,544 matches during his career and officiated at two World Cups, said "referees would be in favour" if they were.
Former Football Association executive director David Davies added: "It is quite beyond me why 90 minutes of conversations aren't always recorded.
"Referees' conversations should be available to fans who pay their wages, if anything, to be aware of what the referees themselves have to put up with.
"If the language is so bad, someone has to do something about improving it without sanitising the game.
"A point has been reached and in this generation, you have got to do something about it."
Former referee Jeff Winter told BBC Radio 5 live: "If a match official has used racial insults or language to a player then he's for the high jump.
"He won't be getting a four-match or an eight-match ban, it'll probably be the end of his career, but that is if indeed he did say anything."
The alleged incident occurred in a highly charged game where Chelsea's Branislav Ivanovic and Fernando Torres were shown red cards in the space of five minutes.
Clattenburg, 37, is viewed by the Professional Game Match Officials (PGMO) as one of the country's leading officials.
He regularly referees games in European competitions and was put in charge of the League Cup final in February and the Olympic final between Brazil and Mexico in August.
In 2009, he was suspended over allegations about his business dealings but Clattenburg appealed and returned to officiating after an eight-month ban.
Chelsea's complaint about him comes in the wake of a number of high profile racism cases involving players during matches.
Liverpool's Luis Suarez received an eight-match suspension for racially abusing Patrice Evra while Chelsea's John Terry got a four-match ban for racially abusing Anton Ferdinand.