Tottenham have been accused of "insulting people's intelligence" by attempting to justify the chanting of the word "Yid" by their supporters.
The head of the Society of Black Lawyers says he will complain formally to police if Spurs fail to take action within a fortnight to stop the chants.
In a statement, Spurs said their fans used the term as a "defence mechanism" against anti-Semitism from rivals.
But Peter Herbert said this "wouldn't make sense to a six-year-old".
Oldham footballer Dean Furman, who is Jewish, has also told BBC Sport there should be a "zero tolerance" approach and fans using the chant should be immediately thrown out of stadiums.
However, Piara Powar, executive director of the Football Against Racism in Europe organisation, has criticised Herbert for being naive and questioned whether he properly understands football.
Some Tottenham fans have historically referred to themselves as the "Yid Army", in reference to the club's Jewish supporters.
The term is used in a chant by some fans at White Hart Lane and can often be seen on flags and banners at the ground.
But Herbert told BBC Sport: "Clearly Tottenham have a distance to travel before they learn of the offence that is being caused.
"If you had a group of Afro-Caribbean supporters using the 'N-word', even as a 'defence mechanism', it would clearly be completely unacceptable. There are young supporters present when the chants are used.
"Use of the word, even in this way, could be seen by some as justifying other people using the term."
He has warned the north London club that unless they take action by 20 November, he will report "a racist incident" to the Metropolitan Police.
Earlier on Wednesday, Spurs released a statement arguing that "real anti-Semitic abuse, such as hissing to simulate the noise of gas chambers, is the real evil and the real offence".
It continued: "Our fans adopted the chant as a defence mechanism in order to own the term and thereby deflect anti-Semitic abuse. Our position on this topic is very clear. The club does not tolerate any form of racist or abusive chanting.
"Our guiding principle in respect of the 'Y-word' is based on the point of law itself - the distinguishing factor is the intent with which it is used, ie if it is used with the deliberate intention to cause offence. This has been the basis of prosecutions of fans of other teams to date.
"They do not use the term to others to cause any offence, they use it a chant amongst themselves.
"We believe this is the area that requires a determined and concerted effort from all parties and where we seek greater support to eradicate."
Oldham and South Africa midfielder Furman backed Herbert by saying the term should not be used at all.
"I understand that it can be used as a nickname in some quarters, but if it's important to stop the use of the word then you can't use it either way," he told BBC Sport.
"You can't use it as a nickname or a chant for Tottenham because abusive chants will come back. The more they use it, the more opposition fans will use it. If it is banned altogether then hopefully we won't get the abuse at all. It needs to be tackled in the same way as racist chanting.
"The sanctions need to be high and then it will be stamped out of the game for good. We need zero tolerance."
"The anti-Semitic abuse is sung on the terraces and it's vital that anyone caught singing these chants is immediately removed from the stadium.
"People will be singing those songs without the knowledge of what went on in the Holocaust, for instance. Knowledge is everything and the sooner we can get these songs stamped out of our game the better."
But Powar criticised Herbert for his intervention. "I think this is where Peter Herbert and the Society of Black Lawyers are naive," he told BBC Sport.
"They perhaps don't know football. To think 20 to 30 years of complex identity will be cleared up overnight is wrong. And I have to say that the Jewish community are split on their use of that term.
"Many young members say it's a badge of appreciation and gives them pride. Some older members says that's not true, it's a derogatory term and goes back to a period they don't want to remember."