Celtic supporters' groups have issued a joint statement calling on Ian Bankier to consider his position as chairman of the Scottish Premiership club.
The Celtic Trust, the Affiliation of Registered Celtic Supporters' Clubs and The Green Brigade accuse him of "an unforgivable slur on Celtic's support".
He had suggested that some social media attacks on fellow board member Lord Livingston were "criminally racist".
Celtic and Bankier say he was only referring to a small number.
The row followed the decision by Tory peer Ian Livingston, whose title is Baron Livingston of Parkhead, to vote in favour of cuts to tax credits in the House of Lords.
An online petition calling for his removal from the Celtic board, saying his actions were against the club's ethos, attracted nearly 10,000 signatures.
Bankier backed Livingston at the club's annual meeting and later issued a statement in which he said: "I have not branded our supporters racist and it is outrageous to suggest that I would ever do that.
"I was only referring to a small number of specific comments which have been made on social media, which I believe are unacceptable, and I know Celtic supporters would agree with me."
However, three supporters groups have expressed their "anger" at what they describe as "disgraceful remarks" by Bankier.
"Mr Bankier's claim that fans opposed to the re-election of Mr Livingston to the Celtic board have engaged in 'criminally racist' social media postings is an unforgivable slur on the Celtic support," they state.
"We are now, since the AGM took place, aware that there were some postings on social media which we condemn and we would support the club, and Mr Livingston, should they take action against those individuals.
"None of this is a justification for the generalised smearing of the Celtic support and, in particular, those of us who voted against Mr Livingston's re-election."
Despite a ballot being called, Livingston is expected to be re-elected.
"Mr Bankier committed an error of judgement and showed a complete lack of control at the AGM and this is not acceptable in a Celtic chairman," the groups added.
"We would like to publically condemn his unprofessional behaviour and the subsequent attempts to suggest that his words had been taken out of context as this patently was not the case.
"We call on all other members of the PLC board to denounce these comments.
"Ultimately, we are of the belief that Mr Bankier is no longer able to redeem himself following this attack on the Celtic support and his failure to retract and apologise immediately afterwards, and we call on him to give serious consideration to his position as chairman of Celtic PLC."
Celtic responded to the fans' statement by defending Bankier further.
"Ian made his position absolutely clear on Friday when he thanked, praised and commended Celtic supporters and shareholders and clarified that he had not branded our supporters as some had suggested," they said in a statement.
"Ian was simply referring to those individuals that the Trust in their statement have also condemned, so we seem to be in agreement.
"We are surprised and disappointed that a public statement has been made without any dialogue on the matter with the club, but as always we will be happy to meet any of our fans groups to discuss."