A Football Association board has rejected an appeal by Gillingham and their chairman Paul Scally after they were fined for "race victimisation".
Both the League One club and Scally were fined £75,000 by the FA in July, in a case relating to the sacking of striker Mark McCammon in 2011.
They appealed against the fines, which were set by an independent regulatory commission, at a hearing this month.
The board reduced the club's fine to £50,000 but upheld Scally's sanction.
Scally previously described the level of the fines as "manifestly excessive, totally disproportionate and completely unjust".
McCammon, now 37, was sacked by the Gills in January 2011 and alleged he and other black players at the Kent club were treated differently from white players.
An employment tribunal found in favour of McCammon and awarded him £68,000 in damages in August 2012, and upheld that decision a year later following an appeal by the Gills.
The FA charged Gillingham and Scally with bringing the game into disrepute in September 2014, heard their case in April and fined both of them this summer - with Scally also ordered to attend an education programme.
A statement from the FA appeal board on Monday said the initial fines were "not excessive in principal" as the penalties had to be viewed "in the context of the gravity of the offending and the message required to be sent to the game".
The board decided to reduce Gillingham's fine by £25,000 because of "the burden the substantial fine would have upon them as a club".