It is a sign of how bad things were that Ilyas Khan actually believes they are better now.
The Accrington Stanley chairman says he has been racially abused and attacked with stones since his takeover of the club little more than a year ago.
He claims he was warned not to name a stand he sponsored after his mother because of her Asian name.
But he says that is a far cry from the 1970s when he said the culture of football was such that he "felt embarrassed" when his mother turned up at a game in traditional dress.
The man who has invested well over £2m in Stanley during the past 20 years to help save the club from financial ruin paints a bleak picture of English football.
"I was in Accrington and someone threw a stone at me and said 'you Paki bastard'," Khan told BBC Sport.
"I turned round and the man who was next to him said 'hang on that's Ilyas', and there was a slightly crestfallen look on the face of the boy who had thrown the stone.
"The thing that disturbed me wasn't the stone itself. It was when I went up to speak to them, they said 'he's the owner of Paki Stanley' and I just felt that we've got so much work to do."
Khan grew up in the Lancashire mill town during the 1970s, before becoming an investment banker based in Hong Kong.
Now back in England after 20 years abroad, despite being a Chelsea season-ticket holder, he is the man responsible for keeping one of the oldest and most famously named clubs in Britain alive.
And, in spite of the personal problems he has faced, Khan believes there is less racism in the game now than there was 10 years ago.
"Do you face issues? Yes, we all have issues," he continued. "My view is that my children are growing up in a society that is far healthier and far more capable of being resilient in the face of these issues than the time I was growing up.
"Football is a tribal game - we can't on the one hand commend it and on the other hand criticise it. I think racism itself is a different issue. We now live in a society that has learned to differentiate between what is and isn't acceptable.
"I think the game is healthy and is becoming healthier. I don't think it's a big feature and, in another five to 10 years, I think racism will be even less of a feature."
The name Accrington Stanley conjures up the best traditions of the game, but Khan admits the homely image has not always fitted the reality.
"Before I was chairman of the club, we got involved in sponsoring a stand that is named after my mum, the Sophia Khan Stand.
"For a number of seasons that stand wasn't named in the way it was meant to be, and I knew why - but I wanted somebody to tell me why. They felt the positives of the association were outweighed by the negatives of people saying 'who's that Paki girl or Paki woman?'
"Today, nobody thinks twice. It could be called the extraterrestrial stand and it wouldn't make any difference, and I think that is a very positive thing.
"The one thing my mum knew was how important football was to me. She would trek to the old Stanley ground and she'd be the only Asian woman there.
"I remember once being in that crowd and my mum came up in traditional Pakistani dress. Somebody said 'that's your mum' and I denied her. I said 'it's not my mum, I'm not a Paki', and she heard it.
"She looked me in the eye, but three or four seconds later I ran after her and, in front of everybody, I gave her a big hug. It's memories like that that make you belong and she's as much a part of the club."
Despite Stanley's steady rise through the non-league structure, culminating in 2006 in regaining the Football League status given up by the club's previous incarnation in 1962, it has been a tumultuous few years at the Crown Ground, and Khan has witnessed plenty of ups and downs, on and off the pitch.
In 2009, the club risked being wound up again when it was revealed it owed over £300,000 to the tax man.
And Khan believes that the club can one day compete at a higher level.
"Do I have the ambition and can I see Accrington in the Premier League? You know people fall off their chairs but I think in a reasonable amount of time, yes," he concluded.
"I love my club and I love my time and everything I have done has been motivated by a love and a passion for my town."