Millwall captain Paul Robinson says the violence that marred their 2-0 FA Cup semi-final defeat by Wigan on Saturday will further tarnish the club's image.
Fourteen people were arrested for fighting in the stands at Wembley.
Robinson told BBC Late Kick Off London & the South East: "There are a lot of good people who have done a lot of hard work to repair the image of our club.
"It's taken a number of years to get to the level we have. It was disappointing to see that work put in jeopardy."
The south London club have tried to distance themselves from an element of their support who were involved in hooliganism in the 1970s and 1980s.
Back in 1985, an FA Cup sixth-round tie at Luton Town saw seats torn out and a mass pitch invasion, which led to a confrontation with police and became known as the Kenilworth Road riot.
However, recent history has also seen supporter unrest at matches involving the Lions.
In 2011, a home game against Middlesbrough was almost called off after missiles were thrown at Boro keeper Jason Steele and a linesman. Two years earlier, hundreds of fans were involved in fighting at a Carling Cup tie against West Ham.
Robinson said the trouble at Wembley was "frustrating to see".
The 31-year-old defender continued: "We felt the community and the club as a whole has gone a way to improving the image [of the club].
"It is a shame that Saturday has really affected that."
Robinson has been sidelined through injury since September, but watched his team-mates at Wembley.
"I travelled with the team and sat behind the dugout and the substitutes," he said.
"It [the violence] was distressing to see as I had family and friends very close to that area.
"A couple were right on the fringes of the trouble that went on and you are obviously concerned for their safety and welfare.
"I'm sure many other players were as well, even though they were concentrating on trying to beat Wigan at the time."