The Rugby Football Union is investigating alleged homophobic and racial abuse of referee Nigel Owens during England's loss to New Zealand.
The RFU was told of the abuse by a supporter who was sitting close to where the alleged incident took place.
"The RFU will take this seriously and deal with it," Owens said.
Speaking to BBC Radio Wales, he added: "It's important to stop this from getting a bigger problem in the game.
"As a referee it's not nice to have stuff like this getting back to me and if I was telling you it doesn't get me down or doesn't affect me I'd be lying, because it does."
Owens was appointed an international referee in 2005 and announced he was gay two years later.
He has since officiated at the 2007 and 2011 World Cups, and was only the second man to referee two consecutive Heineken Cup finals - in 2008 and 2009.
Before alerting the RFU, supporter Keith Wilson wrote a letter to the Guardian newspaper stating a group of supporters shouted "nasty, foul-mouthed, racist, homophobic abuse" at the official during England's 24-21 defeat.
|Who is Nigel Owens?|
|2005: Appointed international referee|
|May 2007: Comes out publicly as gay|
|Sept 2007: World Cup debut, taking charge of Argentina v Georgia|
|Nov 2007: Named gay sports personality of the year by Stonewall|
|May 2008 & 2009: Referees Heineken Cup finals|
|Sept 2011: Referees at World Cup|
"I think whatever has been shouted must have been pretty bad - enough for this person to make this effort and stand up to these people, which he needs to commended for, and send a letter in saying he's never heard anything like it," Owens said.
"If they can prevent it then, yes, they need to be banned from the game and banned from the stadium, because we don't need louts like that in the minority in rugby or in any sport for that matter."
England head coach Stuart Lancaster said: "We applaud the stance the RFU are taking and the investigation.
"We have got to understand the pressures that people are under and be supportive. That is all we want to achieve with referees."
Owens - who attempted suicide before coming out as gay - has been cited as an inspiration for the gay and lesbian community.
He will take charge when France face Australia on Saturday but admits he has considered quitting as a result of the sometimes vicious comments made by supporters.
|England coach Stuart Lancaster:|
|"We have worked hard in rugby to get the core values of the sport ingrained and there is no place for homophobic abuse whatsoever. I applaud the stance and investigation absolutely."|
He added: "Sometimes you think 'do I really need this? Do I really need to do this job?'. It does get you down and I honestly think sometimes like saying 'enough is enough' but you've got to just get up and get on with it.
"Unfortunately you get some people who get to the stadium, who get drunk and who probably haven't even watched the game of rugby, they are just there to get drunk and shout abuse, but they are in the minority."
In a statement, the RFU said it "condemns all forms of discrimination and aims to ensure that all people, irrespective of their age, gender, ability, race, religion, ethnic origin, creed, colour, nationality, social status or sexual orientation, have a genuine and equal opportunity to enjoy rugby union in whatever form, on or off the pitch, at all levels and in all roles".