The leader of Plaid Cymru Leanne Wood has accused the Welsh secretary of peddling "myths and smears", after he claimed members of her party had broken the law over migration into Welsh-speaking communities.
Alun Cairns made the comments on the BBC's Question Time programme.
During the fractious exchange with Ms Wood he said it "wasn't so long ago" that "cottages were being burned down".
Ms Wood called for the "outrageous" comments to be withdrawn.
The Wales Office declined to respond to her criticism of the secretary of state.
Plaid Cymru leader in Westminster Hywel Williams said Mr Cairns should consider his position.
Speaking to BBC Wales, he said: "I think he should apologise and consider whether he really is up to the job.
"I'm not calling on him to resign. It's a matter for him - is he really up to it?"
Labour AM David Rees also called for Mr Cairns to apologise.
He tweeted that the Welsh Secretary should "withdraw" his allegation.
"Unacceptable comments from any politician," he said.
Mr Cairns had told the programme, as recorded in Neath on Thursday evening, that Ms Wood "seems to be very open to immigration into the UK but if it goes into Welsh-speaking communities then she's got something very serious to say".
Ms Wood challenged him, saying: "What are you talking about? Give me a quote, quote me."
The cabinet minister continued, saying "when there's a migration into Welsh-speaking communities, the integration in those communities, and I'm a passionate Welsh speaker... isn't necessarily as it is and many of your members have taken direct action in the past, many have broken the law to that effect".
"I would hope that you'd condemn them bearing in mind the standpoint you're taking now," he said.
Ms Wood protested: "Who are you talking about? What are you talking about?"
The Conservative MP for Vale of Glamorgan replied that there were "communities in Wales where there are nationalist activists that take direct action against people who come in".
Mr Cairns added: "It wasn't so long ago that some of the cottages were being burned down."
Ms Wood said that was "nothing to do with Plaid Cymru", calling the comments "outrageous".
On Friday, Ms Wood accused the Welsh secretary of "distorting history with smears and insinuation", saying he was "in denial over the divisive rhetoric coming from numerous voices in his own Tory party".
"The secretary of state for Wales is guilty of peddling the exact sort of myths and smears which breed hostility and turn people against each other," she said.
"Such comments and insinuations are not what you would expect from a government minister. If he had any integrity, he would withdraw what he said."
Defending her party, Ms Wood said that "Plaid Cymru is an outward-looking and internationalist party" and that since the referendum "we have challenged the Westminster establishment's lurch to the right as well as growing casual racism".
Over a period of 12 years from 1979, what became known as the Meibion Glyndwr Welsh holiday homes arson campaign saw 228 attacks.
One person was convicted of sending fire bombs through the post in 1993.
The campaign came to an end in the early 1990s.