Iain Duncan Smith says the "spin and smear" tactics being used by the campaign to stay in the EU risk long-term damage to the government.
The EU exit campaigner accused the other side - backed by most of his cabinet colleagues including David Cameron - of making "desperate" claims.
Downing Street said it was "important to explain what leaving the EU means to the British people".
Mr Cameron has previously called for a respectful debate.
Britain will vote on whether to leave or remain in the EU on 23 June.
The government's official position is in support of staying in a reformed EU under the "special status" Mr Cameron says he secured for the UK in his EU renegotiation.
'Smears and threats'
But Cabinet ministers have been allowed to campaign on either side of the debate and five of them, including Mr Duncan Smith, the work and pensions secretary, are pushing for an EU exit.
About half of the Conservative Party's MPs - most notably London mayor Boris Johnson - are also supporting a vote to leave.
Mr Cameron and other leading figures in the Remain campaign have warned that to quit the EU would be a "leap in the dark" and endanger Britain's economy and national security.
The prime minister has warned that the Calais refugee camp known as the "jungle" could relocate to Dover if Britain left.
And the government has published an analysis of the UK's options if it left the EU - suggesting they would all be worse for the economy than staying in.
Writing in the Daily Mail, Mr Duncan Smith, a former leader of the Conservative Party, said the Remain camp was basing its campaign on "spin, smears and threats" to try to "bully" Britons into voting to stay.
"The acrimonious manner in which all this has been conducted is troubling, and will I fear have consequences long beyond 23 June," he said.
"After all, such desperate and unsubstantiated claims are now being made that they begin to damage the very integrity of those who make them in the eyes of the public."
BBC political correspondent Alex Forsyth said this was the strongest intervention yet from Mr Duncan Smith and would be seen as implicit criticism of the prime minister and his allies.
Mr Duncan Smith said a "series of highly questionable dossiers" had been produced by the Remain campaign, "threatening almost biblical consequences if we dare to consider a future outside the EU".
He said the campaign had shifted the debate away from the merits of Mr Cameron's deal struck in Brussels to "scaremongering" about whether the UK can stand alone.
"The question being asked seems to be whether we could cope on our own at all. Why they would seek to present our country, and themselves, as so weak is beyond me."
It was "vital", he said, that the debate was conducted in "a respectful manner, where we maturely interrogate the issues".
His fellow Leave campaigner, and former defence secretary, Liam Fox told BBC Radio 4's Today Programme the sour tone of the referendum debate risked making it harder for Conservative MPs to unite again after the vote.
He said: "The Conservative Party is in a position where we will have to come back together to govern the country after the 24th of June.
"And the difficulty we will face is that if the arguments have been ramped up too much in that initial period that will become harder for us."
UKIP leader Nigel Farage launched a strongly-worded attack on Mr Cameron and the other Remain campaigners.
"I got into politics because I did not trust our politicians, I now despise them," he said on his regular LBC Radio phone-in.
"Listening to the lies and arguments that have been made over the course of the last week, 'if we vote to leave everyone in Spain will be rounded up and put on trains and sent back to Britain, millions of jobs will be lost', this sort of apocalyptic vision that we are getting from Cameron and the others.
"The only thing they haven't told us, which they may at some point, is that if we vote to leave a plague of locusts will descend on the country. It is just extraordinary."
He also suggested the government could "just ignore" a vote to leave the EU and try to restart negotiations with EU leaders, saying he did not trust them to "carry out the will of the people".
The row comes a day after Mr Cameron held a summit with French President Francois Hollande in northern France. Mr Hollande said he wanted the UK to stay in the EU - and warned there would be "consequences" for immigration and the economy if it left.
French economy minister Emmanuel Macron earlier suggested his country could end UK border controls in Calais, dismissed as "scaremongering" by Leave campaigners.
Mr Cameron, who said the UK would be "better off... stronger" inside a "reformed" EU, announced £17m to help the French authorities deal with migrants attempting to get into the UK from Calais and Dunkirk.