Sir Eric Pickles has accused a lobby group campaigning for the UK to quit the EU of a "strategy of intimidation".
Vote Leave orchestrated a protest by two students at Tuesday's CBI business conference, registering a fake company to enable them to gain entry.
The ex-minister said people wanted more than "silly stunts and nasty tactics".
He has urged the Electoral Commission not to name Vote Leave as the main "Out" group in the future referendum, but it said it was "desperate stuff".
Vote Leave and another organisation, Leave.eu, are vying to be designated as the lead anti-EU campaign group in the forthcoming in-out referendum, to be held before the end of 2017.
The decision is crucial, as whichever group is chosen will have access to millions of pounds in public funding to campaign.
A group known as Students for Britain, which is affiliated to Vote Leave, briefly interrupted David Cameron's speech to the conference of business leaders, heckling the prime minister and waving banners claiming the CBI had become the "voice of Brussels".
It subsequently emerged that Vote Leave had set up a fictitious company to allow the two students to enter a venue - a tactic they attempted to justified by saying the CBI was misrepresenting the views of business by backing continued EU membership.
'Strategy of intimidation'
Sir Eric, who left the cabinet at May's election, said believing the UK would be better off outside the EU was a "respectable position".
But he said Monday's episode and subsequent comments by Vote Leave suggested it was not fit to lead one side of the argument in the referendum - described Mr Cameron as the most important decision in most voters' lifetime.
"I believe the actions of Vote Leave in disrupting the CBI conference and declaring a strategy of intimidation and protest disqualify Vote Leave from being a designated lead campaigner," he wrote.
Referring to guidance on the watchdog's website, suggesting a campaign's interaction with its opponents would be taken into account when determining whether it was suitable to be designated, he added: "Surely an open campaign of nasty tactics against other campaigners must be taken into consideration and runs counter to your organisation's stated objectives to ensure that the referendum 'should be well-run and produce results that are accepted'."
Vote Leave told the Guardian the intervention showed the "In" campaign was "panicking".
"The [prime minister]'s letter to the EU has bombed, so now they are rolling out former ministers to deflect from their own failure," a spokesman said.
"The establishment does well out of the status quo, and that's why they are wanting to stop campaigns like ours."
But the founder of Leave.eu, businessman, Arron Banks, said he agreed with the MP.
"If we are to gain the trust of the British people in this campaign, then provocative stunts and schoolboy politics is not the answer," he said.
"This is a serious debate about a very serious matter, and we now have to conduct ourselves accordingly stating that this is going to 'get nasty' helps no-one."
The watchdog will not designate lead campaigners until the law paving the way for the referendum is approved by Parliament.
Read more on this story