Former Conservative Defence Secretary Liam Fox says people calling for an early EU referendum are trying to "limit the level of debate".
Mr Fox said some of those calling for the vote to be brought forward had "ulterior motives which are not entirely honourable".
The in/out referendum is due in 2017.
In the meantime, Prime Minister David Cameron is trying to renegotiate the UK's relationship with the EU, saying the "status quo" is unacceptable.
Speaking on BBC Radio 4's The World at One, Mr Fox said he would vote to leave unless a "looser" relationship can be achieved and urged Mr Cameron to take his time so the EU debate "does not keep coming back to haunt us time and time again".
He said: "If that means taking a bit longer to get it right rather than going for a minimal change just for the sake of political expediency, I think that would be the smart thing to do."
There have been calls - including from Labour leadership contender Andy Burnham - for the vote to be held in 2016, and Mr Cameron has previously said he would be "delighted" for it to take place earlier.
But Mr Fox said: "Those who are urging him to go for a referendum in 2016 I think have ulterior motives."
Asked what those motives were, he said: "They are afraid that if we have a very full debate then some of the real unacceptable issues in Europe at the present time will become all the more clearly seen by the British public."