Civil servants told David Cameron in 2012 that it was "impossible" for the government to meet its flagship immigration pledge, the PM's former director of strategy has claimed.
Steve Hilton said Mr Cameron was told "explicitly and directly" that EU free movement rules meant net migration could not be reduced below 100,000.
The target featured in the Tories' election manifesto last year.
But the prime minister said Mr Hilton was "simply not right".
And Downing Street questioned why Mr Hilton - who is backing a Leave vote - had chosen to make his comments for the first time days before Thursday's referendum.
Net migration - the difference between the number of people coming to the UK for at least a year and those leaving - stood at 333,000 in 2015, according to the latest figures.
It has risen by more than a third since Mr Cameron came to power in 2010 with the aim of reducing it to tens of thousands a year.
Leave campaigners have repeatedly said that exiting the EU is the only way the UK can control its borders.
In a Daily Mail article on Tuesday, Mr Hilton recalled meetings towards the end of his time at Downing Street - chaired by Mr Cameron - where ministers and civil servants were "working hard to try to deliver the government's commitment" on migration.
He said: "The news was not good. We were way off target; indeed the numbers were going in the wrong direction. We explored various policy options - and I'm sure that process continued after I left the government in May 2012.
"But I recall very clearly one of the points that was made to us by the expert officials in the room. We were told, directly and explicitly, that it was impossible for the government to meet its immigration target as long as we remained members of the EU, which of course insists on the free movement of people within it."
Mr Hilton, who was formerly one of the prime minister's closest aides before he left Downing Street in 2012, was a driving force behind Mr Cameron's flagship "Big Society" project.
He said that when he saw the commitment in the 2015 manifesto he assumed Mr Cameron was either certain he could "negotiate a solution within the EU" or was assuming the UK would vote to leave.
Referring to leading Leave campaigners Michael Gove and Boris Johnson, he added: "For the government to continue to make the promise today, after no negotiated solution was achieved and while campaigning to stay, is, I think, what Gove and Johnson meant when they described this as corrosive of trust in politics."
'Fibs and nonsense'
In a BBC interview, he added: "This question of immigration isn't about whether or not you want lower immigration, or higher immigration.
"It's about whether the government that people elect in this country can control it.
"And when I was working in government, we were told, by officials, that as long as we were in the EU we couldn't control the level of immigration. And that's why I think we need to leave the EU."
Mr Gove told the BBC that he wanted the British people, via Parliament, to be in charge of deciding who came into the country although he accepted that "full control" would not be possible until 2020 - the date earmarked by the Leave campaign for exiting the EU.
Asked about the manifesto pledge, Mr Gove rejected claims it had been made in bad faith and said he had "hoped and prayed" that Mr Cameron would have been able to secure fundamental reform of free movement rules.
But he added: "Making promises and then saddling yourself with a political union that means you cannot deliver those promises does not contribute to an atmosphere of trust and confidence in politics".
The prime minister has insisted migration can be controlled from inside the EU by limiting access to benefits and other "pull factors" attracting people to the UK.
He said his former aide's recollection of events was "simply not right".
"When Steve Hilton left Downing Street in 2012, net immigration had actually fallen quite substantially and had got down, just after he left, to about 154,000 - not far away from the ambition I set, " he told ITV's Lorraine.
Labour's pro-EU campaign leader Alan Johnson, told BBC Breakfast most immigration was "coming from outside the EU".
He added: "This is not a referendum about immigration, and is still less [about] David Cameron's rather vainglorious promise.
"If people vote to leave on the basis of immigration, I'm afraid they'll find that they're going to be in exactly the same situation, because the issues are complex and leaving the EU won't resolve them. Indeed, in terms of legal immigration and immigration from outside the EU, it will make our situation worse."
A Downing Street spokesman said the Conservatives' 2015 manifesto had made it clear reducing net migration to the tens of thousands remained a government ambition.
He said that since 2008 there had been an extraordinary period when the British economy was growing while many other EU countries were struggling economically.