David Cameron must accept the failure of the government's manifesto pledge to reduce migration into the UK, two leading Vote Leave MPs have said.
Michael Gove and Boris Johnson said the pledge was "corrosive of public trust" while Britain remained in the EU.
But Number 10 said their claim was an "attempt to distract" from the fact that an EU exit would be "disastrous".
It comes amid growing Tory turmoil over the EU, with some MPs threatening a post-referendum leadership challenge.
Meanwhile, a survey of economists has suggested that nine in 10 of those who responded believe leaving the EU would be damaging.
There are less than four weeks to go until the UK decides whether to stay in or leave the European Union on 23 June.
Immigration is one of the key battlegrounds in the referendum debate - and it is the focus of an open letter to Mr Cameron, published in the Sunday Times, by Mr Gove and Mr Johnson, who say the government's pledge to get migration below 100,000 was unachievable.
Net migration - the difference between the number of people coming to the UK for at least a year and those leaving - rose to 333,000 in 2015, according to Office for National Statistics estimates.
The figure for EU-only net migration was 184,000, equalling its record high, and 188,000 for non-EU.
Mr Johnson and Mr Gove said: "Voters were promised repeatedly at elections that net migration could be cut to tens of thousands.
"This promise is plainly not achievable as long as the UK is a member of the EU and the failure to keep it is corrosive of public trust in politics."
They also said they were "particularly concerned about the impact of free movement in the future on public services".
"Class sizes will raise and waiting lists will lengthen if we don't tackle free movement," they wrote.
In response to the letter, Number 10 said: "This is a transparent attempt to distract from the fact that the overwhelming majority of economists and businesses believe leaving the single market would be disastrous for jobs, prices and opportunities for people."
By John Pienaar, deputy political editor
The Brexiteers point is that EU open borders make immigration control impossible.
Its effect, though, is to accelerate a descent into internecine warfare which now threatens to make the Conservatives ungovernable if the referendum ends in anything but a decisive victory for the Remain campaign.
So bitter has the conflict become, so taut the tension between the rival factions, that angry Eurosceptic Tories talk privately of challenging the prime minister's position even if Britain votes to stay inside the European Union.
The row over the migration target comes as the Conservative in-fighting over the EU referendum intensifies.
David Cameron and most of his cabinet are campaigning for a vote to stay in the EU, but about half of his MPs support an exit.
Conservative Leave campaigner Andrew Bridgen told 5 live's Pienaar's Politics that as many as 50 Tory MPs were ready to back a vote of no-confidence in the prime minister over his handling of the debate.
"If there's a small Remain vote... I think there probably would be 50 colleagues who'd be very dissatisfied with the prime minister's performance," the MP said.
One MP, Nadine Dorries, said she had already sent a letter to Graham Brady, chairman of the 1922 committee of Tory MPs, calling for a vote of no-confidence in Mr Cameron if Remain narrowly won.
Speaking on ITV's Peston on Sunday, she accused David Cameron of "outright lies" over the EU and said trust in him and George Osborne, the pro-Remain chancellor, "has been absolutely shattered".
But former cabinet ministers and Leave campaigners Liam Fox and Iain Duncan Smith played down the prospect of a coup, and insisted they would want Mr Cameron to stay on as leader.
Speaking on the BBC's Andrew Marr show Mr Fox said the referendum should not be turned into an internal Tory party debate.
Also on Marr, former Labour Prime Minister Tony Blair, who backs Remain, warned that leaving the EU would not solve migration problems.
"If you actually break down the figures on EU migration, many of these people come in on short-term contracts, and then will go back out again. Many of these people work in vital public services.
"And we also get the benefit. The reason we can travel around Europe, without restrictions, is because of the freedom of movement of people," he said.
Ex-Lib Dem Business Secretary Vince Cable said public trust had been corroded by the government's failure to cut immigration, and said it was a "foolish" target for the Conservatives to have set.
"You can not directly control, in a market economy, levels of net migration. It's not just free movement of labour in Europe, we can't control emigration," he told Radio 4's The World This Weekend.
Former Conservative Prime Minister Sir John Major criticised the Vote Leave campaign in the Mail on Sunday for the "inaccuracies and falsehoods they are peddling to the British people".
He said the Leave campaign's focus on "raising fears" rather than "setting out facts" on immigration was "distasteful".