A UK exit from the European Union could mean the UK misses out on up to 5.6% of GDP growth by 2019, the IMF has warned.
Brexit is the "largest near-term risk" to the UK economy, the IMF said in its annual UK economic outlook.
It added that the net economic effects would probably be "negative and substantial".
But economists for the Brexit campaign said the consensus that a UK exit would be bad for the economy was "based on flawed EU-centric models".
Meanwhile, Russian President Vladimir Putin has accused UK Prime Minister David Cameron of holding the referendum to "blackmail" and "scare" Europe, and France's economy minister says Britain would become as significant as Guernsey if it voted to leave.
'Substantial Brexit costs'
There are five days to go until the UK votes on whether to stay in or leave the European Union on 23 June.
In other developments:
- Field Marshall Lord Guthrie, a former Chief of Defence Staff, has switched sides to Leave, saying he is worried by the prospect of "a European army"
- The Times newspaper has said it supports Remain
- The Remain and Leave groups have suspended their campaigning until Sunday following MP Jo Cox's death
The IMF said that under its least adverse scenario for Brexit, by 2019 UK GDP would be 1.4% below what it would be should the UK vote to stay in the EU.
Its most adverse Brexit scenario predicts 2019 growth 5.6% below what it would otherwise have been, and also a drop in GDP in 2017 of 0.8%, which an IMF official described as a "recession".
Under this scenario the UK would return to GDP growth of 2.9% in 2021. But the UK would have missed out on 4.5% of growth by then, it said.
Following a Brexit, the IMF said the UK would have to negotiate new trade terms with the EU if it wanted to stay in the single market.
If not, the UK could rely on World Trade Organization rules, but this would "significantly raise trade barriers", the IMF said.
A long period of uncertainty would also lead to market volatility and discourage investment, it said.
Concerns about the economic effects of a Brexit may have already begun to affect markets such as the housing sector, the IMF added.
"On balance, the net economic costs of an exit are likely negative and substantial," it said.
By Andrew Walker, BBC News
What do these losses predicted by the IMF mean?
If the analysis is right then we would be worse off than we would be if we were to remain in the EU.
In the adverse short term scenario we would for a while be worse off than we are now, but even in that situation growth would resume and take average living standards to a level higher than today.
It's just that they think if we remained they would be even higher.
Which raises (at least) two questions. Is that missed opportunity to be better off a price worth paying for whatever other advantages you might think leaving would bring?
And the other question, of course - is the analysis right? Most economists would say, broadly speaking, "Yes, it is".
But there are some serious members of the profession who say no; ultimately we would do better economically outside.
And many campaigners say this is a profession that didn't see the financial crisis coming.
'Doom and gloom'
Campaign group Economists for Brexit said the widespread view that Brexit will be bad for the economy in the long term because of bad trade effects and in the short term because of uncertainty was "based on flawed EU-centric models".
Campaign co-chairman Patrick Minford said: "The IMF report, like the Treasury's, uses flawed models and makes wrong, deceitful assumptions to project doom and gloom from Brexit whereas with solidly based models and assumptions Brexit gives the UK more growth and better living standards."
Mr Minford said the best outcome would be for the UK to use World Trade Organization rules "under which the UK would leave the EU, freed from both Single Market regulations and from free EU movement of people."
"Under this option the UK would get rid of EU trade barriers on the rest of the world, so ridding our consumers of a huge burden of EU protectionism of food and manufacturing; consumer prices would fall 8% and GDP would rise 4% in the long term due to the dynamic response of the economy to lower costs."
Matthew Elliot, chief executive of Vote Leave, said: "If we Vote Leave we can create 300,000 jobs by doing trade deals with fast growing economies across the globe."
Vote Leave said that the IMF analysis was "partial" and did not provide any analysis of a new bilateral free trade agreement between the UK and the EU.
It added that "the real risk to the economy is staying tied to the failing eurozone".
Speaking to reporters in St Petersburg, Mr Putin said of Mr Cameron: "Why has he set up this referendum? To blackmail Europe? Or to scare it? What is the purpose if he himself is against" Britain leaving the EU.
He declined to give his own view on Brexit, saying: "It's not our business - it is the business of the British people. I have my opinion on the subject but I will not speak about it in advance."
He also said it would be "inappropriate" to try to predict the result.
Meanwhile, French minister Emmanuel Macron told Le Monde newspaper: "Leaving the EU would mean the 'Guernseyfication' of the UK, which would then be a little country on the world scale.
"It would isolate itself and become a trading post and arbitration place at Europe's border," he said.