He told BBC News: "They take direct funding from the European Union.
"So, once again, it's the same old game. It's taxpayers money being used to tell us what we should think and what we should do. And frankly the scale of this now is outrageous.
"The government and all their friends - taxpayer-funded friends - are frankly cheating in this referendum by selling this constant negative message."
IFS director Paul Johnson told BBC Radio 4's Today programme, his organisation got "about 10% of our income from something called the European Research Council, which is an independent arms-length body which funds world-class academic research".
But he added: "For the last 30 years, the IFS has really built its reputation on the independence and integrity of our work, and actually there is no sum of money from anywhere in the world which would influence what we said".
Prime Minister David Cameron said the IFS was the "the gold standard in independent, impartial economic forecasting and commentary in our country".
The IFS assessment comes as a dozen former senior military officers have warned that the EU's policies are undermining the UK's combat effectiveness.
Speaking out in favour of Britain leaving the EU, they said Nato, and not the EU, should remain the cornerstone of Europe's defence.
More than 300 historians, meanwhile, have written to The Guardian saying Britain has an "irreplaceable role to play in Europe" and warning against "condemning ourselves to irrelevance" by leaving.
The IFS said a vote to leave could result in a £20bn to £40bn hit to the public finances in 2019/20, if GDP was 2.1% to 3.5% lower over the period, as predicted by the National Institute of Economic and Social Research (NIESR).
Paul Johnson, IFS director and an author of the report, said: "Getting to budget balance from there, as the government desires, would require an additional year or two of austerity at current rates of spending cuts."
Trade and economy
How trade and the UK's economy are affected by membership of the EU.
What the leave and remain sides are saying about trade and the economy in the #EUref campaign
About half of UK overseas trade is conducted with the EU
The EU single market allows the free movement of goods, services, capital and workers
Trade negotiations with other parts of the world are conducted by the EU, not individual member states
UK companies would be freed from the burden of EU regulation
Trade with EU countries would continue because we import more from them than we export to them
Britain would be able to negotiate its own trade deals with other countries
Brexit would cause an economic shock and growth would be slower
As a share of exports Britain is more dependent on the rest of the EU than they are on us
The UK would still have to apply EU rules to retain access to the single market
EU referendum issues guide: Explore the argumentshttp://www.bbc.co.uk/news/uk-politics-eu-referendum-36027205http://www.bbc.co.uk/news/special/2016/newsspec_13606/content/iframe/english/index.inc.app.htmlExplore all the issuesChoose an issue:What both sides are sayingAll issuesMain viewsShare this pageEU referendum issues guideWhat the leave and remain sides are saying in the #EUref campaign
The IFS said the UK could use its contribution to the EU - estimated at £8bn a year - to help shore up its finances if it voted for Brexit.
However, it said this could be overshadowed by the negative impact on the UK economy, with a 0.6% fall in national income offsetting the benefits.
Mr Johnson said: "Leaving the EU would give us an immediate £8bn boost to the public finances, but the overwhelming consensus is that the economy would be smaller than otherwise following Brexit.
"If the economy is just a few percentage points smaller than it otherwise would have been, we will be a bit worse off."
Analysis by Kamal Ahmed, BBC economics editor
The IFS's report on the impact on the public finances of Brexit is different from the myriad of economic reports flowing from organisations as diverse as the International Monetary Fund and Economists for Brexit.
Rather than do its own economic modelling, the IFS has taken the mid-point of most of the major reports on the possible referendum impact and built its analysis from that.
It judges that mid-point to be closest to the National Institute of Economic and Social Research study which suggested that Brexit could leave the UK economy between 2% and 3.5% smaller than under a remain scenario.
From that the IFS uses a simple economic model that suggests that for every 1% decline in GDP, the government needs to raise an extra £14bn due to lower tax receipts.
Of course, if the economic impact of Brexit is - in the long run - positive, as some economists argue, then the impact on the public finances would also be positive.
The UK "could perfectly reasonably decide that we are willing to pay a bit of a price for leaving the EU and regaining some sovereignty and control over immigration and so on", Mr Johnson said.
"That there would be some price though, I think is now almost beyond doubt."
But Vote Leave's John Redwood said the IFS was in the "cosy establishment" and the UK was better off out of the EU.
Speaking on BBC Radio 4's Today programme, he disputed the forecasts used in the report, and said savings made from quitting the EU would mean austerity would not be needed.
Patrick Minford, co-chairman of a separate group, Economists for Brexit, said that the IFS analysis acknowledged that the free trade approach recommended by his organisation "would be the best option for the UK following an exit from the EU".
Gerard Lyons, his co-chairman, added: "The UK needs to break away from the short-termism and the groupthink that had dominated UK policy making and embrace Brexit as this is the best way to position the economy for longer-term future growth."