Maintaining the UK's membership of the EU's single market could add an extra 4% to its economy, according to the Institute for Fiscal Studies (IFS).
The think tank weighed up the benefits of staying in the single market compared with membership of the World Trade Organization alone.
The UK voted to leave the EU in the June referendum.
Paul Johnson, IFS director, said there was a big difference between access and membership of the single market.
"We've heard a lot of people saying of course we'll have access if we leave the single market union.
"Broadly speaking, yes, we will, as every other country in the world does. You can export into the EU wherever you are from, but there are different sorts of barriers to doing so."
The IFS report said access to the single market was "virtually meaningless as a concept" because "any country in the World Trade Organization - from Afghanistan to Zimbabwe - had 'access' to the EU as an export destination".
The IFS report argued that the special advantage of being an EU member was that its single market reduced or eliminated barriers to trading in services, such as the need for licences or other regulations.
The IFS said that the absence of trade barriers for services was far more important than removing tariffs on the trade in goods between EU members, such as customs checks and import taxes.
It said that while leaving the EU would free the UK from having to make a budgetary contribution of £8bn, loss of trade could depress tax receipts by a larger amount.
It found new trade deals would be unlikely to make up for lost EU trade, which accounts for 44% of British exports and 39% of service exports.
The government has yet to start negotiating the UK's departure.
The IFS issued stark warnings over the impact of Brexit ahead of the EU referendum, which have made some question its views.
Mr Johnson said he hoped the IFS was proved wrong.
"We wait to see what the economic consequences are going to be, but we've already seen the Bank of England significantly reduce its predictions of growth over the next couple of years and increase its view of where unemployment will be," he added.
The report said UK services would be particularly vulnerable if the government were unable or unwilling to negotiate a replacement deal and become a member of the European Economic Area (EEA), like Norway.
The IFS report added that financial services, which generate 8% of the UK's economic output, might suffer in particular if a final Brexit deal meant they lost their so-called "passporting rights" that allows them to be sold directly to EU customers and businesses.
"To maintain these rights would likely require membership of the European Economic Area (EEA)," it said.
"But that would come at the potentially considerable cost of submitting to future regulations designed in the EU without input from the UK. The UK may have to make some very difficult choices between the benefits from passporting and the costs of submitting to external imposed regulation."
The IFS explored the possibility of the UK signing its own free trade agreement with the EU, or simply adopting WTO rules.
But in both cases, the report argued, such deals would still involve tariffs or other barriers to free trade in goods and services.