A Brexit involving staying in the customs union would leave the UK £80bn worse off a year than if it had remained in the EU, a report says.
The National Institute of Economic and Social Research (NIESR) said tax income would fall by £13bn a year.
The research was commissioned by the People's Vote campaign for a further EU referendum.
Some pro-Brexit economists have argued the UK has "nothing to fear" from leaving the EU customs union.
If the UK agreed to remain in a customs union with the EU after leaving, the two sides would not put import taxes - tariffs - on goods they trade with each other.
In addition, they would agree to apply the same set of tariffs to goods entering their market from non-EU countries.
The NIESR study compared what would happen to the UK economy if it stayed in a customs union but left the single market, compared to if the UK remained an EU member.
It predicted long-term economic growth in both scenarios, but said GDP growth would be 3.1% lower a decade after exit from the EU.
NIESR said remaining in a customs union would not offset the negative impact of trade frictions the UK would face outside the single market, particularly on the service sector.
Countries in the EU's single market share rules in a wide range of areas in a bid to make it easier for firms to trade goods and services with each other.
It added that leaving the single market would also slow economic growth because of reduced foreign investment and productivity.
Labour policy on customs union
Labour plans to remain in a permanent customs union with the EU after Brexit, a demand it has been making in cross-party talks with the government.
The party says this is the only way to keep trade flowing freely and protect jobs, and help avoid the need for a hard border in Northern Ireland.
Labour MP and People's Vote supporter Rachel Reeves said a customs union exit would not be as damaging as a no-deal exit or the PM's current deal.
"But it is a mistake to regard it as a soft option, let alone a fix-all for a political crisis that has its roots in promises that can't be met," she added.
In a report last November, the Economists for Free Trade group said the UK had "nothing to fear" from a "clean break" with the EU involving both a departure from the single market and the customs union.
It argued that using World Trade Organisation rules could give an £80bn boost to the tax base and cut prices by 8%.