Post-Brexit farm subsidies would have to be heavily curtailed or abolished to mix free trade with meeting green targets, according to a think tank.
The Chatham House report said a market-orientated model, such as the one in New Zealand, would lower food prices and increase productivity.
It warned many smaller farms would probably fold but said the scale of job losses could be "exaggerated".
The UK government said it would continue to support farmers.
The report predicts livestock farming, which makes up most of the Scottish sector, would be particularly vulnerable in this scenario with rural upland communities "likely to suffer the most commercially".
NFU Scotland said free trade presented "huge risks to domestic agriculture".
Researchers considered the promises made on farm policy during the EU referendum and the 2017 election campaign, then reflected on the UK's "traditional reformist and liberal position".
It said: "For the UK, only a market-orientated model - aligned and integrated with a more effective commitment to the environment and climate change mitigation - would enable the country to benefit from free trade while keeping the government's promise to improve the environment for the next generation."
The model operates in New Zealand, where farmers have not received subsidies since 1984 and production has become more efficient.
Larger scale farms have created a boom in exports but the transition to the current system was not done without significant pain.
The Chatham House report suggested some businesses would fold or substantially change but said the impact of job losses "can be exaggerated".
It added: "Jobs will be lost and moved to other sectors: by way of comparison, the manufacturing sector has typically shed 100,000 jobs per year - more than a fifth of the current agricultural workforce.
"Similarly, the savings possible from reducing agricultural subsidies are large enough to fund significant worker transition costs."
Farmers in the UK currently operate a "decoupled subsidy" system which means payments are not linked to levels of production.
NFU Scotland President Andrew McCornick said: "There are huge risks to domestic agriculture, the food and drinks sectors it underpins and the public interest from a sudden and significant shift towards free trade and unilateral action on the environment.
"Exposing Scottish and UK agriculture to cheap food imports merely exports responsibility for animal health and welfare and the environment, as well as threatening jobs in our fast-growing and increasingly important food and drinks sectors.
"NFU Scotland is very clear that farming in Scotland and the UK does face inevitable change as a consequence of any Brexit scenario.
"However, as with other leading elements of our economy, it is vital that change is controlled rather than cliff edge and chaotic, and pace of change is determined by how negotiations and trade deals are to be put in place."
The report suggested that if the UK did decide to adopt a market-orientated agricultural model it could trigger a shift in attitudes globally, particularly with the World Trade Organisation (WTO) and the G20.
But it said there seems "no real prospect of substantial reform of the EU agricultural model in the next decade".
A spokesperson for the UK government's Department for Environment, Food and Rural Affairs (Defra) said: "Leaving the EU is a golden opportunity to introduce a new approach that supports our farmers to continue producing high quality food while strengthening environmental protections.
"Outside the EU we can move away from an over-bureaucratic and unresponsive system of subsidy and instead design an approach that helps boost farm productivity and innovation.
"We will match the £3bn that farmers currently receive in support from the CAP until 2022, and we intend to go on supporting farmers for many more years to come where the environmental benefits of that spending are clear."
The Scottish government said the report backed up its claim that Brexit was the biggest threat to rural Scotland.
A spokesperson said: "This paper demonstrates that Scotland's agricultural sector stands to be disproportionately impacted.
"As we have repeatedly made clear, it's vital that our farming and food production sector gets clarity and certainty from the UK government about very real and practical concerns the sector has over issues like future funding and access to labour.
"It is essential for full decision-making over farm policy to rest here so we can consider how best to support communities and maintain population in remote areas."