It was the 1970s in the United States. The country was in the grip of an epidemic: a crime epidemic. From the East Coast to the West, drug use, violence and property crime were at an all-time high. In the state of New Jersey, the police decided to do something about it.

They figured that the best way to deter young men from getting involved was by example. Surely if they saw what life is like in prison, they’d be on their best behaviour.

The authorities set up a series of visits to Rahway State Prison, in which juvenile delinquents were introduced to “lifers” serving time for offenses ranging from bomb building to selling information to the Mafia.

The visits were designed to be as traumatic as possible; inmates swore, screamed and regaled their pupils with graphic accounts of life on the inside – including embellished tales of rape, suicide and murder. The programme became known as “Scared Straight”.

Before long it was being copied across the globe, including in over 30 states and Australia, the United Kingdom, Germany and Canada.

The introduction of seatbelts had resulted in either no change or a net increase in road accident deaths

It was popular, it was intuitive and it was cheap. There was just one problem: no one ever checked if it actually worked. In 1982, a follow-up study of the young subjects revealed the truth: not only did the programme fail to reduce their likelihood of turning to crime, it actually increased it.

Scared Straight is by no means the only policy which has backfired. Take seatbelts. Before they became mandatory back in the 1980s, many expected them to save countless lives – after all, how could safety harnesses possibly be harmful? Some suggested they’d save as many as 1,000 people in their first year.

When they were finally introduced, the number of cyclists killed on the roads is estimated to have risen by 40%, while pedestrian deaths increased by one in five. A subsequent analysis of the legislation in 18 countries found that the introduction of seatbelts had resulted in either no change or a net increase in road accident deaths. The counter-intuitive effect has been attributed to “risk compensation”; people typically change their behaviour in response to the perceived level of risk. When the risk is low, people are more reckless.  

“Sometimes what seems like a fairly good plan may become problematic due to factors you’re not aware of,” says Michael Hallsworth, who works at the UK government’s “Nudge Unit” – but more on that later.

In the real world, our actions are a minefield of hidden motivations and psychological quirks. “We used to think that people take into account all of the available information and then weigh up the costs and benefits of different options. And there’s increasing evidence from the last 40 years that that’s not correct,” says Hallsworth.

Perhaps the visits to Rahway made the delinquents feel that ending up in prison was inevitable, or merely desensitised them to its horrors. Worse, the programme may have helped to romanticise the “lifers” or set up the harsh realities of incarceration as a challenge.

Many nudges take merely minutes, yet their effects may last for decades

But it’s not all bad news. As it happens, there may be a way to use these idiosyncrasies to the public advantage. Enter “wise psychological interventions”, popularly known as “nudges”; techniques to bump people’s decision-making in the right direction by unlocking how we really think.

They’re subtle, they’re ordinary, they certainly wouldn’t make a compelling documentary – but, crucially, each is rigorously tested to make sure it actually works. The result is an extraordinarily powerful technique which can raise millions or save lives.

Many nudges take merely minutes, yet their effects may last for decades. In one study, completing a single one-hour exercise improved students’ grades for three years.

In another, citizens were asked about their motivations for voting the day before an election. Simply changing the phrasing of the question increased voter turnout by 11%.

This is big: for the 2016 US presidential election, it’s estimated a staggering $4bn will be spent on TV advertising. But even among those who spend the most time glued to their sets, adverts only increase turnout by around 10%.

Back in 2010, former UK prime minister David Cameron set up the “Nudge Unit”; a crack team of scientists, psychologists and policy experts tasked with using the technique to improve education, health and the state of public finances.

Governments have had difficulty getting people to pay their taxes for millennia – every advanced civilisation has grappled with this question – John List, University of Chicago

To solve the latter, they found themselves pondering a perennial problem. They say only two things in life are certain: death and taxes. So why do so many people delay paying them – and how can they be coaxed to cough up?

“Governments have had difficulty getting people to pay their taxes for millennia – every advanced civilisation has grappled with this question,” says John List, an economist from the University of Chicago who was involved in the study.

In ancient Rome, farmers suffering under their tax burden would abandon their land and flee to the city of Constantinople (now part of Turkey), which was an early tax haven. By the Middle Ages, many men had resorted to joining monasteries – resigning themselves to a life of servitude not out of religious devotion, but the desire to avoid paying tax.

With this long history in mind, the team decided to take a fresh approach. At the time, indebted UK citizens could expect to receive a letter telling them how much they owed and how to pay.

It wasn’t working. Though the UK has one of the lowest “tax gaps” in the world, last year the amount of unpaid tax was still a hefty £34bn. That’s nearly enough to pay for the UK’s contributions to the European Union three times over.

The HMRC could have tried to claw it back by sending people out to bang on doors, pleading with the “criminals” over the radio or spamming their letterboxes with ever more threatening letters. But they didn’t. Instead they took the standard letter and made some subtle additions.

To the team’s surprise, this modest increase also occurred when people were reminded of what they gain from taxes

In August 2011, around 100,000 people woke up to an envelope on their doorstop. They had all opted to self-report their taxes – rather than the money coming directly out of their pay packet – and owed the HMRC between £400 and £100,000. In case you were wondering, most of the debtors were men of about 50 years old.

While one half of the group received the usual letter, the others received a copy with a single extra sentence. The team tried several options, each intended to invoke a different psychological quirk.

One of these was “loss aversion”, or the tendency to feel losses more keenly than gains. It has plagued public policy for years: citizens who stand to lose $100 as a result of new legislation are likely to be disproportionately more vocal than those who stand to gain $100.

When not paying their taxes was framed in terms of loss “Not paying tax means we all lose out on vital public services like the NHS, roads, and schools”, payment rates increased by 1.6%. That’s equivalent to around £760,000 of extra taxes.

To the team’s surprise, this modest increase also occurred when people were reminded of what they gain from taxes. “Paying tax means we all gain from…” Though it wasn’t the result they expected, this is the entire point of testing out policies in clinical trials first.

The next set aimed to trigger the citizen’s deep, in-built fear of being different, known as conformity bias. It’s thought to stem from the desire to avoid being rejected by our social group – a factor which had life-or-death significance in our evolutionary past. They noted; “Nine out of 10 people pay their tax on time”.

The real breakthrough came from making these statements even more specific

But the real breakthrough came from making these statements even more specific. In a later experiment, when the team observed that not only had most people paid, but most people in their area – ie their neighbours – payments shot up. Adding that most people with similar debts had also paid increased their earnings even further.

Compared to the control group (who received the normal letter), adding this message – just 26 extra words – increased payment rates by 15%; equivalent to an extra £15.4m in just 23 days.

When they were rolled out across the UK, the letters raised an extra  £210m in their first year. Based on that estimate, as of today they’ve raised more than £630m.

 “What’s new about this approach is not just the knowledge of human psychology but it’s that it’s evidence-based policy making,” says List.

Since the tax experiment, the team has applied the letters to solve numerous other problems, including tackling the over-prescription of antibiotics by doctors, which is thought to be contributing to rising antibiotic resistance. 

Discovering that their levels of prescribing were abnormal was so powerful, it reduced the number of prescriptions by 3.5% that winter

The traditional approach to encourage doctors to use fewer antibiotics is to pay them. “That’s what we do when we want GPs to do anything right now – we pay them,” says Hallsworth. Instead, the team wondered if they could achieve equivalent results with their letter technique. “So we wrote to them at the start of flu season and told them that they were high prescribers,” he says.

Incredibly, discovering that their levels of prescribing were abnormal was so powerful, it reduced the number of prescriptions by 3.5% that winter – equivalent to around a 1% reduction over the course of the year. “That same year we spent £23m on payments to GPs to get the same result,” he says.

As of 2016, Nudge Units have been set up in the US, Canada, Australia, Singapore and Germany. And unlike the Scared Straight program, this time there won’t be any surprises.

Join 700,000+ Future fans by liking us on Facebook, or follow us on Twitter, Google+, LinkedIn and Instagram

If you liked this story, sign up for the weekly bbc.com features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Future, Earth, Culture, Capital, Travel and Autos, delivered to your inbox every Friday.