In 2008 Annie Vernon rowed for Great Britain in the quad sculls at the Beijing Olympics. At 25, she was the youngest member of an experienced group that was set on becoming the first women’s rowing team to win Olympic gold. In a close finish, they were beaten to first place by the Chinese. Vernon was devastated, and remains mentally scarred by the loss. In an interview to promote her book on the psychology of elite sport, Mind Games, she called it “the defining feature of my career”.

For those of us who are not elite athletes, it is hard to understand quite how punishing the experience of a ‘near miss’ can be. Reaching the very top level of performance requires an immense mental effort, and when you care that much about winning, losing feels like cruel punishment.

But top athletes – and, it turns out, many others – have a way of turning pain into rocket fuel. The defeat becomes a reason to push themselves even further the next time.

Vernon recovered from her disappointment to win gold at the 2010 Rowing World Championships.

UK Sport, the British government body responsible for investing in elite sport, published the findings of an investigation into the roots of athletic success. Over the course of in-depth interviews with 85 elite athletes and coaches, they looked for what exceptional achievers have in common. The researchers found that most athletes suffer a significant setback early in their career, but some react differently to others. For the truly exceptional athletes, who went on to win Olympic medals, the setback enhanced their motivation; for the merely ‘good’, the near miss was discouraging.

Top athletes – and, it turns out, many others – have a way of turning pain into rocket fuel. The defeat becomes a reason to push themselves even further the next time

There can be something uniquely energising about the experience of finishing second instead of first. Adam Leive, an economist at the University of Virginia, assembled a database of medal winners in Olympic track and field events, between 1846 and 1948, and looked at what happened to their lives after they had won a medal.

Leive found that the athletes who just missed out on the top podium spot went on to live longer and more successful lives than those who won. Silver medallists were more ambitious in their post-sport careers, finding better paid jobs. By the age of 80, about half of them were still alive, compared to about a third of gold medallists.

The trauma of losing seems to have spurred them on for life.

Just behind the leader

It’s a phenomenon which extends beyond sport. A recently published paper in the journal Physics and Society shows evidence that scientists who suffer setbacks early in their career perform better in their careers than others.

The authors, Yang Wang, Benjamin Jones and Dashun Wang, looked at the data on grant proposals made by junior scientists applying for funds from the US National Institutes of Health. They identified two groups: “near-miss” individuals, whose grant proposals fell just below the funding threshold, and “near-win” individuals, who scraped in just above the threshold.

Just as UK Sport found with athletes, losing out acted like a form of natural selection. About one in 10 of the near-missers disappeared from the system altogether, but those who persevered went on to publish more high-impact papers over the following decade than the near-winners.

Childhood setbacks may have a similar effect on the arc of a life. The psychologist Marvin Eisenstadt, in a study entitled Parental Loss and Achievement, found that from a random sample of 573 eminent people who merited more than one column in encyclopaedias, nearly half had had a parent die before they were 20. Nobody would wish the loss of a parent on any child. It means a higher risk of mental health problems later in life. But it’s also true that a surprising number of high-achievers have suffered bereavement or some other kind of trauma as children.

Three of the most famous examples were in a band together. Paul McCartney’s mother died of cancer when he was 14; John Lennon’s mother was killed in a road accident when he was 17. Richard Starkey (Ringo Starr), did not suffer early bereavement, but certainly suffered. Starkey was a toddler when his parents divorced; he was raised in poverty by his mother. He fell seriously ill when he was six, spending a year in hospital. Now aged 79, he still runs on to stage to perform with his All-Starr band.

Lighting a fire

These are extreme examples, in every sense, but in general we tend to under-estimate the extent to which some kind of disadvantage or setback can, paradoxically, catapult people into higher achievement. Some people can turn the hurt and anger generated by a setback into a fierce will to succeed. By struggling against whatever forces they find pushing them down, they develop anti-gravity powers which lift them higher later on.

The principle is akin to a biological one. Weightlifters know that for a muscle to grow it must first be traumatised. The exercise has to be great enough for thousands of tiny tears to open up, which the body can then repair, strengthening the muscle as it does so. In life, as at the gym, it’s what you do with a trauma that determines whether or not it has a latent benefit.

High achievers seem to find a way to perform a kind of mental alchemy, turning loss and disappointment into motivation. The flipside of this is that some who grow up with all their material needs met sometimes lack drive and direction as adults. That’s why some experts in talent development worry that children are not even being given the chance to experience setbacks.

In life, as at the gym, it’s what you do with a trauma that determines whether or not it has a latent benefit

In a 2012 paper entitled The Rocky Road to the Top: Why Talent Needs Trauma, the sports scientists Dave Collins and Aine MacNamara criticised the approach of most talent development systems in sport, which put an emphasis on maximising support to young athletes and reducing stress. The authors argued that these well-funded and high-tech coaching systems were making life too easy for young athletes, who needed moments of challenge or trauma in order to develop resilience. It’s the rocky road, not the smoothed path, that leads to greatness.

Of course, even with all of this in mind, we shouldn’t over-romanticise setbacks and failures. They are painful and upsetting, and sometimes, a bad experience is just pure, unalloyed bad. Certainly, that is how it feels when you’re experiencing it. But when you are in one of those dark moments of loss, it is worth asking yourself whether you might, one day, turn it into something good.

Perhaps Friedrich Nietzsche was right: what doesn’t kill us makes us stronger.

***********************

To comment on this story or anything else you have seen on BBC Capital, please head over to our Facebook  page or message us on Twitter.

If you liked this story, sign up for the weekly bbc.com features newsletter called "If You Only Read 6 Things This Week". A handpicked selection of stories from BBC Future, Culture, Capital and Travel, delivered to your inbox every Friday.