Everyone can cite famous people from Vincent Van Gogh and Virginia Woolf to Tony Hancock and Robin Williams, who were exceptionally creative and experienced mental health problems. There are so many examples that it seems obvious that there must be a link between mental illness and creativity.

Surely research would support this common wisdom? Not exactly.

In fact there is remarkably little good data on the topic. In a review of 29 studies conducted before 1998, 15 found no link, nine did find a link and five found it was unclear. So hardly a straightforward connection. And some of these were simply case studies rather than rigorous attempts to establish whether there really is a causal link.

Some of the studies were simply case studies rather than rigorous attempts to establish a causal link

One of the difficulties is that it isn’t very easy to define or measure creativity, so researchers often use proxies for it. For example, a study from 2011 simply classifies people by occupation assuming that everyone who is an artist, a photographer, a designer or a scientist must be creative, regardless of their exact job. Using the Swedish government census the researchers did find that people with bipolar disorder were 1.35 times more likely to be in one of these creative jobs.  But there was no difference when it came to anxiety, depression or schizophrenia. Because such a small range of jobs was included, this data can’t tell us whether people in creative professions are more likely than everyone else to have bipolar disorder or whether accountants are unusually unlikely to develop it.

The studies most often quoted in favour of a link include a piece of research by Nancy Andreasen published in 1987 which compared 30 writers with an equal number of non-writers.  The writers were more likely to have bipolar disorder than the non-writers. It’s a small sample, with just 30 writers interviewed in 15 years and although it is cited widely, it has been criticised (see this paper by Judith Schlesinger for a comprehensive review of this and the other oft-cited studies on this topic) because the mental health problems were diagnosed via interviews and it is not clear what criteria were used. Also the interviewer was not blinded to whether or not people were writers, which could skew the results. What’s more, the writers had chosen to visit a writing retreat, known to be a place where people sought sanctuary, so perhaps those writers were more likely to feel troubled in the first place.

Even if the results are taken at face value they tell us little about causality

Even if the results are taken at face value they tell us little about causality. Did the supposed creative benefits of bipolar disorder make the writers more likely to choose their profession or did the symptoms mean it was harder for them to find a traditional job?  It is hard to know.

There are two other studies frequently cited in support of a link between mental illness and creativity. The first was conducted by Kay Redfield Jamison, best-known for her fascinating book An Unquiet Mind. Again the research was based on interviews, this time with poets, novelists, biographers and artists. A total of 47 people took part, but there was no control group, so any comparisons can only be made with average rates in a population. She found surprising levels of mental illness. For example, half the poets had sought treatment at one time or another. This sounds like a high number, but as critics have pointed out, it is based on just nine people.

Then there is Arnold Ludwig’s research which involved a far greater number of people. He studied the biographies of more than a thousand famous people looking for mentions of mental health problems and found that different professions had different patterns of problems. The difficulty here is that although the famous people were undoubtedly exceptional (Winston Churchill and Amelia Earhart, for example) they were not necessarily creative in the strictest sense of word. Although his lengthy study is often quoted as evidence in favour of a link, Ludwig himself admits in the paper that it has neither been established that mental illness is more common amongst the eminent or that it is necessary in order to achieve eminence.

When the evidence is thin at best and according to some studies, lacking altogether, why has this idea stuck? 

Studies of prominent people seem to be quite popular among researchers, but they don’t always bring the same results. Back in 1904 Havelock Ellis studied more than 1,000, and found no relationship between mental illness and genius. And a 1949 study of 19,000 German artists and scientists living across three centuries came to the same conclusion. (An important caveat: these studies have the disadvantage of relying on biographers both knowing of and choosing to mention their subject’s mental health problems.)

So when the evidence is thin at best and according to some studies, lacking altogether, why has this idea stuck? One reason is that it seems to make intuitive sense that thinking in unusual ways or experiencing the energy and determination that mania can bring, might aid creativity. Some argue that the relationship between mental illness and creativity is more complex, that mental health problems allow people to think more creatively than others, but this creativity drops back down to average levels or lower during severe episodes of illness.  Sometimes of course a mental health problem can stop people from being able to do what they want to do at all. Depression soon saps motivation.

Perhaps many believe the link simply because it is salient when it happens. The psychologist Arne Dietrich gives a nice explanation based on the Nobel-prize winner Daniel Kahneman’s ‘availability heuristic’. This is our tendency to focus on whatever is in front of us. Tales of Van Gogh cutting his ear off in a moment of madness (and decades of speculation about what did or didn’t happen) make that story vivid in our minds. We don’t have equivalent mental pictures of artists happily getting on with their lives. We estimate how often something is likely to happen by how easily it comes to mind, so if we are asked to consider whether genius and mental illness are linked, we are struck by the examples we can think of immediately.

Some individuals find their illness does seem to enhance their creativity, and are even deterred from taking medication

It’s possible that there could be some potential downsides to believing in the relationship. Some individuals find their illness does seem to enhance their creativity, and are even deterred from taking medication, for example, for fear that that creativity might be extinguished. But is there a risk that people credit their creative success to the illness rather than to their own talents? And what about all those who live with mental health problems, but don’t find they have exceptional talent? Is there pressure on them to feel they surely must excel and to feel bad if they don’t?

Ultimately, I wonder whether the idea persists because it is comforting. Comforting if we have a mental health problem because it opens up the possibility of a positive side to it (and I interviewed many people over the years who have described positives to me) and comforting if we don’t because it makes us think that if we were a creative genius there would be a price to pay. Perhaps the link between mental illness and creativity endures simply because we want it to.

Join 500,000+ Future fans by liking us on Facebook, or follow us on Twitter, Google+, LinkedIn and Instagram

If you liked this story, sign up for the weekly bbc.com features newsletter, called “If You Only Read 6 Things This Week”. A handpicked selection of stories from BBC Future, Earth, Culture, Capital, Travel and Autos, delivered to your inbox every Friday.