We all love a neat science story, but even rock solid facts can be less revealing than they seem. Let's take an example of a piece of psychology research reported recently: the idea that people drink faster from curved glasses.
Hundreds of news sources around the globe covered the findings, many of them changing the story slightly to report that people drink more (rather than faster) from a curved glass. At first it seems like a straightforward piece of psychology research, with clear implications: curved glasses will make pacing yourself harder, so you'll end up drinking more than you should. Commentators agreed with the research (funded by Alcohol Research UK) – beverage manufacturers were probably onto this before, and will now be rushing to make us take our favourite tipple out of a curved glass.
But before we change our drinking habits or restock our glass collections, let's look at what the scientists actually did.
Luckily for us the team of researchers from the University of Bristol, UK, published their paper in an open access journal, which means the research details are free for all to read.
The Bristol team invited participants into the lab and asked them to drink lager (or lemonade) from a straight class or a curved one, while watching a nature documentary (a BBC one, I'm happy to report). They also asked their volunteers to judge when the glass was half full. The results of both were clear, participants finished their drink of lager sooner in the curved glass. They also judged the halfway point as being lower down the curved glass than the straight glass – suggesting a reason for the faster drinking: if people thought the glass was fuller than it really was when then they would underestimate the rate at which they were drinking.
Now this is all well and good, but there are many reasons why the results don't mean that we can make people drink more by changing the shape of their glass. Importantly, none of these reasons would have to do with this research being wrong or inexpertly done. I'm absolutely certain that if we did the study ourselves we'd find exactly the same thing.
No, the reasons you can't jump to conclusions from this kind of study is because, inevitably, a single study can only test one aspect of the world, under one set of circumstances. This makes it hard to draw general conclusions of the sort that get reported. Notice how the psychologists measured one thing (rate of drinking lager), for just two different glasses, over a single drink, for one set of people (volunteers in Bristol, in 2012), and yet a generalised truth stating that "people drink more from curved glasses" emerged from a specific set of circumstances.
Now obviously, the aim of science is to come up with answers to questions that become generalised truths, but psychology is a domain in which it is fiendishly hard to establish them. If you are studying a simple system, then cause and effect is relatively easy to establish. For instance, the harder you throw a rock, the further it tends to travel in distance. The relation between the force you put in and the acceleration of the rock you get out is straightforward. Add a human factor into the equation, however, and such simple relations begin to disappear. (Please don't experiment by throwing rocks at people.)
To see how this limits the conclusions that can be drawn from the drinks study, think of even the most trivial factor that could change these results. Would you get the same result if people drank ale rather than lager? Probably. If they drank two pints rather than one? Maybe. If they drank in groups rather than watching TV (arguably closer to the circumstances of most drinking)? Who knows! It seems to me perfectly plausible that a social situation would produce different effects than a solo-drinking experience.