

The second lesson is that causal explanations are oversimplifications. This is what makes them useful -- they help us grasp the world at a glance. For instance, after watching the short films, people immediately settled on the most straightforward explanation for the ricocheting objects. Although this account felt true, the brain wasn't seeking the literal truth -- it just wanted a plausible story that didn't contradict observation. There's a fundamental mismatch between how the world works and how we think about the world.

The good news is that, in the centuries since Hume, scientists have mostly managed to work around this mismatch as they've continued to discover new cause-and-effect relationships at a blistering pace. This success is largely a tribute to the power of statistical correlation, which has allowed researchers to pirouette around the problem of causation. Though scientists constantly remind themselves that mere correlation is not causation, if a correlation is clear and consistent, then they typically assume a cause has been found -- that there really is some invisible association between the measurements.

But here's the bad news: the reliance on correlations has entered an age of diminishing returns. At least two major factors contribute to this trend. First, all of the easy causes have been found, which means that scientists are now forced to search for ever-subtler correlations, mining that mountain of facts for the tiniest of associations. Is that a new cause? Or just a statistical mistake? The line is getting finer; science is getting harder. Second -- and this is the biggie -- searching for correlations is a terrible way of dealing with the primary subject of much modern research: those complex networks at the centre of life.

While correlations help us track the relationship between independent measurements, such as the link between smoking and cancer, they are much less effective at making sense of systems in which the variables cannot be isolated. Such situations require that we understand every interaction before we can reliably understand any of them. Given the byzantine nature of biology, this can often be a daunting hurdle, requiring that researchers map not only the complete cholesterol pathway but also the ways in which it is plugged into other pathways. Unfortunately, we shrug off this dizzying intricacy, searching instead for the simplest of correlations. It's the cognitive equivalent of bringing a knife to a gunfight.

These troubling trends play out most vividly in the drug industry. Although modern pharmaceuticals are supposed to represent the practical pay-off of basic research, the R&D to discover a promising new compound now costs about 100 times more (inflation-adjusted) than it did in 1950. It also takes nearly three times as long. This trend shows no sign of letting up: industry forecasts suggest that once failures are taken into account, the average cost per approved molecule will top $3.8 billion (£2.46 million) by 2015. What's worse, even these "successful" compounds don't seem to be worth the investment. We are witnessing Moore's law in reverse.

This returns us to cholesterol, a compound whose scientific history reflects our tortured relationship with causes. At first, cholesterol was entirely bad; the correlations linked high levels of the substance with plaque. Years later, we realised that there were multiple kinds and that only LDL was bad. Then it became clear that HDL was more important than LDL, at least according to correlational studies and animal models. And now we don't really know what matters, since raising HDL levels with torcetrapib doesn't seem to help. Although we've mapped every known part of the chemical pathway, the causes that matter are still nowhere to be found. If this is progress, it's a peculiar kind.

Latest on wired.co.uk