Half the Science You Know May Be Wrong

Here’s an interesting piece from (of all places) ScienceNews. Evidently lots of scientists don’t understand statistics, and abuse of this mathematical tool may have lead to huge numbers of unreliable scientific conclusions:

“It’s science’s dirtiest secret: The “scientific method” of testing hypotheses by statistical analysis stands on a flimsy foundation. Statistical tests are supposed to guide scientists in judging whether an experimental result reflects some real effect or is merely a random fluke, but the standard methods mix mutually inconsistent philosophies and offer no meaningful basis for making such decisions. Even when performed correctly, statistical tests are widely misunderstood and frequently misinterpreted. As a result, countless conclusions in the scientific literature are erroneous, and tests of medical dangers or treatments are often contradictory and confusing.

The study that recently raised this issue came to the conclusion that more than half of statistical studies are untrustworthy. That number may not be exactly right – apparently that study may suffer from some statistical issues of its own, but the philosophical and methodological issues it raises seem to be sound. Certain sciences rely much more heavily on statistics than others – most conspicuously the social sciences, such as psychology, economics, and political science – so this article isn’t saying that every scientific finding is bogus. You can probably remain pretty confident in the soundness of physics, for instance.

But many studies are conducted under various statistical flaws and misunderstandings (confused interpretations of p-values and neglect of Bayesian approaches, for instance), and that’s not something we can ignore. In fact, this all may lead us to reexamine several issues normally surrounded by a broad consensus. The negative effects of obesity*, the positive effects of pharmaceuticals*, and (dare I say it) the consensus on causes of global warming could all be called into question since each of these issues rests on a large body of statistical evidence.

Of course, it’s also possible that all of those conclusions are absolutely right. The thing is we just don’t know. If the statistical foundations of many of these studies simply collapse, we lose a lot of the former certainty we had about these issues. At the very least, this news is probably a good ego check for science. After all, science is an imperfect, constantly evolving human endeavor, and we should never be too sure we have all the right answers.

*More to come on these topics soon