Science: tried and true? [Perspective]

Image: PD

Disclaimer: The views and opinions expressed in the Perspectives Series are those of the author and do not necessarily reflect the official policy or position of 2 Minute Medicine.

I. The Good?

Science has an image problem. On the one hand, science has never been more dominant. We apply the methods of science and statistics to all parts of life, including education, psychology, economics, and politics. On the other hand, public distrust of science has never been greater.

I suspect this distrust arises from the very ubiquity of scientific data. When seemingly credible science is used to support both sides of every argument, including very public and contentious debates about global warming and evolution, it becomes hard to see how science is helping us choose the right answer. And when scientific conclusions seem to be in constant flux, with the findings of one year supplanted the next, it can be easy to believe that science generates more noise than Truth.

Of course, I still believe that science can and does advance our knowledge, albeit in small increments. But it is also true that it can be very difficult to apply scientific data to a specific question, especially in the hospital, where somehow the data never quite tell us definitively what we should be doing for our patients. So the question becomes, if science can tell us true things about the world, why is it so difficult to use?

II. The Bad

One simple answer is that there is good science and bad science, and that most science is simply bad.

The most persuasive voice behind this argument is a Greek physician-scientist named John Ioannidis. Ioannidis is a meta-researcher: he researches how we research. Twenty years ago, he set out to understand why the vast majority of medical research, including many of our gold-standard randomized clinical trials, turns out to be wrong.

His conclusion? Bias. When he went through and analyzed specific studies for how they had gone wrong, he found bias at every stage. Biased study design, improper patient recruitment, fudging of data analysis, and inappropriate presentation of the results. Sometimes the problem went as deep as the research question itself.

It’s not that Ioannidis thinks most researchers are dishonest or liars. Rather, he thinks research gets distorted by a system that rewards positive findings. Because we are more excited by trials that, for example, show that a vitamin helps prevent cancer rather than by those that show no link, we skew our whole system. Researchers tend to look for those results, publishers tend to publish those results, and newspapers tend to write about them. In one of his most widely cited studies, Ioannidis showed that this ‘rate of wrongness’ was actually predictable from factors such as the size of the study and the study design. In another, he demonstrated that, of the 45 most-cited studies published by the highest impact factor journals, 14 were later shown to be wrong.

In some ways, Ioannidis’s work is profoundly discouraging. It suggests that most of the research that we use to prevent or treat human disease is likely wrong or exaggerated. Yet, seen another way, his work is actually deeply affirming. The methods that he uses to critique bad science are the same statistical methods that others often misuse. That is, Ioannidis is not saying that how we do science is in and of itself flawed; rather, it is the way these principals are misused in practice that is the problem. Put differently, Ioannidis is not criticizing science so much as scientists. The problem he has identified is difficult to fix because it requires large-scale reorienting of our incentives and expectations, but he leaves intact the possibility that science can lead us to truth. This is reassuring.

It is also, however, only part of the story. Science is difficult to use not only because it is hard to tell good science from bad science, but because the application of scientific data hits up against science’s fundamental limitations.

Click for part 2

Interested in submitting a Perspective? See this link

© 2013 2minutemedicine.com. All rights reserved. No works may be reproduced without written consent from 2minutemedicine.com. Disclaimer: We present factual information directly from peer reviewed medical journals. No post should be construed as medical advice and is not intended as such by the authors or by 2minutemedicine.com. PLEASE SEE A HEALTHCARE PROVIDER IN YOUR AREA IF YOU SEEK MEDICAL ADVICE OF ANY SORT. Content is produced in accordance with fair use copyrights solely and strictly for the purpose of teaching, news and criticism. No benefit, monetary or otherwise, is realized by any participants or the owner of this domain.