Why Do Bad Medical Treatments (and Training Aids) Spread?

Feedback bias means we're more likely to tell others about successes.

Why do ineffective (and sometimes harmful) medical treatments persist? That's the question posed by Swedish researchers in a paper I just came across in (this is a new one to me) the Journal of Medical Internet Research. It's an interesting paper, and I think it applies equally well to the question of why ineffective training aids are so ubiquitous. The full paper is freely available here.

Advertisement - Continue Reading Below

There are several experiments described in the paper. The first two involve comparing Amazon reviews of two sets of products (an Atkins diet book and some fertility aids) to actual clinical trial data on the effectiveness of those products. For example, here's the weight loss data from clinical trials involving reading the Atkins book, and the experiences of Amazon reviewers:

This is not particularly surprising, and the fertility comparison shows similar results. People appear to have a feedback bias in how they share their experiences, much like the publication bias that afflicts academic journals. After taking an ineffective treatment, some people improve while others deteriorate, but the people who improve are far more likely to share their experiences. When you have a process with a delayed and/or stochastic (i.e. subject to random variation) outcome, you judge its effectiveness based on the feedback you receive from others, and that feedback is skewed.

Advertisement - Continue Reading Below

What I found even more interesting was their argument that this effect (i.e. the likelihood that you'll share your experiences depends on how positive they) can, in some circumstances, actually make bad treatments seem more effective that better treatments (rather than just making everything seem uniformly better than it is). Here's a simplified example, in graphical form, of how that might happen:

At the top, you have a "good" treatment that makes half the people better and leaves half of them unchanged. The people who got better are more likely to share their experiences, so the treatment gets a reputation that's somewhat better than it deserves, with a greater proportion of people reporting that it helped them than reporting no change. At the bottom, you have a "bad" (or at least ineffective) treatment that gives you a 50-50 chance of getting better or worse. The people who get better are, once again, more likely to share the results; the people who got worse are much less likely to share, even compared to the people who stayed stable under the good treatment. As a result, nearly everyone who shares their experiences with the bad treatment reports a positive outcome.

I like the thought experiment above, but I'm not entirely convinced it's relevant in real life. How many "bad" treatments actually give you a 50-50 chance of getting better or worse with no one staying the same? I'd need to see some examples to be convinced. That said, there's no doubt this sort of feedback bias does play some sort of role in spreading ineffective treatments. In many cases, people are comparing "conventional" treatments (where the outcomes, as measured by clinical trials, are often not as great as we'd like) with alternative treatments whose reputation is determined by word-of-mouth rather than clinical trial. Take another look at that graph at the top to see the difference between word-of-mouth and reality. It's easy to see why lots of companies selling treatments and training aids have no desire to run clinical trials -- it's not just expensive, it's a serious buzz-kill.



Advertisement - Continue Reading Below
More From Health & Injuries