Science —

Navigating the Wild West of non-peer-reviewed science

Many science news stories are based on papers found in the arXiv, a site that …

Peer review serves as a critical sanity check for the scientific literature. It is by no means a perfect system—flaws ranging from outright fraud to subtle errors can easily slip past reviewers—but peer review can generally identify cases where a paper's conclusions aren't supported by the underlying data, or the authors are unaware of other relevant papers, etc. As a result, peer review acts as a key barrier to prevent scientifically unsound ideas from attracting undeserved attention from the scientific community.

So, does it make any sense to push unreviewed material onto the public? For better or worse, science journalists have discovered preprint repositories like the arXiv, in which scientists post both final papers and works-in-progress—papers that haven't yet cleared the hurdle of peer review.

The lure of the arXiv is pretty understandable. Unlike the tightly controlled world of scientific journals and their press releases, where editors give everyone the same tips on what stories to cover and embargo limits on when the coverage can appear, the arXiv is the Wild West. If you find something obscure there, you're likely to get the jump on covering it, instead of being part of the herd that releases variations on the same story within minutes of each other.

And there's some pretty mind-blowing stuff lurking in the arXiv. Ars recently received a tip to check out a paper that suggests Bell's Inequalities are simply a big misunderstanding, and we've spent decades chasing a phenomenon—quantum entanglement—that doesn't actually exist. These sorts of fringe ideas sometimes do make it into the scientific literature, but they generally don't, and rarely have much of an impact if they do. (In fact, if a paper lingers in the arXiv for years without ever finding a publisher, that's probably saying something about its science.)

So, does it make sense to forward it on to the public as science news? Doing so runs the risk of emphasizing hype instead of solid science, and can substitute the writer's expertise for that of actual scientists.

We struggle with these decisions as part of preparing the news for Ars. We regularly see difficult-to-believe stories being widely reported, but decide against covering them simply because the source is an arXiv paper that hasn't yet made it through peer review.

That's not to say that there aren't ways to handle arXiv publications well. A recent story by our sister publication Wired included the sort of thing that peer review is supposed to provide: a scientist not involved in the work is quoted as saying that the paper lacks some key details, making it impossible to evaluate. Of course, that seems a significant enough issue to raise questions about whether the paper merited coverage at all.

In other cases, the arXiv is the only way to provide in-depth coverage. News broke that NASA would be doing a major data release from its Kepler probe before the space agency even got a press release out; given a choice between arXiv papers and third parties or a press release, the arXiv wins hands down. And, in this case, the instrument and analysis pipeline for its data went through a great deal of peer review during their design phase, which involved large teams of scientists and engineers.

The Kepler article provides a model for what you can expect when Ars has to dive into the arXiv: we'll warn you about the fact that the paper hasn't been through peer review, and we'll consider its reliability carefully. And, in any case where we don't feel we can accurately perform that evaluation, we won't cover it.

The downside is that you probably won't be hearing that gravity doesn't exist from us. But hopefully, we'll make up for that with a bit of reliability.

You must to comment.

Channel Ars Technica