Extraordinary claims require extraordinary evidence, as astronomer Carl Sagan once said. Sagan was talking about UFOs and aliens, but his words now stand as a watchword for skepticism in science.
But how do we know when a claim is extraordinary? Say, maybe when the aliens don't arrive from space?
Consider the controversial "arseniclife", short for arsenic-based life, bacteria study. Rather than arriving on a UFO, the microbe was unveiled at NASA headquarters, announced at an "astrobiology" news briefing on Dec. 2, 2010, as "the first known microorganism on Earth able to thrive and reproduce using the toxic chemical arsenic."
Arsenic is a poison. How did the finding that the bacteria called GFAJ-1 replaced phosphorus, a basic chemical constituent of biochemistry, with arsenic even in its DNA make its way into the hallowed journalScience, and onto the stage at NASA? The idea turned out to be too good to be true, as two studies also published by Science that refuted the claim showed this summer. Both showed that the microbe, discovered in California's Mono Lake, in fact seems to need some minuscule amount of phosphorus to survive. It's just tough, not completely alien, in its biochemistry.
"Science magazine is perhaps the most important publication in the United States for communicating science to the public. Therefore, when Science publishes something funded by NASA that is likely to broadly interest the public, NASA is effectively required to hold a press conference," says chemist Steven Benner of the Foundation for Applied Molecular Evolution in Gainesville, Fla. Benner served as the outside skeptical voice at the NASA press conference announcing the finding. "This all assumes, of course, a very high quality of peer review at Science magazine," Benner says.
And that might be the rub. A USA TODAY investigation suggests that the problems with the arseniclife story started with Science's "peer review" of the findings before it ever took the NASA stage.
Peer review is the basic process in science where anonymous outside experts eyeball a study for journal editors, acting as gatekeepers on publications. In the case of the 2010 GFAJ-1 study, three anonymous reviewers, along with Science editor Caroline Ash, asked the authors 32 questions about the study, which were answered by the authors a month ahead of the press conference. And they made a few general comments on the paper. Here are some:
"The results are exceptional," said Reviewer 1.
"It's a pleasure to get a well-received and carried-out study to review," said Reviewer 2.
"Reviewing this paper was a rare pleasure," said Reviewer 3, adding later on: "Great job!"
Most scientists don't often see such upbeat peer review comments on their papers, notes Princeton genome expert Leonid Kruglyak, an author on one of the 2012 papers refuting the arseniclife results. Kruglyak was also one of a collection of outside experts asked to review the reviews, which NASA released to USA TODAY in January after a Freedom of Information Act request. "In retrospect, what is the interesting aspect of them is there wasn't anything different, or out of the ordinary in them (compared) with how most papers are handled," Kruglyak says. "This was a par-for-the-course review."
In particular, Kruglyak notes Peer Reviewer 2 actually pointed out the problem in proving whether phosphorus was really absent from the GFAJ-1 bug's biochemistry. The concern seems muted and is not drawn as a reason to block publication, however, amid a series of other more positive comments.
Looking at the reviewer comments, roughly half were requests for added explanations or spelling fixes. Another half-dozen were questions on the chemistry or biology of the microbe. The rest involved inquiries about further experiments, which original lead study author Felisa Wolfe-Simon often suggested were "beyond the scope of the paper." Kruglyak says such requests for new experiments are not uncommon in reviews, nor is it unusual for study authors to demur, wanting to save something for a future study and leaving it up to the journal editor to adjudicate the dispute.
Peer review expert Elizabeth Wager, former chair of the Committee on Publication Ethics, says the review looked entirely typical for scientific journals. "The only thing that is surprising is how strongly positive the reviewers are," Wager says in an e-mail. "They clearly think this is an important piece of work and also comment that it is clearly presented."
"Judging from the quotes, the three reviewers were enthusiastic about the paper. Indeed, these reviews would be described by most scientists as 'glowing,' " says biologist Patricia Foster of Indiana University in Bloomington. In particular, she notes at least three striking things about the reviews lost in these glowing comments.
"First, there is little biology mentioned - the reviewers questioned only mildly the authors' assumption that there was too little phosphorus in the medium to support growth. This assumption was a major point of disagreement voiced by scientists after the paper appeared." Second, the reviewers didn't question wide variation in chemistry analysis of the medium that the bugs grew in (specifically, how much contaminating phosphorous was in it), something that later critics saw as a major shortcoming. Finally, they only commented, rather than inquiring, on the biochemistry implied by the results, the central extraordinary claim that arsenic was acting in place of phosphorus in the metabolism of the GFAJ-1 bacteria.
"In conclusion, I believe that NASA would have had no reason to doubt the results of the paper based on these reviews. In fact, NASA officials would have felt encouraged to publicize the paper," Foster says.
Foster does note that the study authors added a figure (and a few more authors) to the paper, suggesting the GFAJ-1 bug had arsenic in its DNA, as a result of the review wanting to see more evidence of arsenic inside the bacteria. The figure became another point of controversy in the debate over the study. (Wager says that adding a new figure and authors is "perfectly acceptable" during peer review and wouldn't necessarily trigger a re-review of a study.)
Basically, the reviewers took at face value the fundamental claim by the study authors that the GFAJ-1 bug was growing without any phosphorus, says microbial ecologist Norman Pace of the University of Colorado. "Once you accept that, everything else follows," Pace says. "You just have to have a certain expertise to know that is nearly impossible; removing phosphorus is just very hard."
In general, Pace says that he believes peer review improves studies and works as a "pretty strong" corrective to error in science. "There is lots of poor science out there, but important claims like this one are checked up on and proven true or false, as this (arseniclife) one was, so I think things are not actually so suspect out there in science. In essence, this was all found out in due course. My belief is the peer-review system is fundamentally sound."
Regardless, Pace says there was "poor judgment at multiple levels" in the arseniclife case, from an "overly exuberant" interpretation of the study results by the authors to the peer reviewers missing "the big crux of the results: the claim of absence of phosphorus," to NASA repeating some of the mistakes that caused the agency trouble in 1996, when it publicized results suggesting a Martian meteorite contained microscopic signs of life. Still, Pace had actually recommended the publication of that 1996 paper, he says. "I do think it is important to get noteworthy results out there."
In 2011, Science editor-in-chief Bruce Alberts echoed that comment in a statement on the arseniclife study. "We hope that the study and the subsequent exchange being published today will stimulate further experiments - whether they support or overturn this conclusion. In either case, the overall result will advance our knowledge about conditions that support life, an important outcome for science and education," Alberts says.
Worth noting is that NASA scientist Michael New and original study lead author Wolfe-Simon, now of the Lawrence Berkeley (Calif.) National Laboratory, both still supported the original 2010 study's findings this summer when the refuting studies were published.
"Science is continuously evaluating its peer-review policies and procedures with the goal of a rigorous and fair process," magazine spokeswoman Ginger Pinholster said in response to questions about whether the journal has changed its peer-review practices since 2010. This year, the journal added an additional step to the review process. Once all reviews are in on a manuscript, all the reviewers are invited to read them and comment. "This step allows the reviewers to react to the comments of the other reviewers and may help the editor to calibrate the reviewer comments," Pinholster says by e-mail.
Kruglyak cautions against looking back too harshly at the reviewers of the arseniclife study.
"In hindsight we can see what went on, but that's how hindsight works," Kruglyak says. "It was a pretty spectacular claim. In the big picture, I'm not surprised about it not working out."
Update: One other scientist asked to review the reviews, microbial ecologist James Cotner of the University of Minnesota in St. Paul, makes the point that it is up to the journal to ensure that peer reviewers are the most appropriate experts to review a study. "Part of the problem with this paper may have been that it is a very interdisciplinary topic (molecular biology, microbial ecology, physical chemistry, etc.) so it may have been hard to make sure all fields were appropriately represented," Cotner says, by email. "Without knowing who the reviewers were, it's hard to say if the editor insured that the best and most appropriate folks were reviewing it. But when there are only three reviewers, it can be very difficult."
Copyright 2013 USATODAY.com
Read the original story: Glowing reviews on 'arseniclife' spurred NASA's embrace