In a unique experiment, The Guardian published online the full manuscript of its major investigation into the climate science emails stolen from the University of East Anglia, which revealed apparent attempts to cover up flawed data; moves to prevent access to climate data; and to keep research from climate sceptics out of the scientific literature.
As well as including new information about the emails, we allowed web users to annotate the manuscript to help us in our aim of creating the definitive account of the controversy. This was an attempt at a collaborative route to getting at the truth.
We hoped to approach that complete account by harnessing the expertise of people with a special knowledge of, or information about, the emails. We wanted the protagonists on all sides of the debate to be involved, as well as people with expertise about the events and the science being described or more generally about the ethics of science. The only conditions are the comments abide by our community guidelines and add to the total knowledge or understanding of the events.
The annotations - and the real name of the commenter - were added to the manuscript, initially in private. The most insightful comments were then added to a public version of the manuscript. We hoped the process would be a form of peer review.
It is hard to believe that tree trunks dragged from frozen bogs in Siberia could undermine the argument about man-made climate change. But that is the claim that has been made by sceptics in recent months.
The claim is wide of the mark, but in the 1,073 emails stolen from the University of East Anglia last November the row over what the trees tell us about climate change is played out in detail. The scientists are shown clinging to their data to prevent it getting into the hands of sceptics even as at least one colleague advised openness to avoid the charge that "bogus science" was being hidden.
Measuring the width of annual growth rings in trees is a sensitive measure of temperatures. And the secrets of those Siberian trees, some of them thousands of years old, have assumed an important place in the reconstruction of past temperatures for the whole planet.
Steve McIntyre, a Canadian former minerals prospector and climate sceptic who has analysed the data, suggests that one tree, known as YAD06, could be "the most influential tree in the world".
In the hacked emails from the Climatic Research Unit at UEA, one word looms large: Yamal. The first and last emails and more than a hundred in between include it. When I phoned Prof Phil Jones, the director of CRU, on the day the emails were published online, he said: "It's about Yamal, I think."
On 6 March 1996, a Russian scientist, Stepan Shiyatov, contacted Dr Keith Briffa, CRU's top tree-ring researcher. Shiyatov wanted money to take a helicopter to measure tree rings in timber hauled from the permafrost of the Yamal peninsula on the Arctic ocean's shores.
Briffa was keen, and he published papers on what those tree rings showed. But by late last year, in the final emails, he is mired in allegations of fraud, and the Yamal data had become a virus infecting past climate reconstructions.
The Yamal data turned up in many studies of global temperature that were cited by the UN's top climate science body, the Intergovernmental Panel on Climate Change, in a report published in 2007, where the relevant section was authored by Briffa. It supported the conclusion that temperatures followed a "hockey stick" shape, with stable temperatures for a thousand years, then sharp 20th-century warming.
By then, McIntyre was on the trail. He claimed that Briffa had not used all the tree ring data available, only a subset. Briffa said there were technical reasons for that. But McIntyre complained Briffa hadn't spelled out those reasons clearly.
In 2008, when Briffa published some data after a long delay, McIntyre charged that Briffa's analysis of the most recent warming was based on just 12 trees: the "Yamal-12". McIntyre said this was too small a sample to draw any conclusions, and claimed if the analysis was redone with other tree ring data from the region, the hockey stick shape disappeared.
It looked like a stalemate. But last year the bloggers moved in. Ross Kaminsky, a columnist on American Spectator, claimed: "One implication, supported by Briffa's near decade-long refusal to share his data, is that he cherry-picked the dataset that supported the conclusion he wanted to find."
Worse was the charge that other scientists had used the suspect Yamal data in their reconstructions of past climate. Ross McKitrick, a climate sceptic and environmental economist at Canada's University of Guelph, wrote that they are "the key ingredient in most of the studies that have been invoked to support the hockey stick". The Daily Telegraph blogger James Delingpole went even further in an article headlined: "How the global warming industry is based on one MASSIVE lie."
Briffa denies any wrongdoing. He said "we would never select or manipulate data in order to arrive at some preconceived or unrepresentative result". And there is nothing in the emails or anywhere else to suggest that isn't true. In September last year Briffa put out a statement on the CRU website defending his research. "We do not select tree-core samples based on comparison with climate data. Chronologies are constructed independently and are subsequently compared with climate data to measure the association and quantify the reliability of using the tree-ring data as a proxy for temperature variations."
One British colleague of Briffa wrote to me last month: "Why should Briffa – one of the world leaders in this field – have to explain himself to people … who are in fact amateurs?"
But others believe Briffa has a duty to explain himself. In October last year, Briffa's old boss at CRU, Tom Wigley, said in an email to Briffa's current boss, Phil Jones: "Keith does seem to have got himself into a mess." Wigley felt Briffa had not answered McIntyre's charges fully. "How does Keith explain the McIntyre plot that compares Yamal-12 with Yamal-all? And how does he explain the apparent 'selection' of the less well-replicated chronology rather than the later (better replicated) chronology? …
"The trouble is that withholding data looks like hiding something, and hiding something means (in some eyes) that it is bogus science that is being hidden."
The Yamal data has become important for scientists trying to analyse past climates. But it is not true that the Yamal rings are omnipresent in climate reconstructions. They were not in the data that produced the "hockey stick" graphs. According to Jones, of the 12 reconstructions of temperatures over the past 1,000 years used in the last IPCC assessment, only three included Yamal data. Other reconstructions were based on retreating glaciers, or water temperatures in boreholes, or core sunk into ice sheets – but they too reproduce a hockey stick shape.
Even McIntyre denounces the more vocal sceptics with their conspiracy theories. In an apparent response to a challenge from the climate scientists' website RealClimate, he wrote to the American Spectator last October: "While there is much to criticise in the handling of this [Yamal] data, the results do not in any way show that AGW [anthropogenic global warming] is a 'fraud', nor that this particular study was a 'fraud'. There are many serious scientists who are honestly concerned about AGW and your commentary … is unfair to them." Sadly, when checked last week, there was no sign of this comment on the magazine website, though the magazine had found room for another feature on "The great hoax" of climate change.
The text below consists of invited comments made on the Climate wars articles. They can be accessed in the main body of the article by clicking on the text to which they refer, which is highlighted in yellow.