Research Cycle

 Vol 5|No 1|October|2008

The Evidence Gap, Truthiness and the Wicked Witch

By Jamie McKenzie

Many times our students will encounter claims and statements poorly substantiated by evidence.

They are contending with a broad social trend that Stephen Colbert has called "truthiness."

Merriam Webster's definitions:

1. truthiness (noun)

We must develop their recognition skills so that an alarm goes off when they read or see an unsubstantiated claim, whether that claim be a biographical note, the ad for a drug or the platform of someone running for office.

Once the alarm sounds, they must learn to ask for the evidence or data that is missing.

A Biographical Example

A few years back while looking for Web evidence of Captain James Cook's navigational skill, I encountered a passage on the site of the Canadian National Library that I found astounding for its lack of evidence and its distorted history.

Many believe that Captain James Cook was the greatest ocean explorer ever to have lived, and that he mapped more of the world than any other man. It cannot be denied that he combined great qualities of seamanship, leadership and navigational skill. He mapped many parts of the Pacific, including New Zealand, Hawaii and the west coast of North America.

He also set an example of how to treat a ship, its crew, and the people he met on his explorations.
Passageways: True Tales of Adventure for Young Explorers

This passage has been altered since I lodged a complaint (removal of underlined segment above), but it remains a bold statement of opinion about a complicated man that oversimplifies his record, distorts the truth and relies upon heavy handed language to pressure us into accepting its questionable point of view. There is still an "evidence gap."

Cook routinely took hostages during his three voyages through the Pacific Islands when the native people "borrowed" his equipment or possessions. He preferred the chiefs as hostages and died in Hawaii on the day a chief refused to come along when summoned. Stabbed to death on the beach, Cook was far from a good example of how to treat the native people he encountered. His treatment of his crew is also open to question.

The previous comments are based on information from "The Death of Captain Cook" By Richard P. Aulie at and The Trial of the Cannibal Dog by Anne Salmond. One wonders if the person who wrote the passage about Cook above actually knew much about his encounters with his crew or the natives he encountered during these voyages.

A Pharmaceutical Example

The New York Times has been running a series of articles, "The Evidence Gap," examining the use of drugs that have been prescribed by thousands of doctors to millions of people despite the lack of evidence that these drugs will prolong life or reduce risks from heart attacks. In some cases, these drugs are now thought to increase the risk of cancer.

When the Food and Drug Administration approved a new type of cholesterol-lowering medicine in 2002, it did so on the basis of a handful of clinical trials covering a total of 3,900 patients. None of the patients took the medicine for more than 12 weeks, and the trials offered no evidence that it had reduced heart attacks or cardiovascular disease, the goal of any cholesterol drug. Source: "For Widely Used Drug, Question of Usefulness Is Still Lingering" By Alex Berenson Published: September 1, 2008 in the New York Times.

This series so far offers six examples of drugs that have been widely adopted without compelling evidence.

The willingness of doctors to prescribe such drugs is a serious issue, especially given the lavish attention and benefits provided to many of them by the drug companies. See article on these benefits "MEDICAL MARKETING -- Treatment by Incentive: As Doctor Writes Prescription, Drug Company Writes a Check." June 27, 2004 - By Gardiner Harris in the New York Times.

At the heart of the various investigations into drug industry marketing is the question of whether drug companies are persuading doctors -- -- often through payoffs -- to prescribe drugs that patients do not need or should not use or for which there may be cheaper alternatives.

But the willingness of patients to accept these drugs without checking doctors' advice is an important aspect of the problem. A patient, sadly, must understand that the old saying "Buyer Beware!" now applies to the medical field in which it must be restated, "Patient Beware!"

After a decade of urging seniors to take flu shots, new studies are challenging the design of the original studies that seemed to support the benefits of such shots. Evidently, the seniors in those studies who took shots were generally better off to begin with, were characterized by healthier habits and received care superior to that available to those not taking the shots, so their survival rate may have been explained by factors other than the shots.


"Doubts Grow Over Flu Vaccine in Elderly." Published: September 1, 2008 By Brenda Goodman in the New York Times.

A Political Example

Modern campaigns often rely upon sound bites, mind bites, eye candy and mind candy to sell candidates, whether they be running on the left or the right. Much campaigning is done with 15 and 30 second campaign ads that compress complicated issues into simple one line solutions.

It seems that few citizens have the time or the inclination to visit the candidates' Web sites where they would usually find lengthy white papers outlining the candidates' positions on the issues of the day. The thirst for simple, brief answers to complicated problems is met by simplistic solutions that often make little sense. Combined with voters' general dislike of pain and sacrifice, the policy options offered by hopeful candidates rarely involve either.

The current issue of high gas prices is a good example. Voters see this as a crucial issue in most countries, but politicians have a difficult time selling conservation as a viable policy option. In the USA presidential election, both major party candidates who previously opposed offshore drilling have shifted their stances as polls have shown citizens want more oil now and want more drilling now, even though most experts have testified that it might take 6-8 years or longer before any of this new drilling would bring more oil to market and even though the oil involved might be a "drop in the bucket."

"Suppose the US produced all its oil domestically," said Robert Kaufmann, director of the Center for Energy and Environmental Studies at Boston University. "Do you think oil companies would sell oil to US consumers for one cent less than they could get from French consumers? No. Where oil comes from has no effect on price." Source: "New offshore drilling not a quick fix, analysts say." June 20, 2008 in the Boston Globe, by Lisa Wangsness.


Furthermore, the article claims, "It would take at least a decade for oil companies to obtain permits, procure equipment, and do the exploration necessary to get the oil out of the ground, most industry analysts say."

Both Obama and McCain now have energy policies that seek independence from foreign sources over the next decade, trusting to new alternatives but suggesting little in the way of conservation or sacrifice. It is interesting to compare their comments about energy in their acceptance speeches:

Senator Obama's Comments about Energy Senator McCain's Comments about Energy













When their ideas are placed side by side, it is difficult to distinguish between them, in part because their positions are thin on specifics, another version of the "evidence gap" that is the focus of this article. What is missing from their words are clear facts about their track records, other than the attacks made by their opponents.

Obama attacks McCain for his lack of support for higher fuel-efficiency standards for cars, investments in renewable energy, and renewable fuels.

McCain attacks Obama for his lack of support for offshore drilling.

Numbers Do Not Lie, But Some People Lie with Numbers

Obama promises to spend "$150 billion over the next decade in affordable, renewable sources of energy," but the number, while impressive, is abstract and difficult to grasp. McCain counters with talk about an ambitious project that will, among many other things, ". . . increase the use of wind, tide, solar and natural gas. We will encourage the development and use of flex fuel, hybrid and electric automobiles." He not does explain how these things will happen. He does not provide specifics or details, but neither does Obama.

To be fair to both candidates, an acceptance speech is expected to be inspirational - a dramatic and moving piece of oratory that cannot be weighted down with too much in the way of specifics or evidence without endangering the opportunity to "rally the base." Appeals to emotion seem strategically wise, while sharp edged specifics would seem risky.

Simple ideas, neatly packaged, have great appeal in such speeches. Thus we heard avid party loyalists chanting "Drill, Baby, Drill!" as the McCain outlined his position.

Big numbers like Obama's promise of a $150 billion investment in alternative energy sources are impressive but unnerving. They resonate. They impress. But they also inspire skepticism. Where will he find this money? If we go to his campaign Web site looking for specifics, we do not find them, even if we download the full text of "NEW ENERGY FOR AMERICA" at It is a big number, for sure, but there is no place in the four page document that clarifies how it will be spent or how it will be funded. There seems to be an evidence gap.

McCain also threw around some big numbers, promising that his plan would help us, "stop sending $700 billion a year to countries that don’t like us very much." Astute commentators quickly pointed out that the number he used is the total trade deficit, not the trade deficit for oil, and that much of the number was spent on goods produced by so-called friends.

There seems to be an evidence gap.

CBS News reported on September 11, 2008 that, "The big rise in oil prices, with the average barrel of imported crude jumping to a record $124.66, pushed overall imports up by 3.9 percent to a record $230.3 billion." "U.S. Trade Deficit Soars; Oil Imports Rise." "Countries that don’t like us very much," is a clever and emotionally appealing sentiment, but it clouds over the facts. Much of the oil we import comes from Canada and friends, as explained by the U.S. Government Web site, The Energy Information Administration.

About Half of U.S. Petroleum Imports Come from the Western Hemisphere


* Canada (18.2%)
* Mexico (11.4%)
* Saudi Arabia (11.0%)
* Venezuela (10.1%)
* Nigeria (8.4%)

It turns out that the biggest problem is our dependence on oil, not our dependence on oil from "countries that don’t like us very much." Once we close the "evidence gap," a very different picture emerges than the one McCain hoped to evoke.


When politicians throw around large numbers or purport to believe in things they have never acted upon, when they wish to be seen green when they are really gray, or when they hope to be seen as well informed when they are really ignorant, they are flirting with what Stephen Colbert has called "truthiness."


When students and citizens learn to poke below the surface claims and demand evidence to substantiate claims and positions, truthiness melts away like the wicked witch (image) in The Wonderful Wizard of Oz. Closing the evidence gap is like throwing water on the witch.


