« Can Better Teachers Close the Achievement Gap? | Main | What NAEP Long-Term Trend Scores Tell Us About NCLB »

Test Scores and Reinforcing the Wrong Connections


Dear Diane,

The good news is that most of the American people haven’t lost their common sense. And, above all, those closest to “the action”—parents, teachers, kids, and their families, plus a majority of those who work closely with schools or are “students” of schooling—haven't. What they have lost is the power to be widely heard. The Education Equality Project et al are doing their best to “brainwash” us into thinking we all agree.

If there ever was a time when I appreciate the existence of organized “teacher voices,” it’s the days we’re living through. Thank goodness for teachers' unions—weak as they are. The attacks on “big labor” are always intriguing. The labor leaders relish it because it’s hard to boast about powerlessness. Their opponents like it so they can blame unions.

You are right: The McKinsey report presents an argument for ending poverty. But you wouldn’t know it by the press it has received. A nation known worldwide for its egalitarian ideals is now least equal among modern societies; it's even near the bottom on social mobility. So, it’s no surprise that it shows up also on test scores, as we’ve known since the invention of standardized testing a century ago. For some of them (Joel Klein) to now claim that NAEP is less reliable than the N.Y. State tests is absurd (For more, see the blog Skoolboy, on the Gotham Schools Web site, by Aaron Pallas).

At a time when the Big Boys have devastated our economy, in ways that affect us all, but are hardest—of course—on the most vulnerable, they are upping their propaganda: "It’s not our fault?” Poverty has been redefined: it’s the side effect of poor schooling! All those side effects would be cured—cheaply—if we required schools to perform properly. Schools are suffering from attracting the wrong people to teaching—“sub-par” people, in Chancellor Klein’s words—and too much allowed leeway in the practice of their mission.

They may succeed—Campbell’s Law suggests that possibility. Test scores could go up if we get the right set of tests, well-aligned with universal lesson plans, and the right incentives for sticking to them! But we might not be one inch closer to eliminating poverty or raising the level of intelligent decision-making. That’s where McKinsey gets it wrong. The magical effect on our economy of higher test scores depends on the existence of enough jobs that pay better; jobs that “require” better-educated people and that can’t be outsourced more cheaply. And scores that don’t rank! Their calculations are unbelievably naïve—at best. It’s as though if everyone’s scores went up, then everyone’s wages would, too. Who will make the hospital beds, sweep the floors, and mow the lawns? And why shouldn’t they be well-paid, too?

Andrew Delbanco’s story in the latest New York Review of Books notes that “more than 400,000 students nationally from families with incomes below $50,000” met the standards for four-year college admissions and yet were unable to attend because of financial barriers. A rich kid with low scores is more likely to go to a four-year college than a poor student with high scores, he says.


As long as we use test scores as our primary evidence for being poorly educated we reinforce the connection—and the bad teaching to which it leads. If by some course of action we could get everyone's score the same—even by cheating—I’d be for it, so we could get on to discussing the interactions that matter in classrooms and schools: between “I, Thou, and It.” I’ve spent 45 years trying, unsuccessfully, to shift the discussion to schools as sites for learning. Such a “conversation” might not produce economic miracles, but it would over time connect schooling to the kind of learning that can protect both democracy and our economy. Because that’s where schools are (or are not) powerful.

My definition of being “well-educated” varies from day to day—but it focuses on a more equal capacity for everyone to be heard in the discourses of power. For rich people, money (and connections) can substitute for smarts. They can “buy” lobbyists, bribe politicians, not to mention the influence of just hobnobbing. But the rest of us need to organize and deliberate wisely, so that we, too, can pay lobbyists and “bribe” politicians with our more numerous votes and voices.

So I judge a school in part by the preparation it provides for entering into the arguments that shape our politics and our economics. “The academics” at best should give us the tools and habits of mind to act wisely. Mission Hill laid out five such “habits:” What’s the evidence? Is there another viewpoint that fits the evidence as well or better? Is there a pattern? What if…? And why does it matter? We organized our schools, our curriculum, faculty culture, and graduation requirements to match such habits. No student leaves our schools without publicly demonstrating them over and over again.

It doesn’t level their test scores—although it improves them. But it levels some real-life challenges over the long haul. I don’t want to pretend that even our greatest schools will by themselves reverse the wisdom of an old Billie Holiday song: “Them that’s got shall get; them that’s not shall lose. So the Bible said and it still is news”…….to some folks. Unless both the "gots" and "nots" decide they have a common interest in changing the rules of the game, gross inequalities will over time erode our economy and our democracy.



Deb, ‘Tis time come, I think, to consider here the U.S. soldier.

Cataloging my issues with the US school systems, with its structures and processes and out-of-balance power centers, those issues are many and broad, and sometimes they give little hope of future progress. It’s easy to get discouraged.

To belie this dismal picture, however, comes a shining ray of hope, a reality check to me, a slap of sorts to say, ‘all is not as bad as one might think’.

This educational success story is the US soldier/Marine, tested beyond all previous tests of history, in the cities and fields of Iraq and Afghanistan.

This is not an insignificant reflection of the quality of public education across the nation.

What we asked our soldiers to do in the Middle East is unprecedented in the history the world. Never before have Captains and Lieutenants been asked as a matter of course to serve as mayors and county area viceroys. Lieutenant Cols make the strategic innovations that change the course of a war. And more to the point here, the average private, corporal, specialist is charged with making split second decisions which may both save his brother’s life and put the world on a cable news cycle for a year.

The evidence is that these young men and women, products of the public schools and little more advanced education, have proved amazingly adept at what I would call real critical thinking.

In short, no matter what the tests scores say, in one of the most robust measures of the quality and smarts of a nation’s young people, ours have stood out head and shoulders and torso above all others. This is not a small nod of approbation to the nation’s teachers. If critical thinking is the measure of a young person, ours have shown that the ed system works better than any might have imagined. Our teachers should be very proud indeed of the cognitive maturity of every soldier/marine they send to patrol and live amongst the peoples of the world.

That these twenty-something’s come from every avenue of US life should not be lost on us. From urban blight and suburban comfort to rural life of many kinds. Together they show that by one really, really important measure, we have plenty of schools doing fine.

The caveat I would throw in is this: if you talk with a young Marine, they are amazed and thankful for the US history they learn as part of their Marine training. They ask why the schools did not share this great story. And they talk much of learning teamwork instead of each person on his own, each for himself.

To their wisdom I would add I wish there were a school of economics to parallel such training. Matters economic discussed even among the thoughtful people here are replete with myth and oversimplification that just bars any forward progress of discussion. Unlearning such myths is the only path to the equality you seek.

As to young people doing critical thinking, though, at least one widely representative set is doing A OK!

Hi Deb. You wrote:


I argued for 8 years that NCLB was not about eliminating the achievement gap... it was about eliminating public schools. Those who engineered NCLB, could care less about that gap or the children it most affects. If they cared-- there would be a uniform way to assess what kids REALLY know and can do... those things would reflect real and relevant learning... and "proficiency" would be measured in the same way across all states in the union. (As it is, the bar is much higher in some states than others!)

Thus I am not at all surprised that we would blame victims of the economic meltdown for being so poorly educated... and blame the schools for failing to bridge a socio-economic/racial chasm that has existed-- by design-- for centuries.

Schools that excel in high poverty neighborhoods do so in spite of all the variables stacked against us. We do so out of commitment to the children and neighborhoods we CHOOSE to serve. No thanks to a federal government that has done precious little to address that one, consistent, decade-after-decade, predictable outlier that so profoundly influences achievement: socipo-economic status.

And even if we can't end poverty... how about the effects of poverty on learning? Thus the renewed enthusiasm so many of us have about the portents of universal health care.

Empire by armies has been practiced for centuries; the US in Iraq and Afghanistan is hardly the first country to implement civil government under military occupation. The British, French, Spanish, Chinese, Russian empires all routinely sent their young people (often from the military) out to rule. The Americans did it in the Philippines in the early twentieth century. The Allies did it in Germany and Japan after World War II. I don't seen how the American soldiers have been tested all that differently than these past military occupations.

Irrespective of the capacity of individuals to "think critically" of individual officers or enlisted Marines, they are doing an old task which in my view is out-dated in a modern democratic capitalist world. And, I really don't see how the fact that today's military is able to undertake such a task is related in any way to the quality of K-12 or post-secondary education that they received in the United States during the last 15-20 years.

Tony Waters

Tony, I would encourage you to read for not for critical evaluation, but for constructive understanding.

In your defense, the news agencies have done a dreadful job of reporting how uniquely responsibility has been pushed down to the individual soldier/marine. In WWII we often got stories of soldiers at work (think Ernie Pyle), but not so much now (exceptions: LongWarJournal and Michael Yon).

When I ask officers their opinions of the recruiting pool they face, they have two responses: first, way too many kids are not qualified for the service of their country. In general, a diploma is required, as are certain fitness standards. They prefer high school graduates, as studies show they will stay longer than those with a G.E.D. Still, the army will pay for GED classes. At present their goal is 90% actual graduates; reality has slipped to 71% hs grads.

The second thing the officers tell me is that they are pleased as can be with the young soldiers they employ. In fact, they are more than pleased, nearly awed. These veterans of past conflicts are amazed at how well their soldiers learn. And, how committed they are.

Last summer I had the pleasure of asking that question of a Civilian Aid to the Secretary of Defense. Two parts to that title: one, Civilian. His real job was with the Dayton Public Schools, in assessment. two: aid to SecDef. In this case, the Secretary paid him to fly to the Army's training centers around the country to assess needs and progress. So his opinion of the young men & women there was fairly well informed.

There are two phases in which we look at the job we asked soldiers to do. The first is the pre-surge phase where the rules of engagement tied their hands. (It is common now to blame the top commanders for these rules; honest memories will recall that world, Arab, and US opinion necessitated just such strict ROE.)

In the surge phase, units were given much more leeway to engage the enemy. In this phase, it became easier to control the Jihadi violence. Yet, it also put a much heavier burden on the individual soldier to make the right call.

The critical difference between this and all past wars is the camera on the scene. The smallest mistake in perception or judgment can instantly become an international news event used by al Quaeda and its supporters to draw sympathy for their cause, and opposition against the US led transition to democracy in these countries.

Yet, the case for mental agility is much more than just reacting to enemy action. Throughout the training, planning, and execution of all phases, we rely to an unprecedented degree on decision making at the lowest levels.

In the old Soviet era armies, for example, one could hardly get a trench dug without officer supervision. Indeed, teaching the Iraqi army the value of a highly competent non-com core was a major delay in the fielding of the new Iraqi army. They just did not get why you would take normal soldiers and give them leadership responsibility!

People who cited history as evidence that we would loose in Afghanistan from the outset; that the invasion of Iraq would take months and 100's of thousands of US lives, can in one sense be forgiven for they could not conceive of the mental agility of the modern us soldier. The unimaginable speed of the invasion was not just helped by advanced radios and satellites; it was in reality the US soldier in the fuel tanker who could be counted on to be exactly where he was needed at exactly the right time, and to offload his fuel with the maximum of efficiency.

War is always about innovation; and our engineers brought much to this one in advance. Yet it is the hundreds of thousands of small innovations by individual soldiers and squadron leaders which made these the low casualty/ high success wars they were. Few countries so raise their young people to take responsibility and innovation upon themselves. It is unique from our schools; it is the goal most teachers talk about in ed forums, and it works.

"Poverty has been redefined: it’s the side effect of poor schooling!"

Yep. And this is balanced (compounded, really) by the belief that poverty causes poor student performance.

Poverty is caused by lack of income.

Poor student performance is caused by inadvertent toxic instruction and by instructionally insensitive tests.

If one examines NAEP and ECLS-K data, the striking differences are not in the differences racial/SES differences, per se, but in the differences in performance across tasks and items.

For example very few 8th graders can handle fractions, while most can handle whole number arithmetical operations, irrespective of race/SES. There is little difference in how the "foils" in test items seduce kids to fill in a non-keyed bubble.

The opportunities for transparent instructional productivity are going un-noticed, while attention is directed to diversionary rhetoric that serves to maintain the instructional status quo.

Not only do we have to combat the insidious assumptions that poor test performance means inability to learn or worse, there's an even more pernicious association lately. I heard John Merrow speak at Teacher's College in New York this past weekend, and before showing a video clip of elementary children in a New Orleans school, he described it as "crummy, you know, 99% free lunch" even though the point of the clip was to show energetic enthusiastic children at work in school. In this respect I feel we sometimes get caught between a rock and hard place, and many people, including Klein and others, try to infer that educators use poverty as an excuse for what they claim is inadequate teaching. I wish we could help everyone to see that it's not just about raising test scores, nor is it helpful to be dismissing "crummy" schools and slowly replacing them with charters. Moreover, no matter how many hero teachers get magically pulled out of a hat, their accomplishments depend on acceptable working conditions and a social surround supportive of teaching and learning in meaningful ways, not just pick-the-right-answer all day long.

Hi All.... hope this finds you well.

I stumbled across this article and thought that Wolk rasied some interesting thoughts........


The Legacy of Five Faulty Assumptions
By Ronald A. Wolk

Most of the people running our public education systems and leading the reform movement are
knowledgeable, dedicated, and experienced.
But they are so committed to a strategy of standards‐based accountability that different ideas are marginalized or stifled completely.

Assumption One:
The best way to improve student performance and close achievement gaps is to establish rigorous content standards and a core curriculum for all schools‐preferably on a national basis.

Assumption Two:
Standardized‐test scores are an accurate measure of student learning and
should be used to determine promotion and graduation.

Assumption Three:
We need to put highly qualified teachers in every classroom to assure
educational excellence.
A great idea! If we could do that, we'd be a long way to solving our education problem.
But it won't happen for decades, if ever.
Highly effective teachers are more crucial to the success of standards‐based accountability than
anything else.
More accountability is again seen as a major part of the solution: more‐rigorous certification,
tougher teacher evaluation, and higher teacher pay. But certification guarantees a high‐quality
teacher about as much as a driver's license guarantees a good driver (emphasis added by JPB).
Tougher evaluation would help get rid of ineffective teachers, but it's hard to see how it would
produce more good teachers. Higher pay is fine, but it is no more likely to improve teaching any
time soon than raising pilots' pay would make flying safer.

Assumption Four:
The United States should require all students to take algebra in the 8th
grade and higher‐order math in high school in order to increase the number of scientists and
engineers in this country and thus make us more competitive in the global economy.

Assumption Five:
The student‐dropout rate can be reduced by ending social promotion,
funding dropout‐prevention programs, and raising the mandatory attendance age.

What do you think of his assumptions.

be well... mike


That pretty well summarizes the Beltway + Governors consensus that has held since at least 1989. To these you could add, Charter Schools, "Value added" test reporting [NB: value not specified], "merit pay" for teachers [NB: "merit" not specified, and state longitudinal data bases [NB: "data" not specified].

Meanwhile, robust data, such as the Haan Foundation Study, the IES Reading First Impact Study, The ECLS-K data base, and NAEP longitudinal data are dismissed, spun, or shoved under the rug.

The unaccountability is at the top of the el-hi enterprise.

Hi Ed,
I'm happy to read for a more constructive understanding, but I still don't see how the anecdotes people from the Pentagon have related to you give us much hope (or despair for that matter) about the state of the American education system.

The political scientist James Q. Wilson actually made a similar point about the need for militaries to delegate to field units, just like you do. In his book "Bureaucracy" he credits the effective delegation of such autonomy to German field units in 1940 as being one of the reasons that Germany overwhelmed the more centralized French army. Despite this successful application of military doctrine, the subsequent military occupation of France by the same German military did not of course lead to much success...

Anyway, we are wandering from the purpose of this blog. My central point would be to agree with Deb and Diane that schools as institutions need to be evaluated on their own criteria, and not those of business, or for that matter the military. Certainly there are things that can be learned about school organizations from business and the military organization, and vice versa. But neither offers a broad model to be emulated by public education. The tasks are simply too different.

Tony Waters

Tony, Sweet! Thanks! ... Lest we lose any more of Deb&Diane's readers, let me promise in a sec to explain this as one measure of US schools.

First, though, finish up your historical inquiry: The autonomy and skill of those German troops in France worked until... what? Until they met up with the greater autonomy and greater ingenuity of the American troops. Right?

So, its all the rage now to compare US students to their European counterparts. And well we should. Yet while we're comparing PISA scores, lets not forget to look around at other measures. In particular, those types of results that teachers claim to value most: creativity, ingenuity, soft subject analytical and synthesis ability, communications, the ability to think on their feet. Right? We've heard it said so often in forums like this?

How do we measure such intangibles? The worldwide success of Silicon Valley and Hollywood and Yum Brands and Levi's seem to indicate we're still doing something right. We're still the lead partner in the International Space Station, and holding our own in the Olympics. Our artists still take the world stage.

I'd argue, though, that the ultimate test of how a nation schools its youth is in how they are ready to defend its national interests and values.

While one can define that defense in many ways, Afghanistan and Iraq offer a great current window. Take the European response to Afghanistan. The war there was considered so vital to Europe's interest that NATO took over full responsibility. So vital that, for the first time in its 50 year history, NATO took command of an operation outside the north Atlantic area. (Indeed, Sept. 11 caused the first ever invocation of Article 5 of the NATO charter.)

How do we then compare Europe and the US in this test vital to both? In the time after NATO took command, the situation there devolved steadily. The Taliban encroached deep into the country. In 2008, command was given to US General David McKiernan; command of Regional Command East, to include half the PRT's in the country as well as operations against the Taliban in Pakistan, also goes to the US.

More to our measure here, the 20,000-30,000 additional troops now required by to stabilize that country will be coming almost exclusively from the breasts of the public schools of the USofA.

They will perform admirably, with compassion and restraint but also with ferociousness when necessary. They will be creative and ingenious when necessary, as were their grandfathers fighting in Europe before them.

Its not by far the only test of a nation's youth, but its a big one, and one where I happen to believe US teachers have done their job well.


While I do not believe as many do that there is little to be learned from business, or the military, or any other organizational structure (symphony orchestras? synchronized swim teams? hospitals?) involving human beings (or others, I suppose, if we consider bees or ants as highly socialized, porpoises as communicators, etc); I am not opposed to schools evaluating themselves, or being evaluated on, their own criteria. Of course that only begs the question of what those criteria might be.

I have spent many years in smallish non-profits dependent on the kindness of strangers in the forms of snippets of government funded programs, grants of varying lengths and purposes, local community chest efforts and local and national church giving, as well as the occasional ability to "sell" services to people with money or a payer of some kind. I have watched the funding environment change from one in which reports of the ways in which we were "doing God's work" could be supported with anecdotes, through years in which head counts (with little concern for the value of anything provided--and an absolute cacophony of costs per unit) became supremely important, to the most recent focus on service outcomes.

This has required a head-change for most agencies, who were accustomed to trotting out individuals to give testimony, or those who ran up their numbers by signing up lots of folks--many of whom dropped off after the registration drive, or delivered lots of "one-shot" services. Suddenly funders were asking substantiation for many un-examined claims of earlier years. You are "saving the neighborhood?" How long with that take? What progress have you made? You are working to prevent teen pregnancy? What was it before? What is it now? You are "building self esteem?" How do you know? You are tutoring kids? How's that working for you? Are they doing better in school? Do they know more?

It is certainly annoying to have to "prove" that something that seems good, is in fact good. But at the same time, I have seen many squeeky wheels get greased in some dubious ways. I have watched established agencies asked after a few years to rescue start-ups who sold themselves to funders as meeting some need that no one else was taking care of. I have seen agencies that should have been providing quality services to meet a legitimate need flounder and become more significant as employment agencies than anything else. In a world where all resources are finite, and particularly those dollars assigned to help the poor, unhealthy, illiterate or otherwise left-out, the annoyance is, at a minimum worth it.

More importantly, however, such efforts can motivate and agency to examine carefully what it is about and to determine what measures ought to be used to evaluate progress and needs. I suppose that education could move forward in such a way--with highly localized individual measures keyed to exactly the things that a school is about. Habits of Mind is sure to have, or there could be developed, measures that would pass muster with regard to reliability and validity. Certainly softer things have been studied over time. And given the freedom to utilize such measures in a reponsible way, I would wager that some small percentage of excellent schools might elect to go that way. I would also wager that the majority would elect to keep on marching in the direction already laid out. There might also be a percentage of scoundrels who make attempts at using the system to cover things that they should be doing but are not--selecting weak indicators of inconsequential things.

Perhaps a mixed model might serve better. Continuation of standardized measures (which if nothing else provide for comparative analysis), allowing for additional measures to be combined, and development of reliable measures of some of the so-called "soft skills." Some localized measures may provide impetus for further research and development as long-term results are tracked.

There are lots of ways to dream. But, I don't see any way around some kinds of highly public measures for which all are accountable--if we are to move forward with any concerns for equality and equity in education.


I think that all are agreed that highly public measures are important in evaluating schools. For that matter, so are aggregated test scores, like NAEP, SAT, PISA, assessments tests or whatever flavor you may choose. But while these tests can inform values and judgment, they are not a substitute for them.

So I would agree with you that if there is to be concern about equality and judgment in education, reliable measures should be taken of achievement. But, in addition, sound judgment needs to be exercised in how such statistics are evaluated and used.

Perhaps I am a little sensitive on the subject because I just spent some hours reading Stephen Jay Gould's book "The Mismeasure of Man" about the uses and misuses of intelligence testing. Too often such exams are "reified" as a real thing which has real consequences for how money is allocated, and children tracked. When this happens, despite the apparent transparency, the goal of educational equity is frustrated.

Comments are now closed for this post.


Follow This Blog


Most Viewed on Education Week



Recent Comments