Arnold Kling  

Short-term Success, Long-term Fizzle

Except Seasteading... We All Wanna Change the World...

Gordon Dahl and Lance Lochner write (I can't find an ungated 2008 version of the paper, and the 2005 version seems to differ),

Our baseline estimates imply that a $1,000 increase in income raises combined math and reading test scores by 6% of a standard deviation in the short run. The gains are larger for children from disadvantaged families and are robust to a variety of alternative specifications. We find little evidence of long-run income effects, with most of the effects disappearing after one year.

In studies of interventions designed to improve education outcomes, a common result is to find statistically significant effects in the short run, but not in the long run. Researchers will report that "X improves test scores in 4th grade, but by 8th grade the difference has gone away." Why is that? Possibilities:

1. Genetic factors are too strong. You can fool mother nature for a little while, but not for long. People tend to revert to their genetically-determined level of ability.

2. Other environmental factors are too strong. You can overcome environmental determinants for a little while, but nor for long. People tend to revert to their environmentally-determined level of skill development.

3. Over time, more random factors are introduced into the lives of the subject populations. Note, however, that this should lower the R-squared in the regression and might bring down the statistical significance of treatment effects, even if it does not cause the magnitude of treatment effects to disappear But researchers are fixated on statistical significance, so they interpret a drop in statistical significance as if it were a fall-off in the effect of the treatment.

Comments and Sharing

COMMENTS (5 to date)
OneEyedMan writes:

This is also consistent with intertemporal substitution of education combined with lifetime inelastic demand. If there is a lifetime amount of education that the students want to acquire then paying for more of it now reduces how much the want later.

Just as some argue that raising wages doesn't influence our lifetime labor supply, decreasing the cost of education may not influence the lifetime investment in education.

Wm Tanksley writes:

Are you saying that long-term dropoff is a problem totally unprecedented in statistics (and therefore completely specific to educational results), or that educational statisticians in specific are ignoring crucial prior work in their field?

I'm not sure what conclusion to draw from this post. I certainly see that you're not looking deep enough into root causes; a study with such a simple statistical error is a deeply flawed study, and to have such a profound flaw be so common... Well, usually it suggests that the person who's found the flaw doesn't understand the study.

floccina writes:

Could it be that it is harder to bias the long term data?

ed writes:

I thought of number 3 as well.

I think it's a real possibility...even people with PhDs confuse statistical significance and substantive significance all the time. It's just very very common to see someone report "X has no effect," when all we really know is that the effect of X is merely imprecisely estimated such that we can't even be sure of the sign.

OneEyedMan nearly has the answer. However, the it is not the students who demand education; most 4th-8th grade students have little independent desire to learn the things school tries to teach them.

It is the school which demands that students acquire a certain level of knowledge. In middle school students are typically retaught the same topics in math each year. The schools rarely expect them to actually learn the material and retain any knowledge.

The basic problem is that the amount of math knowledge demanded by the school is both price-inelastic and very, very low.

Comments for this entry have been closed
Return to top