GothamSchools — daily independent reporting on NYC public schools

A few years ago, the New York State lottery’s slogan was “Hey, you never know.”  In its original formulation, the slogan sought to motivate New Yorkers to play the lottery, a game of chance, on the grounds that you never know unless you play if you are a winner.  But the slogan is a double entendre when applied to Caroline Hoxby’s highly-publicized study of the effects of attending a charter school in New York City.  Propelled by Hoxby’s forceful claims about the superiority of lottery-based research on charter schools, much of the mainstream media has concluded that we now know definitively that New York City charter schools outperform their traditional counterparts—in spite of the fact that her study has not undergone a rigorous peer review process that might identify problems in the study and ways of addressing them.  Today, however, an equally forceful critique prepared by Sean Reardon of Stanford University argues that Hoxby’s research is anything but definitive.  Citing flaws in the statistical analysis of the report, Reardon writes that it “likely overstates the effects of New York City charter schools on students’ cumulative achievement … It may be that New York City’s charter schools do indeed have positive effects on student achievement, but those effects are likely smaller than the report claims.”

Reardon is careful to point out that it’s not possible, based on the information provided in Hoxby’s report and associated documents, to judge the extent of the bias in Hoxby’s estimates of charter school effects on student achievement.  More than anything, he calls for reserving judgment until more information about the study, its data and methods are available, and until the study has undergone rigorous peer review.  Until then, he maintains, it would be unwise to rely on the statistics reported in the study, and the inferences Hoxby and her colleagues draw about charter school effects in New York City.     

Here I’ll mention two of the features of Reardon’s critique that I find particularly persuasive.  The first is that Hoxby used an inappropriate set of statistical models to analyze the data, which likely distorts the charter school effects.  You might be surprised to learn that Hoxby used statistical models at all.  If her results are based on comparing students who won a charter school lottery with students who lost the lottery, and the lottery was fair, balanced and random, why would a model be needed?  It seems like the charter school effect would simply be the difference in the outcomes observed for the lottery winners and the lottery losers.  But comparing lottery winners and losers isn’t really estimating an individual causal effect, because an individual student can’t simultaneously be enrolled in a charter school and a traditional public school.  Even in the context of a lottery, or any other kind of study that can capitalize on a randomization process, such as a clinical drug trial, statistical models come into play to allow for inferences about cause-and-effect relationships.  These inferences are always made in relation to a particular statistical model, and all such models have assumptions.

One of the assumptions that is widely recognized is that a statistical model for causal inference should take account of factors that precede selection into the “treatment”—in this case, enrollment in a charter school versus a traditional public school.  If, hypothetically, charter school attendees were wealthier than traditional public school attendees, we’d want to control for wealth to make the charter and traditional school attendees as comparable as possible.  But it’s just as widely recognized that such a statistical model should not take account of factors that are measured after, and hence potentially influenced by, the treatment.  If attending a charter school increased a student’s motivation, and heightened motivation yields better test scores, then we wouldn’t want to control for motivation in a statistical model for the causal effect of going to charter school on test scores.  That kind of control means that the charter and traditional school attendees are no longer comparable at the time that they began attending a charter versus traditional school, which is the critical time.

Reardon demonstrates that this is precisely what Hoxby and her colleagues do in most of their statistical analyses.  For the analyses of charter school effects on test scores in grades four through eight, she controls for achievement in the prior year—achievement that was observed after the lotteries that determined whether a student enrolled in a charter or traditional public school.  The effects of charter school attendance on test scores in grades four through eight are therefore distorted, but to an unknown degree.  This is not a problem for estimates of the cumulative effect of charter school attendance in grades K-3 on third grade test performance, because the statistical models don’t include prior test scores (as there aren’t any before grade three.)  Reardon therefore finds Hoxby’s estimates of the effect of going to charter school in grades K-3 to be more credible than those for grades four through to eight.  However, the K-3 effect is only one-half to one-third as large as the estimated annual effect of charter school attendance in grades four through eight.    

The second issue is estimation of the cumulative effects of charter school attendance.  Hoxby’s report gets a lot of mileage out of the claim that the effects of attending a charter school from kindergarten to grade eight are large enough to close the performance gap between (predominantly white, upper-class) children in Scarsdale and (predominantly minority, low-income) children in Harlem by 66% in English and 86% in math.  Reardon points out that these figures are based on unrealistic extrapolations.  You can’t simply add up the annual effects of attending a charter school from year to year because the gains decay over time.  Moreover, most of the students in the Hoxby study have been in charter schools for only three or four years;  virtually none have been enrolled in charter schools for as many as nine years, and those would only have been enrolled in the very small number of charter schools that have been open that long, and cannot tell us about the long-term effects of attending the much larger number of newer charter schools.  Reardon’s analysis suggests that Hoxby’s estimate of the cumulative effect of attending a charter school from grades four through eight could be exaggerated by as much as 50%.

Are Caroline Hoxby’s estimates of the effects of attending a charter school rather than a traditional public school in New York City accurate?  Maybe.  But based on Sean Reardon’s critique, probably not.  Hey, you never know.