All a study like this can do, since it cannot be completed on a scientific level, is hand people like Mr. Hirsch something to point to, as evidence for what they have already decided. Despite, of course, the study showing nothing.

Yes, the conflict of interest is immediately obvious. But the only good research I have seen recently in education has been Eduwonkette’s anti-research, showing the world that the data the DoE (and others) pump out is full of holes.

I don’t think that ed schools preparing teachers have, as a group, been very successful in making the case either that they transform the knowledge, skills and dispositions of prospective teachers, or that these qualities increase the likelihood that children and youth will learn what society expects. The lack of compelling evidence makes ed schools vulnerable to criticism. But I’m not sure that the existing literature is all that relevant to key policy issues regarding teacher supply and demand. Alternative certification is here to stay.

eduwonkette: I haven’t figured out how I feel about the effect size argument made by the Pathways group. Sure, if you adjust for random measurement error, you’ll see bigger effects–there’s nothing exotic about that. But there’s a difference between modeling and predicting outcomes, and it seems to me that the predicted and observed outcomes will still reflect the dampening effects of measurement error.

Jonathan: There’s more than enough blame to go around on the Mathematica study, but one thing I’d point out is that the report was prepared under a contract from the Institute of Education Sciences. This means that Mathematica was contractually obligated to deliver a report to IES, which could then release the report at its discretion. In my view, the main problems with this study were in its design more so than its execution, but it can be difficult to reconstruct how and why particular design decisions were made.

This study was too poorly designed to draw conclusions.

With a sense of integrity, a researcher would have declined to publish. But they know that there are advocates out there who will treat the null conclusion as a conclusion, and run with it.

Many scholars are now reexamining certification and coursework studies in light of two considerations: 1) that the issue of measurement error has confounded estimates of the effects of teacher observable characteristics on student outcomes, and 2) that the selection of teacher candidates into different kinds of training and education experiences makes identifying the effects of these factors difficult.

Here is the critical part of an abstract from a study that is addressing issue 1:

When teacher attributes [i.e. certification, college attended, SAT scores, experience] are considered jointly, based on the teacher attribute combinations commonly observed, the overall effect of teacher attributes is roughly half a standard deviation of universe score gains – even larger when teaching experience is also allowed to vary. The bottom line is that there are important differences in teacher effectiveness that are systematically related to observed teacher attributes. Such effects are important from a policy perspective, and should be taken into account in the formulation and implementation of personnel policies.

http://www.wcer.wisc.edu/news/events/VAM%20Conference%20Final%20Papers/MeasuringEffectSizes_BoydEtAl.pdf

Thanks for this post: it summarizes a long paper in a clear manner and gets to the heart of the results. I wasn’t aware of some of the specifics of the sample set.

You write: “… if you’re a Texas principal interested in hiring an early elementary grade teacher with a few years of experience into a small classroom based on generic standardized test scores, this is the study for you.” The study could also be useful to policymakers, principals, and others that are willing to consider the data to be relevant to, if not the same as, their own circumstances. Your list of particular facts about the data can help them to judge the relevance.

Regardless, if I read it correctly, the study gives an example of traditional certification and extra coursework having no impact on effectiveness. My understanding (perhaps incorrect) is that the vast majority of studies on this general subject reach similar conclusions. I would be curious to hear about your favorite studies that reach different conclusions.