It sounds from your analysis that all those principals would be better off running charter schools.

They could cap their enrollments, balance their own budgets to produce the class sizes they want, and generally remove the impediments that you describe.

Diane Ravitch

I understand the tradeoff that the DOE faces in constructing comparisons that are simultaneously meaningful and easily communicated to diverse audiences, some of which are sophisticated about research and statistical analysis, and others of which are not. Value-added approaches are challenging to explain; the DOE’s peer index, for better or worse, is not. But the evidence is clear that the peer index does not adequately adjust for real differences in a school’s resources, and therefore results in a distorted picture of how schools are faring head-to-head.

I’m not sure I understand your point about propensity scores, which generally work best when there are two clearly-defined treatment and control groups, and the researcher models the propensity for a case to be selected into the treatment condition. Generally, the propensity model is indifferent to the particular outcomes to be examined, because those outcomes are observed only after selection into treatment or control condition has occurred. You wouldn’t have separate propensity models for different outcomes.

But it’s true that the attributes on which we would want to match schools might be multidimensional. That’s why I’m suggesting a latent-class approach, in which schools are classified into latent classes that are as homogeneous as possible on the set of attributes.

Class size is a good example of some of the challenges involved in doing this kind of matching. The idea of this kind of matching is to control for factors that are logically prior to a student’s entry into a school, but *not* to control for factors that are under the school’s control as a matter of school policy or practice. The DOE might argue that average class size is a choice made by principals about how to allocate the resources under their control. On the other hand, you might argue that class size reflects a school’s enrollment, the amount of physical space available (i.e., capacity) and a school’s budget, all of which are outside of the school’s (and principal’s) control, because they’re imposed on a school by the DOE. Adding class size as a factor wouldn’t add any new information if it really is highly determined by enrollment, capacity and budget, but if there were differences across schools in their average class size even when enrollment, capacity and budget were equal, then we might think of that variation as something under the school’s control, and hence not to be matched on in constructing peer groups.

I strongly believe that, if we are to make inferences about the relative performance of schools, we should take account of the different resources available to different schools, and these certainly include enrollment, capacity and budget measures.

Incidentally, the use of class size in the teacher data reports is less ambiguous, because even if class size is under the control of the school and principal, teachers generally don’t have the ability to choose the size of their classes. That’s one of the limits of the teacher value-added approach: if teachers are assigned to classes of different sizes in a nonrandom way (i.e., a principal wants to reward or punish a teacher with a small or large class, or just thinks that a particular teacher might do better with a smaller or larger class), then simply controlling for class size may not adequately explain why a teacher winds up with one class versus another.

When different factors are put together like this, can’t they end up offsetting and counteracting in unexpected ways? I know that propensensity scores compress many attributes into one dimension, but is that what we want to do here? I mean, do we we want to create propensity scores for doing well on one test or the other?

If we did that, wouldn’t we need multiple peer groups for the different factors we want to look at? Like a peer group for attendence, a peer groups for teachers experience, a peer group for test scores, etc..

While methodologically more rigorous and more useful for those who want to be careful about meaningful comparisons, that might work. But that doesn’t seem to be the goal of this DOE or such public reporting generally.

So, what is the alternative? A single peer index, like this? How good can such an approach ever really be?

Correlated factors actually make things easier, because the correlations reduce the number of dimensions on which schools must be compared. The approach would be quite a bit more complicated than the peer index that the DOE uses, but generally the idea is to use the observed factors to classify schools into a set of groups. A technique known as latent class analysis would sort schools into groups in which the schools are as similar as possible within a given group, based on these observed measures, but with each group as different as possible from other groups.