GothamSchools — daily independent reporting on NYC public schools

good question. I haven’t read it either, but from the Edweek article:

“The study—the first independent examination of the program’s effectiveness—includes principals who remained at the same school for three or more years. Using the data from those years, researchers compared the scores of the average student in each of those schools with the citywide grade-level average.”

Hi everyone -
I’m one of the co-authors of the Leadership Academy study, and would just like to contribute my two cents to this discussion.

First off, the headline above is probably not the best choice of words to describe our study — we actually do have conclusions. The “inconclusive” part of the study relates primarily to high schools. We had hoped to do as thorough an analysis of high school achievement, but at the end of the day there were too few eligible HS principals to say much with confidence. At the elementary and middle level, a better word to describe our results might be “mixed.” After a few years, APP schools were improving at a slightly faster rate than comparison schools in ELA, but in math they improved at comparable rates.

On the subject of selection bias, this is always an important issue. There are several varieties of selection bias, and we were not able to address them all in this study. For example, those who sign up to participate in the APP program are not a random draw from those who might become principals, and principals are definitely not randomly assigned to schools.

In addition, we focused only on principals who stayed in place 3 years or more. It could be that those who stay are more (or less) effective than those who leave early, as Prof. Pallas said. One could call this “attrition bias.” The question is whether this was more or less true for APP principals than comparison principals. We don’t address this in the study, but it’s not obvious to me that it should work one way or the other. It’s also not obvious to me that the least effective are the most likely to leave their school early. It could be that more effective leaders are hired away by other schools, are promoted to district positions, or are charged with phasing out a closing school (which some did–they had to be excluded from the study).

Unfortunately, I can only speculate. We didn’t look specifically at these questions in the study, and doing so is probably harder than it sounds. It’s important to know if/when principals stay in place for only a year or two (or less). But when they do, they don’t leave much of a track record behind for us to assess.

As is true for all research, this study should not be considered the last word on the APP program. There is much in the study we did not (or could not) address, and still much to learn.

Thanks to everyone for their interest -
Sean Corcoran
New York University

Thank you to Drs. Pallas and Corcoran for contributing on a public website and your comments on this study in particular.

I’ll throw an anecdotal piece of information into the statistical stuff just to bring the overall findings into street-level view. P.S. 194, which has been in recent news for its potential closing and reopening under the Harlem Success Charter School banner, had a revolving door at the top of the school several years ago. One such principal was from the Leadership Academy. He lasted less than one year. His next destination? The rubber room. In research terms it’s “n=1″ but to those who worked at the school it was another failure – one which didn’t make the cut in the study.

I appreciate the hard work that goes into number crunching. This is a difficult area to study and it’s even harder to practice in a politically charged atmosphere. However, my instinct tells me that there is a lot of data that is missing from the conclusions, and that missing data renders the findings highly suspect…or unreliable if I may use a statistical term. Knowing what I’ve observed in other schools I’d be hard pressed to find a LA principal who could demonstrate a significant advantage above what other principals without LA training might show. True, current data from Dr. Corcoran’s study yields the opposite conclusion, and I’m just objective enough to give it credence, but I’ve yet to note a difference at the locations I’ve seen with my own eyes.

As an aside, please throw all the wonkiness at us. It may be obvious that your educational and professional credentials are unassailable in the academic world but in the blogosphere it always helps to establish street cred. And it’s extra motivation to dust off my old stats books to get a better feel for the real meaning underlying the narrative.

OK, I’ll let the wonks draw their SPSS swords and watch the battle from afar.