GothamSchools — daily independent reporting on NYC public schools

“I think there’s nothing wrong with anything.”  So spoke Chancellor Joel Klein at yesterday’s release of the 2009 elementary and middle school progress reports.  As Anna Phillips reported, 84% of the schools received a letter grade of A, and an additional 13% received a B.  Only two schools out of 1,058 received an F, and just five more were awarded a D.

The letter grades were driven by the remarkable/suspicious gains in 2009 on the state’s ELA and math tests.  Schools weren’t actually compared to one another on their performance this year to derive the letter grades.  Rather, they were compared to last year’s peer and citywide benchmarks.  To use a football metaphor, because test scores rose across the board, virtually all schools moved up the field, but the goalposts didn’t move.  I wasn’t sure that the progress report letter grades could actually be less useful this year than last, but Chancellor Klein’s administration has achieved that dubious feat.  When 84% of the schools receive an A—the top grade, which everyone understands to signify excellence—what useful information about the school’s relative performance is being conveyed to parents, students, educators, and others with a stake in our schools?  Not much, in my view.

Last year, my blogging partner Jennifer Jennings (who, for those keeping score at home, is now Dr. J) and I were sharply critical of the 2008 school progress reports.  Writing on Jennifer’s eduwonkette site, we demonstrated that student achievement growth over the past year—which makes up 60% of the overall progress report letter grade—was highly unreliable.  Schools that demonstrated high gains in student achievement from 2006 to 2007 were no more likely to show gains from 2007 to 2008 than schools that showed low gains in 2006 to 2007.  We concluded that the measure of student progress making up 60% of the overall progress report grade was picking up chance fluctuations from year to year.  And if 60% of the score is random, there’s not much genuine information about school performance in the progress report grade.

It wasn’t a fluke.

I’ve now replicated our analysis using the 2009 progress report grades.  Many measures of school performance—its environment, based on parent and teacher surveys, its attendance, even the percentage of students achieving Level 3 or Level 4 on the state ELA and math tests—are fairly stable from year to year.  But once again, the student progress measure is wholly unpredictable.  The schools that showed big gains in student achievement from 2007 to 2008 were no more likely to show big gains from 2008 to 2009 than those schools that had small gains from 2007 to 2008. 

Here are some simple graphical representations, using the roughly 600 elementary schools in New York City that received progress report grades in 2008 and 2009.  The first graph plots a school’s academic expectations in 2009, based on surveys, with that school’s academic expectations in 2008.  There is a strong positive correlation of .75, which indicates a great deal of stability from 2008 to 2009 in the relative position of a school compared to other schools.

The second graph plots a school’s attendance rate in 2009 with that school’s attendance rate in 2008.  Attendance is highly stable from year to year:  the correlation between 2008 attendance and 2009 attendance is .95.  There’s almost as much consistency in the percentage of students scoring at levels 3 or 4 on the state ELA test over these two years.  The correlation between the 2008 percentage and the 2009 percentage is .94, which is a very strong association indicating that schools that are relatively high in 2008 are likely to be relatively high in 2009.    

As you look at the three graphs above, you can see a clear pattern in the data:  elementary schools that are relatively low compared to other schools in 2008 are also relatively low compared to their peers in 2009.  Similarly, elementary schools that are relatively high compared to other schools in 2008 are likely to be relatively high in comparison to other schools in 2009.  Now, look at the graph below, which plots each school’s student progress from 2008 to 2009 by that school’s progress from 2007 to 2008.  Do you see a pattern?

 If you do, there’s probably a psychologist somewhere salivating at the idea of administering a Rohrschach inkblot test to you.  The correlation between student progress from 2007 to 2008 and student progress from 2008 to 2009 is -.02, which is indistinguishable from zero.  This means that there really is no pattern to the results, and certainly not a pattern that demonstrates consistency or stability from one year to the next.

It’s no surprise to see this, since it follows from what we know about the instability of test score gains from one year to the next in schools across the country.  As was true last year, there’s a bit more consistency in progress in math, but the state ELA and math tests on which these scores are based are coming under increasing scrutiny for their predictability, easiness, and poor content coverage.  The New York City Department of Education has hitched its accountability wagon to a runaway horse.

When it comes to annual test score gains, randomness is not a fluke.