It’s all about “regression to the mean.” While focusing on the streaks.

Derek Jeter just tied Lou Gehrig for most hits by a Yankee. Ever. Last week, he went 0 for 12.

Same thing.

Next week, odds are he’ll go 1 for 3 at any given at bat, regardless.

Klein would give him an F, then an A. Or send him to a charter and give Girardi a C.

Then there’s the no. and %age of kids to whom principals, in their sole, unreviewable discretion, could grant testing modifications and/or accommodations, such as extended time, etc., because the principals decided the kids had experienced temporary medical disabilities … . I understand that NYCDOE’s %age of kids given these mods and accoms on the NAEP tests were off the charts. Why would one assume it was just limited to NAEP testing?

And how do scores which (allegedly) show whether a kid has attained standards for one grade tell whether the kid has made a year’s worth of progress measured by whether the kid has answered questions relating to a totally different set of content standards the following year? It seems to me that this is like comparing apples and oranges.

It’s not as though one must master every area of the ELA curriculum in 4th grade for one to master every area of the ELA curriulum in 5th grade, is it?

If these were diagnostic reading and math tests, comparison of year-to-year scores would show something significant. But if fractions aren’t in both the 4th and 5th year math curricula, and if knowing 4th grade fractions isn’t necessary in order to master the 5th grade math curriculum, or at least score well on most, but not all, of the 5th grade math questions … progress? I don’t think so, but I’m not a psychometrician.

But then, I don’t recall that Joel Klein-Bloomberg is, either.

However… DOE then uses the negative percentawhatevers to pull down — below zero — what should be re-normed to zero. Seems like a double-hit of sorts.

Next, look at how non-intuitive a number of the so-called “progress” result bars are in the only two failing schools.

Repeating myself from a prior post’s comment: How does it pass the sniff test that at one school, there can be *simultaneously* above the 100% of range (average change in ELA proficiency for kids at Level 3 or 4) and below the 0% of range (similar for kids at Level 1 or 2). That’s some dichotomy.

And yet… the actual change in proficiency was 0.10 and 0.09 respectively. That’s some dead heat.

Above true for both peer and citywide spectra.

This happened at Harlem Link Charter, one of the two that got an “F.”

Time to bring back the predictor monkey, only this time get the monkey drunk first.

The second %, which is used for calculations of scores for each of the components of the Progress Reports, is not a percentile. Instead, it represents the location of a given school in relation to its peer group or in relation to schools citywide. Consider attendance. Each school has a peer group of 40 similar schools, each of which has an average attendance rate. Suppose that the lowest of the peer schools is 83%, and the highest is 96%. (There’s some way of tossing out outliers that are judged to be unusually low or high, but that process isn’t explained in detail.) The “peer horizon” thus spans the range from 83% to 96%. Now suppose that School X had an attendance rate of 86%. School X’s attendance covers 3 percentage points of the 13-percentage-point range from 83 to 96. 3/13 is .23, or 23%. So School X has traveled 23% of the distance of the peer horizon.

Now, suppose that School X has an attendance rate of 97%–it actually exceeds the attendance rates of all 40 of the schools in its peer group. School X has covered 14 percentage points above the peer minimum, which exceeds the 13 percentage points of the peer horizon range. 14/13 is 1.08, or 108%. So School X’s attendance is 108% of the peer horizon range. But it’s not a percentile. Similar logic if School X were to have a value that’s below the minimum of the peer horizon range; that’ll result in a negative percentage of the amount of the range that’s covered by School X’s value.

Jennifer: I’m using the percentage of students making at least one year of progress, as defined by the DOE. This uses the so-called Proficiency Ratings, which are the DOE’s transformations of the state’s four proficiency levels. No reputable psychometrician I know approves of this peculiar transformation, but it’s probably not at the root of the problem of unstable year-to-year student “progress”.

There once was a young educator,
Said “‘Progress Reports’ are simply inflators!
But Chancellor Joel Klein
Says, “There doing quite fine…
We’ll bump the other 16% later.”

I was reading the negative and over-100 percentiles from within the 2008-9 School Progress Reports — off a mix of the various performance and progress sub-metrics — for the only two schools that got an “F” this year and first year reported; from the top UWS school that went D-A-A over the last three years; and from PS8 in Brooklyn that went C-F-A. In order:

Manhattan’s Washington Heights Academy
Manhattan’s Harlem Link Charter School
Manhattan’s MS243
Brooklyn’s PS8

Email me (witzeroo at yahoo dot com) if you want the links via the DOE’s school data portals. I don’t want to get stuck in moderation.

I get the “fluke.” I do NOT get why the DOE is resistant to FIXING it. Specifically, I have called on DOE to REDUCE the weight on the flawed progress metric. Instead, between Yr 1 and Yr 2, they increased it from 55% to 60% at the expense of the performance metric, which went from 30% to 25%.

Dr. P (or Dr. J),
I need some professional help.
Sincerely, please look at my comments on related stories.

1) What is the impact of negative percentiles, or over-100th percentiles on the resulting scores? You touch on this above with your reference to the lack of resetting the benchmarks. That intuitively contributes not only to inflation, but I believe to the undercutting of the internal roll-up math as well.

2) Clearly, no one at DOE understands calculus, specifically derivatives — rates of change OF rates of change can get pretty tricky.

But we need to measure student progress somehow, right? As a teacher, I want my students to make a year’s worth of progress in a year in my class, and I want the students in my school to demonstrate that sort of progress in all their classes. How can we do better measurement? Are there different criteria we can use?

I also notice that this links with Daniel Koretz’s work on the unreliability of value-added measures. These failed progress reports show that you can’t judge a school on the basis of year-to-year changes in test scores, even when you put together the work of dozens of teachers. If this is so, how could you possibly evaluate individual teachers on the basis?

The “accountability” geeks have got a lot of work to do. Hope they’re listening to what you’ve got to say.