I only care about test results insofar as they are truly reflective of increased student learning. If students fail bad tests but are learning, I am happy. If students do better on tests, but are not learning, I am not.

Do I want students to fail? I want the tests to accurately reflect student learning. If that means they fail the tests, then they should fail the tests. If these tests have any education value, it would be to draw attention to differences in learning and the levels to which students are learning. If tests hide or obfuscate those things — by design, by happenstance or by inappropriate test preparation — they actually can be net-detrimental to education.

******************************

Let me give a (rather simplified) analogy:

Nine of the top ten on the career “assist” list in the NBA played in the last 20 years, out a 50+ year league history. There have been rule changes that have made it easier to get an “assist.”

Does this mean that nine of the ten the best passers played in the last 20 years? Should we celebrate the improved passing in today’s game compared to the 1960′s?

*******************************

When the rules for measuring change (e.g. rule changes or different tests), we shouldn’t celebrate higher measurements as necessarily indicating improved underlying performance. If that was appropriate, we could virtually eliminate achievement gaps by making all the tests ridiculously easy. But that would be too obvious and no one would believe that it was significant, right?

Well, what if a test is a little bit easier (due to design, happenstance or inappropriate test preparation)? That wouldn’t be quite so obvious, right?

I generally question the meaning any quantitative result that is not accompanied by a researched explanation of the mechanism(s) that produced that result. And as improved test performance is not what I consider the real goal of education/schooling, I do not celebrate it for its own sake.

When scores (pardon the pun) of school district superintendents throughout New York State can boast of comparable — or greater — increases in their students’ test scores as can NYC’s Chancellor Klein, it should call into question whether or not his policies are the cause of the test increase. (Or perhaps even holding back NYC results from reaching Buffalo-level results.)

Does not a rooster take credit for the sunrise?

As to the comment in the main post re “gains by black and Hispanic students drove the improvements,” note that these two groups (along with Spec Ed and ELL though in smaller quantities) are disproportionally represented in the overall group struggling to achieve proficiency. So OF COURSE, any closure of that gap would have to be led by those groups. If they weren’t improving, the average wouldn’t either. I’m glad that they are, but we need to see through some of the circular logic.

Spot on re “ceiling effect.”

In NYC and NYS, and perhaps elsewhere, the “Performance Index (PI)” formula treats 3′s the same as 4′s.

Bragging about raising 1′s to 2′s, or 2′s to 3′s, and declaring victory is nonsense.

As long as it is possible that hordes of 3′s made it to 4 and the TRUE gap may have actually gotten wider, that’s simply cooking the books.

Campbell would say the PI is nothing but soup.

Closing achievement gaps can be done without with closing real learning gaps. All you need is an easy enough test.

There is something called the “ceiling effect.” The idea is that when enough members of a group start to top out on a test, the average of that group stops increasing — or at least slows down. Then, lagging groups can catch up some. I believe that Texas is infamous for having done this. That is, an easy state test had a strong ceiling effect, and minority groups average scores narrowed the gap with white scores.

Any time I hear that achievement gaps are narrowed, I wonder if that is part of what is going on. That is, I wonder if it is an artifact of the test rather than improved teaching and learning.

My second guess is a greater increase in emphasis put on test preperation with the lagging group than the leading group. This could happen if the leading group already did a lot of test prep, or if the lagging group gave up more other things to focus on test prep.

Yes, I am cynical about about reports of narrowing gaps. Until changes in teaching and/or conditions have been shown, I think that campbell’s law is probably disproportionately at work.