GothamSchools — daily independent reporting on NYC public schools

This is not quite right, though it is very close. The statement “But the NAEP test, which is given only to a sample of students across the country, not to every child, does not permit the kind of detailed comparison Klein’s statistic would demand,” is actually highly misleading.

1) This problem is NOT a product of the fact that the test is given to only a sample of students, or even because no single student gets the entire set of questions. You could test every kid (in a given grade) in the entire country with the entire test, and you still couldn’t make this kind of comparison.

2) This is an issue of test design, not test administration.

3) The issue is called “linking.” That is, are the different tests (i.e. the 8th and 12th grade tests) linked? And and they are not. (I’m stretching my memory a little bit. Linking MIGHT not be the right term, but I think that it is.)

4) This mistake is easy to make because they scores are reported on very similar looking scales, with the 12th scores generally greater than the 8th grade scores. However, this is neither a design nor an administration issue. They could report the scores on any scale they want. It’s an arbitrary decision, and one that has been made poorly. This mistake is not actually that uncommon, because of the dumb way the scores are reported. But the people who run the NAEP chose to report them this way, anyway.

5) I believe that they actually broke the link between sucessive administrations of the NAEP at one point. That is, we want to be able to look at this year’s results and compare them to the results of last time, and the time before that and the time before that. And you can generally do that, and are supposed to be able to do that. But one time, you couldn’t. So, you can compare the results of every administation before that poin to each other, and every administration after that point to each other. But you can’t compare across that broken link. But because they kept the same scale, people make this mistake all the time.

******************

This is all incredibly important because of the way the the country — the policy makers, the practitioners, the public, the media and everyone else — looks to test scores as the most important educational outcome. (Heck, I would suggest that most poeple look to test scores as the only important educational outcome.) And yet, a tiny fraction of a tiny fraction of a tiny fraction of people know about testing to understand the the reports might actually mean. Even policy professionals, knowledge of educational measurement is quite poor.

Alas…

Yes, “vertical scaling” as the term I was looking for.

Thank you, Anon.

************************************

A little bit more on vertical scaling.

There are plenty of tests out there that are vertically scaled across many grades, even all the way from K through 12. Obviously, the different tests for each grade will focus on different material. This does not prevent them from being vertically scaled.

There are technical reasons why the 8th grade NAEP might not be scaled with the 12th grade NAEP, but there are a bunch of practical reasons, too. It costs time and money to do that kind of thing, for example. Also, there are assumptions you have to make about that material in order to vertically scale, but I think that these assumptions have already been made to generate the scores that exist. (We make assumptions in our work all the time, based on experience and judgement. These are not unwarranted assumptions or rash assumptions. Rather, they are define the problem we are working on. Others might view the problem differently, and therefore have different assumptions.)

I don’t buy Goldstein’s explanation for why the two scales are not equated, because what he talking about does not make it impossible. I do not know if he is trying to give an explanation that is easy for laypeople to understand, or if he is intentionally trying to obfuscate, or if he thinks that this is actually the reason. But I am fairly confident that he is wrong.