GothamSchools — daily independent reporting on NYC public schools

Maybe you’re right that politicians are taking time out from their fundraising activities to write questions like this, but I’d be willing to wager at least a nickel that there are psychometricians involved, and that they have some reason for thinking this question a useful proxy for mathematical knowledge. What are those reasons, and why are they wrong?

Well, I know that in Arkansas (where I live), the Department of Education requires that a committee of “nationally recognized testing experts and psychometricians selected by the Commissioner of Education . . . shall advise the Department in all technical aspects of the assessment system.” To be sure, New York has a reputation for being a bunch of rubes and hicks compared to Arkansas, but I find it hard to believe that New York doesn’t involve psychometricians somehow.

Anyway, maybe that’s a bad question — it strikes me as way too easy, although 33% of kids still managed not to know the answer even when given a choice. So are you trying to say that they should have been asked to prove the quadratic formula by completing the square, or something like that?

Stuart & Ken,

Interestingly, this particular standard has not appeared on any other NY state assessment since 2006, which is when the technical reports began indicating which standard a particular item was to represent. I think the item is problematic because it’s simply a statement of the standard. A student could memorize the words without ever knowing what a polynomial or algebraic expression was. A better item, I imagine, might present an algebraic expression — say, x^2 -5x + 4 — and then ask which of the following terms best describes the expression, with polynomial being one of the item options.

The broader point has little to do with this particular item. It’s rarely difficult to cherrypick an exotic item from an assessment and poke fun at it, but the persistent patterning of problems is much more problematic. In the New York State assessments, that’s taken two forms: first, the fact that some standards are much more likely to appear on the assessments than others, thereby narrowing the curricular coverage of the assessments, and second, that from year to year, the standards which do recur are represented by items that are “clones” of the items that have appeared in previous years, which makes them overly predictable.

The presence of technical advisory groups populated with psychometricians is rarely an adequate guard against problems such as these. Having served on such groups for the federal government (although not for testing and assessment projects), I would characterize them as largely symbolic. The advisory groups are convened intermittently; the groups rarely have independent resources to explore issues; often the key decisions have already been made; and the agency staff are free to disregard the advisory group’s advice on cost grounds, on political grounds, or on no grounds at all.

In many ways, the lack of control in the organizational system managing the state assessment system reminds me of Diane Vaughan’s wonderful book The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA. The involvement of scientists and engineers in the decision-making process did not prevent the Challenger disaster. In the same vein, the involvement of psychometricians in the item and test development process will not — did not — preclude the development of a badly flawed assessment system.

Aaron, is there a state test you think is good?

For extra credit…
http://www.youtube.com/watch?v=4e_gbzuuG0c