I haven’t studied the features of state tests systematically, so I don’t think I know enough to suggest any exemplars.

Interestingly, this particular standard has not appeared on any other NY state assessment since 2006, which is when the technical reports began indicating which standard a particular item was to represent. I think the item is problematic because it’s simply a statement of the standard. A student could memorize the words without ever knowing what a polynomial or algebraic expression was. A better item, I imagine, might present an algebraic expression — say, x^2 -5x + 4 — and then ask which of the following terms best describes the expression, with polynomial being one of the item options.

The broader point has little to do with this particular item. It’s rarely difficult to cherrypick an exotic item from an assessment and poke fun at it, but the persistent patterning of problems is much more problematic. In the New York State assessments, that’s taken two forms: first, the fact that some standards are much more likely to appear on the assessments than others, thereby narrowing the curricular coverage of the assessments, and second, that from year to year, the standards which do recur are represented by items that are “clones” of the items that have appeared in previous years, which makes them overly predictable.

The presence of technical advisory groups populated with psychometricians is rarely an adequate guard against problems such as these. Having served on such groups for the federal government (although not for testing and assessment projects), I would characterize them as largely symbolic. The advisory groups are convened intermittently; the groups rarely have independent resources to explore issues; often the key decisions have already been made; and the agency staff are free to disregard the advisory group’s advice on cost grounds, on political grounds, or on no grounds at all.

In many ways, the lack of control in the organizational system managing the state assessment system reminds me of Diane Vaughan’s wonderful book The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA. The involvement of scientists and engineers in the decision-making process did not prevent the Challenger disaster. In the same vein, the involvement of psychometricians in the item and test development process will not — did not — preclude the development of a badly flawed assessment system.

I would love to hear a bit more about what is wrong with the question and how you suspect politics was involved in its creation.  

I don’t know if it is a good question or a bad one, although I feel comfortable that just about anyone that is literate and has a rough idea of what a polynomial is should get that one correct.  Of course, that doesn’t necessarily make it a good question. 

Anyway, maybe that’s a bad question — it strikes me as way too easy, although 33% of kids still managed not to know the answer even when given a choice. So are you trying to say that they should have been asked to prove the quadratic formula by completing the square, or something like that?

If you can point me to these psychometricians, I’ll be happy to ask them. I’m quite serious. I invite you to read the technical report for the 2008 New York State Grades 3-8 assessments; see if you can understand the process by which items appear on the assessments. It’s downloadable at http://www.emsc.nysed.gov/osa/pub/NYSTP_2008_MAT_OP_TechReport.pdf

As an historian, I’ll just note that the administrative progressives of a century ago also claimed to be taking schools out of politics. Of course, we consider education to be a right of citizenship. How did that happen without politics, and how can we expect anything we attach to citizenship to be without politics?