GothamSchools — daily independent reporting on NYC public schools

What does it mean for tests to get easier? And is that really what happened to New York’s tests?

The analysis that has spurred that idea in the last few weeks actually found something slightly different. The tests aren’t necessarily easier, in the way that a kindergarten spelling bee is easier than the SAT. Instead, between 2007 and 2009, students who hadn’t learned much came out looking like they had.

This is an important distinction because it points to a different culprit behind the dropping standards than simply the individual test items themselves. Instead, Harvard professor Daniel Koretz – the lead author of the analysis commissioned by the state education department — names two possible causes: a phenomenon called “score inflation” and a possible psychometric error tied to an obscure state law.

The actual questions on the test play a role in both, but just as important is the practice of prepping students extensively for tests. Another key is a state law that forces New York to release all test items publicly, making it easier for teachers to practice test prep and making it harder for officials to keep tests consistent over time. (more…)

Here’s one more reason state tests might be getting easier to pass: a longstanding State Education Department practice of publicly releasing every question on each year’s exam.

The unusual practice makes it harder for test-makers to gauge how difficult a test is, said Howard Everson, chair of the state’s Technical Advisory Group, an oversight committee that monitors state testing.

Many states release some test questions but keep others private so they can be used again to compare one year’s test to another’s, said Daniel Koretz, a Harvard University education professor who studies testing. But New York has long had a practice of releasing every single test question to the public soon after students sit for the exams. (more…)

State Board of Regents Chancellor Merryl Tisch is calling for state exams to be more “defensible,” but a study investigating test score credibility requested a year ago by the state’s testing oversight board has still not received a go-ahead.

The committee first formally asked the state education department to join an academic study on the state tests in the fall of 2008, said chair Howard Everson. The education department declined but did not rule out future participation. Since then, Everson has received no requests to revisit the idea, he said in an interview yesterday.

“It’s hard to trust the data right now,” said Everson, a psychometrician who is also a senior fellow at the City University of New York. Everson’s committee, the state Technical Advisory Group, is charged with monitoring the state testing process. (more…)

One highlight of the mayoral control panel put together by the parent commission Friday night was testimony by Robert Tobias, the former city testing czar and now New York University professor. Tobias has often been quoted expressing concerns that the Bloomberg administration inflates its record of educational improvement.

But the analysis Tobias presented Friday, explaining exactly what progress he thinks happened (“real” improvements in math) and what he thinks did not (any narrowing of a longstanding gap between the state and city students’ scores on reading tests), was the most succinct summary I’ve ever heard him deliver — not to mention a striking counterpoint to the sanguine evaluations of Chancellor Joel Klein, Mayor Bloomberg, and even Caroline Kennedy.

Here’s what Tobias said:

Tobias also tempered the fact of the improvements in math scores with a warning about score inflation, the phenomenon by which test-prepping, in his words, can “undermine” the meaningfulness of the test as an indicator of what students know, versus how well they have been prepped. (Harvard Graduate School of Education’s Daniel Koretz has written the most on score inflation that I know of. For more on the topic, see this story I wrote for the Sun and these posts by Eduwonkette.)

Tobias’s remarks on score inflation are below the jump. Thanks to David Bellel for sending me the video. (more…)