GothamSchools — daily independent reporting on NYC public schools

Here’s one more reason state tests might be getting easier to pass: a longstanding State Education Department practice of publicly releasing every question on each year’s exam.

The unusual practice makes it harder for test-makers to gauge how difficult a test is, said Howard Everson, chair of the state’s Technical Advisory Group, an oversight committee that monitors state testing.

Many states release some test questions but keep others private so they can be used again to compare one year’s test to another’s, said Daniel Koretz, a Harvard University education professor who studies testing. But New York has long had a practice of releasing every single test question to the public soon after students sit for the exams.

“In the interests of transparency the State Education Department has a decades long practice of making our tests publicly available soon after their administration,” education department spokesman Tom Dunn said in an email. Dunn said the department has never re-used test questions from year to year as a way of comparing the difficulty of the exams.

Releasing every question soon after the test is given prevents test makers from field testing exam items in an actual test setting, Everson said.

Field tests allow test makers to figure out how hard questions should be and set the scale used to judge students. Exams like the SAT include field-test questions folded into the actual exam, but students don’t know which questions won’t count toward their scores. The experimental questions are then used again on future tests to gauge their difficulty.

Unable to field test questions in this real-world setting, the state must rely on no-stakes tests given to a sample of students on a different date. Dunn said that students who sit for the field tests are told that the exams are only experimental.

But when students know they won’t get a grade for the field test, they might be less motivated to do their best, Everson said.

As a result, field tests often suggest that questions are more difficult than they actually are. And because they’re used to set the scale by which the real tests are graded, the end result is an easier state exam, Everson said.

“This is not ideal,” Koretz wrote in an e-mail. “What we don’t know is how much of an impact this has had.”

Koretz and Everson have both been calling for an investigation of test score credibility for over a year. Koretz has frequently said that until such an investigation happens, it’s impossible to know whether the test score gains in recent years truly reflect real learning.

Releasing some or even a large portion of test questions to the public in the interests of transparency is common and has its benefits, testing experts said this week.

When the New York legislature first considered requiring the publishers of post-secondary admissions tests like the SAT and GRE to make their test questions publicly available, testing companies initially lobbied against the law, said Paul Kanarek, founder of the Princeton Review of California who was also involved in New York’s legislative discussions of the requirement.

But test-makers have since acknowledged the benefits of the practice. And releasing real test questions is good for students, he said, both because it helps them prepare for exams and also because it help ensure that the tests are equitable.

“Whenever you are forced to show what you are giving kids the light of the day, all of the sudden a lot of the obnoxious style questions drop out of the tests,” Kanarek said. “They’re no longer asking questions that are so class-based, for example.”

But both Kanarek and Michael Zieky, a test developer for the Educational Testing Service, said that it was very unusual for a test-maker to release its experimental field-test questions.