Post a comment about the budget cuts at your school on our interactive comment map. more »
GothamSchools — daily independent reporting on NYC public schools
guesswork
August 21, 2009
Test analyst: Reading exam bar even lower than critics say
Passing the state reading test might be even easier than recent criticism has suggested, a former Department of Education testing analyst is arguing.
Independent statistician Frederick Smith examined the way free-response questions were graded and found that virtually every student received enough points on that section to then pass the test by guessing randomly on the multiple-choice questions.
Smith’s finding expands on an informal study by a city teacher who concluded that arbitrarily filling in a pattern of multiple-choice answers and leaving the open-response section blank could yield scores high enough to promote a student to the next grade in New York City. That amounts to eight correct multiple choice answers on the fifth-grade English Language Arts exam.
Smith found that students who answered only 6 multiple choice questions correctly almost certainly would also pass the Level 2 bar. That’s because an overwhelming majority–99 percent–of fifth-graders who took the exam in 2009 received at least two points on the open-ended essay and free response sections, which would boost them to the eight-point cut-off level.
In addition, more than 97 percent of students received three or more points on that section, meaning that they would have had to answer just five multiple-choice questions correctly to receive a Level 2 score.
“There’s no point in setting a cut-off score that you can guess your way through,” Smith said. By effectively reducing the number of correct multiple choice answers needed, he said, the tests significantly increased the odds that a student could guess randomly and still hop the Level 2 bar.
Columbia University Teacher’s College professor James Corter calculated the likelihood that randomly guessing students would score at these levels. He found that more than 57 percent of students guessing would answer at least six answers correctly. Just over 75 percent of guessing students were likely to receive five or more correct answers.
The statistical calculations apply only to students who are truly guessing randomly, but could apply to a student with no knowledge of test content.
The number of students receiving failing Level 1 scores has declined dramatically over the past four years. On the ELA exams Smith analyzed, the number of students scoring at Level 1 declined by 95 percent between 2006 and 2009.
Corter cautioned that even though almost all students are receiving at least partial credit on the free-response questions, it doesn’t necessarily mean the questions are easy.
But Smith said the low scoring bar harms the tests’ credibility.
“The fact remains that this is the first year you can waltz your way to a level 2,” Smith said. The scoring effectively removes the failing Level 1 as a meaningful tier in the test evaluations, he said.
Smith has been publicly calling for an independent testing ombudsman in New York City since the early 1980s. He said the results of his examination highlight the need for a truly independent auditor of the state testing program.
“It has to have real authority, real teeth, and be able to get data without having to beg for it from the Department of Education,” he said. “This is the era of accountability. It’s time for some real answers here.”
-
August 21st, 2009 7:27 pm
I have been doing some more experiments since the initial one, and my results coincide with Smith’s. I am delighted that Smith has been looking at the scoring of the written portions. It is indeed easy to score a few points on those, and with 2 or 3 points in hand, the chances of passing by guessing are very high indeed. Also, I found that the number and proportion of students at level 1 is lowest where the odds of attaining a 2 by guessing are highest (i.e., on the third grade math exam, the fifth, sixth, and seventh grade ELA exams, and the seventh grade math exam).
-
August 21st, 2009 8:04 pm
Question: are Fred Smith’s findings true of all the state exams 3-8 in all subjects; or just some? and what about the probability findings of Prof. Corter?
-
August 21st, 2009 9:21 pm
Fred Smith looked at the ELA exams in grades 5, 6 and 7 and reported the same pattern in those tests. Prof. Corter’s calculations apply only to the fifth grade ELA exam. The calculations are based on the specific number of multiple choice questions on a test, so it will vary from exam to exam.
-
August 21st, 2009 11:40 pm
Do you have a link to Smith’s results?
-
August 22nd, 2009 11:46 am
Thank you for providing a cogent and statistically accurate summary of some of Fred Smith’s insights into the use of these statewide tests for a purpose for which they were apparently never intended. I am especially delighted that you corroborated Smith’s analysis with the results of Professor Corter’s calculations of the probability of achieving a particular raw score without a student having any knowledge and simply guessing. For those who doubt either of these two experts I suggest utilizing a table of probabilities associated with the binomial distribution (i.e., the mathematical distribution which deals with two and only two outcomes; in this case success or failure on a given multiple choice item) and all disbelief shall be washed away. These somewhat subtle mathematical realities are easily overlooked by those for whom the political rather than the educational merits are the coin of the realm. In the nature of disclosure I was a colleague of Mr. Smith during the the early 1980s when we bought were employees of the (former) NYC Board of Education.
-
August 24th, 2009 9:28 am
Thanks for an excellent article. Does the State Education Department have any thoughts on this? Additionally, the State might want to tell us how and when the standards were set.
Mr. Smith’s continual urging fior an ombudsman must be taken seriously. Parents and students need and deserve an advocate.
Leave a Reply
Mapping the Budget Cuts
6 Comments
Subscribe to comments with RSS or TrackBack