You are confusing a number of test design parameters. The order and structure of questions is not typically random for a number of reasons. The order of answers is almost always entirely random, even when such answers are themselves ordered (i.e. in ascending numerical order). You also are confusing the psychological aspects of test design (i.e. having easy answers) with the statistical aspects (i.e. random choices should be easily discernible from non-random). No matter whose test design principles the State supports, the test can be effective from a statistical standpoint.

There are indeed well-known techniques to neutralize guesses, but most of them do not revolve around attempting to minimize the score of any particular pattern, because there are too many patterns. For some reason, tests are generally designed to have a roughly equal number of correct answers assigned to each answer choice which does minimize the effect of “all C’s”.

As for your point regarding the “the cut score has been dropped too low”, I agree. I would note that rather than blaming the mechanism, you must simply state the problem: “students who guess every single answer stand a very high chance of being promoted, a small amount of effort on their part increases those chances to near-certainty”. In Diana’s post in the comments, I think we have much more well-supported evidence that this is true in several instances and I feel much more confident relying on that versus data from only two trials.

I think that you fundamentally misunderstand the argument.

We can predict how many questions students will get right with random guessing. Sure, there is a variance to the actual result, but random guessing will generally result students get 1/n of the questions correct (where n is the number of answers to choose from).

The issue here is NOT that the questions have been made too easy. The issue is that the cut score has been dropped too low. Imagine needing just 1/2n of the questions correct to pass. In that case, we would expect almost all the kids to pass, especially if they just guessed for each question. Well, in this case, the cut score is not nearly enough above 1/n – if one takes into account easy points on the written response section — to make the entire test credible. Add in some easy question, (and there SHOULD be some easy questions, right? I mean, that would would be good test design.) and the test overall simply becomes too easy to pass.

Yes, Ms. Senechal’s dramatic and true examples make for a great story while perhaps backgrounding the fundamental problem. But that doesn’t mean that she is, at root, wrong.

There are well-known techniques for dealing with guessing. Yes, they make statistical analysis more difficult, and scoring slightly more difficult, but many are well established. I would argue that a practical late step to take in test design would be to make sure that the most common guessing patterns do not result in passing scores.

Moreover, true random assignment of answer orders and questions orders is actually a bad idea, because it can results in bad outcomes. Randomization is a weak answer. We already know that if you have too many of the most difficult items early, you can depress scores on the test. So, if randomization is one of your strategies, you cannot stop there. Once you randomized, you’ve GOT to check for known issues, and then re-randomize if former roll of the dice fell into any of them.

The experiment was informal, but in fact the odds of getting a 2 by guessing on these tests are quite high. I did some followup, and these were some of my findings.

For each test, I calculated the mean by dividing the total number of multiple-choice questions by 4 (since there are 4 options for each question). I calculated the standard deviation by taking the square root of (total number of questions x 0.25 x 0.75). I used a binomial distribution probability calculator from there. Consulting the score conversion tables (raw to scale and scale to performance level), I arrived at the following:

1. On the third-grade math test, there are 25 multiple-choice questions. A student need only answer 11 correctly to get a 2 (without doing anything whatsoever on the written portion of the test). There is approximately a 3 percent chance of doing this through random guessing.

2. On the fifth-grade ELA test, there are 24 multiple-choice questions. A student need only answer 8 correctly to get a 2 (without doing anything whatsoever on the written portion of the test). There is approximately a 23 percent chance of doing this through random guessing.

3. On the sixth-grade ELA test, there are 26 multiple-choice questions. A student need only answer 7 correctly to get a 2 (without doing anything whatsoever on the written portion of the test). There is approximately a 48 percent chance of doing this through random guessing.

4. On the seventh-grade ELA test, there are 30 multiple-choice questions. A student need only answer 9 correctly to get a 2 (without doing anything whatsoever on the written portion of the test). There is approximately a 33 percent chance of doing this through random guessing.

5. On the seventh-grade math test, there are 30 multiple-choice questions. A student need only answer 11 of them correctly to get a 2 (without doing anything whatsoever on the written portion of the test). There is approximately a 20 percent chance of doing this through random guessing.

Now, this assumes that the student only guesses on the multiple-choice questions and does nothing on the written parts. But what if the student knows a few of the answers—or earns a few points on the written section? Here are some scenarios.

1. Let’s take the third-grade math test. Let’s say that in addition to guessing on the multiple-choice part, the student earns two points on the written portions. His or her chances of getting a 2 go up to about 15 percent.

2. On the fifth-grade ELA test, if the student earns two points on the written portions of the test and guesses the rest, his or her chances of getting a 2 go up to about 58 percent.

3. On the sixth-grade ELA test, if a student earns 2 points on the written portions and guesses the rest, his or her chances of getting a 2 go up to about 82 percent.

4. On the seventh-grade ELA test, if a student earns 2 points on the written portions and guesses the rest, his or her chances of getting a 2 go up to about 65 percent.

5. On the seventh-grade math test, if a student earns 2 points on the written portions and guesses the rest, his or her chances of getting a 2 go up to about 33 percent.

And now compare the odds with the numbers and proportions of students at level 1. Is it a coincidence that, in 2009, 0.1 percent (yes, one-tenth of one percent) of sixth-grade students in New York State scored at level 1 in ELA? In New York City, a total of 146 sixth graders scored at level 1 in ELA.

On a well-designed test, more questions mean that the questions will range in difficulty, and thus increase the chances of differentiating each level of skill, but note that it will NOT increase change the random probability of passing the test. Specifically, a significant number of easy questions help lower the chances that a student has literally not read the test and is randomly choosing answers. 

All you have shown is that you have found two out of the N^Mth ways of completing this test that result in a pass. There are a number of ways to change these factors but absolutely no way in hell that you can control for a “random answer” strategy of passing, any more than you can control for a randomly chosen lottery number being a winning ticket. There will always be at least one, and in a good test, many more. Tests are evidence of a student’s ability, not a final judgement.

Any of these issues would be covered in the most basic courses a teacher should take in statistics. Whether teachers are able to pass their statistics classes in university by adopting a random choice strategy without understanding statistics, I cannot say, but I believe that any of them would be sufficiently competent to understand the above.

Thank you for your comments and your perspective.

My younger brother, now in his 40′s, has CP and significant physical handicaps, the term back then. If it were up to standardized testing (without accomodations), he’d never have made it to grad school. People, especially those facing severe challenges, are themselves not standardized.

I have seen the disaggregated data for ELA and Math for D2 elementary and middle school students. The scores for Special Ed kids are lower than those of many other sub-groups. Between the implication of the scores, and quite possibly the testing structure, we’re doing children with disabilities a disservice twice over.

How true. But I’ve looked – hard – since NCLB’s inception and so far, I’ve failed to find one state which has a system of what you’d call “high quality testing.” I’m certainly not implying that this can’t be done – I’m simply noting that our current American education industry and infrastructure are either unwilling or unable to produce one and then let the education industry in the states live with the results of those high quality tests. And as with most things in the education industry, NY is both the most expensive and the most corrupted.

So given the hand that we’ve been dealt, for me the issue is whether any of the data purportedly euchered out of the testing system is valid and reliable for any purpose whatsoever.

Unfortunately, I think not.

Now, for the other side … I always have fun telling folks about the USDOE-sponsored, longitudinal, national, large-scale research done re kids with disabilities at the elementary and secondary levels. These show, without exception, that there is “almost zero” correlation between the subjective grades teachers give kids with disabilities and those kids’ objectively-assessed reading and math levels. I’m talking about diagnostic testing for real reading and math levels, not the NCLB-mandated tests.

So for the 15% of kids who are classified as having some disability or another, if we don’t have a legitimate testing system and we don’t have legitimately reported results … we wind up with what we have now: massive numbers of kids with mild disabilities who’ve been graduated from high school with allegedly “regular” high school diplomas and who can’t read or do math at anything close to the high school level. I suspect that these are many of the NYCDOE’s kids who have been given the benefit of bogus “credit recovery” in order to get their diplomas.

For me, these are the kids who are the real victims of the ridiculous tests NYS provides and the even more ridiculous scores our State and NYC spinmeisters have massaged so well.

Dee Alpert, Publisher
SpecialEducationMuckraker.com

I agree with most of what you have written, until the end. While Bloomberg has given this even more political significance than it usually has, its significance is *not* limited to politics.

There *are* political implications, and these things do impact the education of children and educational reform — even apart from the political avenue. This testing regime goes back further than Mayor Mike’s tenure as mayor. High quality testing does have educational value, and weaknesses in our assessment programs have at *at least* an opportunity cost for students and educators.

That’s why I have no idea what I think about social promotion in general. But rarely do we face these issues in an abstract setting.

Decisions regarding social promotion, remediations, interventions, enhancements, credit recovery or whatever need to be discussed honestly given the messy realities of what situation is being addressed.
But santimonious fig leafs, that are the forte of Bloom/Klein are destructive. I don’t think we disagree,