GothamSchools — daily independent reporting on NYC public schools

This and next week I am raising objections to the idea that new standards — particularly new national standards — are worth the attention they get. It is ridiculous to think that they can be a meaningful lever of broad educational improvement. In fact, I do not think that they can have any significant impact at all.

Problem #5: Tests Matter; Standards Do Not

As much as they may hate them, teachers do respond to tests. Not always well or in good faith, but teachers and schools feel the pressure of high stakes and public reported tests.

Tests, of course, are usually supposed to be based on standards. State tests are specifically supposed to be based on state standards. State tests might, in the future, be based on the Common Core standards.

But that’s not really true, not exactly. You see, tests only include the standards that we know how to test and are capable of testing relatively cheaply.

The recent draft of the Common Core ELA standards actually begins, “A crucial factor in readiness for college and careers is students’ ability to comprehend complex texts independently.” However, psychometric requirements for reliability, combined with reasonable limits on the time and money it costs to take an exam, prevent the inclusion of complex texts if only because text length is an issue. Later, the draft says, “Students must be able to revisit and make improvements to a piece of their writing over multiple drafts when circumstances encourage or require it.” There is also a whole section for “Speaking and Listening.” Does anyone expect that that standardized tests based upon the Common Core standards will include a speaking section or will include students’ ability to revise their own work?

And so, even if tests do a good job of evaluating students on the standards that they attempt to include, they do not actually represent the larger set of standards fairly. Moreover, in the absence of real advances in testing, I do not see how changes in the standards will lead to changes in the tests. Test developers will continue to test what they know how to test, regardless of what the standards say or how they have been changed.

Previous: Problem #4 — Classrooms
Next: Problem #6 — Local Control

Number of students retained or needing academic intervention services, 2004-2008

A highly anticipated independent research study on the effects of Mayor Michael Bloomberg’s promotion and retention policies says that fifth graders benefit from the promotion practices — at least through their seventh grade year.

The policy requires that students in several grades reach a certain level on state math and reading tests before going on to the next grade. Citing years of research, critics have charged that the new rules wouldn’t help students and could possibly hurt them or cause them to drop out of school later.

But researchers at the RAND Corporation, which conducted the study, said that hasn’t happened.

The lowest-performing students who took tests under the new promotion policy did better later than earlier students who weren’t held to the new standards. The study compared the first three classes of fifth-graders held to the promotion standards to the previous class of students who were not affected by the new policy.

The report said students benefit because their schools identify them as at-risk earlier and give them extra help.

Students surveyed for the report also said being held back didn’t make them less confident about school. (more…)

NYC Gifted and Talented Information Session Reminder Postcard

In the mail yesterday we received the notice below from the NYC DOE reminding us about the gifted and talented program information sessions that began yesterday in the Bronx. I’m sure there were many parents in the Bronx who would have liked to attend but didn’t get their notice until it was too late. Hopefully those parents can attend the sessions in the other boroughs over the next week.

• A new study says Mayor Bloomberg’s social promotion policies are working. (Daily News, Post)
• Math NAEP scores are flat in nationwide. (Times, Wall Street Journal, Christian Science Monitor, AP)
• They’re flat in the state, even though state tests are up. (GothamSchools, Times, Post, Daily News, NY1)
• Nick Kristof: The city’s rubber rooms are one reason to support policies teachers unions don’t. (Times)
• Students who graduate without skills can and do sue the city for “compensatory education.” (USA Today)
• A children’s advocate says Los Angeles’s school reforms should include parent training. (L.A. Times)
• Mayor Bloomberg says school siting and overcrowding aren’t parent issues. (The Villager)
• Tuesday, the City Council questioned how the DOE has spent money meant to reduce class size. (NY1)
• The Washington Post sees good news in D.C.’s NAEP scores, which rose but are still the nation’s worst.
• Some city teachers are working in the same schools they attended as students. (Daily News)

• Today’s NAEP results make it look like Richard Mills diluted the tests, Peter Murphy writes.
• Merryl Tisch wasn’t surprised by the gulf between the NAEP and state test results.
• Kevin Carey: the disappointing NAEP scores tell us that most national edu policy claims are overwrought.
• Assemblyman James Brennan casts doubt on claims that NYC’s test scores went up more than state’s.
• Diane Ravitch says the NCLB law needs to be replaced, not made over.
• Chaz says school aids are the “lifeblood of the system.”
• D.C. may have the lowest NAEP scores, but news that they’re increasing is good for Rhee.
• Philly says only 3 percent of its major budget cuts will impact school children.
• Rhode Island wants to make its teacher entry tests the most difficult in the country, but how?
• CA’s decision to link test scores and tenure is meaningless.
• And after math scores come out, some wonder if we need specialists in the classroom.

Reacting to differences between the state’s own testing data and the results of a national math assessment, Commissioner David Steiner called for the state to review and redesign its tests to make the questions less predictable.

“The New York State NAEP scores in mathematics, released today, are of great concern to the Board of Regents and to me,” Steiner wrote in a statement. “We are struck by the contrast between results on the NAEP and on New York State’s own math tests.”

The call from Steiner is the strongest language a state education official has used since critics began challenging the state tests in 2007. (more…)

Today’s New York Daily News published a bold editorial on the progress of New York City schoolchildren under the administration of Mayor Mike Bloomberg and Chancellor Joel Klein.  “You would be better off arguing that the world is flat, or that the sun revolves around the Earth, than to dispute that New York City kids are performing better and better in school,” writes the Daily News, crowing that there are “fresh and incontrovertible data” pointing to what the newspaper refers to as a “sea change” in New York City. 

They might have wanted to wait a day.

This morning, the U.S. Department of Education released the 2009 results of the National Assessment of Educational Progress assessments of fourth-grade and eighth-grade mathematics in each state and for the nation overall.  Nationally, fourth-grade performance held steady from 2007 to 2009, and there was a slight but statistically significance over this period in eighth-grade math performance.  In New York State, the small declines in fourth-grade and gains in eighth grade were not statistically significant, leading to the conclusion that there has been no change in the performance of New York students on the NAEP math assessment from 2007 to 2009. 

This is a very different story than the one told by New York’s own assessment system, on which the Bloomberg and Klein administration has staked its claims about the great progress in student achievement.  The average scale score in fourth-grade mathematics increased from 680 in 2007 to 689 in 2009, a hefty 9 points;  the jump in eighth-grade scores was even more dramatic, as the average scale score rose from 657 in 2007 to 675 in 2009, a remarkable increase of 18 points.

To put these two sets of numbers in context, the chart below shows the gains in fourth-grade and eighth-grade math performance from 2007 to 2009 expressed in standard deviation units (i.e., the amount of variation among individual students in 2007).  According to NAEP, fourth-graders’ performance fell .07 standard deviations from 2007 to 2009, a difference that is not significantly different from zero.  In contrast, fourth-graders gained .23 standard deviations on the New York State assessment from 2007 to 2009.  Similarly, the NAEP results indicate that eighth-graders in New York gained .08 standard deviations from 2007 to 2009 in math performance, a difference that is not significantly different from zero, but they gained .47 standard deviations over this period on the New York State test.

Another way of comparing the implications of the two different sets of test results is to think about where the average student in 2009 would have scored in 2007.  Based on these standard deviations, and assuming that the scores follow a bell-curve distribution, the New York State scores indicate that the average fourth-grader in 2009 scored at the 59^th percentile of the 2007 fourth-grade distribution, which is a pretty big jump.  The increment for eighth-graders is even more striking:  the average eighth-grader in 2009 scored at the 68^th percentile of the 2007 eighth-grade distribution, based on the New York State tests.  In contrast, the NAEP data indicate that the average New York fourth-grader in 2009 scored at the 47^th percentile of the 2007 distribution of fourth-grade math performance in New York State, and the average eighth-grader in 2009 scored at the 53^rd percentile of the 2007 eighth-grade distribution.
 
How can we explain these differences?  There are lots of possible explanations, but most of them don’t hold up under close scrutiny.  The two tests are taken by similar populations of students under similar conditions, and the grade-level mathematics standards on which the two assessments are based do not differ dramatically.  The NAEP test is a low-stakes test, which might result in students not taking it seriously, but the statisticians who oversee the NAEP testing program look for patterns suggesting this, and find little evidence of it.  It’s extremely unlikely that there’s rampant cheating going on in the New York State testing system that could explain the differences. 

 It’s possible that the New York State tests have been getting easier over time.  I have yet to see definitive evidence ruling this out.  There also is strong suggestive evidence of “score inflation” in the New York State tests, because there are predictable patterns in the standards which appear on the state tests year after year, with some standards showing up repeatedly each year, and some standards having never been tested at all during the life of the testing program.  Schools and teachers can make use of these patterns, which also show up in the format of test questions covering particular standards, to focus their instruction on the subset of standards that crop up again and again.  Because the New York State tests never test some standards, we have no idea about whether students have mastered them.  In contrast, the design of the NAEP assessment allows for a much broader picture of mathematics performance, because so many more standards and test item formats are incorporated into the test.

 Whatever the reason, the discrepancy between the NAEP trends and trends in the NewYork State test scores raises serious questions about what the New York tests are telling us about the academic performance of students in New York State.  The same, of course, goes for New York City.  We’ll see NAEP scores for New York City in a month or so, but it’s unlikely that they will yield a different story than what I’m describing here.
 
Is the Earth flat?  No.  But New York State test scores, and probably New York City scores, are.