Teachers College sociologist Aaron Pallas tackles the subject in the Community section of GothamSchools today, by taking a closer look at two middle schools that the Post has recently highlighted for exceptional performance and finding that both schools admit their students selectively. He writes:

Due to their selective admissions, IS 187 and, to a lesser extent, IS 364 were born on third base.  The New York Post thinks they hit a triple.

Some schools might have hit something closer to a home run. Manhattan’s citywide Anderson School, for instance, admitted every single one of its students in grades 3-8 on the basis of their scores on an IQ test and in-person interview. Not a single student at Anderson failed the math test, and in fact it was the only school citywide with a clean 100 percent of all students in a single grade scoring at the very highest level, in the sixth grade.

Not all successful schools handpick their students. Another school the Post profiled, Carl C. Icahn Charter School in the Bronx, had just about every student in grades 3 through 8 pass the test. (Aaron didn’t include the school in his Post coverage criticism.) That school admits students by random lottery, in accordance with state law for charter schools. It does have a low proportion of students with special needs, just under 4 percent, according to data obtained by Insideschools.

Another school touting itself as a test-score success story, although it didn’t make the Post today, was Harlem Success Academy, the charter school operated by Eva Moskowitz. I got a press release from the school yesterday that said Harlem Success “slammed the 2009 New York State Math exam,” with 71 percent of students scoring at the highest level and 100 percent passing the test. The 71 percent figure made Harlem Success the top-performing charter school in the state and the 10th-highest school in the city overall, the release said.

But Harlem Success, actually a chain of four schools (so far), only had one set of kids taking the test this year, so the 71 percent figure is based on the scores of just 58 kids. In fact, 118 grades at 67 schools had higher proportions of students testing at the highest level than Harlem Success.

Schools Chancellor Joel Klein has for years been a proponent of paying some teachers more based on their performance, and he has made some headway in introducing merit pay in the city schools. But the policy has plenty of critics, from teachers who say merit pay divides them to statisticians who point simply to flaws in the measures on which pay calculations are based.

In the video above, University of Virginia psychology professor Dan Willingham gives six reasons in three minutes why paying teachers based on their students’ test scores isn’t statistically sound. But Willingham doesn’t totally rule out the prospect of paying better teachers more: “Merit pay can’t work until there’s a way to measure teacher performance that’s fair,” he concludes.

At best, there is no correlation over time in the DOE’s reports of which schools are good at inducing growth in ELA achievement. At worst, the DOE’s system finds that the schools that were better than average in 2007 were actually worse than average in 2008.

Over at Eduwonkette, Harvard education professor Daniel Koretz offers a lengthy critique of the progress report methodology. He notes that test scores alone are not a legitimate way to evaluate schools; New York State’s tests were not designed to be used in “value-added” analysis like that behind the progress reports; and the progress reports, like all accountability systems, place pressure on school administrators that likely leads to score inflation. In addition, he writes that the DOE’s formula does not take into account “interval scaling,” or the reality that different amounts of “value” are required to move students from one proficiency level to the next at different points on the proficiency spectrum. (In June, I wrote about how interval scaling might contribute to the finding that No Child Left Behind has helped high-performing students less than their low-performing peers.)

But those problems exist in many test-based, value-added accountability systems — Koretz writes that New York’s progress report system has its own set of errors. The tremendous variation in schools’ grades from last year to this year probably has less to do with school improvement than sampling and measurement error, he writes.

Here’s an illustration of the effect of error. I first calculated the variation in schools’ grades between last year and this year and then graphed it against their enrollments. It’s obvious that larger schools were less likely to see sizable changes in their grades than smaller ones. No school with more than 1,500 students went up or down more than one grade, while all schools whose grades changed the maximum amount possible had fewer than 1,000 students; most of those that increased by that amount had 500 students or fewer.

A substantive explanation for this distribution might be that large schools don’t do a good job moving their students forward, and smaller schools can give more attention to each student’s individual needs. But I’m with Koretz that correct explanation is more likely to be methodological — and rooted in error. A school with 400 students sounds like it would produce stable results. But consider that elementary progress reports only look at two grades’ worth of students — those with two years of test scores. The progress report grade for a school with 400 students could depend on just 100 students’ test scores — hardly a sample that allows chance differences among students, and in each student’s year-to-year test experiences, to wash out.

Four-Year Outcomes for the Class of 2007

When the state released graduation figures earlier this month, I wondered what the city’s old formula for determining graduation rates would have said about the class of 2007. Yesterday, Edwize pointed us to a 276-page report available on the DOE’s website that includes the answer to that question and much, much more.

Although the state’s graduation figure of 52 percent is the official one thanks to an agreement between the city and state last year, the DOE still calculated the graduation rate for the class of 2007 using its old formula, which gave credit for students graduating in August and for students completing a GED or IEP diploma rather than a local or Regents diploma. According to this formula, 62 percent of students entering the city’s high schools in the fall of 2003 graduated on time, an improvement of 2.3 percentage points over the class of 2006.

In the report, the DOE attributes a reduction in the four-year dropout rate — to 13.8 percent, the lowest ever
recorded in the 20 years since the city started keeping these statistics — to a new policy that requires school officials to conduct exit interviews with some students who indicate an intention to drop out. These interviews, during which students must be reminded of their legal right to remain in school until the year in which they turn 21, may have prompted some students to stay in school or to delay graduation, according to the report.

The report contains detailed information about school completion at each of the city’s high schools, including information about the types of diplomas students earned, graduation rates by ethnicity, and outcomes for students in special education. What’s missing from the report is information about what happened to the 26.3 percent of students who entered high school in the fall of 2003 who were “discharged” by the city’s schools “primarily to enroll in another educational program or setting.” They represent nearly twice the number of students who officially dropped out.

Mind the gap, by Marcin Wichary

The internet has seen a flurry of activity recently over the DOE’s claim that it has reduced the achievement gap between black and Hispanic students and their white and Asian peers. Testing that claim, the New York Sun submitted the ELA and math scale score data for students in grades 3-8 to three independent analysts, who concluded that the gap has decreased in ELA, but has stayed flat since 2002 in mathematics, confirming much of Eduwonkette’s analysis.

The new analysis emphasizes the difference between closing the proficiency gap by comparing the percentage of students who score at a level 3 or 4 on state tests, and closing the achievement gap by comparing mean scale scores.

The experts consulted disagree as to which of these measures is most useful, though in the end, both clearly matter. Closing the proficiency gap is important as it reflects a larger number of students performing at a level considered acceptable by the state, that is, more students with more knowledge (at least the kind of knowledge tested). Closing the achievement gap is important because it evens the playing field for students applying to college and in other situations where rank against others matters.

Aaron Pallas, an education and sociology professor at Columbia University’s Teachers College, sums it up in his memo to the Sun:

There are a great many social institutions that sort and rank individuals on the basis of test scores and the competencies they represent. Most of these institutions don’t have an unlimited number of positions or slots—rather, individuals are competing against one another for access. When these institutions rely on test scores, and there is an achievement gap among racial/ethnic groups on these tests, the lower-scoring group will be underrepresented. Raising everybody’s scores doesn’t change the rankings of individuals, which is the only way to change the representation of minority groups among those who are selected. Only by reducing the achievement gap can we increase the chances that members of racial/ethnic minority groups can get ahead in society via selective social institutions.

Eduwonkette points out that Klein’s downplaying of the achievement gap in the Sun piece contradicts his commitment to closing the achievement gap outlined in the Education Equality Project.

The National Center for Education Statistics also concludes that upward trends in the reading scores of black and Hispanic fourth-graders lauded by Mr. Klein are not statistically significant.

Mr. Klein criticized the National Center on Education Statistics analysis.

“Those are just confidence levels. Nobody is saying this is a science,” Mr. Klein said. He added: “If three points is flat, and four points is statistically significant, then what you’re doing is, you’re playing something of a game.”

Um, Chancellor Klein? Statistical significance isn’t game-playing – it’s a standard measurement of how likely a result is to be non-random. Did test scores rise because of DOE policies, or random chance? Statisticians don’t just draw a line wherever they want – they use widely-accepted significance tests that you learn in any basic stats class. Or perhaps you’re suggesting that the NCES plays with confidence levels to downplay New York’s progress?