GothamSchools — daily independent reporting on NYC public schools

Two schools whose students have identical test scores would seem to perform differently if they have different enrollment practices, according to a chart produced by a city charter school leader.

It’s not only the teachers union that is arguing that charter schools’ enrollment practices can influence their apparent test performance.

Unlike district schools, charter schools can choose whether to replace students who leave. Charter schools that do not practice “backfill” can end up posting scores that make it look like their performance is better — or worse — than it really is, argues the founding principal of Harlem Link Charter School, Steven Evangelista.

In the Community section, Evangelista explains that when schools opt not to fill empty seats, “survivorship bias” skews test scores toward the results of students who remain enrolled.

The bias renders test scores meaningless, even dangerous, if the scores are not presented alongside context about a school’s enrollment practices, he writes:

Strong schools take the time required to plan, assess, and tweak new initiatives until they become standard operating procedures. The lack of information provided alongside scores obscures this type of growth, creating perverse incentives for schools to “push out” students who are low performers and to “quick fix” by whittling down large original cohorts to smaller groups of survivors, uncompromised by new admittees.

Evangelista says Harlem Link replaces students who depart, knowing that test scores could be adversely affected, in order to keep its budget stable and fulfill its mission of serving needy students. Last year, he writes, the school got lucky: The students who left were, on average, lower-performing than the students who left the previous year, so the appearance of large test school gains was easy to come by.

It’s a phenomenon that the teachers union has been particularly eager to put onto the agenda. After the city released elementary and middle school progress reports for last year on Monday, the union distributed a fact sheet noting high student attrition rates at several top-scoring charter schools. At South Bronx Classical Charter School, for example, between 20 and 40 percent of students that originally enrolled left before they were tested, and no new students replaced them, the union pointed out.

The fact sheet was meant to hammer home the same insinuation — that some charter schools’ strong results were the result of enrollment practices, not superior teaching and learning — that the union leveled at the charter sector’s self-assessment report this spring. ”The report fails to quantify just what the impact is on test scores when students leave charter schools and are not replaced,” said Dick Riley, a UFT spokesman, at the time.

In an even more extreme example of student attrition that the union did not highlight, Harlem Village Academy had 62 eighth-graders sit for state tests last year. But 104 students had been in the fifth-grade class in 2008, meaning that the test-takers represented just 60 percent of the original cohort. If the students who left were more often ones who struggled — which founder Deborah Kenny said was the case in her recent book — the aggregate scores would have been skewed upward.

A self-assessment report by the city’s charter sector released in April said school leaders “have mixed opinions” about backfilling. But it acknowledged that charter schools’ performance probably benefits from the flexibility not to backfill, especially if it is low performers who leave most often. Charter middle schools, which backfill seats least often, post the strongest performance.

Yet backfill doesn’t always cut in a single direction. Some charter schools fill spots that open up and still post consistently high scores, such as the schools in the Icahn network, which had two schools rank in the top 15 progress reports citywide. Other charter schools, including the UFT Charter School, do not fill always all seats that open up yet do not see their scores rise. South Bronx Charter School, which does not practice backfill, lost nearly 50 percent of the students in its first kindergarten class before their fifth-grade year, yet it posted only middling scores.

Indeed, Evangelista writes, a different array of students coming and going would have resulted in totally different scores at Harlem Link. That’s why he is not arguing for or against backfill as policy, but instead for more transparency about how who the students are can influence what the test scores say.

“What I want is for the public to have some understanding of the context behind test scores, so alleged miracles can be put in their proper place, and year-to-year statistical swings that have nothing to do with a school community’s actual performance can be put into their proper perspective,” he writes.

The methodology used in the city’s progress reports, released for last year on Monday, both mitigates against this effect and exacerbates it. A significant portion of the grades are based on the straight test scores, but an even larger portion looks at the improvement of individual students from year to year. Students who left before the most recent year aren’t factored into the progress component, so schools don’t get a boost if lower-performing students leave.

But attrition also means that a school’s progress score is based on improvements among a smaller number of students, often ones who are higher-performing. Plus, as classes shrink and “peer effects” — or the influence of having a higher proportion of high-performing students — kick in, gains might be easier to achieve.