GothamSchools — daily independent reporting on NYC public schools

fuzzy math

A statistician offers a caveat on single-school score celebrations

It’s not news to report that statistics can be deceptive. But when a new set of test scores come out, it’s worth repeating nonetheless.

Teachers College sociologist Aaron Pallas tackles the subject in the Community section of GothamSchools today, by taking a closer look at two middle schools that the Post has recently highlighted for exceptional performance and finding that both schools admit their students selectively. He writes:

Due to their selective admissions, IS 187 and, to a lesser extent, IS 364 were born on third base.  The New York Post thinks they hit a triple.

Some schools might have hit something closer to a home run. Manhattan’s citywide Anderson School, for instance, admitted every single one of its students in grades 3-8 on the basis of their scores on an IQ test and in-person interview. Not a single student at Anderson failed the math test, and in fact it was the only school citywide with a clean 100 percent of all students in a single grade scoring at the very highest level, in the sixth grade.

Not all successful schools handpick their students. Another school the Post profiled, Carl C. Icahn Charter School in the Bronx, had just about every student in grades 3 through 8 pass the test. (Aaron didn’t include the school in his Post coverage criticism.) That school admits students by random lottery, in accordance with state law for charter schools. It does have a low proportion of students with special needs, just under 4 percent, according to data obtained by Insideschools.

Another school touting itself as a test-score success story, although it didn’t make the Post today, was Harlem Success Academy, the charter school operated by Eva Moskowitz. I got a press release from the school yesterday that said Harlem Success “slammed the 2009 New York State Math exam,” with 71 percent of students scoring at the highest level and 100 percent passing the test. The 71 percent figure made Harlem Success the top-performing charter school in the state and the 10th-highest school in the city overall, the release said.

But Harlem Success, actually a chain of four schools (so far), only had one set of kids taking the test this year, so the 71 percent figure is based on the scores of just 58 kids. In fact, 118 grades at 67 schools had higher proportions of students testing at the highest level than Harlem Success.

  • http://www.eduwonkette.com eduwonkette

    Hi Philissa, I agree that not all successful schools are schools that pick their students, but I think the choice of two charter schools is not the best example to make that point. Students have to select into charter lotteries, enroll given that they’ve won the lottery, and persist until test time. Each of those steps introduces non-random elements into the process, and results in a population that is unlike the population one would have if one took a true lottery from the neighborhoods in which these schools are located. That is not to say that these schools may not be doing better than their neighborhood schools – I have no idea if they are or not – but the point is that none of us can know that without comparing lottery winners’ and losers’ outcomes.

  • http://www.gothamschools.org Philissa Cramer

    Absolutely, eduwonkette. I picked the Carl Icahn example because it was a school that the Post also looked at. But there are a number of zoned elementary schools that do very well on state tests, and not just in the neighborhoods you would expect. For example, the 4th grade at PS 172, which is in Sunset Park, Brooklyn, was the second-highest scoring grade in the state. PS 172 is a zoned school that sometimes has room for children from outside its zone. (It also used to have a magnet program; students in the upper grades now might have been admitted through that program, I’m not sure.) Many top-performing schools do have gifted or magnet programs, but most of them don’t admit all of their students through that program. And of course there are many excellent zoned schools in parts of eastern Queens.

    One other note: I wasn’t comparing schools’ scores to scores at other schools in the neighborhood. Instead, I took the state’s entire excel file of score distributions by school and grade level and arranged it in descending order of proportion of students scoring at a level 4 on the math test. The very top of that list is full of NYC schools — charter, gifted, magnet, and neighborhood.

  • Matthew

    Philissa,

    With 1,400 schools NYC is probably over-represented at every level of your excel file. Scroll down to the bottom and I will bet you a coffee that you find a lot of NYC schools as well.

    This doesn’t make NYC a better or worse performer than other districts per se. Just bigger.

    On a % basis I assume NYC continues to suffer in comparison to districts in Westchester and Nassau, but as Jennifer points out, that’s unfair in comparison as the selection of students is non-random. Maybe you could look at Rochester or Buffalo if you wanted another urban system with chalenges that is comparable, if a lot smaller, than NYC.

  • http://bit.ly/X3rT7 gadfly1974

    Even more statistically-concerning to me is the lack of transparency in CTB/McGraw Hill’s generation of the raw score to scaled score conversion charts.

    Because of “proprietary software” restrictions, the public does not have access to information explaining why my 7th grade math students only had to earn 44% of available points to pass the test this year.

    Please see my website for additional analysis and a writing style way crappier than eduwonkette’s!

Tips, questions, feedback?

Contact us at .

Follow GothamSchools

RSS

Feb. 10: You’re invited!

Chalk It Up

Recent Comments

51 comments so far today

Events Calendar

Our Twitter Updates

  • Despite some tense confrontations between protesters and police, nothing ever got physical and a lieutenant just said there were no arrests. 1 hr ago
  • He's been frozen in that stoic position all night MT @lisafleisher: A protester speaks with his middle finger. http://t.co/xLar4NRU 1 hr ago
  • Last of the occupy protesters just walked out together, shouting expletives and insults on their way out. #toughcrowd 1 hr ago
  • Frank Thomas, DOE spokesman just told me no arrests have been made tonight at PEP despite confrontation between protesters & police earlier. 1 hr ago
  • RT @leoniehaimson: It's been shown repeatedly that as one schl closes another overwhelmed w/ high needs kids that small schls won't take 1 hr ago
  • More updates...

Archives

February 2012
M T W T F S S
« Jan  
 12345
6789101112
13141516171819
20212223242526
272829  
?>