Post a comment about the budget cuts at your school on our interactive comment map. more »
GothamSchools — daily independent reporting on NYC public schools
guest perspective
November 20, 2009
In a post on this page earlier this week, “Comparing Small Apples to Large Apples,” Teachers College Professor Aaron Pallas raised several important issues with respect to New York City’s high school progress reports. A frank dialogue about the strengths and weaknesses of our accountability system is important as it helps us make improvements while deepening the public’s understanding of how the system works. There are several areas in Dr. Pallas’s argument that I’d like to address to clarify our approach and avoid potential misconceptions.
The high school progress report accounts for multiple data measures: four and six-year graduation rates; attendance; five Regents exams; and credit accumulation at the end of ninth, tenth and eleventh grade.
Pallas questions the usefulness of credit measures. Credit data is derived from the grades teachers assign to students. As a former high school principal, I know firsthand that the progress students make in earning credits is a key predictor of graduation. Academic success in ninth grade in particular better predicts graduation than either demographics or prior academic achievement.
Pallas concedes that the use of peer schools to contextualize each school’s performance and progress is a “good feature” of the progress report; however, he criticizes our method for identifying peer schools. The student characteristics most predictive of high school success — as measured by ability to earn credits, pass Regents, and graduate — are students’ incoming proficiency levels and special education and over-age status. Our peer index controls for these factors. Pallas suggests that we should also control for socio-economic factors like race and Title I status. While these are correlated with achievement outcomes, they are less significant indicators when we control for a student’s incoming proficiency. Socio-economic factors are typically used as a proxy for academic achievement level in the absence of reliable data on student achievement levels. By contrast, we have and use real data on incoming achievement levels for high school students. Our method guards against unduly rewarding academically screened schools with high-minority or low-income populations. It’s also important to note that while English language learners are not part of the peer group determination and Pallas faults progress reports for “systematically penaliz[ing]” these students (among others), schools serving high numbers of ELL students tend to outperform their peers on progress reports.
Pallas recommends that we include school size and per-pupil expenditure as components in identifying peers. In a comment, Leonie Haimson adds class size to the mix. In fact, neither budget nor class size are correlated with a school’s progress report score. With respect to school size, there is a small correlation between the enrollment of a school and its progress report score: Smaller schools perform slightly better on average (though there are many cases of large schools outperforming smaller schools — Francis Lewis, our largest non-specialized high school, earned an A, for example).
So why don’t we include school size in the peer index? Because if we did, we would merely be identifying, for ourselves, the city’s best-scoring big school or small school. We don’t want to control for size; we want to compare what all schools contribute to student learning irrespective of their size. We identify where the creation of smaller units, either in the form of small schools or through small learning communities in large schools, leads to better student outcomes — and where it does not. Similarly, we identify cases where larger schools have achieved better outcomes than smaller ones. We believe this is the information families and school leaders find most useful.
Let’s step back for a moment and consider more broadly the equity concerns Pallas raised.
Everyone understands that there is a shameful achievement gap in New York City. The DOE is focused on eliminating it. We’re making progress after decades in which no one was held accountable for the profound failure to educate our neediest students. We are working to create a school measurement system that compares schools on a level playing field and explicitly rewards their efforts to close the achievement gap and move our highest-need students forward. Each year we have been incrementally more successful. We identify peer schools based on similarly performing student populations. We create incentives in many of our measures that reward schools for outstanding performance and progress with special education, over-age, and lower-performing students. We have measures dedicated to closing the achievement gap in the school (credit measures in the student progress section) and in the city (additional credit measures tied to exemplary progress among special education, English language learners, black students in the lowest third citywide, Hispanic students in the lowest third citywide — a separate measure not linked with black students — and all other students in the lowest third citywide). These measures have schools focused on traditionally underperforming groups of students that for too long were being overlooked by our system.
In other words, we evaluate success using a system that controls for much of the demographic differences across schools and we have established incentives that we hope will reduce what remains of those demographic differences. Almost every other accountability system in the country has far more severe demographic skews in the results because a) these systems don’t take the steps we do to control for the differences in performance among groups of students and b) don’t measure progress.
Pallas closes by suggesting that we have stacked the deck in favor of the new small schools created in this administration. Here is the demographic composition of students in the 125 new high schools opened since 2002 and the composition of students in the other high schools receiving grades:
This doesn’t seem so inequitable. The new small schools have higher percentages of minority students, higher percentages of Title I students, lower incoming proficiency, and parity with respect to English language learners and special education students. Despite these demographics, these new small schools have received higher progress report scores on average.
Look at one example: the Van Arsdale campus, which was the site of the press conference for the high school progress reports earlier this week. The three new small schools on that campus have a collective graduation rate above 80%. The school they replaced had a 35% graduation rate in 2003-04. The demographics for the campus then and the campus now are similar, though not exactly the same (variances of 4-5 points in high-needs categories). It’s implausible to suggest that very modest demographic differences led to a difference of 45 points in the graduation rate between 2003-04 and 2008-09. By any standard, this campus is a terrific educational success story.
Progress reports have helped to increase the focus on student achievement and are holding principals accountable for student performance in a way that never existed before. We are not claiming they are perfect. We continually look for ways to improve them and solicit feedback from our educators in the field as well as external proponents and critics of our system. We have been doing this for the last three years and will continue to do so going forward. We are also reviewing the elementary-middle school progress reports to address concerns noted by Pallas and others.
It’s essential to remember, however, that for us this is not an academic exercise. We are not interested in creating the most statistically complex measurement system. That system would sacrifice a critical goal of the Progress Report: to serve as a tool with clear measures that educators in the field can understand and use. Since 2007, when the Progress reports were first released, principals have focused more intently on student learning and achieved notable gains as a result. Principals know that if they can enable their highest-need students to succeed, they will earn significant rewards on the Progress Report from the additional credit section, the diploma weights for Special Education and over-age students, and credits earned by lower-performing students. Dr. Pallas’s attempt to dismiss the credibility of the system suggests a desire to explore the perfect design from a statistical perspective without serious consideration for what is effective in practice. In fact, the Progress Report is a much better school evaluation and performance management tool than anything we’ve ever had before in New York City and has become a model for other districts nationally and internationally. We’ll continue to work to improve it and look forward to a continued dialogue on the best ways to do that.
Shael Polakow-Suransky is the Department of Education’s chief accountability officer.
-
November 20th, 2009 3:41 pm
Great post! It’s good to hear both sides of these debates. I look forward to Aaron’s comments.
-
November 20th, 2009 4:14 pm
Let/s cut right to the chase on this. How does telling a student who is a good student trhying his best that he is attending a school with a grade of D? How does that make him feel with his peers? How does this effect his college applications. No matter how much spin they try to give this, these grades are a fraud and most detrimental to kids. And the fact that 75% of schools got an A or a B is of course further evidence of just what nonsense these grades are as much as the inept, incompetent, uncertified civil rights lawyer masquerading as an educator would have us believe they are a wonderful innovation.
-
November 20th, 2009 8:32 pm
Polakow-Suransky’s first and main point about the importance of “credit meaures” fails to include the inflationary incentive when this metric becomes so critical. When credits are not earned (see http://gothamschools.org/2009/06/10/credit-recovery-joel-kleins-race-to-the-bottom/) and Regents grades are artificially skewed upward we end up with undereducated graduates, as revealed by CUNY remediation rates and other post-secondary data. All the Progress Reports seem to measure now is the sophistication of principals in making things look good and, as pointed out in Pallas’s post and many other sources, the variables that the DOE fails to acknowledge.
-
November 20th, 2009 9:39 pm
So to summarize this “frank dialogue about the strengths and weaknesses of our accountability system” , the writer dismisess all criticisms. What are the weaknesses? Please tell us, rather than again defending the system. What changes would you like to see? I would like to know what evidence supports the DOE’s contention that poverty plays no role in funding and evaluating middle schools.
-
November 21st, 2009 12:04 am
I simply don’t believe that if all factors were properly controlled that class size wouldn’t matter. It’s not a coincidence that small schools have class sizes significantly smaller than the large schools; it’s not meaningless that principals and teachers say that this is a critical factor in student success. It’s not chance that Klein, Bloomberg, and Photo all sent their own children to schools where class sizes are half the size or less that they are in our public schools. But then I have little confidence that the grades generated by this system are meaningful, or that the people running this office understand statistics sufficiently or the workings of Campbell’s law — or they would never have come up with such an untenable system in the first place. I also believe that any system in which staff at schools are grading the Regents of their own students, teachers are being encouraged to pass their students no matter what, and credit recovery is so widespread is ripe for abuse.
One question: Jim Liebman said repeatedly in testimony to the City Council when the progress reports were first developed that he understood that one year’s changes in test scores at the school level were statistically unreliable, and that the progress reports would ultimately be based upon three years of data. He is even quoted in Beth Fertig’s book saying this. So why, Mr. Polakow-Suransky, did he go back on his word? What’s the justification for that?
-
November 21st, 2009 6:28 am
Leonie,
As to why Liebman went back on his word and its justification: he/they did it because it served their interests (however they perceived them at that moment), and it’s justification is that they could.
As your important work consistently reveals, in the Alice In Wonderland world of corporate ed deform, facts don’t matter. Once anointed by the Bloombergs, Gates’, Broads and Waltons, you may do anything with impunity, for theirs is the power (if not the glory).
-
November 21st, 2009 12:10 pm
Leonie and MF,
Per the passage in the Fertig book (in which I am the nominal foil for Mr. Liebman), history has proven him not only to be short on follow-through per above, but the “acceleration vs. speed” trap in my coinage, that I and others predicted, is indeed what came to pass.
Liebman’s filibuster that night could not control the march of subsequent facts.
Sweet comeupance would be to MAINTAIN the one-year view as the tests get de-inflated. Or… restate (downgrade) the last two report years based on three years of data, not just current and prior.
And for gawd’s sake, reduce the weights on both “peergroup” and “progress.” (Thinking of ES/MS scoring; HS a bit different.)
-
November 21st, 2009 12:22 pm
Thanks Michael - but I’d like to hear it directly from DOE. What could possibly be the justification for maintaining this accountability system based on one year’s variations in test scores, when the entire academic establishment says that this is inherently unreliable? Mr. Polakow-Suransky — can you please answer the question?
I find it also strange that he should argue on one hand that small schools only slightly outperform the large school according their metric system, yet Joel Klein at every possible opportunity brags about the huge success of the small schools over the large. Why can’t they maintain a consistent line?
Even within this posting, there appears to a be an ambiguous message. First, he says that the system is fair comparing small schools vs. large in the peer indexes because ” Smaller schools perform slightly better on average (though there are many cases of large schools outperforming smaller schools — Francis Lewis, our largest non-specialized high school, earned an A, for example). ” Then he goes on to write that the ”
… The three new small schools on that {Van Arsdale] campus have a collective graduation rate above 80%. The school they replaced had a 35% graduation rate in 2003-04. … It’s implausible to suggest that very modest demographic differences led to a difference of 4-5 points in the graduation rate between 2003-04 and 2008-09. By any standard, this campus is a terrific educational success story.” Either the system is fair that compares large schools against small, or its rigged, to show off the small school’s advantages. You can’t have it both ways.Moreover, in the first comparison chart, he is comparing small schools against all schools in the system; the second he is comparing the outcomes in terms of the graduation rate of the small schools on the Van Arsdale campus to the previous school — which I would bet you had a very different student body on the whole — w/ far more high-needs students. He says the variance is only 4-5 points, but is he counting the number of ELL students, which is not included in the progress reports? is he counting 8th grade attendance factor, which I dont think the progress reports measure? This sort of sleight of hand that the DOE constantly attempts, mixing up comparison groups in order to build their case. This is also why none of the evaluations of the NYC small schools were considered rigorous enough to show positive results, in the view of the What Works Clearinghouse of the IES — because there was no attempt to fully control for the differential background of the students at the small schools compared to the large schools that they replaced. I dont think many people would doubt that if you take less needy students and give them smaller classes they will do better than high needs students who attend overcrowded schools with large classes. This should not be a surprise to anyone. But whether it serves the cause of equity is another question. And without a more reliable, independent system of measuring student learning than passing Regents exams that are being scored by the teachers and principals at the same schools, as well as credit accumulation through discredited practices like credit recovery and like, I don’t think we can really tell much of anything about what is happening at these schools.
-
November 21st, 2009 6:05 pm
You can’t deny Bloomfield’s point that a school improves its grade by lowering its standards. So, I’m not sure why we need to discuss the more complex issues. But I’ll throw in another objection anyway: I’m concerned about comparing schools that screen applicants against those who take anyone who applies. Will you acknowlege that there are many mildly or deeply troubled young people in our system who read at or above grade level and that such students are less likely to apply, much less be accepted at “screening” schools? Would you also acknowledge that such students are less likely to accumulate credits in the ninth grade than those with similar reading scores but much better attendance and academic records - the ones more likely to be accepted at screening schools?
If we can agree on those two points, and I’m correct in believing that small schools are more likely to screen their applicants, we can agree that the progress reports were designed to make small schools look good and large schools look bad. -
November 22nd, 2009 10:02 am
Point of contention Leoni.. I just started teaching in a “small” school this year.. My average class size is 32. I cam from a high school that is being phased out where my average class size was 28. The school I am at now has seen its grade drop from an A when i first opened 5 yrs ago to a C this year. In that time a new principal has taken over and the principal pointed out at the SLT meeting that level of students has also dropped. When the school opened all 9th graders were 3’s and 4’s now the majority of 9th graders are at level 2. I agree with everything you said just wanted to point out though that not all small schools are truly small.. They face the same issues as large schools face as they develop.. They have students who need to repeat classes and they have a limited amount of space. So in order to service those at grade level and those that need to retake a class they increase class size to maximize limited space. I never expected to have classes this large when I took the position this summer.
-
November 22nd, 2009 12:24 pm
Insider: It is true that small schools are seeing a needier population over time and also getting more students that they can handle, in many cases leading to increases in class size and worse outcomes for students. Unfortunately, the Gates crowd and the DOE did not make small class size a necessary ingredient for their small schools, even though it was for the earlier generation of small schools. Instead they droned on about the three Rs: Rigor, relevance and relationships (with the latter for some reason not linked explicitly to smaller classes, though it should have been.) But many analyses show that in general, large schools still on average have larger class sizes than most small schools. We will check the latest data when it is released to see if that is still true — though unfortunately, again, the HS class size data if very unreliable.
-
November 22nd, 2009 12:49 pm
The efforts by Gates, Klein, et al. were primarily focused on destroying large high schools and scattering their constituencies, diminishing resources, multiplying administration and control over labor They were a bait-and-switch operation, in which the goals of the original small schools movement - small scale and classes, intimacy, personal attention - were hijacked to implement the broader strokes of the ed deform agenda. As these schools struggle and fail, they will make easy targets for the impostion of charters, since they don’t have a rooted history in the community or a critical mass of political support within those communities.
Corporate ed deform, as executed through the vehicle of mayoral control, is about disenfranchisement, the privatization of school governance, and the creation of separate-and-unequal schools. It’s a mirror image of the hyper-polarized society, in which losses are socialized and surpluses are privatized, that is rapidly evolving around us.
The legitimate desires of parents who are desperate to find decent schools for their children are being manipulated, and parents in minority communities are being pitted against each other in a despicable divide-and-conquer effort - masked with civil rights rhetoric - that avoids the need, and the overclass’s refusal, to improve all public schools.
-
November 22nd, 2009 12:52 pm
I have another question for Mr. Polakow-Suransky: When you testified before the City Council, you were asked about the fact that 84% of elementary and middle schools earned A’s, and 97% received either “A”s or “B”s. Robert Jackson suggested that these grades indicated that you had set the standards too low. According to Gotham Schools, you replied that “In hindsight, if we could have predicted the future, we would have set the cut scores differently.”
At the time, I thought you were meant that you hadn’t known how many schools would receive As, because you didn’t know how high the state test scores would be.
But didn’t you change the cut scores that determined the grades between June 5 and August 6, long after the state test scores had been released — indicating that you determined how many schools would receive As?The EMS Guide posted on the DOE website with the earlier set of cut scores is dated June 5; while the Aug.6 version has the new cut scores. Meanwhile, the state scores were released on May 7 and June 1, and within days there were multiple reports on how inflated they were.
Or did you mean by saying that “if we could have predicted the future” you would have set different cut scores, meaning if you knew how much ridicule the grades would receive? In essence, theby setting the cut scores differently each year, the DOE controls how many grades of each category will be awarded. Doesn’t that highlight the subjective — and some might say, political — nature of this exercise?
-
November 22nd, 2009 3:05 pm
Another question about the formula: Let’s say schools A and B are in the same peer group. A is a new, “everybody passes” small school and B is a “you must earn your passing grade” large school. If B has a moderately higher graduation rate, but A has much better credit accumulation among the bottom third of its ninth and tenth graders isn’t it likely that A will receive a better score on the progress report?
-
November 22nd, 2009 3:07 pm
Leonie, Mr. Suransky explained at the hearing that the cut scores for this year’s elementary and middle school grades were sent to principals shortly after the last round of reports were issued in September 2008–long before students even took the tests that determined the 2009 grades. He said the goal was to give principals a target to work towards at the beginning of the school year, and therefore he thought it would be unfair to change the targets at the end of the year when the test scores were available (even though this meant almost every school would get an A or a B).
-
November 22nd, 2009 4:19 pm
I think you are confusing the individual school targets — which specify what goals schools need to reach to get bonuses — and the school grades. From the DOE handbooks, it is apparent that the cut scores that determined the schools’ grades were changed sometime between June 5 and August 6, 2009.
-
November 22nd, 2009 8:01 pm
Leonie,
I don’t quite understand your point about small schools and large. It seems clear to me what Shael is saying above, but tell me if I’m getting it wrong. By and large, small and large schools get the same population of students. Many small schools outperform large schools, although some large schools outperform some small schools. This rings true with my own experience as an educator in the city and I appreciate that there isn’t a purely dogmatic response (eg, all small schools are better than all large schools). So while the small schools at Van Arsdale are a success story, we shouldn’t rush to shut down Francis Lewis or other successful large schools.
I don’t think all our graduates are ready for college, and that’s a shame that schools across our country share. I tend to be skeptical of folks who believe it’s because unscrupulous teachers and principals are handing out credits so they can get bonuses. Sure it happens somewhere in this huge system, but get to know NYC teachers and principals. Most of the ones I meet are hard-working and really do want the best for students. They’re not in it for the money.
Maybe one solution would be to track our students through their first year after HS and award high schools points for how well their students do in they year after graduation. That’s a complicated task but would hold teachers and principals accountable for making sure students are college-ready.
-
November 23rd, 2009 2:03 pm
Mr. Fiorillo,
I find your critique of the privatization cabal compelling. I normally don’t think conspirators are smart enough to pull off elaborate plans (i.e, I tend to believe we give them too much credit), but, in this case, the picture you paint of Gates, Broad. etc and their Bloomberg/Klein agents booming behind the curtain makes sense.
I also whole-heartedly agree with your deconstruction of Thomas W. Carroll, who has carte blanche on the op-ed pages of the tabloids. Murdoch and Zuckerman are the megaphone he is using to prop up his anti-union, pro-charter school agenda. Thank you for piercing Carroll’s veil of sanctimony.
Leonie, you are heroic. If sincerity and interest in children were criteria you would be Schools Chancellor–or damn close to it.
By the way, I discovered the research study that Polakow-Suransky bases his defense of School Progress Reports on. It appeared in the Journal of Burmese Neurology and was re-printed in the Review of Non-Significant Findings. The title of the report is: Are There Any Differences in Size between Large and Small Schools? [Just kidding.]
-
November 23rd, 2009 2:37 pm
Roget,
-
November 23rd, 2009 2:52 pm
Roget, Thanks for the positive feedback. The only thing I would add is that I really don’t see this as a conspiracy - a loaded term that is used to trivialize and dismiss critiques of corporate ed deform - but rather a ruling class consensus. Corporate America, and Wall St. in particular, has cannibalized much of the productive economy of the US, and has now set its sites on a hostile takeover of the public schools and the public sector. From their own perspective, there’s little else they can do, although of course they must couch it in altruistic terms. But then again, that’s what a big chunk of what so-called philanthropy is: unpaid tax dollars used to reinforce the personal or class interests of those who underwite the foundations, and to burnish the reputation of those who get to buy social policy. It’s almost like a perpetual motion machine (except that it must ultimately collapse for having excessively weakened its host, which is what we have before us now): the overclass lobbies to reduce or eliminate income, capital gains and inheritance taxes, then uses those unpaid tax dollars to attack and privatize the public sector further.
-
November 23rd, 2009 8:29 pm
Tilllie, it depends on your definition of “same population”. As far as I’m aware, the small schools aren’t getting the kids with the worst academic and discipline problems. Either their applications are rejected and or they don’t apply. They end up in large schools that are placed in the same peer group as the small ones. Does anyone (Shael?) think that 8th grade attendance and behavior don’t matter - that only test scores need to be taken into account when deciding that students are “similar” in terms of the number of credits they are expected to accumulate?
-
November 23rd, 2009 10:12 pm
Smith, I only worked in one small school, so I can’t speak for the whole of them, but definitely, we got the same kids that would have gone to the large school that had been broken up. (Their siblings, cousins, etc. had gone there before them.) We got kids that were non-attenders in MS and we got kids that had repeated grades on numerous occasions before they got to us. I know that as a Gates-funded school, we were not allowed to “cherry pick” our students, nor was that part of our mission as a school. Do you think the stats Shael quotes aren’t accurate?
-
November 24th, 2009 6:35 pm
I’m not saying his stats aren’t accurate, I’m saying they don’t tell us which schools are taking in students with serious attendance and/or behavorial problems. I don’t know how typical your experience was, but I know I worked in two schools in the same peer group and one was a screening school where everyone showed up - maybe late for first period but stayed the whole day - and the other school took zoned kids, a significant number of whom had serious attendance problems.
-
November 24th, 2009 9:30 pm
Smith, I totally agree with you that things like 8th grade attendance make a huge difference, and there’d definitely be an unfair comparison between a screened school and a large zoned HS. Our school was small but we weren’t allowed to screen. In fact, I know our school could LOOK at incoming students in terms of their 8th grade attendance, but we couldn’t rank them or anything.
I wonder how many small schools are allowed to screen. That would be an interesting comparison. Are screened schools being compared to nonscreened? I know the schools in our comparison group (there were large and small, I think) were not screened.
-
February 1st, 2010 1:18 pm
I’m sorry, I don’t mean to be rude or narrowminded, but it seems to me that the real issue is why are we allowing the fox to measure the chicken coup?
Leave a Reply
Mapping the Budget Cuts
25 Comments
Subscribe to comments with RSS or TrackBack