Responding to Readers’ Comments

guest perspective

November 25, 2009

Responding to Readers’ Comments

by Shael Polakow-Suransky, at 1:22 pm

I’d like to respond to some of the reactions to Friday’s post:

1. Cut scores: Contrary to Leonie Haimson’s allegation, we did not determine the percentage of A grades after learning the results of the 2009 state tests. The cut scores for the elementary and middle school progress reports were set in September 2008 and communicated to principals in the Sept. 23, 2008, mailing of Principals’ Weekly (pasted at the end of this post) — long before the state tests were even administered. The two educator guides Ms. Haimson cites correspond to different years — one is for the 2007-08 progress report and the other is an updated version for the 2008-09 progress report.

We raised the cut scores significantly from 2007-08 to 2008-09 to reflect the progress schools had made. However, the gains our schools achieved in 2008-09 surpassed anything we had seen during the last few years. Had we been able to forecast this growth, we would have set the cut scores even higher.

2. Multiple years of data — We use three years of data to establish benchmarks for comparing schools’ performance and progress. Put another way, the range of scores that determine each school’s peer group are based on three years of achievement data.

That said, we look only the most recent year’s results when determining progress report grades because it is critical that schools focus on their students’ achievement every single year. Using three years of results would allow schools that performed well for one or two years to mask poor results in a third year. Also, our high school progress reports are based on hundreds — and in many cases thousands — of individual student-level outcomes across multiple measures for each school. In this regard, we are very comfortable with the level of statistical rigor reflected in the results.

3. Small schools vs. large schools — In my original post, I explained why we do not control for a school’s size when determining peer groups and also provided data showing that on average the new small schools opened under Mayor Bloomberg and Chancellor Klein have outperformed all other schools on the progress report. This data is not inconsistent with the fact that many other schools — large schools as well as small schools opened prior to this administration — also performed well.

In addition, data do not support the claim that small schools serve less challenging populations. In fact, as the chart below shows, small schools serve more challenging populations in every high-need demographic category with the exception of special education, where there is parity. Small schools are not always better than large schools, but small schools in New York City are more likely to be successful with high-need students. Based on my experience as a teacher and a principal, I would argue that this has a lot to do with the size of the principal’s class. In a small school, the leader is likely to be supervising only 30 teachers, compared to large schools where there are often 150+ teachers. The small school structure has fewer administrative layers, making it much easier to know and support each teacher’s individual needs-which in turn enables teachers to do the same for their students.

4. Class size — Class size is not correlated with progress report score. As the chart below shows, the average class size at schools receiving each letter grade on the high school progress report varies little.

This is not to say that adjusting groupings of students and teachers should be ignored as a possible strategy to increase teacher effectiveness and student achievement. Again, in my own experience as a high school principal, I found that a powerful way to increase the effectiveness of my teachers was to reduce the total number of students for whom each of my teachers was responsible throughout the day and week. As UCLA Professor William Ouchi has demonstrated, when teachers’ total student load decreases, student performance increases.

Note: Class sizes reported here were calculated by dividing the number of students in a course by the number of official classes in that course, regardless of the grade level of the students enrolled. More information about methodology for calculating class size is online here.

5. Credits and Regents — Passing five Regents Exams and earning 44 course credits are the core requirements defined by New York State for graduation, and the New York City accountability system is based on these measures. Failing to measure credit accumulation would return us to a system where principals are not accountable for student learning.

Unfortunately, there have always been charges of cheating in our system, which are dealt with when they are substantiated. The vast majority of school leaders and teachers approach this part of their work professionally and report it when they see a colleague do otherwise. Staff from New York State and New York City randomly monitor the administration and scoring of Regents examinations. As with the Grades 3-8 Math and ELA examinations, any reports of mis-administration or security breaches are immediately reported for appropriate action. Schools where such reports have been made receive additional monitoring during the scoring and administration process.

Regarding credit recovery, our schools follow the state’s guidelines for awarding these credits. When a student doesn’t pass a required course or doesn’t complete all of the necessary coursework, the student must make up that work; this is the practice we refer to as “credit recovery” and it is a sensible and longstanding practice in schools nationwide. Credit recovery can be achieved in several ways, including retaking an entire course during the school year or attending summer school. In addition, as the State Education Department recently explained: “Sometimes students may come close to passing a course and may have deficiencies only in certain clearly defined areas of knowledge and skill. In those cases, it may not be necessary for the student to retake the entire course. Instead, the student might be permitted to make up those deficiencies, master the appropriate standards, and receive credit.” Like any other process, credit recovery can be abused. This abuse hurts students and is cause for disciplinary action. To that end, we have been working with the state to establish clear guidelines and processes for credit recovery. In October, the Board of Regents adopted a policy for making up course credit and directed districts to draft regulations to implement that policy. We will continue to work with the state to implement the policy and regulations in New York City.

Finally, I’d like to echo the suggestion that we focus our efforts on preparing our students for college. We are beginning to work on the complicated task of tracking students’ performance through their first years after high school and look forward to using this kind of data in the future to increase the rigor of the progress reports.

***

From the Sept. 23, 2008, Principals’ Weekly:

2008-09 Progress Report Cut Scores Elementary and middle schools

When they released elementary and middle school Progress Reports last week, Mayor Bloomberg and Chancellor Klein celebrated the accomplishments of schools in increasing student progress across New York City and noted that those gains, along with the work that remains to be done, make it appropriate to raise the bar – something we said we would do as Progress Report grades rise. We are setting new cut scores that elementary, middle, and K-8 schools need to achieve to earn a grade of A, B, C, or D on next year’s Progress Report. The new cut scores are as follows:

A – Progress Report score of 68+
B – Progress Report score from 54 to 67.9
C – Progress Report score from 43 to 53.9
D – Progress Report score from 33 to 42.9
F – Progress Report score less than 33

In the October release of the ITT, schools will receive a new Progress Report data file that contains the school’s Progress Report “modeler.” These new cut scores will be built into the modeler to allow schools to run scenarios that help them forecast how they may perform on next year’s Progress Report using these new cut scores.

http://www.davidcbloomfield.com David Bloomfield

Shael’s comments re: credit recovery (“my” issue, I guess) reek of sanctimony. Head studiously buried in the sand, he doesn’t even say they check for abuse in this area but are simply working with the State on a mutual policy that is a blank check for abuse; but, since now State-sanctioned, won’t be abuse! And who would report abuse anyway, the student? the school?, both so profitably credit-engorged? A sure sign that the DOE encourages the practice is its refusal to track credits earned this way, inflating its own self-designated record of supposed student progress.
Smith

I know you had a lot to respond to, but you didn’t answer my question about “screening” schools. My question was about attendance and behavior, not about demographics or test scores. Could you please answer the following question “yes” or “no” and briefly explain your answer?

If two black, incoming 9th graders have identical reading and math scores, both receive free lunch, and are the same age, yet one accumulated 40 absences and 3 superintendent’s suspensions in 8th grade while the other had perfect attendance and no record of behavior problems, would the DOE consider them “similar” when determing their schools’ peer groups for the progress reports?
Smith

It’s quite a stretch to equate credit accumulation with student learning. There’s a big difference between a 65 and a 95, and I’m sure you’re aware, notwithstanding your perfunctory comments about cheating, that this difference has become a lot bigger in the last few years.
http://www.SpecialEducationMuckraker.com Dee Alpert

This past Friday, the NYS Comptroller issued an audit of State Ed.’s supervision and monitoring of the grading of Regents exams. http://osc.state.ny.us/audits/allaudits/093010/08s151.pdf.. Essentially, the Comptroller reported that when State Ed.’s outside consultants found that districts and schools had inappropriately inflated grades on their students’ Regents exams, State Ed. did absolutely nothing about it. And the Comptroller’s own regrading of Regents exams showed the same inappropriate grading result. The audit also noted that State Ed. failed to investigate most of the complaints regarding this type of adult cheating it received, and had nothing good to say about how such allegations are handled within the NYC DOE. Other, prior reports reveal that the same is true for the grading of the alternate assessments for severely disabled students and the regular grades 3-8 ELA and math exams. Unfortunately, any system for evaluating schools’ performance, or that of individual teachers, and which is based on these grades in whole or in part fails the most basic tests because the data used is unreliable in extremis.

Insofar as schools are evaluated based on graduation data, and graduation requires students’ passing Regents exams, the evaluations are similarly unreliable. NYC Comptroller Thompson issued an audit this past July which raised the most substantial questions regarding the validity of graduation numbers, http://www.comptroller.nyc.gov/bureaus/audit/PDF_FILES/ME09_065A.pdf.

While the NYC DOE claims – repeatedly – to be transitioning to a lean, data-based organization, and claims to expect its employees to do the same, the fact remains that the vast majority of data used in its various, highly-touted evaluations, calculations and reports to parents and the public is both unverified and open to significant, legitimate, and very pressing question. As long as such verifications are not performed, and their results made promptly and fully available to the public, one must seek data which the NYC DOE does not control in order to assess its functioning and claimed improvements. To date, while the Tweed and City Hall press releases would make one think that things have gotten substantially better, the fact is that both the NAEP and SAT scores are flatlined. If individual schools’ performance was improving at the rate the recent Progress Reports would indicate, SAT scores would have risen significantly. They did not. Period. End of story.

The whole student, teacher, school and program evaluation system the NYC DOE operates is nothing short of a house of cards. One real audit, done by auditors over whom the NYC DOE has no control, and it simply falls of its own weight. One can only hope that the Independent Budget Office, or the US Dept. of Education’s Inspector General, will do it. Soon. Please.
Marty

Shael, Here’s my other question you didn’t respond to: Isn’t it possible for a school with a higher graduation rate to receive a lower grade than it’s peers simply by having tougher grading policies (what we used to call “higher standards”) and thereby awarding fewer credits to 9th and 10th graders? I’m hoping you’ll explain the reasoning behind this formula.
Michael M.

Shael,

Thank you for dedicating a column to responding to readers’ comments. A few more, if I may:

1) My (and others’) previous comments about multiple years’ data STILL STAND. Your response above is in contrast to a number of schools having their scores along a spectrum fall OUTSIDE the nominal min-max range. It would seem that if the current year (and two priors) is included in establishing the bounds of a range, that simply could not happen. That it DOES happen suggests something that, at best, still needs explainifying.

2) My previous comments about the problems with peer groups STILL STAND, and are not addressed above. (An “A” school amongst a weak peer group vs. a “B” school amongst strong schools, etc.)

3) My previous comments about the double-benefit of too much weight on “progress” (which was increased DESPITE parent requests that it be REDUCED), plus state score inflation… STILL STAND.

These points remain unaddressed — some for years. I look forward to your reply.
Smith

Shael, shall we assume that whatever questions you ducked in your post will not be addressed in the comments section?
Michael M.

Thanks, Smith.

Helloooo Shael?

Multiple Years’ Data?
Peer Groups?
Progress Weighting?

Irony re “accountability?”
Smith

You know, sometimes I wonder if I go a bit too far in making claims like “the DOE uses progress reports to make small schools look good and big schools look bad.” Couldn’t it be possible that these people mean well, but don’t really understand the implications of what they’ve done? But then we try to ask the DOE a few simple questions and get ignored.

Elizabeth, how about some ground rules for guest bloggers? Shouldn’t they have to respond to legitimate questions?
Michael M.

I’m still tittering over the title of this essay, given my first round comments were not answered, nor was my challenge here on round 2.

Yet UHgain, irony, not to mention doublespeak, is alive and well at Tweed!

DING! Can we call it a TKO yet?
Shael Suransky

New questions have come up since I last posted, and there are some questions that I didn’t get to in my first two posts. Here are responses:

1. Prioritizing student progress – As many readers have noted, we put significant weight on year-to-year student progress. While some on this page have expressed a preference for putting additional weight on student performance, we believe that schools should be judged, in most part, on the progress their kids are making. The current progress report weighting policy reflects that belief.

2. How schools scored outside of the range relative to peer and city horizons – A fair question has arisen about how a school could score outside of the range relative to the school’s peer and city horizons. The answer is relatively simple. The maximum and minimum values on the range are not absolute values, but derived from the distribution of achievement results. To calculate the range, we find the mean score using the previous three years of achievement data. The maximum and minimum values on the range represent two standard deviations above and below the mean. By definition, roughly 2.5 percent of schools should then fall outside of the range on both sides. Undoubtedly, we saw more schools falling above the range this year because of the unprecedented gains on state exams our students made relative to the prior three years.

3. Multiple years of data – The range of scores that determine each school’s peer group are based on three years of achievement data. Peer groups for the 2008-2009 progress reports were calculated based on data from the 2005-2006, 2006-2007, and 2007-2008 school years. While it’s true that in past years we have used current year achievement data to set peer horizons, it is a priority of the Department to give principals clear targets to meet prior to a given school year. Using current year data precludes us from achieving that objective.

4. Attendance as a factor in peer index methodology – We have previously considered using attendance in the high school peer index methodology. Certainly, attendance is highly correlated with student-level outcomes in high school. In the absence of eighth grade proficiency scores, attendance would work very well in helping us to develop peer groups. However, a student’s proficiency is an even better predictor of student-level outcomes in high school. And because attendance is so highly correlated (~80%) with eighth grade proficiency, it does not provide us with any additional benefit in “peering” schools (i.e., proficiency essentially controls for both attendance and academic ability). Special Education and over-age (on entry) are far more important to control for in the peer index than a variable that has such a high degree of correlation with another factor we’re fundamentally controlling for already.

5. Credit accumulation and graduation – The progress reports are designed to reward schools that are successful in helping their students make progress toward graduation and in measuring the graduation outcomes themselves. If students are not passing their courses or passing Regents exams, they will not be in a position to graduate following their senior year. The correlation between credit accumulation in years 1 and 2 (and year 3 for that matter) and graduation is over 70 percent, so schools where students are earning more credits in the early years are also graduating more students. Students who fall behind in credits often find it difficult to catch up and are less likely to graduate.

6. Admissions processes for small schools – Several readers have suggested that small schools are more likely to screen, and are therefore more likely to succeed on progress reports. However, this is not true. In fact, over 80% of the small high schools opened under Mayor Bloomberg and Chancellor Klein since 2003 are unscreened, compared to approximately 50% of HS’s citywide.

At the end of this calendar year, I will respond to any additional questions that remain unanswered. Until then, happy holidays!
http://www.davidcbloomfield.com David Bloomfield

#5 ignores the issue of credit recovery as well as the other watered-down educational standards students now need for graduation. Shael’s explanation of the predictive value of credit accumulation is a tautology: students need credits to graduate so we give them the credits they need (and weakened Regents); presto, they graduate!
http://www.SpecialEducationMuckraker.com Dee Alpert

Shael writes that: “Regarding credit recovery, our schools follow the state’s guidelines for awarding these credits.”

Shael – Perhaps you’d be kind enough to give me a link to the “state’s guidelines for awarding” credit recovery credits – those which were in effect before the Bd. of Regents formally approved this process in October 2009. Frankly, I’d never heard of them and never been given the chance to analyze them.

Of course, if I’m wrong in stating that diplomas granted to NYC DOE students on the basis, in whole or in part, of “credit recovery” before October 2009 were illegitimate, I’d be more than happy to publicly say so. I review State Ed.’s web site frequently, but never came across any such “guidelines.” And frankly, the “credit recovery” schemes described to me by various NYC DOE high school staff in the past few years have appeared to be arbitrary, at best. But I realize that these are mere, albeit numerous, anecdotals which may not be representative of what NYC DOE high schools were doing overall. On the other hand, maybe they were.

So, again, if you could just give me a link to the “guidelines” which existed prior to October 2009, I’ll be prepared to read and digest them – and if necessary, eat my hat as well if I’ve been wrongly thinking that something substantial stinks in Denmark.

Dee Alpert, Publisher
SpecialEducationMuckraker.com
Michael M.

Shael,
Thank you sincerely for the additional responses. I will have more to say later. Cheers.
alim

http://gothamschools.org/2009/04/22/regents-are-weighing-procedural-rules-for-credit-recovery/
http://www.alimcourse.com aalim courses

Thank you for your opinion. Although I only agree with it partly, can you tell me how you arrived at this conclusion? Thanks.
http://www.SpecialEducationMuckraker.com Dee Alpert

One more time – Please provide a link to the State Ed. Dept.’s guidance, which the NYCDOE allegedly followed, regarding “credit recovery” prior to October 2009, when regulations governing credit recovery were formally passed by the Board of Regents. If the document(s) containing this guidance are not available on line, please provide the date(s) of the document(s) to which you refer and the author(s) and title(s).

I asked for this back when you wrote attempting to refute Ms. Haimson’s position back in November 2009 … and am still waiting. You cannot expect people, including journalists, to simply rely on your recitation regarding available State Ed. “guidance” when the underlying documentation is not made available for analysis.

If you don’t provide the links or titles, etc., I’ll FOIL them and then distribute the NYCDOE’s responses widely. I realize that this would result in delay after delay, but I have a long memory and have no intention of dropping this issue.
Matt Mittenthal

The links below lead to memos released by the New York State Education Department with information on credit recovery. These memos demonstrate that, while the State’s regulations did not specifically address credit recovery prior to the adoption of current policy, it was common practice for schools to allow students to make up credit in courses they had failed.

Recognizing this, the State Education Department took steps to formulate a policy on make-up credit, which became effective July 1, 2010.

For the April 2009 memo: http://www.regents.nysed.gov/meetings/2009Meetings/April2009/0409emscd2.htm

For October 2009 memo: http://www.regents.nysed.gov/meetings/2009Meetings/October2009/1009emsca5.htm

For April 2010 memo: http://www.regents.nysed.gov/meetings/2010Meetings/April2010/0410brca1.pdf

Matt Mittenthal
Deputy Press Secretary
NYC Department of Education
http://www.SpecialEducationMuckraker.com Dee Alpert

Actually, what these memos show – clearly – is that there was no state law or regulation allowing “credit recovery.” As with many other important things, NYSED merely turned a blind eye to what was going on. The NYCDOE didn’t formally report what it was doing in this area … and NYSED didn’t look. Claiming that NYCDOE conduct in this area was somehow legally authorized by NYSED is like claiming that since some cop, somewhere, might have had an idea that you might have had a gun and might to use it, since he didn’t stop you, shooting your noisy neighbor was fine. And, of course, you shouldn’t be prosecuted for doing so.

In other words, to be blunt, credits awarded pursuant to credit recovery schemes (a/k/a scams) prior to October 2009, when credit recovery was specifically authorized by the Regents, were awarded without legal authority and were – and should be – deemed void. Similarly, high school diplomas awarded which were based in any way on credits granted pursuant to credit recovery schemes prior to October 2009 were not properly awarded and should, at a minimum, be ignored when official graduation and dropout statistics are calculated and reported. I can’t see punishing individual students who were granted “recovered” credits, but it is appalling that the NYCDOE’s officials would claim that they should get credit for having done so unlawfully.
Ed

Dee Alpert, as McNulty would say, is “real p-o-l-i-c-e.”

We need a new “Wire” that focuses solely on the racket of urban public education, the egos, fear, paranoia, and gamesmanship involved. David Simon go so much right about local politics and law enforcement, but glossed over how public education fits into the racket with mere mention of “juking the stats.”

There just wasn’t enough time or space.

Our Twitter Updates

Public comment period ends. Moving much faster than what was expected. Most protesters didn't speak and UFT got here too late to sign up. 7 mins ago
Update: they're back in now and there is a.considerable police presence near the stage. Crowd has thinned considerably #warofattrition 17 mins ago
An activist just drew gasps from his fellow protesters. ”I hope your kids get their asses kicked by all the kids here” he tells the panel. 47 mins ago
NYPD has closed auditorium doors. Students protesting in lobby are now locked out of the meeting. 1 hr ago
... & a mom says the city transferred her son from KAPPA VII, which is staying open, to ABCD, which could close, after he was bullied. 1 hr ago
More updates...

M	T	W	T	F	S	S
« Jan
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29

GothamSchools — daily independent reporting on NYC public schools

guest perspective

November 25, 2009

Responding to Readers’ Comments

Tips, questions, feedback?

Follow GothamSchools

Feb. 10: You’re invited!

Recently Posted Jobs

Chalk It Up

Recent Comments

Events Calendar

RSS Feeds

Our Twitter Updates

Archives

Copyright Notice