GothamSchools — daily independent reporting on NYC public schools

grading on a curve

March 6, 2012

Integral to “value-added” is a requirement that some score low

by Sarah Butrymowicz, Hechinger Report, at 9:00 am

Add one more point of critique to the city’s Teacher Data Reports: Experts and educators are worried about the bell curve along which the teacher ratings fell out.

Like the distribution of teachers by rating across types of schools, the distribution of scores among teachers was essentially built into the “value-added” model that the city used to generate the ratings.

The long-term goal of many education reformers is to create a teaching force in which nearly all teachers are high-performing. However, in New York City’s rankings — which rated thousands of teachers who taught in the system from 2007 to 2010 — teachers were graded on a curve. That is, under the city’s formula, some teachers would always be rated as “below average,” even if student performance increased significantly in all classrooms across the city.

The ratings were based on a complex formula that predicts how students will do — after taking into account background characteristics — on standardized tests. Teachers received scores based on students’ actual test results measured against the predictions. They were then divided into five categories. Half of all teachers were rated as “average,” 20 percent were “above average,” and another 20 percent were “below average.” The remaining 10 percent were divided evenly between teachers rated as “far above average” and “far below average.”

IMPACT, the District of Columbia’s teacher-evaluation system, also uses a set distribution for teacher ratings. As sociologist Aaron Pallas wrote in October 2010, “by definition, the value-added component of the D.C. IMPACT evaluation system defines 50 percent of all teachers in grades four through eight as ineffective or minimally effective in influencing their students’ learning.”

At a time when the rhetoric around new teacher-evaluation systems has focused on removing ineffective teachers from the classroom, some question whether the ranking structure makes sense. Even William Sanders, a researcher known as the “grandfather of value-added,” is concerned about the bell-curve shape of the ratings generated by the city’s system and many others.

“If you just continue that in the future, you will always have a very high group and a very low group,” said Sanders, a former University of Tennessee professor now at the SAS Institute Inc., who has spent decades developing and refining value-added formulas. “If your population of teachers [is] improving, you basically will not be capturing [that].”

Other researchers, including Doug Harris, say a bell curve is a necessity of value-added models. Harris is a value-added expert at the University of Wisconsin-Madison, where some of Harris’s colleagues designed New York City’s formula, though he was not involved in its development.

Harris said the requirement that 5 percent of teachers be rated as “far below average” provided further evidence that value-added scores should not be used alone in making decisions about teacher tenure, dismissal or pay. Instead, he said, being in the bottom 5 percent should trigger things like more classroom observations.

As New York State rolls out its new teacher evaluation system, value-added models will play an important role. In the new system, 40 percent of a teacher’s rating will be based on student test-scores or similar quantitative measures of student performance — and at least half of that 40 percent will use state standardized test results for some teachers. (Exactly what value-added model the state will use hasn’t yet been decided.)

Even though other measures (such as observations) make up the majority of each teacher’s evaluation, student performance measures could have a big impact on a teacher’s final rating. “Teachers rated ineffective on student performance based on objective assessments must be rated ineffective overall,” said a press release by the New York State Education Department that celebrated last month’s evaluations agreement. “Teachers who remain ineffective can be removed from classrooms.”

Sanders disagrees with Harris that a bell-curve distribution of teacher ratings is necessary. In the Tennessee ranking system that he helped design, there’s no limit as to how many teachers can be placed in each category.

“You can allow for — at least theoretically — everybody to be rated very effective. We think that’s appropriate,” Sanders said. “There is so much emphasis and effort to try to get more and more people to get more and more effective. You don’t want your metrics stuck where you always have a high half and a low half.”

This story was produced by The Hechinger Report, a nonprofit, nonpartisan education-news outlet based at Teachers College, Columbia University.

Print Print
  • BP

    This system is so explicitly unfair. In reality teachers who are rated as “minimally effective” are simply less effective relative to other teachers. This says nothing about their “absolute effectiveness ” as teachers.

  • http://ednotesonline.blogspot.com/ Norm

    This is the application of Jack Welch rules — remember he was brought in by Joel Klein to advise the Leadership Academy. “Replace the bottom 5-10- or 15% people.” Thus even if everyone is a good teacher you still need the curve. This is part of the ed deform scam. How else will you be able to go around tenure rules to remove senior high-paid teachers? Did you see the story in San Francisco where they will get rid of senior teachers to keep “valued” first year TFAs using some bogus excuse?

  • Flerplunk

    “How else will you be able to go around tenure rules to remove senior high-paid teachers?”  Exactly.  The flip side of this coin is that such data-driven methods are what you end up with when you resist any effort by administrators to fire bad teachers by using non-data-driven methods.  

  • Anonymous

    The irony of this whole situation is the probability that the school district will lose its most talented teachers and not the “bottom 10%.” The teachers who are young, well-educated and very talented will be the first to go. From what I’ve read, this is already happening. Good.

  • Ken Hirsh

    Exactly.  We can remove ineffective tenured teachers provided we don’t use data-driven methods and we don’t use non-data-driven methods.

    With that said, I agree with Norm’s comparison to the Jack Welch approach, unless I am misunderstanding the systems discussed.  

  • Flerpcount

    It’s amazing there are times when I think you get it and then you say something that shows you really don’t have a clue. If someone wants to get rid of a bad teacher all that needs to be done is for an administrator to do their job. If you worked in education this was done all the time but you never worked in education.

  • http://twitter.com/JimDevor Jim Devor

    As already noted. the WHOLE POINT of the VAM as implemented by Tweed is to create castes including the untouchable bottom five (or ten) percent.  What is even more disturbing (and AFAIK an unreported issue) is the very small spread in results between the various percentiles. 

    At least as I saw it on the WSJ site, one of my daughter’s former teachers scored in the 15th percentile in one year and in the 93rd percentile over three years.  Most importantly, in both instances, that teacher’s “score” varied only minutely from the “predicted” score that presumably would have placed her around the fiftieth percentile!.

    That evidence, at least in part, explains the absurd “margin of error” many have complained of.  Indeed, the tiny variations in outcomes could more easily lead one to conclude that ALL teachers in a given cohort are either either HIGH or LOW performing (or just as easily average). 

    If “proficiency” is the target (and at least until recently, that is what the tests were designed to measure), then ranking teachers on the basis of tiny variations from the “norm” in student outcomes makes no sense.  After all, I daresay, NO ONE would seriously consider the Bar Exam test scores of lawyers (or even the average scores of students at particular Law Schools) would be particularly useful in evaluating them.  The ONLY relevant question is whether or not they passed

    If “professionals” like Joel Klein have to be evaluated holistically, why shouldn’t teachers?

      

  • Flerplunk

    Presumably your solution would be to force administrators to do their jobs (i.e. fire more teachers) or lose their jobs, then.  Or is your point just that it’s not your fault that bad teachers aren’t fired more often?  If so, consider yourself absolved. 

  • Flerpcount

    My solution isn’t to force anyone to do their jobs. I want good teachers to do their jobs and bad teachers to either get better or move on. I want bad administrators to do a good job or else they should move on.
    It certainly isn’t my job that bad teachers are hired especially when 70% of the teachers who are working now were hired under Bloomberg. Maybe the job he is doing certainly isn’t working. What do you think we should do about that?

  • Alex Messer

    Some fellow NYC teachers and I have just started a very important petition to help give us voice against this:   
    http://www.change.org/petitions/stop-the-public-shaming-and-unjust-firing-of-teachers

    Please visit, read, repost and support! 

  • Nychistoryteacher

    “If your population of teachers [is] improving, you basically will not be capturing [that].”

    This is a critically important story and it doesn’t end with teacher evaluations. Almost all standardized tests are norm referenced. We won’t be able to see an improvement in student performance either if we use standardized assessments. When too many students start doing well the state will change the cut scores (move the goal posts.) When you use norm referenced assessments you can’t see improvement over time.

  • Michael M. (parent still)

    If the Margin of Error on a student’s score was as big as for the teacher’s “score,” this topic would be entirely different.

    DOE: “Your child scored in the 30th percentile.  Give or take 75 points.  So we’re holding him or her back a year.  AND we’re going to help the ComPost run a front page story on what a terrible student she or he is, so that other students will know we mean business.”

  • NUFF SAID

    One thing you can count on is “the law of unseen consequences”–consider this –what would stop an elementary school from actually setting the students up to score low on the tests just so they can meet AYP in the following years? Why would they want the students to score top 4′s if that meant the next year it would be impossible to meet AYP? The entire thing seems poorly thought out and harmful to students in the long run, it certainly does not encourage excellence

  • Michael M. (parent still)

    I’ve been mulling the pun in the headline:

    a) As pointed out by others, any “curve” has a low end, even if the entire curve has moved up.

    (One year’s student achievement growth should nominally make a teacher look like he or she has done exactly what was expected, but I’m not sure it works that way.  In other words, I’m not sure the “curve” is “centered” on 1 year’s student growth.) 

    b) The only way to show “value added” is to take low performing students, and raise them to higher levels.  So I’m having a hard time understanding how — in a standards-based  system — teachers would be adequately rewarded for helping students truly excel beyond the standards.  It seems there’s a bias that the STUDENTS score low (the prior year) for teachers to score well in a subsequent year.  (But not too low, or perhaps there’s something going on keeping them stuck low, and who’d want to risk THAT?)

  • nuff said

    THEY LIKE TO YELL PEOPLE ARE ENTITLED TO SEE THE SCORES. AND YET NOONE HAS ASKED FOR RAW SCORES–THE REAL DATA–SCATTERGRAMS!!–YOU ARE ONLY SEEING WHAT THEY WANT YOU TO SEE–WHY IS THAT?

  • Guest

    law of unintended consequences.

  • Flerplunk

    “Maybe the job he is doing certainly isn’t working.  What do you think we should do about that?”

    Using your model, I’d do nothing.  Just wait for him to “either get better or move on.”

  • http://ednotesonline.blogspot.com/ Norm

    Reality: all non-tenured teachers can be removed for anything. Or nothing.
    Reality: 70% of all teachers currently teaching came in under Bloomberg. And probably a much higher percentage of principals.
    Reality: Many of the teachers had tenured extended and are in their 4th and even 5th year of working where they can be fired at any time.

    Exactly how many people are we talking about rooting out here?
    Bloomberg will close 33 schools to get $55 million even if it costs him $200 million. Do you see what the prime directive is? Remove all the high salaried people you can. Low scores for 2 yrs go for it.
    I found that a teacher can have a perfect record for 25 years but have one incident and they can face dismissal. History doesn’t count when you have a political agenda. THus a teacher might have 20 yrs of “good” data but end up with some doctored classes — a new principal half your age wants to dump salary — here is the trigger.
    In fact admins have the tools to remove a tenured teacher — it seems I’ve heard from an awful lot of them over the years so it is not as if this is not happening.
    Does it cost? Yes and it should not be a snap. But even that system is cheaper than massive tests and monitoring — where principals have to do useless obs.

    Instead of all these expenses and attention to the potential removal of —- and I love this term — ineffective — teachers why not just offer a buyout. Or aren’t there all kinds of talents needed in the system that a person could be put in?
    I was in the system for almost 40 years and saw hundreds of teachers in a number of schools. And administrators. The percentage of ineffective admins has always been much higher than ineffective teachers which is a major root of the problem. I have always maintained that teachers and parents at the school level should chose the principal because they have the biggest stake in competency.
    But in this country we will never go there.
    I imagine people like Ken assume the worst teachers in charter schools are removed without a problem because they have no union or tenure protection. But the reality in both public and charters is there are other protections. Loyalty, relatives, buddies, etc. This is a fact at all levels in schools and in every job. Just as I saw in schools I taught in, the principal would sooner be rid of someone viewed as a non-team player no matter how good a teacher.

  • Guest

    This is intended.

  • guest

    This makes me think of what happened with summer school for the past two years, in 09-10 “oops, your child was supposed to be in summer school” and 10-11, “oops, your child wasn’t supposed to be in summer school”

  • Vote NO!

     To  those  inside  the  classroom,   this  is  not  a  debatable  issue.  What  has  happened  over  the  past   two  years  with  this  new  teacher  evaluation  emphasis  has  taken  public  education in  a  disastrous  new  direction.  If  the  emphasis  of  the  new  evaluation  system  is  trying  to  create  a  more  effective  teaching  force,  it  is  having  just  the  opposite  effect. It  is  TURNING  many  good  teachers  away.  I’m  sure  it  will also  drive  many  prospective  teachers  away  as  well.  People  are  not  going  to  invest  time  and  money  in  a college  education  to  work  under  an  evaluation  rubric  where  they  are  just  being  set  up  to  be  fired.  The  1990s  “one  foot  teachers”  are  back!

  • guest

    “The flip side of this coin is that such data-driven methods are what you end up with when you resist any effort by administrators to fire bad teachers by using non-data-driven methods”. What are you talking about?! The poor administrators who stand around and BULLS**T and gossip but don’t completely do all their work and observe and compile the proper paperwork? Or are you talking about unsubstantiated vengeance that cannot be backed up with any paperwork? That’s the resistance that you are talking about? Everywhere I go, a hospital, a government office, Staples, I can observe several people doing their jobs and several people walking around frantically looking busy and doing NOTHING!  I bet you are the one walking around frantically out there somewhere, looking busy, making lots of noise..AND DOING NOTHING! 

  • Flerpcount

    You used your words not mine just like a lawyer would do. I want people to do the job that they are paid to do. If someone is not doing their job and someone’s job is to judge someone work then they should do their job or be fired.The mayor isn’t doing his job but you say nothing about that. Bloomberg hired most of these working teacher’s and hired most of the administrators who didn’t do their job’s, say something about that.
    Who told you to use my model certainly not me?Do yourself a favor before you comment on something you really should research your subject material and have some facts. You should understand what tenure is all about but I really believe you don’t really care because you are here just to stir the pot.
    In a year Bloomberg and his education legacy will be gone and it will take 50 years for the system to recover. 
    Ed deformers like yourself will attack tenure and it might go away but in the end you will have destroyed a system that for many years worked to ensure that educators could work in a system that they weren’t afraid of. They could speak their minds and advocate for what they thought was right. More and more great educators are running to retire and get away from this broken system. I hope you are very proud of yourself you are destroying something you really know nothing about. 
    You say you have children in the system you certainly aren’t making it better for them but I don’t think you really care.

  • Vote NO!

    The  question  no  one  is  asking is:  In  the  zeal  to  root  out  the  “ineffective  teachers”  are  we  creating  an  inferior  teaching   force  with   the   Danielson  “gotcha”  observations,  and  the  unreliable  VAM  assessments?  How  many  good  teachers  are  going  to  be  forced  out,  or  terminated?  Who  will  replace  them?  Do  you  really  think  the  “replacements”  will   perform  any  better  once  they  are  subjected  to  the  Danielson  “checklist”  or  the  VAM  frustration,  and  humiliation?

  • First year teacher

    That’s why a rubric system must be used to objectively evaluate performance. This is the only fair approach.

  • no2privateprofit taxpayerloss

    Here’s my take:

     

    The goal education reformers are claiming to make is that
    they want to reform only the poorest performing schools.  If that were true, the D.O.E. would simply
    reform schools with children performing below grade level in math and reading.

     

    I believe the reformers’ actual goal is to CONVERT the ENITRE
    PUBLIC EDUCATION SYSTEM into the for-profit-charter school model (or “non-profit”
    charters with for-profit management model). To achieve that goal, they would
    need a rating system so that even the best teachers in the best schools could
    not get a good rating 2 years in a row.

     

    To illustrate, here is an example rating from an article
    in the NY Times: (In New York Teacher Ratings, Good Test scores Aren’t Always
    Good Enough by S.Otterman & R. Gebeloff):

    “In one extreme case, the formula assigned an
    eighth-grade math teacher at the prestigious Anderson
    School on the Upper
    West Side the lowest possible rating, a zero, even though her
    students posted test scores 1.22 standard deviations above the mean — normally
    good enough to rank in the 89th percentile. Her problem? The formula expected
    her high-achieving students to be 1.84 standard deviations higher than the
    average — roughly the 97th percentile.”

    This link has similar findings:   

    http://boogiedowner.blogspot.com/2012/03/nyc-teacher-speaking-out-on-teacher.html

    The reformers say that the ratings give parents information, yet no one can
    tell what grade level the students achieved this year vs. last year, or expected
    improvement % over the previous year. WHERE IS THE TRANSPARENCY?

     

     

     
     

Tips, questions, feedback?

Contact us at .

Word from Our Sponsor

Chalk It Up

Recent Comments

2 comments so far today
View all comments »

Events Calendar

RSS Feeds

Newsroom posts Community posts All GothamSchools content

Archives

June 2013
M T W T F S S
« May  
 12
3456789
10111213141516
17181920212223
24252627282930

Copyright Notice