GothamSchools — daily independent reporting on NYC public schools

research shows

January 6, 2012

Gates Foundation study paints bleak picture of teaching quality

by Elizabeth Green, at 6:49 pm

The study measured teachers against the criteria in Charlotte Danielson's Framework for Effective Teaching rubric, which is used in New York as a tool for observing teachers. Teachers scored better at classroom management than they did on measures of higher-order instructional challenges, such as asking productive questions.

A historic look inside the nation’s classrooms, including some in New York City, painted a bleak picture, according to a report released by the Bill and Melinda Gates Foundation today.

The second installment of the foundation’s ambitious Measures of Effective Teaching study, the report focuses on the picture of teaching yielded by five different classroom observation tools. It also scrutinizes those tools themselves, concluding that they are valuable as a way to help teachers improve but only useful as evaluation tools when combined with measures of student learning known as value-added scores.

The conclusion is a strong endorsement of the Obama administration’s approach to improving teaching by implementing new evaluations of teachers that draw on both observations and value-added measures. New York State took this approach to overhauling its evaluation system when it applied for federal Race to the Top funding.

Among the group of five observation tools the foundation studied is the rubric now being piloted in New York City classrooms as part of stalled efforts to implement the changes to teacher evaluation, Charlotte Danielson’s Framework for Effective Teaching.

Through all five lenses, instruction looked mediocre in an overwhelming majority of more than 1,000 classrooms studied, the report concludes. There were some bright spots. Many teachers were scored relatively well for the aspect of teaching known as “classroom management” — keeping students well-behaved, making sure they are engaged.

But teachers often fell short when it came to other elements of teaching, such as facilitating discussions, speaking precisely about concepts, and carefully modeling skills that students need to master. These higher-order skill sets, the report notes, are crucial in order for students to meet the raised standards outlined in the Common Core.

The study is the most expansive known examination of instruction in the U.S., reviewing more than 1,000 teachers for this report and nearly 3,000 for the study. Its lead authors are the economists Thomas Kane, of Harvard, and Douglas Staiger of Dartmouth, although more than a dozens researchers contributed to the study.

The evaluations were conducted by trained evaluators, who watched clips from videotape of more than 1,000 teachers around the country and then judged whether the teaching exhibited certain traits outlined in the observation tools.

One complicated aspect of the study is that it doesn’t just ask what the observation tools have to say about teaching; it also asks whether those observation tools are good ways to measure teaching at all. The question is crucial to the contentious teacher quality debate.

Motivated by the Obama administration’s focus on improving teaching by improving the way teachers are evaluated, the teacher quality debate has been dominated by a search for a better evaluation tool. The idea is that if school districts could have a better way to sort teachers, then they could increase quality by rewarding those who are most effective and improving or removing those who are less effective.

The study offers a qualified endorsement of the five observation tools it studied, saying that they should be one of multiple evaluation measures but that no one observation tool should be a sole measure. While the study found that all five observation tools had a positive association with student achievement as measured by value-added scores, the associations were not perfect.

And the tools’ reliability was relatively low — lower, in some cases, than the famously volatile judgments of value-added measures. When different observers used the same tool to evaluate the same teacher, they sometimes gave very different scores.

But the report does endorse using the observation tools in combination with value-added measures, as New York’s new evaluation system does. When researchers combined multiple observation tools’ judgments of teachers together — and then combined those with the teachers’ value-added scores, the result was a view of a teacher that was more able to predict future student achievement, the report says.

A final complication worth noting is that the study’s ultimate arbiter of what makes a good evaluation tool is itself under heavy scrutiny. That arbiter is a teacher’s value-added score, an estimate that attempts to extrapolate the amount of student learning for which a teacher can be held responsible, excluding other factors such as a student’s family income level.

A study that was the subject of a story in today’s New York Times found that value-added scores indeed are useful predictors not only of student achievement, but other measures of life success. Researchers have cast doubts on value-added measures’ validity, citing a host of concerns from the measures’ volatility to whether a high value-added score reflects true student learning or simply effective test prep.

Though an overhaul of teacher evaluation in New York has been stalled by the failure of teachers unions and school districts to agree on how to conduct it, both the New York City teachers union and the Department of Education agreed to participate in the Gates Foundation study when it launched in 2009. The union helped recruit teachers to join, and ultimately, teachers from about 100 schools signed up to have their lessons videotaped and analyzed.

“It takes the politics out of what’s being measured,” UFT president Michael Mulgrew said when the union first agreed to participate. “Teachers are very frustrated with the political debate. They are always saying, ‘why don’t you just come into the classroom?’ That’s what this is doing.”

Since then, the politics over teacher quality has grown even more heated.

Last summer, a GothamSchools reader who had worked in a school piloting the Danielson evaluation said it was very hard for teachers to be rated “effective.”

Print Print
  • Tiredofyou

    wow what a surprise the Gates foundation found something wrong with teachers.
    Its so amazing asking the fox about who ate the eggs in the henhouse.

  • I noticed that…

    Why should we be surprised that the MET project funded by the Gates Foundation and endorsed by the UFT would not be used to disparage the teaching profession in the public school system?  How many teachers signed up to be involved in the MET project, not knowing that they were drinking a poisonous, career-killing kool-aid.  First it was the 2005 contract, then the TDR, then the ATRs, and now the MET!  Next DOA for all senior teachers. 

  • guest

    exactly what I was thinking – as soon as I saw that Gates sponsored  this study -I knew I could stop reading it-  Wonder how much the “trained professionals” watched the tapes got paid and who deemed them “trained professionals?? Maybe he could pay me to observe those morons too.

  • Vote NO!

    The  Danielson  framework  is  totally  unrealistic  for  inner-city  schools  with  a lot  of  high  needs  students.  If  implemented  the  way  it  is  being  piloted  in  the  PLA  schools,  it  will  decimate  the  teaching  profession.  Many  teachers  in  these  schools  are  seriously  considering  leaving  teaching..  Some  have  retired  mid-year,  others  have  quit  mid-year.  This  has  left   the  administrations  scrambling  to  find  teachers  for  those  classes.  That  is  NOT  helping  the  students.  This  is  a  profession  with  40%  attrition  after  5  years  when  “S  or  U”  is  the  evaluation  rubric.  Danielson  and  this  new  APPR  evaluation  law    will  cause  the  attrition  rate  to  skyrocket.  High  teacher  turnover  is  NOT  in  the  interest  of  students.

    Districts  will  have  to spend  a  lot  more  money  to  recruit  teachers.   More  monetary  incentives,  and   more  teaching  fellow  type  programs.  These  are  quite  costly,  and  only  serve  to  keep  the  profession  staffed  with  inexperienced,  inferior  teachers.  It  takes  most  teachers  3  to  5  years  to  really  learn  how  to  manage  a  classroom,  and  teach  a  curriculum.  In  NYC,   many  times  it’s   even  longer,  due  to  the  many  obstacles  to  instruction.  Obstacles  such  as  a  lack  of  resources,   and  often  little  administrative  support.  The  new  evaluation  rubric,  will  force  most  people  out  well  before  they  reach  5  years  of  service.

  • Unsure

    Vote NO!, You seem well schooled in what the possible new evaluation may bring in terms of specifics . I must admit that I am not.  Can you please answer this question: What protocol must administration follow after administration administers a first “ineffective” on a teacher?  Isn’t the DOE then required to support this “ineffective” teacher with support?  Thank you.

  • Vote NO!

    Unsure,

    Evidently  that  was  one  of  the  “sticking-points”  between  the  UFT  and  DOE  last  week.  I  think  the  DOE  sees  Danielson,  and  the  new  evaluation rubric  as   a  way  to  “unload”  a  lot  of  veteran  teachers,  and  their  salaries.  The  UFT  had  to  see  the  reports  coming  from  the  PLA  schools,  and  realized  what  is  going  on.  The  APPR  law  does  require  an  “action  plan”  to  assist  teachers  rated  “developing  or  ineffective”  after  the  first  year.  But  two  consecutive  “ineffective”  ratings  and  a  teacher  WILL  BE  FIRED.

    There  is  more  to  the  new  APPR   law  than  just  the  observations.  The  Danielson  framework,  along  with  the  yet  to  be  negotiated  “student  progress or  achievement”  components of  the  law,  are  looking  as  if they  will  cause  an  awful  lot of  unintended  consequences,  and  chaos  for  schools  in  NYC.

  • Daskala

    My principal asked me to attend a Network seminar this past summer about the Danielson framework. The presenters reviewed the various rubrics with the cluster of teachers and then asked us to grade videos of teachers teaching in their classroom. The rubrics are organized from unsatisfactory (level 1) to distinguished (level 4) much like the rubrics we create for our students. Every video we watched was given a grade of 1-3 depending on the observation focus. When we asked to see a video of a teacher who received a “distinguished” across the board, we were told they didn’t have one—it didn’t exist. The reason being that teachers should always be seeking to improve themselves. Therefore a level 4 was impossible because it was a constant work in progress.

    Can you imagine giving your students a rubric for a project and telling them that there was no way anyone was going to get a level 4 because it didn’t exist—it was impossible? After all, if you did receive a 4 then it would mean that you don’t have to work to improve anymore ( being sarcastic).

    Needless to say, this is a motivation killer. What a great way to deflate the spirit and lower morale! And people wonder why most teachers leave the profession after a few years.

    I feel disheartened at times. I entered teaching to follow my passion. I was sold the idea that if I pursued a career that encompassed what I loved, reading and writing, I would be happy.

    Sometimes I want a refund.

  • Marty

    Are we still supposed to be differentiating?  I haven’t heard that word a single time this year.  It disappeared existence and seems to have been replaced at staff meetings by the equally mysterious and panic-inducing word “Danielson”. 

  • Marty

    “from” existence

  • 15 year Veteran

    See this points straight to what I feel Daskala.  This may come off to many as infantile but why can’t our Union leaders convey this same message to the both the public and the media? 

  • Daskala

    Why yes, Marty. We still need to figure out how to “reach” 30 different kids using 30 different approaches—and that is just one of your classes.You need to do that with ALL your other classes too.

    Also, keep in mind,
    if they don’t even try to complete homework, class work, projects,
    etc., and they read and write 3-4 years below level, it is all YOUR
    fault! Don’t even think of expecting the same quality work from ALL your students. That is not fair! I mean, they are all different from each other; they are unique individuals; they have different needs and you need to fulfill them so they feel safe and emotionally supported. Oh, but don’t forget—they ALL take the same exact city and state exams at the end of the year and you are judged on their performance. What would we do without this data?  Make sure your lessons are rigorous, but not tooooo rigorous!

    You need to do that with ALL your other classes too. Also, keep in mind, if they don’t even try to complete homework, class work, projects, etc., and they read and write 3-6 years below level, it is all YOUR fault! Either your tap dance routine and drama skills have grown stale or you must be a lazy and incompetent teacher. You just aren’t doing your job of motivating the kids to learn. I mean, what else could it possibly be? What didn’t YOU do? What aren’t YOU doing? What are YOU doing wrong?

    It must be YOU, Marty! There is no way we should lay a smidgeon of blame on the students or the parents!

    Can you imagine if students were TRULY held accountable for their academic performance or lack of? ( money issue) Can you imagine if parents had to pay any consequences for supporting their child’s “learned helplessness”? (voting issue)

    No, no, no!

    A scapegoat is needed—and society (with the help of the media)— choose the teacher! They are soooooo easy to replace after-all.

  • Daskala

    I don’t know. I am hoping that maybe this venue will help open eyes and teachers will gain support. We need to stop being vilified. By the way, you got me by one year. I am a 14 year veteran….:-)

  • Daskala

    I don’t know. I am hoping that maybe this venue will help open eyes and
    teachers will gain support. We need to stop being vilified. By the way,
    you got me by one year. I am a 14 year veteran….:-)

  • great report!

     i dont really see what the gates study found wrong with teachers. this title misses the point. the study is looking at instruments that can assess teachers. the fact the instruments generally found that many teachers are not asking higher order questions and  is not pushing kids thinking shouldn’t surprise anyone..

  • Anonymous

    No surprises here, but all of these studies tend to point to mixtures of what we already know and horse crap. Reality check: today’s tests do not assess critical thinking, and we are quite far from assessing the deepening and expansion of critical thinking with even modest accuracy. Value-added gains are superficial, like judging people by their salaries and social standing. Not too surprising to see the link in that 20-year study. Teachers who successfully engage students in lots of lower-order-thinking activities may fare way better on observations, test results and student feedback than those that struggle to engage students in higher-order thinking.  

  • Ask

    “And the tools’ reliability was relatively low — lower, in some cases, than the famously volatile judgments of value-added measures. When different observers used the same tool to evaluate the same teacher, they sometimes gave very different scores.”

    There is a vote of confidence. Sound like a winner to me.

  • Donnie

    They should do a study of education in NYC under Bloomberg.  Then they can see how his micromanaging, via Klein, during the mid 00′s forced teachers to be more concerned about what color their bulletin board was and we had to teach the students in groups of four every day.  Cooperative learning they called it then.  The precious few students who cared about learning would work and the others would just talk and every teacher knew that cooperative learning would never be effective when used every day. 

    Then, once stats became the main focus, you have teachers giving pass grade on regents exams when students just a few years ago were receiving grades (on average I would say) about 15-20 points lower for the same quality of work. 

    Now, we’re basically required to pass everyone, doesn’t matter if the students never show up and/or never does any work, because graduation rates are so important to the survival of your school. 

    Of course, the latest is the test score percentages which students earn on standardized exams.  These are supposed to play a critical role in our evaluations and ultimately, our jobs!  Who would ever take a job in a school with challenging students?!  Why the hell has teaching because a career that is as obsessed with stats as major league baseball?  The average baseball player’s professional career is 5.6 years… soon, the average professional teacher’s career will be the same (or less)- if it isn’t already. 

  • VGW

    Thoughts based on my experiences (9 yrs teaching in Title I high schools):

    1) When I first read through the Danielson framework, I (like Vote NO!) thought it seemed wholly unrealistic, given the students I teach and the content I need to cover.

    2)  To prep for some teacher-directed PD, I subsequently looked up videos of exemplary Danielson stuff, specifically a demo of engaging students through discussion.   I was not impressed; I didn’t think the class in the video was as engaged as Danielson thought it was.   (Maybe her standards aren’t that unrealistic!)

    3)  When I teach to the Global Regents, I have very little time to foster higher-order thinking.  The exam does not require it, though it may require a student to distinguish between the Maurya and Gupta empires.

    4)  When I’ve taught electives, I have been able to go into more depth with fewer topics, and have had students engage in debates, pose critical questions, take a position on a historical controversy using serious primary and secondary sources for support . . . but I and our school won’t be judged on that, we’ll be judges on those Regents test scores!

    Never a dull moment . . .

  • VGW

    I mean we’ll be “judged” on the Regents scores, but you knew that.

  • Vote NO!

    VGW

    BINGO!   Assessments  that  are  supposedly  aligned  with  the  Common Core  standards  are  not  going  to  be  available  until  the  2014-15  school  year.  Until  then,  teachers  in  NY  state  will  still  have  to  teach  a curriculum  aligned  with  NY  state  standards,  and  assessments.  Lessons  prepared,  and  taught  according  to ” Danielson”  are   detrimental  to  teachers,  and  students  preparing  for  Regents  exams.

  • Marty

    Regarding your last point:  Have you seen those videos the state produced about Common Core standards? They’re pricelessly condescending!  Apparently these geniuses have figured out that history students should be spending more time on fewer topics.  [What idiots we were not to have figured this out!]  In the one I saw, they suggested spending 3 to 5 days on the Gettysburg address.  No mention by the right hand what the left hand (the folks writing the Regents exams) had to say these suggestions.

  • Marty

    “about” these suggestions.

    I need to start proofreading before I post.

  • Marty

    I just read your reply after I posted mine.  Are you saying the history Regents exams will be different in 3 years?

  • Supercyberblonde

    Thanks for doubling an already impossible work load Miss (gotta be) Danielson. You look like you have no life and are determined to make sure all teachers dont either. The artifact notebook was MURDER and we all hated it. Does she have any ideas how to teach kids who dont do homework, study, pay attention or attend school regularly? If so, I missed that. I didnt take one course in college as to that effect.

  • Supercyberblonde

    You should go to Greenville AL where MANY teachers (CIP TEAM) received distinguished across the board. Riiiiiiiiiiiight.

  • Supercyberblonde

    In Alabama they take PAY OFF after PAY OFF. We got a pay cut this year and 134.00 for supplies. Stinks having to take food off your table and spend your OWN money on other people’s kids instead of your own. We may as well have George (Dubbya) Bush as our union rep. Pitiful.

  • Suercyberblonde

    Yes, like me, Im out of here!!!! Going back this summer for Master’s because I cant seem to make a living at this because I spend more and more and more money trying to make a iving and it just doesnt cut it. When a teacher is penalized for doing the very thing they were told to do? I cant stomach the profession. I have no idea why anyone would knowingly pursue education as a viable income source.

Tips, questions, feedback?

Contact us at .

Word from Our Sponsor

From Our Jobs Board

Featured Employers
Recent Jobs

Chalk It Up

Recent Comments

21 comments so far today
View all comments »

RSS Feeds

Newsroom posts Community posts All GothamSchools content

Archives

June 2013
M T W T F S S
« May  
 12
3456789
10111213141516
17181920212223
24252627282930

Copyright Notice