The laws of New York recognize the labor-management fault line, but they do little to guide a collective-bargaining process toward agreements in the many districts in which teacher-evaluation systems are contested. Each side brings a powerful public value to bear on the disagreement.

For the employers, it’s all about efficiency. It’s in the public interest, they argue, to recruit, retain and reward the best teachers, in order to maximize the collective achievement of students. A teacher-evaluation system that fails to identify those teachers who are effective, and those who are ineffective, can neither weed out consistent low-performers nor target those who might best benefit from intensive help. Rewarding high-performing teachers can, in the short run, help keep them in their classrooms, they claim, and, in the long run, can help expand the pool of talented individuals who enter the occupation.

For teachers, the key concern is fairness. Fairness is primarily a procedural issue: Teachers, and the unions that represent them, seek an evaluation process that is neither arbitrary nor capricious, relying on stable and valid criteria that they believe accurately characterize the quality of their work. In this view, an evaluation process is unfair to the extent that it can be manipulated by a building administrator or school district to yield a particular rating for a teacher’s performance. It is also unfair if random factors beyond a teacher’s control unduly influence the evaluation of his or her performance.

The values of efficiency and fairness collide head-on in New York’s Education Law §3012-c, passed as part of the state’s efforts to bolster its chances in the 2010 Race to the Top competition. The law requires annual professional performance reviews (APPRs) that sort teachers into four categories—“highly effective,” “effective,” “developing” and “ineffective”—based on multiple measures of effectiveness, including student growth on state and locally selected assessments and a teacher’s performance according to a teacher practice rubric.

The fundamental problem is that it’s hard to assess the efficiency or fairness of an evaluation system that doesn’t exist yet. There are too many unknowns to be able to judge, which is one of the arguments for piloting an evaluation system before bringing it to scale. The properties of the state tests that are to be used to assess teachers’ contributions to student learning are a moving target; the tests have been changing in recent years in response to concerns about their difficulty, predictability and coverage of state curricular standards. And in a couple of years, those standards and assessments will change, as New York and many other states phase in the Common Core standards and new assessments designed to measure mastery of them. The models to estimate a teacher’s position relative to other teachers in contributing to students’ test performance are imprecise at the level of the individual teacher, and different models yield different results for a given teacher. There’s been little to no discussion of how to incorporate this uncertainty into the single numerical score a teacher will receive.

The evaluation of teachers’ practices via classroom observations using New York State Education Department (NYSED)-approved rubrics, such as Charlotte Danielson’s Framework for Teaching or Robert Pianta’s Classroom Assessment Scoring System, is another unknown. There’s evidence that with proper training, observers can reliably rate teachers’ classroom practices, but the nature of the training is critical, and there is no evidence to date of New York City’s ability to prepare more than 1,500 principals, or the principals’ “designees,” to carry out multiple observations of many teachers, teaching many different school subjects, each year.

Amazingly, there is even uncertainty about whether the evaluations can or should be based solely on a teacher’s performance in a single year. The statute creating the new evaluation system in New York describes it as an “annual professional performance review.” But is this a professional performance review that occurs annually, or a review of annual professional performance—that is, a teacher’s performance in the most recent year? The guidance provided by the NYSED suggests that it has no idea. “For 2011-12, only one year of teacher or principal student growth percentile scores will factor into each educator’s evaluation,” the guidance states. “When more years of data are available, NYSED will consider whether each evaluation year should include more than one year of educator student growth results. Empirical and policy considerations will determine the decision.”

Well, that certainly clarifies matters. In other words, a “bad” year where a teacher is ranked relatively low compared to other teachers might reverberate, affecting his or her ranking in subsequent years. But a good observational rating in a given year seemingly will have no spillover effect into subsequent years. If, as has been true in Washington, D.C.’s IMPACT teacher-evaluation system, teachers generally score higher on observational ratings than on their value-added or growth-score rankings relative to other teachers, the carryover for value-added performance—but not observations of teachers’ professional practices—appears unfair. And in D.C., this evaluation system has resulted in the termination of hundreds of teachers based on one or two years of performance.

Teacher-evaluation systems have multiple purposes, which might include certifying teachers as competent or selecting some for particular forms of professional development to enhance their professional practice. For most of these purposes, it’s essential that those with a stake in the education system view these evaluation systems as legitimate—and the perceived efficiency and fairness of an evaluation system are central to such judgments. It’s not hard to see why a great many teachers, in New York City and across the state, have serious doubts about the fairness of New York State’s APPR process. And if future teachers do as well, the process could have the unintended consequence of reducing, rather than increasing, the pool of individuals willing to consider teaching as a vocation. This, coupled with the more than 1,300 principals across the state who have raised questions about the efficiency of the process, illuminates the challenges confronting the state as it seeks to implement the APPR system and avoid a scolding from U.S. Secretary of Education Arne Duncan.

William Blackstone, an 18th-century English legal scholar, wrote “better that ten guilty persons escape than that one innocent suffer.” Benjamin Franklin, one of the founders of our country, later upped the ante to 100 to one. The principle captures squarely the trade-off between the value of efficiency and the value of fairness. A legal system that lets the guilty go free is inefficient, as these offenders are free to continue to transgress against the common good. But to Franklin and others, that was still preferable to a legal system that did not provide adequate procedural protections for all, whether innocent or guilty, because such a system would be inconsistent with the principle of fairness so central to the American polity.

It’s important to note that Blackstone and Franklin were concerned with the workings of government; fairness in the private sector was not a central concern, and efficiency was taken for granted as a consequence of market forces. Civil servants, as agents and employees of the state, arguably are subject to a different set of rights and responsibilities than those working in the private sector, and teachers are one of the largest groups of such public servants. What’s an acceptable tradeoff between efficiency and fairness in the mix of teachers’ rights and responsibilities? It’s a lot easier to speculate about percentages in the abstract than to confront the possibility that you, or someone close to you, might be out of a job because of an untested teacher-evaluation system that cuts corners on fairness.

This post also appears on Eye on Education, Aaron Pallas’s Hechinger Report blog.

But it is entirely a matter of opinion as to whether the particular statistical analyses involved in the production of the Teacher Data Reports warrant the inference that teachers are more or less effective. All statistical models involve assumptions that lie outside of the data themselves. Whether these assumptions are appropriate is a matter of opinion. Among the key assumptions that are necessary to make inferences about teacher effectiveness from student performance on the state tests are the following:

• The tests are valid measures of students’ mastery of English Language Arts and mathematics.
• A student’s performance on the test, which is taken on a particular date, reflects how that student would perform on the test on other dates.
• The student, classroom and school-level variables taken into account in the value-added model underlying the Teacher Data Reports are appropriate for inferring that a particular teacher caused the test-score gains experienced by that teacher’s students.
• Test-score gains observed on tests administered in the middle of one year and the middle of the following year can be properly apportioned to the prior-year teacher and the current-year teacher.

The fact that reasonable people might disagree about these assumptions makes clear that they are a matter of opinion. For example, research by testing expert Dan Koretz, Jennifer Jennings and others shows that the tests at issue were subject to score inflation, because they covered an increasingly predictable and small subset of the curricular standards set by New York state, and failed to predict whether students were well-prepared for college and life after high school. Researchers such as economist Jesse Rothstein have questioned whether value-added models such as the ones used in the production of the Teacher Data Reports are able to simulate a “level playing field” in which teachers can be assumed to have equivalent classes of students.

Even the Department of Education’s own contractors have been of different minds about how to apportion gains when students are exposed to two different teachers between last year’s test and this year’s test. Initially, the gains were apportioned based on the number of months of exposure to last year’s teacher and this year’s teacher. But the most recent technical report for the production of the Teacher Data Reports attributes all of the gains between last year’s test and this year’s test to the current-year teacher.

Value-added measures such as the Teacher Data Reports are constructed through a social process involving expert judgments, and there may be a great deal of consensus around many of those judgments. But that doesn’t make the Teacher Data Reports “facts” that are somehow removed from the realm of opinion and assumption. The data don’t create the categories used to label teachers as above or below average; the labels are a matter of opinion.

There are many definitions of the term “fact,” and perhaps the definitions relied on in legal reasoning differ substantially from those used in social and educational research. But State Supreme Court Justice Cynthia Kern’s argument that the Teacher Data Reports are “facts” makes little sense. In my opinion.

This post also appears on Eye on Education, Aaron Pallas’s Hechinger Report blog.

The state of New York has a different conception of the expertise needed to be a school district superintendent. State law specifies that to obtain a professional school district leader certification, school district leaders (i.e., superintendents) must have completed a School District Leader program authorized by the state; accumulated a minimum of 60 semester hours in graduate courses approved by the state commissioner of education; and have at least three years of teaching experience. The certification also includes a full-time, 15-week clinical component of school-building leadership experience or its equivalent, and requires passing two written School District Leader assessments.

The content of the School District Leader assessments provides some purchase on the kinds of expertise that the state views as necessary to successful practice. The standards expressed in these assessments include applying knowledge of skills for engaging building leaders, board members, community members, parents/guardians, students and school staff in an ongoing dialogue regarding core values, goals, policies, practices and achievements; demonstrating knowledge of the New York State Code of Ethics for Educators and the role of values and ethics in district leadership; demonstrating knowledge of factors to consider in comprehensive, long-range planning, including the importance of involving all key stakeholders in planning processes; analyzing concepts, principles and best-practice applications of developmental and learning theories, curriculum development, instructional delivery, and classroom organization and practices with regard to the diverse needs of all students (e.g., special-education students, English-language learners, gifted and talented students); analyzing strategies for developing staff capability through the supervision and evaluation of teachers and building leaders, effective staff assignments, and systems of mentoring, support, and development; and demonstrating knowledge of processes of collective bargaining and contract management that support and extend the educational vision, to name just a few.

If the various requirements of the School District Leader certification are indicators of the expertise that New York state requires of school superintendents, and Cathie Black has not met those requirements, how are we to judge if she has the requisite expertise? Mayor Bloomberg sought a waiver to the requirements, which provides an alternative route to the credential. State regulations allow the state commissioner of education to issue a professional certification to exceptionally qualified individuals “whose exceptional training and experience are the substantial equivalent of such requirements and qualify such persons for duties of a superintendent of schools.” State Commissioner David Steiner convened a screening committee of eight education professionals to advise him on whether to grant the waiver. To the surprise of many, the screening committee did not recommend granting a waiver to Black. Four of the eight members voted no; two voted yes; and two voted “not now.”

In inviting the screening committee to consider the option “not now,” Commissioner Steiner paved the way for the education version of “jury nullification.” Jury nullification is the term used to describe when a jury reaches a verdict that is contrary to the facts in evidence, typically because of a belief that the law is immoral or improperly applied to a defendant. Cathie Black is not, of course, on trial, although many may feel that she is, by virtue of the high-handed way in which she was selected from Mayor Bloomberg’s mental Rolodex to be chancellor without any signs of a search. Steiner has now brokered a compromise in which an insider to the New York City public school system, Shael Suransky-Polakow, will be appointed Senior Deputy Chancellor and Chief Academic Officer. Press reports indicate that with this proviso, Steiner is now prepared to grant the waiver to Black.

I describe this as a form of jury nullification because Commissioner Steiner’s willingness to grant Cathie Black the professional certification needed to be appointed school superintendent is based on criteria other than those specified by the state’s education code. Steiner has already determined that, on the merits of her application, Black was not qualified for the position. In suggesting that Black would be acceptable if accompanied by a chief academic officer, he is saying, as have Mayor Bloomberg, Chester E. Finn, Jr., and others, that the rules should be ignored, mainly because New York City is a special case.

In my view, Commissioner Steiner’s decision should be independent of the context of New York City. The state’s requirements for school district leaders do not state that there’s one set of rules for New York City, because it’s so big and complicated, and another set of rules for the 700 other districts in the state. If the state wanted to create a different set of qualifications for the New York City schools chancellor, it could have done so.

It has not.

Therefore, it’s hard to see how Steiner should take account of the exceptionality of New York City’s educational system. He has found a political solution that further undermines the view held by most professional educators — and, I dare say, the overwhelming majority of school superintendents in New York state — that there is a body of expertise they apply to their daily work that cannot be picked up overnight.

In the long run, trying to assess the expertise necessary to be a school district leader without taking local context into account may be shortsighted. The compromise brokered by Steiner proposes that Black’s lack of experience in public schooling can be offset by the fact that she will be joined by a chief academic officer, and there will be others in the new chancellor’s “cabinet” who have specialized knowledge of educational issues in general and the New York City school system in particular. And if we take seriously the shift in leadership studies from “great man” theories that emphasize the distinctive charisma and personality traits of individuals in positions of formal authority to a view that focuses on the interactions among leaders, followers, resources and context, we’d want to pay close attention to these features in assessing the qualifications of Cathie Black to lead the New York City schools.

But doing so raises some tough questions. Once we acknowledge the notion that expertise is distributed among individuals in a setting, why would we rely on credentials that emphasize individual accomplishment? Why would we seek to isolate the contributions of individual teachers to students’ learning when teaching is an activity distributed among the educators in a particular school? Why would we even assess students’ learning via methods that preclude students from using tools in concert with other students? If, as Mayor Bloomberg has asserted, Cathie Black’s appointment is justified because she’ll be learning in concert with others, why don’t schools assess students’ preparedness to do just this? In the spirit of the season, what’s good for the goose is good for the gander.

This post also appears at Eye on Education, Aaron Pallas’s Hechinger Report column.

That’s not necessarily a bad thing.

The term “propaganda” has gotten a bad rap, ever since its association with 20th-century totalitarian governments promoting troubling political objectives. But there is a long and honorable tradition of propaganda in the genre of documentary films. In its original formulation, “propaganda” is simply a deliberate effort to change what people know, understand and value, for a particular purpose. Propaganda can rely on many different media and symbols to carry its message. Documentary films have often sought to activate a sense of urgency about a social problem or condition that needs our attention. The medium of film is especially powerful because propaganda often appeals to emotion as much as reason, and film is very effective at evoking an emotional response. Much better than, say, a speech by Al Gore, Arne Duncan or Bill Gates.

I had the opportunity to view Waiting for “Superman,” the new documentary by Academy Award-winning filmmaker Davis Guggenheim, at a pre-release screening at Teachers College last week. Based on the early buzz from proponents and detractors alike, I expected to see a film that lived up to its billing as “stirring” or “moving.”

Although Guggenheim, who also directed “An Inconvenient Truth,” is a skilled filmmaker, I didn’t enjoy the film as an aesthetic experience. But that’s because I found myself second-guessing the director’s choices. The audience at Teachers College, which was quite diverse, seemed to like the movie, laughing at predictable times but growing silent as the film built to the climactic scenes in which five children’s futures were portrayed as riding on the outcome of charter-school lotteries.

Over the course of 100 minutes, “Waiting for ‘Superman’” follows the fortunes of five families who believe that their children are trapped in an educational system that fails to meet their needs. The film features two prominent and colorful actors on the contemporary education scene whose fortunes have diverged dramatically in the past week: Geoffrey Canada, whose Harlem Children’s Zone just received a $20 million gift from Goldman Sachs Gives to expand its operations, and Washington, D.C. Schools Chancellor Michelle Rhee, who, on the heels of the defeat of D.C. Mayor Adrian Fenty in the Democratic primary last week, is widely expected to step down after a tumultuous three years in office.

Canada and Rhee are vivid figures, but neither has shunned the media spotlight nor is new to national media attention. Rhee, for example, has appeared on the cover of Time magazine wielding a broom, and allowed herself to be filmed firing a school principal in what would have otherwise been a private meeting — if the cameras weren’t present. (The clip is reprised in the movie.) And HCZ, with Canada as its founder, spokesman and chief fundraiser, has received plenty of national attention — and deservedly so, because it’s an innovative model for human development in blighted urban areas, even if we don’t yet know its eventual success. It is an open question whether ubiquitous figures such as Canada and Rhee will be new to the intended audience of the film.

To return to the theme of a propaganda campaign: There is an inherent contradiction in propaganda as simultaneously broadening and narrowing. On the one hand, it seeks to open the minds of audience members to new ideas and information. But on the other, it seeks to lead an audience to a particular conclusion. ”Waiting for ‘Superman’” may well broaden public understanding of the condition of public schooling in America. But it is far less successful in communicating a clear vision of why so few American students are achieving at high levels, and in creating momentum for collective action by the film’s audience.

A lot of the film reduces to a morality play that pits the good-guy reformers and charter-school leaders (e.g., Canada, Rhee) against the bad-guy unions, without taking into account the broader political and social context in the U.S that should frame the debate. The film gives no sense of the complexity of our goals for public education, reducing outcomes to test scores (and maybe going to college). Even the film’s animation-far less incisive than some past “Simpsons” episodes covering education, to be honest-contributes to the oversimplification. Virtually the only portrayal of classroom teaching and learning in ”Waiting for ‘Superman’” consists of an animated teacher opening up a child’s head and dumping “knowledge” — an alphabet soup of letters and numbers — into it. What does that say about the nature of teaching, learning and curriculum? Perhaps that test scores are a good measure of what schooling is all about?

And this brings me to another central criticism of the film. All documentary filmmakers must make choices, since what appears on the screen can only be a representation of the world that they perceive. The film portrays the voices of parents and children who feel let down by the educational system as well as those campaigning to upend the current educational system. But there is a conspicuous silence. The film and book (yes, there’s a tie-in book, published before the movie’s release) ignore the voices of teachers talking about their day-to-day work. Had Guggenheim chosen to include teachers talking about classroom teaching, he might have further illuminated some of the contextual factors that make urban schools a problem-concentrated urban poverty; communities segregated by race, ethnicity and social class; the lack of high-quality programs for infants and young children; and families which lack the resources to support their childrens’ schooling, to name a few. But the narrative would have become less tidy.

“Waiting for ‘Superman’” and its tie-in are shot through with contradictions. Teachers are, in the words of Newsweek editor Jonathan Alter, “great, a national treasure” — except when they’re being lampooned as “lemons,” “turkeys” or “trash.” (And, apparently, they hire themselves, and give themselves tenure. Or maybe the teachers’ unions do that; the film isn’t clear.) Winning a charter-school lottery is the only route to success — but only one in five charter schools is achieving “amazing results.” Fixing failing schools is offered as the key to fixing failing communities-but, as my colleague Jeff Henig pointed out at the screening, two of the initiatives featured in the film (the Harlem Children’s Zone and the SEED school in Washington, D.C.) take very seriously the primacy of community, either by seeking to transform a community through the provision of social services to families, in the case of HCZ, or by removing children from troubled communities, in the case of the SEED school.

Money isn’t a problem, Guggenheim avers, because the United States has higher per-capita spending on education than ever — but the ”Waiting for ‘Superman’” website invites visitors to make a contribution to DonorsChoose to support the work of teachers who cannot receive the resources they need from their school districts, and the tie-in book includes a $15 gift card from DonorsChoose to give to “a classroom in need.” The movie and book tell us that the main problem for Francisco, a first-grader from the Bronx, is that his school is overcrowded and his teacher is “overworked with too many students” — two conditions that go unremarked-upon for the remainder of the film.

Filmmaker Davis Guggenheim was quoted by New York Magazine writer John Heilemann as saying, “Here’s what I’m scared of: that the movie will be misperceived as a pro-charter, anti-union piece.” He must be disingenuous or stupid, and I doubt he’s stupid. How much screen-time is devoted to positive portrayals of charter schools? How much to positive portrayals of successful traditional public schools? (Spoiler alert: a lot, and none, respectively.) If only 20 percent of charter schools are producing “amazing results,” what about the charter schools that are no better, or worse, than the traditional public schools that are the site for educating the vast majority of students in the United States? Similarly, is the portrayal of teachers’ unions one-sided or balanced?

Having seen the film and bought the book, I’m skeptical that the ”Waiting for ‘Superman’” propaganda campaign is going to have much impact on education policy, despite all of the buzz for and against the film. Although a few documentaries or biopics have succeeded in getting viewers to think differently about their subjects, I don’t think that films in general have demonstrated much potential for moving people to action; and ”Waiting for ‘Superman’” doesn’t really lead the viewer to take a particular action. ”We know what works,” “Text this number to help,” and “Get involved” are exhortations that confront the viewer at the film’s conclusion — but they’re hopelessly vague.

And even if one accepts the premise that the message of the film is to support expanding charter schools, or make teachers more accountable for how their students perform, the likelihood of the film actually provoking movement on these objectives is muted by the fact that the nation just went through a Race to the Top competition in which precisely these goals were rewarded. As many states have just passed laws supporting these things, it’s hard to imagine much pressure for even more.

The ”Waiting for ‘Superman’” website has city-specific websites with local “campaign managers” whose job, apparently, is to channel audience sentiment into action. What’s on the New York City page? An invitation to write the candidates for governor to demand “world-class standards” for all students in New York. “The Common Core Standards will help better prepare students for college and the workplace,” the site asserts.  (Ahem: No one knows if that’s true, of course; the Common Core Standards have yet to be implemented anywhere, and standards are but one piece of a complex system that hinges on curriculum and assessments that don’t exist yet.) But New York’s State Board of Regents already adopted the Common Core Standards for math and English language arts in June! This is a call to action?

The other key initiative is to “support efforts to keep the most effective teachers.” Here, the link is to a May column entitled “We’re Firing the Wrong Teachers,” published on The Daily Beast, by New York City schools chancellor Joel Klein. In it, Klein complains that the $500 million state budget cut facing the city would oblige him to lay off teachers according to the rules he negotiated in a collective bargaining agreement in 2005. (But remember, money is not the problem.)

Next up for me: reviewing “Wall Street: Money Never Sleeps.” But, given the corporate support for the key figures in ”Waiting for ‘Superman,’” some might say that I already have …

This post also appeared on Eye on Education, Aaron Pallas’s column at The Hechinger Report.

No, it’s that I think miracles are exceedingly rare phenomena. And the current narrative about miracles in school reform relies heavily on a “great man” theory, replete with outsized personalities. Witness the contemporary stage, on the cusp of the release of Waiting for “Superman”: Geoffrey Canada, Michelle Rhee, even — God help us — Bill Gates and Joel Klein being anointed as miracle-workers who, by dint of their commitments, hard work and personalities, are overcoming entrenched bureaucracies and transforming the life-chances of poor and minority children across America’s urban landscape.

It was against this backdrop that I read Caitlin Flanagan’s stirring op-ed that graced the gatefold of Sunday’s New York Daily News. Flanagan, a former prep-school teacher who now writes for The Atlantic and other publications, singles out Mike Piscal, who founded a charter management organization called the Inner City Education Foundation (ICEF) that now operates 15 elementary, middle and high schools in south Los Angeles. Flanagan and Piscal were colleagues, once upon a time, in the English department of the elite Harvard-Westlake School.

Flanagan’s argument goes something like this: the ICEF schools are extraordinarily high-performing; in fact, the elementary schools have eliminated the achievement gap. But the educational bureaucracy is trying to close them down. These fine schools were unable to benefit from the Race to the Top funds they should have received because California’s RttT application was scuttled by a lack of teacher-union support. These teachers unions, therefore, deserve our scorn: they are single-handedly preventing inner-city children from succeeding.

The gaps in logic are breathtaking. It’s not at all obvious that ICEF charter schools would have gotten a cent if California’s RttT application had been funded. As is true of most RttT applications, California’s emphasized developing new standards and assessments, providing high-quality professional development to principals and teachers, expanding the state’s longitudinal data system and improving its lowest-performing public schools.

Basic operational support for existing schools was never the purpose of the competition. Moreover, California lost more points in the RttT judging for its failure to fully implement a longitudinal data system than for not securing the support of a broad group of stakeholders for its plan. (And teachers unions were not the only stakeholders who were lukewarm about the state’s application.)

But I was most intrigued by Flanagan’s claims about how the ICEF schools have closed the achievement gap. Time and time again, such claims have been shown to be exaggerated. We’d all like to believe the story of how great people, working hard, can overcome the powerful forces that structure inequality in American society. Can it be true?

Here’s what Flanagan wrote: “ICEF has done what we are always told is impossible. All five of its elementary schools have eliminated the achievement gap in reading for its African American students. Eliminated it. That fact alone should cause the Department of Education to send a team of researchers to ICEF this afternoon to keep them there until they learn what Mike’s doing.”

How much of this is true? Well, there are five ICEF elementary schools. Beyond that …

I compared the average performance of students in these five schools on the 2010 statewide California Standards Tests (CST) with the average performance of white students in the Los Angeles Unified School District, expressing the gap as a fraction of a standard deviation. (I estimated the 2010 standard deviations as the average of the 2008 and 2009 standard deviations.) Figure 1 shows the data for English Language Arts for grades 2 through 5. Across the Los Angeles Unified School District, black students score about .85 to .90 standard deviations below white students, on average. A gap of this magnitude indicates that roughly 80 percent of all white students score above the typical black student.

If the achievement gap had been eliminated in the five elementary schools, then the columns expressing the gap should have no height – they’d be at zero. Clearly, that’s not the case. On average, students in ICEF elementary schools have made some progress in closing the achievement gap in reading test performance, but that’s driven primarily by the performance of students at a single school, View Park Prep.

Second-graders at View Park Prep are outscoring the typical white student in Los Angeles, and that’s a remarkable accomplishment. But if one wanted to cherry-pick results to make a point, it would be just as easy to point out that the fourth-grade students at Vista are performing 1.1 standard deviations below the typical white student in L.A. (More than 75 percent of the students at Vista are Hispanic or Latino; there aren’t even enough black students for the state to report their performance on the CST, which makes me wonder why Flanagan felt that she could say that “all five” of the ICEF elementary schools had eliminated the achievement gap in reading for African-American students.)

The story is much the same in mathematics. Figure 2 reports the math achievement gaps for the five ICEF elementary schools. As is true in reading, the district-wide black-white achievement gap in mathematics is substantial – on the order of .9 standard deviations. The ICEF elementary schools have made some progress in closing the gap, and that’s commendable.

But no one looking at this figure would conclude that the ICEF elementary schools have come close to eliminating the achievement gap that separates the test scores of African-American and Latino children from white children in Los Angeles. Test scores are, to be sure, a very narrow representation of what children are learning in school, and I would never want to base a judgment about the quality of ICEF schools, or any other schools for that matter, solely on test scores. But Flanagan flew the achievement-gap flag, and her claims don’t hold up under scrutiny.

I like a good story as much as the next guy. But when it comes to swaying opinion on important matters of public policy, we should demand more. Perhaps Caitlin Flanagan has access to other data that provide more support for her claim that all five of the ICEF elementary schools have eliminated the black-white achievement gap in reading. But until she goes beyond a bald, unsupported assertion, she’s got a credibility gap.

This post also appeared on Eye on Education, Pallas’s column at The Hechinger Report.

To be sure, the editorial page of a newspaper should express a point of view, and a typical reader will likely agree with some viewpoints, and disagree with others.  But it’s a very dangerous thing when the editorials of a newspaper are not informed by the daily reporting of its journalists.  Ignoring the news, reported with a minimum of spin by “beat” reporters, leads to simple-minded and ignorant editorializing on complex matters of public policy.  It’s also insulting to the profession of journalism, and to the many reporters whose goal is simply to understand the news and get the story right.  (I talk to some of the reporters to whom I’m referring.)

A case in point is yesterday’s Daily News editorial, “Truth in testing.”  The editorial is an effort to shore up claims about the success of school reform in New York City under Mayor Mike Bloomberg and Chancellor Joel Klein.  Last week’s revelations that the state testing system was dramatically overstating student growth and the closing of the achievement gap rocked the New York City Department of Education on its heels.  The Daily News editorial board, which has long supported these reforms, came out firing, citing four “facts”:  (1) The State Education Department defrauded parents and students;  (2) Regents Chancellor Merryl Tisch and Education Commissioner David Steiner owned up to the deception;  (3) The drop triggered bogus charges that the schools have made no progress;  and (4) Only radical action will give New York’s kids a shot at the quality education they need.

Interesting points, although they can scarcely be described as “facts.”  The most provocative point is the third one.  To support the claim of great progress, the editorial states, “From 2006 to 2009, scale scores among city kids rose 23 points in math and 13 points in English.  Both held firm in 2010 … And city fourth- and eighth-graders improved on the National Assessment of Educational Progress by far more than kids in the rest of the state and across the country.”  To be sure, scale scores on the state assessment did improve over the period 2006 to 2010, although it’s not possible to calculate an average growth like this, because the scales for the state’s assessment system are not vertically equated.  (And “holding firm” is an artful way of saying that although the Children First agenda was in full sway in 2009-2010, test scores in New York City did not go up.)  If the editorial board had dug a bit deeper into its own pages (see here, here and here), it would have learned that testing experts are unable to determine whether scores rose because student performance improved or because the tests got progressively easier and more predictable over time.

But the question I have is, where did these numbers come from?  I’ve reviewed the reporting of the two primary education beat reporters for the Daily News, Meredith Kolodner and Rachel Monahan, and have found no evidence of these figures in their published articles.  If the editorial board of the Daily News is going to write about education in New York City, shouldn’t they draw the evidence from the news side of their operation?  Surely that would be better than simply parroting a set of talking points provided by the New York City Department of Education. (The figures do appear in a DOE PowerPoint deck released last week.)

Reporters such as Kolodner and Monahan (and their predecessor at the Daily News, Erin Einhorn) strive to learn from different sources, not just their cronies.  Talking to people with disparate points of view yields a more balanced picture of the world.  For example, if the Daily News editorial board had reviewed the paper’s reporting on the National Assessment of Educational Progress, they would have known that the claims that New York City students improved by far more than kids in the rest of the state and across the country was an out-and-out lie.

Below, I summarize the evidence comparing gains from 2003 to 2009 on the fourth-grade and eighth-grade NAEP reading and math tests in New York City with gains observed in nine other large urban school districts.  Overall, scores did rise in New York City over this period, and this is an accomplishment that should not be ignored.  But did scores rise faster than in other large districts?  You be the judge.

In the chart below, a green arrow indicates that the gains from 2003 to 2009 in a particular subject at a particular grade level were significantly greater in New York City than in a comparison district.  A red arrow indicates that the gains in New York City were significantly smaller than in a comparison district. And a grey circle indicates that the gains in New York City were not significantly different from the gains in a comparison district.

Out of 36 comparisons—nine urban districts, two subjects (reading and math), and two grade levels (fourth and eighth)—New York City gained significantly more than a comparison district in only four instances.  Conversely, there are ten instances in which another district gained significantly more than New York City over the period 2003 to 2009.  The remaining 22 comparisons show no difference in the rate of growth of NAEP scores in New York City and the growth rate for other urban school districts. 

It is just not possible to read these results and to conclude that New York City’s fourth- and eighth-grade students improved on the NAEP by far more than kids across the country.

If the Daily News editorial board is going to ignore the careful and thorough reporting of its education beat reporters in favor of talking points provided by the New York City Department of Education, I have a suggestion:  Place a black border around the editorial, and in small type at the top, print “Paid Political Advertisement.”  That way, we’ll all know the score.

 Last week, the Board of Regents voted to adjust the cut scores that determine proficiency on the state’s readingand math assessments in grades through eight.  They didn’t say by how much, but we have a clue from Merryl Tisch’s assertion that the “inflation rate” on the state tests has been about 20% in recent years. Twenty percent of what is not clear.  But I’m going to assume that the cut score for Level 3, which represents proficiency in a subject at a particular grade level, is going to rise substantially at all grades for both reading and math.  What are the likely consequences?

 We’ll see tomorrow, but here’s my prediction, focusing on eighth-grade math.  First, I’m assuming that the distribution of scale scores for 2010 will be the same as it was for 2009.  If the tests were more difficult in 2010, the average scale score might go down a bit;  if students were actually learning more in 2010 than in 2009, the average scale score might go up a bit.  For my little prediction exercise, I’m assuming that these two things cancel each other out.

 We don’t know where the NYSED will set the cut score for Level 3, but let’s assume that it’s 675.  This is a challenging standard, but one that predicts a probability of 80% of scoring 80 or higher on the Math A regents exam and, according to the NYC chart, predicts a probability of obtaining a Regents diploma of 81%.

 Here’s how things looked in 2009:  Across the state, 80% of students were judged proficient in eighth-grade math.  The percentages varied by racial/ethnic group, with the highest percentage (92%) recorded for Asian students, and the lowest (63%) for Black students.  89% of white students were met the standard for proficiency in eighth-grade math, as did 69% of Latino students.

 Here’s what I predict:  Across the state, 50% of students will be classified as proficient in eighth-grade math.  The proficiency rate for Asian students will fall to 73%;  for white students to 60%; for Latino students to 35%;  and for Black students to 29%. 

But perhaps a proficiency cutoff of 675 is too hard for the Regents and the State Education Department to stomach, as they will have to live with the political fallout of plummeting proficiency rates.  Perhaps they will increase the cutoff from 650 to 670.  This would still be a substantial increase, and proficiency in eighth grade would still mean a lot more than it does today.  What would the prediction look like then?

 If the cut score for Level 3 for eighth-grade math were to rise to 670, I predict the following:  Across the state, 56% of students will be classified as proficient in eighth-grade math.  The proficiency rate for Asian students will be 77%;  for white students, 66%;  for Latino students, 41%;  and for Black students, 35%.

 Calculating the achievement gap using group differences in average scale scores, which is what Jennifer Jennings and I have argued for in the past, would not be affected by the shift in the proficiency cutoff.  But for those who calculate the achievement gap in “points” (I’m talking to you, New York City Department of Education), the increase in the cutoff is destined to increase the achievement gap—even if the score distributions for the groups stay the same. 

 For New York State as a whole, the 2009 achievement gap in eighth-grade math, calculated as differences in group proficiency rates, was 26 percentage points for the white-Black difference, and 20 percentage points for the white-Latino difference.  (And, for completeness, the gap between Asian and Black students was 29 percentage points, and between Asian and Latino students 23 percentage points.)  If the state moves the proficiency cutoff to 675, the white-Black difference will rise from 26 percentage points to 31 percentage points, and the white-Latino difference from 20 percentage points to 25 percentage points.  The same increase would be observed if the state increases the proficiency threshold to 670.

  Tomorrow, we’ll see whether I’m a good prognosticator … or whether I should quit my day job and become a meteorologist instead.

Setting aside some of the most remarkable inconsistencies—Charles Murray, 2010 edition, doesn’t think that test scores are meaningful measures of academic performance?  Has he met Charles Murray, 1994 edition, who was quite comfortable in The Bell Curve reducing the whole of human intelligence to a single score on the Armed Forces Qualification Test?—Murray fundamentally misunderstands the historic logic of the charter schooling movement—an exchange of autonomy for accountability.  We can argue over the scope of that autonomy and accountability, but even those who have disagreed on this site about whether charter schools are properly labeled as public or private schools generally agree that it’s appropriate to hold them accountable for their students’ performance on assessments measuring standards that are the de facto public curriculum of the state in which they are located.  Certainly, the charter movement gains energy from studies showing that students in charter schools may outperform their counterparts in traditional public schools on state assessments.  Charter schools may strive to expose students to a curriculum that’s more ambitious, but the standards of the state cannot be ignored.

It’s puzzling, then, that Murray uses the recently-released results of the Milwaukee Program, which is a program providing vouchers to low-income students to attend private secular and religious schools, to motivate an argument for school choice, and especially the expansion of charter schools.  The particular example he cites is a proposed K-8 charter school in Frederick County, MD, where he resides. 

Frederick County, MD is an odd place to make a stand for charter schools.  As of the 2000 Census, 80% of the residents were non-Hispanic whites, and only 4% were foreign-born.  Only 5% of the residents were living in poverty, well below the Maryland state average, and the median household income was more than 10% above the state average.  The 2009 Maryland report card records a 94% high school graduation rate for Frederick County students.

But the real oddity is Murray’s claim that the state should underwrite the costs of a customized curriculum not available in the “ordinary” public schools.  If a rigorous curriculum is what can best prepare our children for the future, why not offer it to all students?  Allowing savvy parents such as Charles Murray to opt into specialized schools subsidized by the state, while consigning most students to a less-challenging curriculum, only perpetuates a system of unequal educational opportunity.  Even Murray might acknowledge that it’s bad public policy for the state to exacerbate inequality of educational opportunity.

It’s true that 2.4% is 60% more than 1.5%.  But both percentages round to the same whole number, 2.  So it’s hard to say that the likelihood of admission is dramatically different for students in charter and traditional public schools.  And, although Winters pays lip service to the notion that these data are solely descriptive, there’s no mistaking his desire to use these data to argue that the quality of charter schools is in fact responsible for this small increase.  “Charter schools could,” he writes, “increase minority access to the city’s esteemed high schools by offering a higher quality elementary and middle school education than is available in the traditional public schools system.”  Yep, that’s true, they could.  They could also be successful in recruiting some talented minority students with families that are highly motivated to help them succeed in school.  In the latter case, the primary dynamic is selection into charter schools, not their academic consequences.

By focusing on the relative rates of minority access to New York City’s specialized exam high schools for students in charter and traditional public schools, however, Winters has buried the lead.  The real story here is the fact that, in a system that is overwhelmingly made up of Black and Latino students, very few are getting into the most prestigious high schools.  71% of the eighth-graders in New York City’s traditional public schools are Black or Latino, but only 16% of the students offered seats in the specialized exam schools are Black or Latino.  Another way of representing the same information is to look at the probability of admission to the exam schools for members of different racial/ethnic groups.  As Winters noted, 1.5% of the Black and Latino eighth-graders in traditional public schools were offered admission to the specialized exam schools.  But 19% of the white and Asian eighth-graders attending such schools scored high enough on the entrance exam to be offered admission.

Either way you cut it, the data demonstrate the persistence of a huge achievement gap between Black and Latino students, on the one hand, and white and Asian students, on the other.  Data such as these give the lie to the persistent claims of Mayor Mike Bloomberg and Chancellor Joel Klein that New York City has closed the achievement gap.  It’s just not true.  The evidence strongly suggests that the reforms introduced by the Mayor and the Chancellor over the past decade have done little to improve the academic prospects of thousands and thousands of Black and Latino children and youth—the vast majority of which attend the traditional public schools for which they are responsible.

The disproportionate emphasis on Lemov’s approach in Elizabeth’s article surprised me.  To be sure, he’s a fine human-interest story, and the schools he works with have shown remarkable performance on state achievement tests.  But Elizabeth briefly acknowledged the lack of a research basis for Lemov’s approach, writing:  “And while Lemov has faith in his taxonomy because he chose his champions based on their students’ test scores, this is far from scientific proof.  The best evidence Lemov has now is anecdotal…”  Why would she and the Times choose to feature an approach with so little evidence to back it up?

Lemov’s book, “Teach Like a Champion:  49 Techniques that Put Students on the Path to College,” was published two weeks ago, and currently ranks #30 on Amazon’s bestseller list.  I wanted to see what he had to say about the research evidence underpinning the techniques.  A thin research base does not, of course, mean that the techniques are not valuable—I expect to learn quite a bit from studying them, and seeing if there are opportunities to adapt them for teaching my graduate students (who will tell you that classroom management is not my strong suit.)  And, of course, who wouldn’t want to be a champion teacher?  Because it is, after all, a competition, right? 

But my motivation runs a bit deeper.  Yesterday, the New York State Board of Regents unanimously endorsed a proposal to pilot new programs for preparing teachers that would allow organizations that are not institutions of higher education to offer programs leading to the master’s degrees that New York requires of certified teachers.  Writing in Monday’s New York Times, Lisa Foderaro saw the parallel between this proposal and State Commissioner of Education David Steiner’s approach when he was Dean of the School of Education at Hunter College, where he “sought to elevate the practical aspects of teaching:  when to make eye contact, when to call on a student by name, when to wait for a fuller answer.”  At Hunter, Steiner pioneered an innovative teacher preparation program called Teacher U, partnering with leaders from Uncommon Schools, KIPP, and Achievement First, three networks of charter schools operating in New York State and elsewhere.  Doug Lemov is an instructor for Teacher U.

I’m on record expressing fear that that the Regents’ proposal will decouple the preparation of practitioners from the colleges and universities where research about practice is produced.  Lemov’s book does nothing to assuage that fear.  It provides no research evidence that these 49 techniques produce high levels of student achievement, either singly or in particular combinations.  Lemov’s book does not describe either the incidence or prevalence of the use of these techniques in any population of teachers—there’s no way to tell if more frequent or intense use of a particular technique by a teacher is associated with higher student achievement. 

This doesn’t stop Lemov from writing, “The techniques described here may not be glamorous, but they work.  As a result, they yield an outcome that more than compensates for their occasionally humble appearance” (p. 6).  The book provides no evidence that these techniques work.  What is clear is that there is a set of schools that Lemov works with or is familiar with that demonstrate exemplary performance on state standardized tests, and that, through observation, he has found evidence of these techniques in some of the teachers who teach in these schools.  But that doesn’t mean that it’s the techniques, singly or in combination, that account for the success of the schools.  There may be any number of other explanations for why these networks of schools are demonstrating high levels of success on state assessments. 

If these practical teaching techniques matter, it should be possible to demonstrate their effectiveness using methods that meet conventional scientific standards.  Before we give out master’s degrees based on mastering them.

No, of course they don’t.  But the people who produce those numbers do.  We would all be wise to recognize that the veneer of scientific objectivity coating most standardized tests is paper-thin.  Politics infuses the form that standardized tests take;  their length;  how they are scored, and by whom;  the content standards that appear on the tests;  and the judgments about which levels of performance are to be labeled proficient.

Here’s what I saw at the data revolution:

One of the seventh-grade algebra standards in New York State’s Mathematics Core Curriculum is the following:

7.A.3  Identify a polynomial as an algebraic expression containing one or more terms

In 2008, the following item appeared on the eighth-grade New York State mathematics test.

24.  Which of these phrases best describes a polynomial?

a.       a decimal that is non-terminating or non-repeating
b.      an algebraic expression containing one or more terms
c.       a close-planed figure formed by three or more line segments
d.      a number greater than one that has exactly two different factors

67% of New York eighth-graders got the item correct.

How does an item that is such a poor representation of the mathematical skill it is intended to measure wind up on the state exam?  Did it write itself?  Or did people, with political interests, write and approve it?

And can people, with political interests, succeed in fixing the New York State assessment system?

(hat tip to my colleague Jennifer Jennings for pointing out the test item)

Chancellor Klein Applauds New York City Public School Students For Six Years of Sustained and Significant Gains in Math on National Exam (Let’s get that “six years” in at the start, to make it look like the growth has been steady, rather than stalled over the past two years.)

City Students Outperform the Rest of the State and Nation on the National Assessment of Educational Progress (“Outperform”?  Only in the sense that NYC fourth-graders scored almost as high as students in the nation overall, and were significantly lower than eighth-graders nationally.  But it’s a headline, and who pays attention to them, anyway?)

Record Number of Students Performing at or Above Proficiency

Chancellor Calls on State to Adopt More Rigorous Standards to Ensure Further Progress

Schools Chancellor Joel I. Klein today applauded consistent and sustained gains by New York City public school students on the 2009 National Assessment of Educational Progress (NAEP) math exam. (Consistent and sustained might be a stretch, but maybe it’ll pass.) Under the Bloomberg Administration, the progress of the City’s fourth and eighth graders-the two grades tested-has outpaced that of fourth and eighth graders in the rest of New York State and the nation. The 2009 results show that New York City students in both grades have made statistically significant improvement since 2003, when NAEP’s Trial Urban District Assessment in math was launched, and have raised their scores each time the biannual exam has been administered. (Maybe no one will catch the fact that the increases in scores from 2007 to 2009 could be an illusion due to chance alone.) New York City fourth-grade scores rose 11 scale score points since 2003, compared to an increase of one point among fourth-grade students in the rest of the State during that time and five points among fourth graders nationally. (Now, here’s the real good news.  But will anybody acknowledge it?) New York City eighth-grade scores rose seven points since 2003, compared to an increase of one point among eighth-grade students in the rest of the State and five points among eighth graders nationally.  (Ditto.  I hope people pay attention to the average gains since 2003.)

In addition, the number of fourth-grade City students performing at or above proficient on the NAEP exam rose 14 percentage points since 2003, from 21 percent to 35 percent, only three points below the national average. For the first time, New York City eighth graders also made statistically significant gains at or above proficiency, with 26 percent of students now at or above proficient, compared to 20 percent in 2003.  (Thank goodness for 2003.  If we had to compare everything to 2007, we’d look like dogmeat.)

The narrowing of the gap between New York City students and those in the rest of the State on NAEP follows a pattern similar to the one established on New York State math exams over the same period. (Gotta claim that the pattern on NAEP is the same as the pattern on the New York State assessment, even though the gains on NAEP have been modest overall, and nonexistent since 2007, whereas the gains on the state assessment have been astronomical, which undermines the legitimacy of the state tests and everything we use them for.) Since 2003, the gap separating New York City fourth graders and their statewide peers on NAEP has decreased from 15 scale score points to five points. During this time, the gap between City fourth graders and those in the rest of the State on the New York State math exam has decreased from 18 scale score points to two points. In the eighth grade, the gap separating City students and their statewide peers went from 21 scale score points to 15 points on NAEP, and from 23 points to 11 points on the State exam.  (Not sure why it’s important to toss in the NYC vs. New York State comparison for NAEP, but let’s throw as many statistics out there as we can, and see what sticks.)

“Today’s results reflect what we have seen in the classroom and what we have seen on State tests since Mayor Bloomberg took control of the schools-that New York City students are making consistent and meaningful progress thanks to their hard work and the dedication and determination of our educators. The fact that more students than ever are at or above proficient is especially encouraging,” Chancellor Klein said.  (“More students than ever are at or above proficient” can deflect attention from the fact that the average score didn’t increase from 2007 to 2009, and that the increase in the percentage of proficient students in both grades four and eight over that period also wasn’t significant.)

“A lot of work still lies ahead of us. The next important step is for New York State to adopt the more rigorous national common core standards so we can help all students reach their potential.”  (Anybody out there against higher standards?  I didn’t think so.  We’ll worry about the fact that 85% of NYC fourth-graders are proficient on the state test, but only 35% are on NAEP, later.)

New York City students are also doing well in comparison to students in other urban districts that share their results publicly as part of the NAEP Trial Urban District Assessment (TUDA). New York City fourth graders rank third among their peers in the 18 participating districts, with City fourth graders eligible for free or reduced-price lunch ranking first among their peers and outperforming the nation by seven points. (Using ranks can obscure the fact that the averages are estimates, and that NYC is really tied for seventh with Miami-Dade, San Diego, Houston and Boston, behind Charlotte and Austin.) New York City eighth graders rank sixth out of the 18 urban districts.  (Sixth out of 18 sounds pretty good, even if New York City scores didn’t differ significantly from large cities nationally.)

Students in nearly every fourth- and eighth-grade subgroup-white, black, Hispanic, low-income-made statistically significant progress since 2003. (“Nearly every” subgroup-except white and Hispanic eighth-graders.  That’s nearly every, isn’t it? And let’s hammer that “since 2003,” because there were no increases in any of these groups from 2007 to 2009 at either grade level. ) New York City students in these grades also perform at nearly the same level, and in some cases exceed, their suburban peers in New York State. The City’s white fourth graders, who gained 10 scale score points since 2003, outperformed white fourth graders in the State’s suburbs by five points. The City’s black fourth graders gained eight points since 2003 and now score four points above their suburban peers.   (Saying it this way can make it seem like NYC is closing the achievement gap, even though the black-white gap has not shrunk at either grade four or grade eight since 2003.  I hope that nobody says that the Hispanic-white and black-white gaps at grades four and eight are the largest they’ve ever been since 2003;  that would be mean.)

NAEP, often referred to as “the nation’s report card,” is the nation’s ongoing representative sample survey of student achievement in core subject areas and reports the educational progress of students in grades 4 and 8. Mandated by Congress, NAEP is administered by the United States Department of Education’s National Center for Education Statistics. Eighteen cities varying in demographic makeup, including New York City, participate in the NAEP Trial Urban District Assessment by allowing their results to be reported publicly. The results for this year’s fourth- and eighth-grade math tests were released in Washington, D.C. this morning. The results of NAEP English and science exams will be made public in the spring.  (Whew!  We made it.  I wonder if anybody will fall for this?

Thank goodness that damned eduwonkette isn’t blogging anymore.)

Last night, Anderson Cooper reported on the Harlem Children’s Zone for the CBS newsmagazine 60 Minutes.  “For years, educators have tried and failed to get poor kids from the inner city to do just as well in school as kids from America’s more affluent suburbs,” he began. “Black kids still routinely score well below white kids on national standardized tests. But a man named Geoffrey Canada may have figured out a way to close that racial achievement gap.”  Cooper asked Canada, “So you’re trying to level the playing field between kids here in Harlem and middle class kids in a suburb?”  “That’s exactly what we have to do,” Canada replied.

As is customary, Cooper spoke with Harvard economist Roland Fryer, who has analyzed the achievement of students attending the HCZ Promise Academy charter schools.  Fryer said, “At the elementary school level, he closed the achievement gap in both subjects, math and reading.”   

“Actually eliminating the gap in elementary school?” Cooper asked.

“We’ve never seen anything like that. Absolutely eliminating the gap. The gap is gone, and that is absolutely incredible,” Fryer said.

I suppose that one can look selectively at the most recent achievement data available—the 2009 state assessments in English Language Arts and math—to draw this conclusion, but boy, is it a stretch.  The figure below shows the 2009 English Language Arts and mathematics achievement of students at the HCZ Promise Academy and HCZ Promise Academy II, for grades three, four, five and eight.  This achievement is contrasted with New York City citywide averages for Asian and white students.  The group differences are represented in standard deviation units, using the citywide standard deviation for the scale scores on the respective tests.

In grade 3, HCZ students score higher than the citywide average for white students in both ELA and math, a remarkable accomplishment.  They also outperform Asian students in ELA, but are about a quarter of a standard deviation below the citywide Asian average on the math assessment.  Were we to limit our attention solely to third grade, one could, without too much hyperbole, claim that HCZ had eliminated the racial achievement gap within New York City.

But there are other elementary and middle school grades on which to compare HCZ and white and Asian students across New York City, and the story is quite different for these other grades.  In grades four, five and eight, HCZ students score consistently about .6 standard deviations below white and Asian New York City students on the state ELA exam.  The gaps are also large in mathematics, although the eighth-grade gap is considerably smaller than those in fourth and fifth grades.  In fifth grade, HCZ students score .9 standard deviations below white students citywide, and 1.1 standard deviations below Asian students.

Taking all of these data together, there is virtually no basis for claiming that the Harlem Children’s Zone has eliminated the racial achievement gap in elementary and middle school.

The data that I’ve presented compare HCZ students with New York City students.  But recall that Geoff Canada’s objective is to level the playing field relative to middle-class suburban kids, who may be higher-achieving than the white and Asian students attending NYC public schools, as a good fraction of the most affluent children and youth in New York City attend private schools.  How do things look if we compare HCZ students with students in Scarsdale, the economist’s suburb of choice for claims about closing the achievement gap?

As the figure below indicates, the math gaps look about the same, since Scarsdale students score in the same ballpark as white and Asian students in New York City.  Thus, HCZ third-graders outperform even Scarsdale third-graders, but there are large gaps in grades four and five, and then a smaller, but still substantial, shortfall in grade eight, with Scarsdale students scoring .36 standard deviations higher than HCZ students.  However, the gaps in English Language Arts are much larger at every grade level, because Scarsdale students score considerably higher on the state ELA exam than do white and Asian students in NYC at every grade level.  In grades four, five and eight, HCZ students score from .97 to 1.22 standard deviations lower on the state ELA exam than do Scarsdale students, a huge gap.  Score differences of this magnitude indicate that the typical HCZ student might score at the 15^th percentile of the Scarsdale distribution of performance in these grades.  

In the 60 Minutes segment, Roland Fryer used a football analogy to describe the accomplishments of HCZ.  “We’re ten touchdowns down in the fourth quarter,” he said. “We kick a field goal and everyone celebrates, right? That’s kind of useless. We’re still 67 points down … What Geoff Canada has shown is that we can actually win the game.”

But here’s the problem.  We’re not in the fourth quarter.  We’re in the first quarter, and most of the game still lies ahead.  The Harlem Children’s Zone is not a mature intervention.  No child has gone through his entire childhood and youth exposed to the intervention, and we don’t know what the outcomes will look like until that occurs.  I am hard-pressed to conclude, based on the most recent data available, that the results are, in Cooper’s terms, “nothing short of stunning,” or that the gap is gone for good.  The 2009 results for third-graders are terrific;  those for students in grades four, five and eight are not.  These latter grades show large and persistent gaps within New York City in both English Language Arts and mathematics, and even larger gaps with the affluent students in Scarsdale, particularly in English Language Arts.  If the third-grade pattern were to persist through the end of high school—on assessments we can trust—that would truly be nothing short of stunning, and well worth celebrating.  But it’s still too early to declare victory.

It’s no surprise that the State Education Department and the Board of Regents have taken up the cause of ensuring an equitable distribution of highly-qualified teachers across New York State.  The key justification for such a goal is the fact that the K-12 education system is shortchanging our children.  Although some students are highly successful, many more are not, and the problems are concentrated in urban school systems serving large numbers of poor children of color. 

If that’s the problem, is improving the education of teachers the solution?  It’s certainly part of the solution, given what we know about the centrality of teaching to student learning.  But it’s by no means the entire solution, as a great many other forces shape student outcomes.  For example, a great teacher can’t compensate for a child coming to school hungry, and great teaching of an out-of-date curriculum only results in great mastery of out-of-date knowledge.  I trust that Chancellor Tisch and Commissioner Steiner are not seduced by claims that the single most important determinant of a child’s achievement is the quality of his or her teachers, because that’s simply not true.  Family background continues to be the dominant factor.  But the quality of teachers is, at least in theory, something that is manipulable via education policy initiatives, and it’s a lot more tractable than addressing the fact that one in five children under the age of 18 in New York State live below the poverty line.

So if we redefine the problem as too few students in New York State are taught by highly qualified teachers, then what’s the solution?  The recent proposal brought by the State Education Department to the Board of Regents offers five recommendations.  In some ways, this package of proposals is a straightforward extension of trends that have shaped the course of public education in the U.S. over the past two decades:  a heightened concern for holding school systems, schools and teachers accountable for student outcomes, with more oversight by the federal government and the state;  an expansion of the role of markets in the operation of public schooling, coupled with the belief that market forces will reward successful enterprises, and drive the unsuccessful out of business;  and a targeted deregulation that allows some institutions to strike a bargain for increased autonomy in exchange for increased accountability to the state.

There’s not a whole lot of evidence that these kinds of policy reforms have led to better educational outcomes for children in the U.S.  It’s challenging to show that accountability systems such as No Child Left Behind or state-level initiatives have led to substantially higher achievement;  and the charter schools spawned by the expansion of markets have, by most accounts, included some great successes and some abject failures, with little overall impact on student performance across the country.  So I am not optimistic that the proposed reforms will result in sharp improvements in teaching and learning in public schools in New York State. 

As is often the case with public policy, the devil is in the details, and I’d like to take a couple of minutes to mention two things that I’m worried about.  The first is the heightened attention to classroom effectiveness in both performance-based assessments for teacher certification, and in the assessment of the institutions which offer teacher preparation programs.  I worry that the State Education Department and the Board of Regents will be obliged to rely on an overly-narrow set of measures of classroom effectiveness, due to the constraints of time, money, and technology. 

Portfolio assessments, including lesson plans, videotapes of teaching practice, collections of student work, and candidate self-assessments are a promising direction, and there are some existing models that may be useful to guide the design of such assessments.  I’d be happier if there were more evidence that the implementation of portfolio assessments resulted in better teaching and learning, and if there were a more explicit theory of how having assessments produces better teaching. 

But it’s striking that the recommendations single out value-added student assessment data as components of both the portfolios of candidates for professional certification and of the profiles of certifying institutions.  Simply put, the technology for using value-added student assessment data for these purposes is not ready for prime time, and likely will not be for many years to come.

One major obstacle is the lack of reliable and valid measures of student performance that can serve as the basis for value-added assessments of teacher effectiveness.  When the 2009 New York State results from the National Assessment of Educational Progress were released last month, they showed a flat level of performance at odds with the sharp growth observed on the New York State exams at every grade level in English Language Arts and mathematics over the past several years.  Commissioner Steiner and Chancellor Tisch have both expressed concern that the state’s testing system is not telling us what we need to know about the academic performance of New York State’s children and youth.  The tests and test items have become predictable over time, and some grade-level standards have never appeared on the exams.  There are, for example, 48 eighth-grade math standards.  This year, just seven standards made up 50% of the points on the eighth-grade exam. 

The state assessment system is broken, and it can’t be fixed overnight.  I strongly urge the Commissioner and Chancellor to take the time to develop a new assessment system that more accurately measures students’ mastery of school subjects, and that is designed to measure changes in students’ learning over time.  And, if the objective is to use the assessment system to evaluate teachers, schools, school districts, and institutions of higher education, it’s important to build that objective into the design of the assessments, rather than using them for a purpose for which they were never designed in the first place. 

The technology for test design is surprisingly complex and fragile, and we do ourselves no favors by assuming that we can just whip up a test overnight. 

The Regents exams, which I assume would be central to measures of value-added effectiveness at the secondary level, are problematic in their own right.  I don’t understand why the State Education Department has high school students’ Regents exams scored by the teachers at their own schools.  One big idea that the Freakonomics school probably has right is that people respond to incentives.  And there are powerful incentives for schools to maximize students’ scores on Regents exams in ways that compromise the validity of those scores.  We see the fruits of this when students go off to college and find that they are unprepared for college-level work.

So in essence, I’m saying to Commissioner Steiner and Chancellor Tisch, “Clean up the state assessment system — and take the time to do it right.  Then we can talk about value-added assessment.”

But beyond the many questions about value-added effects on students’ test scores, we should be asking, how do we assess a teacher’s contributions to other learning outcomes?  Surely we care about more than test scores.  What are good measures of a teacher’s contributions to preparing students to be competent citizens in our democracy?  How much are the Board of Regents and the State Education Department willing to invest in creating measures that will capture how well teachers teach students to think, question and act?

A brief vignette may reveal the challenge.  It’s January, and Ms. Bilsky, a fourth-grade teacher in the Bronx, is teaching a math lesson.  The subject is geometry, and the lesson is about how to classify angles as either acute or obtuse.  The topic is a standard from the state’s math core curriculum.  In the middle of the lesson, Rashid, a boy in the class, audibly aims a racial slur at his classmate Javier.  Ms. Bilsky hears it, but she chooses to ignore it, instead plowing ahead with the lesson.  At the end of the year, the students in Ms. Bilsky’s class did a bit better on the state math assessment than the students in other fourth-grade classrooms in the Bronx.

Now, is that good teaching? 

The value-added assessment will tell us that it is good teaching.  

And a teacher preparation program sending a cohort of Ms. Bilsky’s out into the field might look pretty good too.  But I think we should demand more of our teachers and our teacher preparation programs than simply raising students’ test scores.  And I think we should demand more of the New York State Education Department in developing measures that can capture a broader array of outcomes of good teaching.  I do not doubt that it will be difficult to do so;  important things often are.        

The second issue I want to discuss is the proposal to pilot a new teacher certification model that would enable providers other than institutions of higher education to offer teacher preparation programs, with the Board of Regents awarding Master’s degrees to the candidates who complete these programs.  I am deeply troubled by this prospect, because it seems to be a serious threat to the very nature of graduate education.  Currently, state regulations require that most graduate-level courses in certification programs be offered by full-time faculty holding terminal degrees, with the assumption that these faculty are scholars of the subjects they teach.  Now, we all know that there are schools of education, and other professional schools, around the state, where the quality of research may not be very high, and the contribution to a body of knowledge about real-world practice may not be very great.  But the explicit decoupling of the production of knowledge from the preparation of practitioners is, in my view, a very bad idea.  And one might wonder whether other occupations regulated by the state will be far behind.  Will health clinics be authorized to prepare physicians?  Will mental health facilities train psychologists? 

Moreover, I suspect that a proposal such as this might tax the capacity of the State Education Department to offer appropriate oversight.  Would the Department have to devote staff to ensure that these pilot programs met existing criteria for preparation — libraries, facilities and physical space, qualified faculty, relationships with local schools and field placements, curricula and syllabi, and so on?  Or would these pilot programs not have to meet the state’s existing standards?

I think what’s most baffling in this proposal is the awarding of a Master’s degree by the Board of Regents.  Why is it necessary for the state’s recognition of completion of a teacher preparation program to be coupled with a Master’s degree?  Neither No Child Left Behind nor the Individuals with Disabilities Education Act requires that a highly-qualified teacher have a master’s degree, and there are plenty of undergraduate education programs around the state of New York that lead to teacher certification.  Having the Board of Regents award degrees that are widely understood to be the province of accredited institutions of higher education in a sense turns the State Education Department into a giant ed school.  Doesn’t the Department have enough problems, without tarring it with that particular brush?

On Monday, the Department released the 2008-09 Progress Reports for high schools.  Anna Phillips reported that Chancellor Joel Klein said that the high school Progress Reports were more stable and accurate than those for elementary and middle schools because they’re based on multiple measures.  Huh?  Welcome to the party, Chancellor Klein.  I hate to tell you that measures such as credit accumulation are not necessarily accurate measures of a school’s contribution to student learning and development. 

But the high school Progress Reports have a bigger problem.  Three-quarters of a school’s score comes from a school’s location in relation to a group of 40 peer schools.  The idea of comparing a school to peer schools is to create an “apples to apples” comparison.  It’s actually a good feature of the Progress Reports that they seek to compare a given school to how schools across the city are doing as well as to how schools that serve similar students are performing.

But it only works if the right criteria are used to determine a school’s peer schools.  Wednesday, Jenny Medina and Robert Gebeloff broke a story in the New York Times that high schools with higher percentages of poor, black and Hispanic students received lower grades on the Progress Reports.  In 2009, they wrote, the high schools which received A’s enrolled an average of 77% black and Hispanic students.  In contrast, the high schools which received C’s, D’s and F’s enrolled an average of 91% black and Hispanic students.  This pattern, found in 2007 and 2008 as well, suggested that the school grading system doesn’t adequately adjust for racial and ethnic differences among schools.

A high school’s peer index is based primarily on its students’ average eighth-grade scores on the state ELA and math exams (using the peculiar metric the DOE has developed for converting the exam’s scale scores into a 1.0 to 4.5 proficiency scale), minus two times the percentage of special education students and minus the percentage of overage students.  A high school with an average proficiency of 3.10, 6% special education students, and 12% overage students would have a peer index of 2.86.  One with an average proficiency of 3.70, 2% special education students, and 5% overage students would have a peer index of 3.61.

Although the formula tries to take special education and overage status into account, I suspect that its designers were unaware that it is dominated by the average proficiency value, because there is far more variance from school to school in average proficiency than in special education and overage status.  But a larger question is, why these factors and not others?  Why not the percentage of English Language Learners (ELL’s)?  Why not the percentage of students eligible for a free or reduced-price lunch?  Why not the racial/ethnic make-up of the school?  (And when is the DOE going to wise up that it can’t treat black students as equivalent to Hispanic students, and Asian students as equivalent to white students?  These groups have different learning trajectories.)

And why stop there?  If the goal is to try to isolate the impact of the school on student performance and progress, then logic would dictate that we should seek to control for all factors that are prior to selection into one school versus another, and potentially related to students’ outcomes.  That includes a range of demographic criteria, to be sure.  But there are at least two other factors that I think ought to be taken into account.  The first is school size.  Schools in New York City generally have little control over their size, and if small schools provide certain advantages for students, then we should compare small schools to small schools and large schools to large schools.  The second is per-pupil expenditures.  Even in the Fair Student Funding era, there are disparities in per-pupil expenditures across schools that are not accounted for by demographic differences in the students attending different schools.  I’ve spoken to principals who are indignant that their peer schools have higher expenditures, and yet they are being held to the same performance criteria.

Does all this matter?  You bet.  Let’s look at just one of the many measures in the high school Progress Reports:  the percentage of second-year students accumulating ten or more credits.  (The pattern I’m going to describe is found for many of the performance and progress measures in the Progress Reports.)   Citywide, the 2009 average was 72%, with a standard deviation of 15%.  Schools are compared to their “peer range,” a school’s location in relation to its lowest and highest peers.  Citywide, schools were, on average, 59% of the distance between the lowest and highest peers on their percentages of second-year students accumulating ten or more credits.

But some schools were advantaged in these calculations, and others disadvantaged, even though the peer horizon scores are explicitly designed to compare “apples to apples.”  The figure below compares schools in the lowest quarter of a given demographic feature to schools in the top quarter.  Schools with high concentrations of black and Hispanic students;  large schools;  schools with a higher proportion of special education students;  and schools with more English Language Learners all score lower relative to their “peer” schools than do other schools. 

What these figures suggest is that New York City’s high school Progress Reports systematically penalize some schools and reward others.  So when you see the DOE touting the superiority of the progress made by the small schools opened during the Bloomberg/Klein era, remember that it’s no accident:  it’s built into the accountability system.

Reardon is careful to point out that it’s not possible, based on the information provided in Hoxby’s report and associated documents, to judge the extent of the bias in Hoxby’s estimates of charter school effects on student achievement.  More than anything, he calls for reserving judgment until more information about the study, its data and methods are available, and until the study has undergone rigorous peer review.  Until then, he maintains, it would be unwise to rely on the statistics reported in the study, and the inferences Hoxby and her colleagues draw about charter school effects in New York City.     

Here I’ll mention two of the features of Reardon’s critique that I find particularly persuasive.  The first is that Hoxby used an inappropriate set of statistical models to analyze the data, which likely distorts the charter school effects.  You might be surprised to learn that Hoxby used statistical models at all.  If her results are based on comparing students who won a charter school lottery with students who lost the lottery, and the lottery was fair, balanced and random, why would a model be needed?  It seems like the charter school effect would simply be the difference in the outcomes observed for the lottery winners and the lottery losers.  But comparing lottery winners and losers isn’t really estimating an individual causal effect, because an individual student can’t simultaneously be enrolled in a charter school and a traditional public school.  Even in the context of a lottery, or any other kind of study that can capitalize on a randomization process, such as a clinical drug trial, statistical models come into play to allow for inferences about cause-and-effect relationships.  These inferences are always made in relation to a particular statistical model, and all such models have assumptions.

One of the assumptions that is widely recognized is that a statistical model for causal inference should take account of factors that precede selection into the “treatment”—in this case, enrollment in a charter school versus a traditional public school.  If, hypothetically, charter school attendees were wealthier than traditional public school attendees, we’d want to control for wealth to make the charter and traditional school attendees as comparable as possible.  But it’s just as widely recognized that such a statistical model should not take account of factors that are measured after, and hence potentially influenced by, the treatment.  If attending a charter school increased a student’s motivation, and heightened motivation yields better test scores, then we wouldn’t want to control for motivation in a statistical model for the causal effect of going to charter school on test scores.  That kind of control means that the charter and traditional school attendees are no longer comparable at the time that they began attending a charter versus traditional school, which is the critical time.

Reardon demonstrates that this is precisely what Hoxby and her colleagues do in most of their statistical analyses.  For the analyses of charter school effects on test scores in grades four through eight, she controls for achievement in the prior year—achievement that was observed after the lotteries that determined whether a student enrolled in a charter or traditional public school.  The effects of charter school attendance on test scores in grades four through eight are therefore distorted, but to an unknown degree.  This is not a problem for estimates of the cumulative effect of charter school attendance in grades K-3 on third grade test performance, because the statistical models don’t include prior test scores (as there aren’t any before grade three.)  Reardon therefore finds Hoxby’s estimates of the effect of going to charter school in grades K-3 to be more credible than those for grades four through to eight.  However, the K-3 effect is only one-half to one-third as large as the estimated annual effect of charter school attendance in grades four through eight.    

The second issue is estimation of the cumulative effects of charter school attendance.  Hoxby’s report gets a lot of mileage out of the claim that the effects of attending a charter school from kindergarten to grade eight are large enough to close the performance gap between (predominantly white, upper-class) children in Scarsdale and (predominantly minority, low-income) children in Harlem by 66% in English and 86% in math.  Reardon points out that these figures are based on unrealistic extrapolations.  You can’t simply add up the annual effects of attending a charter school from year to year because the gains decay over time.  Moreover, most of the students in the Hoxby study have been in charter schools for only three or four years;  virtually none have been enrolled in charter schools for as many as nine years, and those would only have been enrolled in the very small number of charter schools that have been open that long, and cannot tell us about the long-term effects of attending the much larger number of newer charter schools.  Reardon’s analysis suggests that Hoxby’s estimate of the cumulative effect of attending a charter school from grades four through eight could be exaggerated by as much as 50%.

Are Caroline Hoxby’s estimates of the effects of attending a charter school rather than a traditional public school in New York City accurate?  Maybe.  But based on Sean Reardon’s critique, probably not.  Hey, you never know.

Kolodner drew attention to the fact that although elementary and middle school charter students had higher rates of proficiency on the state math and English Language Arts assessments this year, charter schools on average had a lower score on the progress component of the School Progress Reports.  And since the progress component makes up 60% of the overall score, charter schools also had lower overall scores on the Progress Reports than did district schools.  She quoted Patrick Sullivan, an appointed member of the Panel for Educational Policy that the DOE describes as its governance body, on the meaning of this pattern.  “Either the progress reports are invalid,” Sullivan said, “or charter schools are lagging.”

The Daily News article and a subsequent posting by Sullivan on the NYC Public School Parents blog prompted a quick reply from Peter Murphy, Director of Policy & Communications for the New York Charter Schools Association (NYCSA), here and here.  Murphy called into question the metric used by the DOE in its Progress Reports, especially the fact that student performance only counts for 25% of the overall score, whereas student progress counts for 60%.  This, he contended, is “woefully lopsided,” and unfairly penalizes schools that have had students scoring high for several years running.  If I read his second posting correctly, he concludes that the progress reports indeed are invalid. 

Murphy went on to argue that charter schools serve more “at-risk” students than do district schools, and the fact that 16% of district students receive special education services, whereas only 9% of charter school students do, is “not at all significant.” He sidestepped the report’s findings about the disproportionately low enrollment of English language learners in charter schools.

Is this anything?  Yes and no.  The relative performance of charter and district schools on the DOE’s School Progress Reports is, in my view, nothing of consequence.  The progress measures have been shown to be highly unreliable from year to year, and I wouldn’t base any conclusions about the relative ability of charter and traditional public schools to promote growth in student learning from these measures.  I think Murphy is probably not correct in arguing that the high performance of students in a charter school in a given year limits the ability of that school to show growth the following year, as the DOE calculations now count persistence at Level 4 from one year to the next to be a year’s worth of growth, regardless of whether a student’s scale score increased or decreased.  But I’ll acknowledge that the state tests that are the basis for the performance and progress measures have shown to be very inaccurate at measuring very high and very low levels of performance.  In their current form, I don’t think that they’re to be trusted for important policy decisions.

On the other hand, I think there’s something to be learned from the composition of students attending charter schools in New York City.  Students attending charter schools are much more likely to be African-American than students in New York City overall.  In charter schools, about 62% of the enrollees are African-American;  30% are Latino;  and 8% are either white, Asian-American, or members of another ethnic group.  In contrast, the distribution of students overall in New York City is approximately 40% Latino;  30% African-American;  15% Asian-American;  and 15% white.  Moreover, 80% of students attending charter schools are eligible for free or reduced-price lunches, in contrast to 75% for students in NYC overall.

The most striking figures in the DOE report are the disparities in the percentages of special education students and English Language Learners in charter and district schools.   Contra Mr. Murphy, I think that the fact that charter schools are enrolling proportionately fewer special education students is worthy of more scrutiny, and can’t be explained away so easily.  So too the disparity in the enrollment of English Language Learners, who represent nearly 15% of students in NYC overall, but only 4% of students in charter schools.  It may be that charter schools have been sited in locations that are heavily African-American, and African-American students are less likely to be English Language Learners than Latino or Asian-American students.  I’m not sure whether or not the evidence bears that out, and it warrants further study.  But Murphy’s contention that “For all the handwringing about special ed students and students with English language needs, charter schools are in fact serving and benefiting a greater proportion of students deemed ‘at risk’ than the City as a whole” doesn’t hold up.  It’s true only in the narrow sense that charter schools serve fewer than 8% white students, whereas NYC schools overall serve about 15%, and charter schools have a slightly higher concentration of students eligible for a free or reduced-price lunch.  It’s certainly not true regarding the very great educational needs of students with disabilities or who are English language learners.

What I found most revealing about the DOE report is that in 2008-09, the New York City public schools served 1.1 million students, but only 24,000 students were enrolled in NYC charter schools.  That’s about 2.2% of the students in the NYC system.  Of course, many of these charter schools are still adding grades each year, such that when they are at capacity, they may enroll three or four times as many students.  But even four times as many students as the current figure would place fewer than 10% of NYC schoolchildren in charter schools.

 Critics wondered why this budget shortfall wasn’t identified earlier, before such widespread hiring, and some have questioned whether this pattern of hirings and layoffs was intentionally orchestrated to replace older, veteran teachers with younger, less-costly ones.  DCPS, under Chancellor Rhee’s name, posted on October 7^th a list of Frequently Asked Questions Concerning the Budget Shortfall and Staffing Reductions.  One of the questions was:  Did you target veteran teachers?

 The response:  “Absolutely not.  This is rumor and completely unsubstantiated.  Such a practice is not only illegal but morally reprehensible.  In addition, since many, many thousands of DPS students benefit from being skillfully taught by veteran teachers, it would be ill-advised for us to ever take such an action. 

•  The percentage of staff members over age 40 (a protected class) separated last week mirrors almost exactly the percentage of veterans within the DCPS work force.
• Employees with three or fewer years of experience are more heavily represented in the pool of separated staff members than they are among total number of DCPS staff.”

 That was the first version of the FAQ.  Later, a revised version was posted, which removed the bullet points above, and substituted the following:

 ”The experience levels affected:

•  Only 7% of teachers affected had experience of 25 years or more
• 54% of teachers affected had experience of 10 years or less
• 39% of teachers affected were in their first five years of teaching
• 17% of teachers affected were new hires;  they were in their first year of teaching”

 I’m no lawyer, but from my reading of the federal rules on equal employment opportunity, it’s actually not illegal to discriminate on the basis of experience.  It is, however, illegal to discriminate on the basis of age, and in some occupations, such as teaching, there is an extremely high correlation between age and experience.

 Rhee’s artful/deceptive response above doesn’t tell us if the individuals who were laid off differed in their experience profiles from those who were not.  The fact that 46% of the teachers who were laid off had more than 10 years of experience needs to be compared to the proportion of teachers in the District overall with more than 10 years of experience to see if there was some kind of disparate treatment. Although Rhee has in interviews repeatedly pointed to the comparison between 7% and 17% to argue that new hires were more likely to be laid off than highly-experienced teachers, I would hope that she would understand that the data say nothing of the sort. 

 In any event, the revision to the document goes on to indicate that the average age of DCPS employees is 43, and the average age of those “separated” is 49.  That is, the teachers who were laid off were, on average, six years older than the typical DCPS teacher.  Is that a little or a lot?

 I think it’s a lot.  We don’t know the details of the age profile of DCPS teachers, but sociologist Richard Ingersoll drew a picture of the profile based on the 2003-04 Schools and Staffing Survey for a 2009 report by the National Commission on Teaching and America’s Future.  His analysis indicates that, in 2003-04, about 25% of DC teachers were 33 years old or younger, and 25% were 54 years old or older.  The median age was 46.  Working with his figure, I estimate the average age to be about 44, and the standard deviation of teachers’ age to be about 12.4.

 What this suggests is that the 266 teachers and other educators who were laid off on October 2^nd were about a half a standard deviation older than the average teacher in the DC public schools.  A difference of this magnitude is extremely unlikely to have occurred by chance.

 In a TV interview on local Fox affiliate WTTG on October 9^th, Chancellor Rhee said, “I do, though, think that it’s troubling to hear when public officials are saying things that just aren’t true.”  Me too, Chancellor Rhee.  Me too.

The reasons for this are pretty clear.  To be a good prospect for scaling up in a Goal Four project, an intervention must previously have been shown to be effective in at least one site, using rigorous methods for assessing cause-and-effect relationships.  Relatively few interventions meet this threshold, because most policies and programs don’t have educationally meaningful effects, even if it seems like they ought to.  Similarly, projects that are good candidates for Goal Three funding must previously have shown at least some evidence of effects on student outcomes in pilot studies in which the intervention received a tentative tryout, but not a full-blown test using rigorous experimental or quasi-experimental research methods.

I was struck by a thought experiment:  what if my panel of distinguished researchers (the other members, at least) had been presented with a proposal based on the Race to the Top criteria that Secretary Duncan talked about at Teachers College, and which have been acclaimed by opinion writers such as Nick Kristof and David Brooks, as well as the editorial page writers for major newspapers in New York City and around the country?  The draft Race to the Top criteria for funding state proposals provide incentives for linking teachers to their students’ standardized test scores, and in his remarks on Thursday, Secretary Duncan drew attention to Race to the Top incentives for states and districts to link student performance to the teacher preparation programs from which students’ teachers had emerged.  Only Louisiana currently does this, the Secretary said.  What if a scale-up proposal for this intervention had been presented to a panel charged with applying the IES criteria to evaluate its fundability?

It would have been laughed out of the room.

Not literally, of course;  the panel members take their work very seriously, and seek to provide feedback to applicants as well as to advise the staff of IES about the merit of the proposals.  But a key criterion for the viability of Goal Three and especially Goal Four proposals is evidence that the intervention has had a positive effect on student outcomes.  There is to date no evidence that the implementation of longitudinal data systems linking teachers and teacher preparation programs to student achievement outcomes has actually improved student performance.

Could such data systems result in improved outcomes for students?  Sure.  However, I have yet to see a full-blown theory of change specifying exactly how the implementation of longitudinal data systems would result in better outcomes, and even theories that seem quite plausible often don’t pan out.  Developing a theory of change would be an essential feature of a Goal Two development and innovation project.

And that, in my view, is where longitudinal data systems linking teachers and teacher preparation programs to student outcomes belong.  As innovative pilot projects developed and refined in local settings over a few years—not as projects rushed to scale in states across the country despite the complete lack of evidence that they will improve student achievement.

They might have wanted to wait a day.

This morning, the U.S. Department of Education released the 2009 results of the National Assessment of Educational Progress assessments of fourth-grade and eighth-grade mathematics in each state and for the nation overall.  Nationally, fourth-grade performance held steady from 2007 to 2009, and there was a slight but statistically significance over this period in eighth-grade math performance.  In New York State, the small declines in fourth-grade and gains in eighth grade were not statistically significant, leading to the conclusion that there has been no change in the performance of New York students on the NAEP math assessment from 2007 to 2009. 

This is a very different story than the one told by New York’s own assessment system, on which the Bloomberg and Klein administration has staked its claims about the great progress in student achievement.  The average scale score in fourth-grade mathematics increased from 680 in 2007 to 689 in 2009, a hefty 9 points;  the jump in eighth-grade scores was even more dramatic, as the average scale score rose from 657 in 2007 to 675 in 2009, a remarkable increase of 18 points.

To put these two sets of numbers in context, the chart below shows the gains in fourth-grade and eighth-grade math performance from 2007 to 2009 expressed in standard deviation units (i.e., the amount of variation among individual students in 2007).  According to NAEP, fourth-graders’ performance fell .07 standard deviations from 2007 to 2009, a difference that is not significantly different from zero.  In contrast, fourth-graders gained .23 standard deviations on the New York State assessment from 2007 to 2009.  Similarly, the NAEP results indicate that eighth-graders in New York gained .08 standard deviations from 2007 to 2009 in math performance, a difference that is not significantly different from zero, but they gained .47 standard deviations over this period on the New York State test.

Another way of comparing the implications of the two different sets of test results is to think about where the average student in 2009 would have scored in 2007.  Based on these standard deviations, and assuming that the scores follow a bell-curve distribution, the New York State scores indicate that the average fourth-grader in 2009 scored at the 59^th percentile of the 2007 fourth-grade distribution, which is a pretty big jump.  The increment for eighth-graders is even more striking:  the average eighth-grader in 2009 scored at the 68^th percentile of the 2007 eighth-grade distribution, based on the New York State tests.  In contrast, the NAEP data indicate that the average New York fourth-grader in 2009 scored at the 47^th percentile of the 2007 distribution of fourth-grade math performance in New York State, and the average eighth-grader in 2009 scored at the 53^rd percentile of the 2007 eighth-grade distribution.
 
How can we explain these differences?  There are lots of possible explanations, but most of them don’t hold up under close scrutiny.  The two tests are taken by similar populations of students under similar conditions, and the grade-level mathematics standards on which the two assessments are based do not differ dramatically.  The NAEP test is a low-stakes test, which might result in students not taking it seriously, but the statisticians who oversee the NAEP testing program look for patterns suggesting this, and find little evidence of it.  It’s extremely unlikely that there’s rampant cheating going on in the New York State testing system that could explain the differences. 

 It’s possible that the New York State tests have been getting easier over time.  I have yet to see definitive evidence ruling this out.  There also is strong suggestive evidence of “score inflation” in the New York State tests, because there are predictable patterns in the standards which appear on the state tests year after year, with some standards showing up repeatedly each year, and some standards having never been tested at all during the life of the testing program.  Schools and teachers can make use of these patterns, which also show up in the format of test questions covering particular standards, to focus their instruction on the subset of standards that crop up again and again.  Because the New York State tests never test some standards, we have no idea about whether students have mastered them.  In contrast, the design of the NAEP assessment allows for a much broader picture of mathematics performance, because so many more standards and test item formats are incorporated into the test.

 Whatever the reason, the discrepancy between the NAEP trends and trends in the NewYork State test scores raises serious questions about what the New York tests are telling us about the academic performance of students in New York State.  The same, of course, goes for New York City.  We’ll see NAEP scores for New York City in a month or so, but it’s unlikely that they will yield a different story than what I’m describing here.
 
Is the Earth flat?  No.  But New York State test scores, and probably New York City scores, are.