Post a comment about the budget cuts at your school on our interactive comment map. more »
GothamSchools — daily independent reporting on NYC public schools
skoolboy
May 8, 2009
Now that I have your attention … Today’s New York Times column by David Brooks touts a new study by Roland Fryer and Will Dobbie of the Harlem Children’s Zone (HCZ) Promise Academy charter schools, two celebrated schools in Harlem. Fryer and Dobbie’s finding that the typical eighth-grader was in the 74th percentile among New York City students in mathematics leads Brooks to state that HCZ Promise Academy eliminated the black-white achievement gap. He’s so dumbstruck by this that he says it twice. Brooks takes this evidence as support for the “no excuses” model of charter schools, and, claiming that “the approach works,” challenges all cities to adopt this “remedy for the achievement gap.”
Coming on the heels of yesterday’s release of the 2009 New York State English Language Arts (ELA) results, in which the HCZ schools outperformed the citywide white average in grade 3, but were well behind the white average in grades 4, 5 and 8, skoolboy decided to drink a bit more deeply from the datastream. The figure below shows the gap between the average performance in HCZ Promise Academy and white students in New York City in ELA and math, expressed as a fraction of the standard deviation of overall performance in a given grade and year. The left side of the figure shows math performance, and the right side shows ELA performance.
It’s true that eighth-graders in 2008 scored .20 standard deviations above the citywide average for white students. But it may also be apparent that this is a very unusual pattern relative to the other data represented in this figure, all of which show continuing and sizeable advantages for white students in New York City over HCZ students. The fact that HCZ seventh-graders in 2008 were only .3 standard deviations behind white students citywide in math is a real accomplishment, and represents a shrinkage of the gap of .42 standard deviations for these students in the preceding year. However, Fryer and Dobbie, and Brooks in turn, are putting an awful lot of faith in a single data point — the remarkable increase in math scores between seventh and eighth grade for the students at HCZ who entered sixth grade in 2006. If what HCZ is doing can routinely produce a .67 standard deviation shift in math test scores in the eighth grade, that would be great. But we’re certainly not seeing an effect of that magnitude in the seventh grade. And, of course, none of this speaks to the continuing large gaps in English performance.
But here’s the kicker. In the HCZ Annual Report for the 2007-08 school year submitted to the State Education Department, data are presented on not just the state ELA and math assessments, but also the Iowa Test of Basic Skills. Those eighth-graders who kicked ass on the state math test? They didn’t do so well on the low-stakes Iowa Tests. Curiously, only 2 of the 77 eighth-graders were absent on the ITBS reading test day in June, 2008, but 20 of these 77 were absent for the ITBS math test. For the 57 students who did take the ITBS math test, HCZ reported an average Normal Curve Equivalent (NCE) score of 41, which failed to meet the school’s objective of an average NCE of 50 for a cohort of students who have completed at least two consecutive years at HCZ Promise Academy. In fact, this same cohort had a slightly higher average NCE of 42 in June, 2007.
Normal Curve Equivalents (NCE’s) range from 1 to 99, and are scaled to have a mean of 50 and a standard deviation of 21.06. An NCE of 41 corresponds to roughly the 33rd percentile of the reference distribution, which for the ITBS would likely be a national sample of on-grade test-takers. Scoring at the 33rd percentile is no great success story.
How are we to make sense of this? One possibility is that the HCZ students didn’t take the Iowa tests seriously, and that their performance on that test doesn’t reflect their true mastery of eighth-grade mathematics. The HCZ Annual Report doesn’t offer this as a possibility, perhaps because it would be embarrassing to admit that students didn’t take some aspect of their schoolwork and school accountability plan seriously. But the three explanations that are offered are not compelling: the Iowa test skills were not consistently aligned with the New York State Standards and the Harcourt Curriculum used in the school; the linkage of classroom instruction to the skills tested on the Iowa test wasn’t consistent across the school year, and Iowa test prep began in February, 2008; and school staff didn’t use 2007 Iowa test results to identify areas of weaknesses for individual students and design appropriate intervention.
If proficiency in English and math are to mean anything, these skills have to be able to generalize to contexts other than a particular high-stakes state test. No college or employer is ever going to look at the New York State ELA and math exams in making judgments about who has the skills to be successful in their school or workplace. I’m going to hold off labeling the HCZ schools as the “Harlem Miracle” until there’s some additional evidence supporting the claim that these schools have placed their students on a level academic playing field with white students in New York City.
-
May 8th, 2009 5:47 pm
I can’t believe that David Brooks gets paid for such sloppy journalism. The NYT obviously needs you as a fact checker. Kudos on a great post.
-
May 8th, 2009 11:42 pm
Is there some way to summarize all this in a scrolling headline on CNN or something? Turn it into a billboard in Times Square?
-
May 8th, 2009 11:51 pm
According to NYSED, Harlem Children’s Zone has 0% special ed students and 1% ELL students. Despite the negative comments by Brook (as always) about class size, the HCZ insists on small classes, with class sizes of 18 through 6th grade and ranging from 12 to 20 in 8th.
According to 60 minutes, the school “raises $36 million in private funds per year. Classes have a ratio of one adult for every six kids as well as state-of-the-art science labs, a first-class gym, and a cafeteria that looks more like a restaurant.”
In addition, in the book about HCZ by Paul Tough, it tells the story about how when Canada first started the school, he had promised the middle school kids and their parents that everyone in the class would be able to go onto college. That is the “promise” in the Promise Academy.
But when the scores didn’t go up, Canada called an assembly one day and he told the students he was throwing them out of the school. The whole class had to scramble and find a new school. He just booted them out. Some promise.
-
May 9th, 2009 11:18 pm
Great post. Have you considered submitting an op ed or even a letter? This information needs to reach a broader public. It’s good to have your analysis here but meanwhile Brooks’ puffery is going across the country.
-
May 10th, 2009 10:29 am
Great post. Quick question though - which report are the ITBS numbers from? I can’t find them in the NYC version: https://www.nystart.gov/publicweb/School.do?county=NEW%20YORK&district=&school=310500860864&year=2008
-
May 10th, 2009 10:45 am
Canada doesn’t have a high school … where do his kids go and how are they doing? Unless the “miracle” is sustained in high school HEZ is “smoke and mirrors.”
-
May 10th, 2009 6:32 pm
Pallas I can’t thank you enough for this response. Exposing the weakness in the data is one thing, but I was also concerned with Brooks’ politicized interpretation that HCZ’s success should be a celebration of paternalistic, middle class inculcation, and government dismantlement of the welfare state! And he conveniently hid the money ball when noting HCZ’s ’sufficient funds’ which in reality are exceptionally high investments.
-
May 10th, 2009 9:19 pm
Brooks was simply pointing out something which is 100% true. The HCZ results were unprecedented. They are, without serious challenge, a very important first step. What the data show is that it is possible to make tremendous progress though the type of educational reform utilized by the HCZ charter schools. Have they solved every last problem? No. Is there much additional progress which remains to be made (and then proven)? Yes. But show me where any conventional public school reform effort has ever achieved even the partial success of his program. It was a legitimate, major achievement, and it deserves the spotlight shown on it by the op-ed column of David Brooks.
-
May 10th, 2009 9:22 pm
There’s tons of precedent for those kind of results. HCZ’s school isn’t even the best among Harlem charters - not even close. And both Brooks and Canada know it. Not sure why they’re acting like this is news.
-
May 10th, 2009 9:27 pm
>>There’s tons of precedent for those kind of results.<<
Citation please. I’d like the opportunity to compare and contrast.
-
May 10th, 2009 10:09 pm
Hey Socrates, There is no precedent at all for 1.3 - 1.4 standard deviation gains that I am aware of - certainly not in any of the charter lottery studies that are out there.
-
May 10th, 2009 10:38 pm
What I’m saying is that there have been many schools showing much bigger gains than HCZ’s school shows. They may not have, however, been part of an official study yet. My point is that among the highest performing charters, many of which are in Harlem, Promise is not among the best.
-
May 11th, 2009 2:49 am
Socrates, I hear what you’re saying. I’m not sure that the results of the Promise Academy are notably different from those of KIPP, Amistad, etc. The difference between them is that Promise Academy is part of larger social services organization — something that Brooks somehow neglects.
I’ve had a few days to think about it, and I can conclude only that Brooks is full of it — and that he owes his readers an apology. http://www.edpolicythoughts.com/2009/05/sunday-commentary-sale-on-snake-oil-at.html
-
May 11th, 2009 8:55 am
Guest, the HCZ ITBS data are reported in http://schools.nyc.gov/NR/rdonlyres/E54DBC8D-CEAC-41F2-A491-105A3FA25949/0/HCZPAI.pdf
-
May 11th, 2009 6:06 pm
Aaron,
As ever an interesting and thought-provoking post.
My question was whether Brooks oped arose from mis-reading the Fryer/Dobbie paper, or whether Fryer’s claims unsupported. My hypothesis was the former, not the later.
Roland Fryer is well-regarded to say the least. While he is associated with the current administration in New York City, I figured surely he recognizes his own reputation and goals are too big to stake on mis-representing the outcomes at HCZ for short term gain.
Yet in the paper’s abstract the authors write “Taken at face value, the effects [of HCZ] in middle school are enough to reverse the black-white achievement gap in mathematics and reduce it in English Language Arts. The effects in elementary school close the racial achievement gap in both subjects” Pretty bold statements, and ones that you seems to contradict in your post.
Corey (see above post) says Brooks fails to observe that on page 4 of the paper Fryer writes, in italics no less, “We cannot however disentangle whether communities coupled with high-quality schools drive our results or whether the high-quality schools alone are enough to do the trick.” So the question seems to center on what your definition of “HCZ” is: Schools alone, or the basket of additional services offered?
I believe that in many middle class districts, researchers suggest both factors are at play. One would think that in creating his “conveyor belt” model (as Paul Tough describes it) Canada recognized that he needed both. That’s the idea behind the kind of behavior I have heard described at other successful charters where administrators intervene in the surrounding social environment in ways that one would never find surprising among middle class parents in the suburbs. It is not that community is not important, but rather that as Annette Lareau describes, people in lower SES have rarely had this kind of intervention modeled for them so they engage in them less often.
I’m not going to pretend to be able to take apart the economics in the paper, and in the authors’ defense I note the paper is annotated “preliminary and incomplete.” But if Fryer hasn’t proven that “schools alone” can “do the trick” how can Brooks reach this conclusion?
Fryer attempts to get there on pp. 22-23, but concludes “either the Promise Academy charter schools are the main driver of our results OR the interaction of the schools and the community investments is the impetus for such success.” (my emphasis) Fryer concludes (p. 24) “High-quality schools or community investments coupled with high-quality schools drive these results, but community investments alone cannot.”
So I’m tending to conclude that the fault lies with Brooks, not Fryer. But then based on your post, I sense you see it differently? Maybe they are both over stating their cases?
-
May 11th, 2009 6:26 pm
Can I get a link to number of ESL and special education students? I’m having trouble finding those numbers.
-
May 11th, 2009 6:59 pm
Vanessa, probably the best source is the New York State School Report Card for 2007-08, available at https://www.nystart.gov/publicweb-rc/2008/1a/AOR-2008-310500860864.pdf. You’ll see that 7 of the 600 students, or 1%, enrolled in HCZ Promise Academy I in 2007-08 were Limited English Proficient. 5 of the 85 students in 3rd grade, or about 6%, were identified as students with disabilities.
The percentage is higher in the middle grades: 14% in grade 7, and 21% in grade 8. In 2007-08, 91% of the students attending HCZ Promise Academy I were Black or African-American, and 9% were Latino. -
May 11th, 2009 10:25 pm
It takes a bit of chutzpah for all of these people to be so dismissive towards Brooks (”gullible,” “sloppy journalism,” “full of it,” etc.) simply because he believed in an article written by Roland Fryer. Are you all really intending to imply that someone with Fryer’s outstanding reputation and analytical power is actually a hoaxster so obvious that even journalists should immediately see through him? Be ready to eat crow, if that’s your claim.
-
May 11th, 2009 11:07 pm
Stuart, I guess you and I have different standards for what counts as good journalism. I expect someone who tosses around terms like standard deviations, and who makes bold claims about the effectiveness of school reform models, to do his homework before putting those claims before a national audience. Are you suggesting that because Roland Fryer has an outstanding reputation that we should forget everything we know about just how intractable the gaps in academic performance between more and less advantaged children and youth have proven to be over the past half-century? Exactly what is the nature of the crow you would have me eat?
-
May 12th, 2009 12:38 am
Matthew, I acknowledge that there’s some ambiguity around the language of “closing” the achievement gap. A reduction is closing the gap, but that’s not the same as eliminating it. I was surprised by the boldness of Fryer’s claims, and, I think, most concerned about the failure to take account of the low-stakes Iowa test results. A footnote in Fryer and Dobbie’s paper states, “…we have the results of low-stakes interim test scores given by the charter schools for internal instruction purposes. Student performance on the low-stakes tests is comparable to the high-stakes tests.” This may not refer to the Iowa tests, but the Iowa tests are part of the school’s accountability plan, and I would think that the authors would be aware of the results. As for Brooks, I like Corey Bower’s take, http://www.edpolicythoughts.com/2009/05/sunday-commentary-sale-on-snake-oil-at.html
-
May 12th, 2009 2:38 am
[...] unsubstantiated claims about what I’m sure are genuinely good schools in many ways. And, as Aaron Pallas’s analysis of Roland Fryer’s claims about the Harlem Children’s Zone illustrate, this is not an isolated incident. It is what happens when you write the marketing copy [...]
-
May 12th, 2009 2:57 am
I must say that I’m surprised that no one has raised the point that even if the gains are real (as opposed to being due to ‘teaching to the test’, or other de-legitimizing factors), they are not what they seem, i.e. evidence of a closure of the black-white achievement gap. That’s because you are comparing an almost certainly non-representative group of black students with the AVERAGE white student in New York state. It’s clearly arguable that for a whole range of traits correlated with academic success these black children, BEFORE entering the ‘miracle’ schools, are at the high end—children of parents who go to great lengths to get their children into such schools are properly assumed by researchers to be so different from the average black child that when making comparisons a researcher must do so only with CHILDREN WHO LOST THE LOTTERY TO ENTER THESE SCHOOLS! These researchers know that you can’t compare them to the average black child at an ordinary public school and draw conclusions about differences in the schools, because you would not have been controlling that crucial variable—THE CHILDREN THEMSELVES!
And for exactly the same reason you cannot compare the achievement of the black children at the ‘miracle’ schools to the AVERAGE white child’s achievement and declare the black-white achievement gap to be closed, let alone that the closure is due to the ‘miracle’ schools. For the black-white achievement gap to be declared ‘closed’, you must compare the AVERAGE black child with the AVERAGE white child. Or, at the very least, compare these atypical black children who have hyper-involved parents to equally atypical white children with similarly hyper-involved parents, and if they perform equally well, then you can say that perhaps a ROUTE to the eventual closure of the achievement gap between the AVERAGE children of each race has been found.
-
May 12th, 2009 4:49 am
[...the gaps in academic performance between more and less advantaged children...]
But the evidence indicates that it doesn’t really have anything to do with advantaged vs disadvantaged, however you may want to define those terms (hopefully in some reasonable way). I say this due to the following facts:
Black children from the wealthiest families have mean SAT scores lower than white children from families below the poverty line.
Black children of parents with graduate degrees have lower SAT scores than white children of parents with a high-school diploma or less.
I think most people would say that in these (very wide) statistical samples the white kids are ‘disadvantaged’ relative to the black kids, and yet on average the white kids still outperform the black kids.
-
May 12th, 2009 9:12 am
As for what we know about the achievement gap from the past half-century: off the top of my head, we know that merely increasing spending hasn’t done much (Hanushek); desegregation didn’t do much either (Cook et al. 1984); class size reduction provided small gains in the Tennessee STAR experiment (Krueger) but that didn’t scale up when implemented across California (Jepsen/Rivkin); vouchers provide small gains in most randomized studies, but it’s yet to be seen what a broader program would do; charter schools nationwide look about the same as other public schools, but certain places (the recent Boston study) or models might be better than the nationwide average.
So I don’t know: maybe you’re right — along with Charles Murray — that journalists should turn up their noses at any new study that purports to have found a promising solution. At the same time, op-ed columnists don’t have time or training to research every issue they write about from scratch . . . and I’ve seen Paul Krugman uncritically report the results of a study (on supposed Republican prosecutions of local Democratic politicians) that was unspeakably less credible than Fryer’s. So I don’t know: How gullible does a journalist have to be to believe Roland Fryer?
-
May 12th, 2009 9:12 am
I had better math and reading scores (higher than 33%tile on average) on the Stanford test (comparable to ITBS) from middle school ESL kids. Our ESL kids have to hit the 40th %tile on ELA to get out of ESL and into mainstream. These results are far from miraculous, more like pathetic.
-
May 12th, 2009 9:20 am
[...] An excerpt from another skeptic: Today’s New York Times column by David Brooks touts a new study by Roland Fryer and Will Dobbie of the Harlem Children’s Zone (HCZ) Promise Academy charter schools, two celebrated schools in Harlem. Fryer and Dobbie’s finding that the typical eighth-grader was in the 74th percentile among New York City students in mathematics leads Brooks to state that HCZ Promise Academy eliminated the black-white achievement gap. He’s so dumbstruck by this that he says it twice. Brooks takes this evidence as support for the “no excuses” model of charter schools, and, claiming that “the approach works,” challenges all cities to adopt this “remedy for the achievement gap.” [...]
-
May 12th, 2009 1:16 pm
Aaron,
Thanks for the response.
Maybe I was too verbose in my original query, but I fear the larger issue is getting lost in all the usual vitriol. (and now that I look at it, the post is not a lot shorter)
Let’s set aside the question of David Brook’s and Paul Krugman’s responsibility to know what they’re talking about before they write. (Lord knows the media are in enough economic trouble these days without requiring responsibility of reporters and essayists. One can only imagine what we’d have to pay in unemployment to support former Fox News staff if such a standard were imposed broadly).
Fryer (as Jennifer observed earlier) is making some pretty dramatic news when he writes about HCZ moving children up by 1.3 or more standard deviations. Whether this is due to the community bundle, the charter schools or the return of Jesus, we’d all presumably love to hear about such a development, and figure out if it can be replicated.
Your analysis above seems to suggest that these gains are not all that dramatic, or maybe they don’t even exist. (Am I missing the point?) You furthermore note there are associated data like the results on the ITBS that seem to hint the HCZ children’s performance on the NY state exams was anomalous. In which case, perhaps there were no (or smaller) gains than Fryer reports.
Now I can see where one could agree simultaneously with both you and Fryer: To whit: the kids come in to HCZ doing so poorly on standard measures that even a gravity-defying improvement of 1.3 STDEVs only leaves them at the point where they are at the 74th percentile relative to the white population
Is that a fair summary of the issue? To me the far larger issue would be Fryer getting it wrong than Brooks. The way I read it, Fryer hedge himself on a number of occasion in the paper, so it seems appropriately cautious, but not unexpectedly, hopeful.
As for Brooks, since I never had any expectations of him, it was hard to be disappointed. (This points us to that large body of research on the benefits of high expectations on student performance. Maybe we need to send David the message that we know he can do better, rather than criticizing him for his failures?)
-
May 12th, 2009 1:41 pm
I’m not sure how you can say there are 5 students with disabilities, that’s just the number of students with disabilities who took the test. Technically, the school could have more students with disabilities who did not take the test, correct?
-
May 12th, 2009 2:28 pm
Vanessa, my understanding of the NCLB rules is that at least 95% of students overall and in accountability subgroups are required to participate in state assessments for a school to achieve AYP. The HCZ Promise Academy 2007-08 state report card indicates that 32 students with disabilities were continuously enrolled in 2007-08 in the three grades for which state data were reported (3, 7 and 8), which is the also the sum of the number of students with disabilities who were tested in those grades. Because there are fewer than 40 students in the disability category, a participation rate is not reported in the state report card, but I’m inferring that all of the students with disabilities did in fact participate in the assessment, and that therefore there were 5 students with disabilities in grade 3 in 2007-08. For the school-administered Iowa Tests, no students were excluded from testing on the basis of their IEP or ELL status.
-
May 12th, 2009 4:32 pm
So, the number you are using is for students in testing grades, but does not include the number of special education students in K, 1, and 2, for example.
-
May 12th, 2009 4:40 pm
$36 million in one year? This reminds me of the film “The Hungarian Orange”
-
May 12th, 2009 9:49 pm
[...] Just How Gullible Is David Brooks? | GothamSchools [...]
-
May 13th, 2009 8:55 pm
[...] Mea Culpa: Aaron Pallas did a terrific analysis of HCZ’ test results last week which I overlooked. Do have a look. [...]
-
May 14th, 2009 11:46 am
Matthew, good question. I share it.
My opinion is that some of the debate has shifted over the last year.
Was: Do kids in No Excuses schools make big gains, or is it all a charade of selection bias and possibly teaching to the test?
Now: More and more people concede kids in these schools make legitimate big gains.
New question: Do any of these schools “fully” close the achievement gap, and what would that mean exactly?
This particular kerfuffle thread reminds me of the one for the Boston Foundation study by Tom Kane. Scholar finds pretty big gains. Hopeful and excited. Someone (Klein, Brooks) touts it, possibly simplifying or overstating.
Aaron P perceives overstatement. Finds some other data that would question whether gap is “fully” closed.
I tend to share this part of his sentiment. In the charter where I work: giant value-add gains and higher scores than white students on state tests, but I’m not sure how exactly to evaluate “fully”…certainly in vocabulary and other ways I think our kids remain behind (though far less so than the lottery losers who don’t attend our school).
Where I suspect I agree with with Aaron is No Excuses schools still have a lot of untapped upside, and the state test mastery might lead to resting on laurels.
But in appropriately questioning perceived overstatement, it seems Aaron chucks baby with bathwater.
I think his message both times could have been stated as “Promising, but let’s hold on, not overreach.”
Instead the language is “Gullible” and “Fugitive from the Facts” or lauding someone else who calls it “Snake Oil.”
-
May 14th, 2009 1:13 pm
GGW-
Great post. I think you’re right about untapped upside in No Excuses charter schools.
Seems that there are 4 levels of success that urban charter schools should be aiming for:
1) Get higher test scores than the surrounding traditional schools. This allows schools to get rechartered.
2) Erase the achievement gap by matching the success levels of white kids (or non-low income kids) across the state.
3) Match or surpass the test score levels of the highest performing districts in the state.
4) Get kids to be comparably well educated as their white, wealthy peers.
Reaching the first 2 levels might be achievable within the current structures of No Excuses schools, namely: high expectations, longer school days, strong school culture, discipline and a lot of regimented instruction. KIPP chants and timed and recorded “mad minutes” come to mind.
For our kids to honestly get to level 4 will probably require a much greater focus on excellent teaching, more years within the same high functioning schools (4 years of HS probably won’t cut it, as you guys at MATCH seem to have figured out with your new MS), and more wrap-around services. It’s not that we won’t be able to get kids to that level, but it won’t be easy.
BTW, I’ve never been particularly impressed with the teaching I’ve seen at some of the highest performing charters in the nation (Community Day in Lawrence, MA and North Star come to mind). Can you imagine what kind of education their kids could get if they developed truly excellent teachers?
-
May 14th, 2009 5:41 pm
GGW, the issue is more than just Joel Klein’s and David Brooks’ embellishments about the extent to which particular schools or reform initiatives have closed the Black-white achievement gap. I am just as critical of the subsequent claims about the features of those schools and reform initiatives that account for their success. Take, for example, Klein’s assertion about “no-excuses schools” and the Boston Foundation study of charter, pilot and traditional schools. In his US News piece, he writes, “The results suggest that the freedom conferred on charters to hire teachers and principals and to shape school culture made a huge difference in subsequent student performance.” The report itself said the following: “Finally, it also bears emphasizing that our study is not designed to uncover why or how Charter Schools and Pilot Schools might change test scores. Rather we focus on the narrower though still important question of whether different types of schools produce significant achievement gains.
For the moment, we cannot say which educational strategies or characteristics are most valuable in each school setting, though that is a question we hope to address in future work. Thus, it’s important to keep in mind the fact that there might be many reasons for a school’s success:
instructional focus, student/teacher ratios, staff qualifications or background, use of tutors,
and length of school day, to name a few.” When columnists, policymakers and politicians ignore caveats about what a study can or cannot reveal about mechanisms, I think terms such as “gullible” and “snake oil” are entirely appropriate. -
May 14th, 2009 8:06 pm
Yes, but Aaron, you devote WAY more time and energy disproving Klein’s erroneous claims than you spend debunking the (at least) equally invalid claims of his opponents. Your one-sidedness betrays an ideology that I feel dictates the results of your analysis, rather than vice versa. It’s not an unusual approach, but it’s a disappointing one given your skill with numbers and your potential to get at some real truths if you were looking for them in equal measure on all sides of the arguments.
-
May 16th, 2009 12:28 pm
I’ve been really enjoying this thread, everyone. Awesome thinking going on.
I want to repeat the question that I see being asked of Aaron by many people: Are we to give up on “no-excuses” schools altogether, or are we just to be more realistic about exactly how much they are closing the achievement gap? You point out that (a) students still under-perform white students and (b) the student body may not be exactly the same at Promise Academy than at control-group public schools. Are you also suggesting that (c) the disappointing Iowa results suggest that the Promise Academy is not meaningfully improving academic outcomes?
In other words, should we take your analysis to suggest that the Promise Academy might have made no meaningful academic improvements at all? I would think that we should be impressed by the standard deviation #s that Eduwonkette says are unprecedented. But are you suggesting that if Fryer and Dobbie repeated their study using Iowa test results, the effect size would probably be much smaller?
In that case, it would seem to me that your suggestion is that not only has the achievement gap not been eliminated, but it may not have been narrowed, either.
-
May 16th, 2009 4:11 pm
Elizabeth, my critique of David Brooks’ claims in his column has little to do with judgments about the promise of “no-excuses” models of schooling. I was focused more narrowly on the logic of an argument that went something like this: (1) the Harlem Children’s Zone schools have produced enormous gains in student achievement, and eliminated the black-white achievement gap; (2) the Harlem Children’s Zone schools are “no-excuses” schools; (3) therefore, the remedy to the achievement gap is to replicate “no-excuses” schools. I pointed out that the evidence for (1) was highly questionable; the enormous gains on the state assessments were observed for a single cohort of students, students who entered the school as 6th-graders in 2006, and the performance of HCZ students on another assessment, the Iowa Tests of Basic Skills, was undistinguished.
As a general rule, I think it’s a bad idea to rely on a single assessment to make judgments about the efficacy of schools, or educational programs or policies. We expect public schools to contribute in so many ways to the social and intellectual development of children and youth, and no single measure can come close to capturing the full spectrum of goals that we have for our public schools. The gains that HCZ demonstrated on the state ELA and math tests are impressive; the lack of progress on the Iowa Tests is a counterweight to that growth. High-stakes tests such as the state assessments are, as Dan Koretz explains in detail in his book Measuring Up, susceptible to score inflation. Given the discrepancy between the patterns of performance of HCZ students on the state assessments and the Iowa tests, I think it is extremely likely that the effect of attending HCZ Promise Academy would be much smaller if Roland Fryer and Will Dobbie had tested HCZ lottery winners and losers.
But the question of the extent to which a model school reform is able to “level the playing field” for Black and white children is one that, in my view, requires evidence beyond effects on test scores. Some of these outcomes — e.g., the possibility that HCZ lottery winners will be more likely to graduate from high school and go to college than lottery losers — won’t be observed for many years. It’s just way too premature to be declaring that this model is the “equivalent of curing cancer for these kids,” as Brooks quotes Roland Fryer.
-
May 16th, 2009 4:55 pm
Gotcha. This clears things up for me - and I hope for others, too.
-
May 17th, 2009 10:02 pm
Aaron, I’m puzzled. In your latest post (May 16 at 4:11 P.M) you say:
“Given the discrepancy between the patterns of performance of HCZ students on the state assessments and the Iowa tests, I think it is extremely likely that the effect of attending HCZ Promise Academy would be much smaller if Roland Fryer and Will Dobbie had tested HCZ lottery winners and losers.”But David Brooks, in the column that sparked all this, says they DID test the lottery winners and losers:
“Fryer and his colleague Will Dobbie have just finished a rigorous assessment of the charter schools operated by the Harlem Children’s Zone. They compared students in these schools to students in New York City as a whole and to comparable students who entered the lottery to get into the Harlem Children’s Zone schools, but weren’t selected.”
Comparison to an appropriate control is the most critical factor in research of this type; the appropriate control here is the HCZ lottery losers. In fact, the reason I paid attention to this study in the first place was David Brooks’s assertion regarding this aspect. If they didn’t have such a control the research is worthless and everyone’s criticisms are valid. If they did, then their claims about the schools are far more potent. So what is the truth Aaron? -
May 17th, 2009 10:13 pm
Christina, sorry, my language was sloppy. Fryer and Dobbie did compare lottery winners and losers on the New York State assessments, but they did not do so for the Iowa Tests, which are observed only for the HCZ attendees. My assertion was that it is likely that the estimated effect of attending HCZ Promise Academy, comparing lottery winners and losers, would be smaller if the outcome were the Iowa Tests than the New York State assessments, since the performance of HCZ Promise Academy attendees on the Iowa Tests was undistinguished. I hope this clarifies things.
-
May 17th, 2009 10:58 pm
Aaron,
If the state test scores are legitimate because they studied lottery winners and losers, and the state score value-add at HCZ is significant, what do you suppose explains their poor performance on the ITBS? I can think of a few possibilities:
1. The tests are testing two completely different sets of skills, and HCZ taught to the state test.
2. The test-prep HCZ did for the state test didn’t translate to the ITBS for whatever reason
3. They cheated on the state test.
4. The testing conditions on the ITBS were different than for the state test (e.g. the students didn’t take the test seriously).
5. A different group of kids took the state test vs. the ITBS.It could, of course, be a combination of these factors. Unless one of #3-5 accounts for the score differences, this would suggest that test prep or teaching to one test can have a dramatic false effect. I’d be interested to hear how dramatic that effect can be. Do you know of any studies of test-prep effects or test-focus effects? Or for that matter, testing environment effects?
-
May 18th, 2009 12:05 am
Socrates,
I’d encourage you to read the chapter “Inflated Test Scores” in Dan Koretz’ book Measuring Up. He describes seven different types of test preparation, several of which—e.g., working more effectively, teaching more, and working harder—are typically viewed as legitimate, and others of which—e.g., cheating, reallocation of instructional resources toward the content of a particular test, and coaching—are potentially more problematic. There are studies that document a strong likelihood of teacher cheating (e.g., Brian Jacob and Steven Levitt’s studies of the Iowa Test of Basic Skills in Chicago). There also are studies that show implausible discrepancies in students’ performance on high-stakes and low-stakes assessments (e.g., Stephen Klein and his colleagues’ studies of the Texas Assessment of Academic Skills and the National Assessment of Educational Progress, and the studies by Ron Hambleton, Dan Koretz and their colleagues of the Kentucky Instructional Results Information System). My colleagues Jennifer Jennings and Andy Beveridge have a paper in the June, 2009 issue of Educational Evaluation and Policy Analysis demonstrating that schools in Houston strategically exempted special education students from the high-stakes Texas Assessment of Knowledge And Skills test in ways that, based on the performance of these students on the low-stakes Stanford Achievement Test, distorted school passing rates and the magnitude of the achievement gap.
I have no reason to think that there is cheating going on at HCZ, and at least in reading, it appears that the same students took both the state ELA test and the Iowa test. Based on my understanding of the content covered by the respective exams, I don’t believe that they are testing two completely different set of reading and math skills. It’s certainly possible that students didn’t take the Iowa tests seriously. I’m inclined to discount this possibility because the Iowa tests are part of the school’s own accountability plan, which suggests that school staff would encourage students to try their best on the test. In my view, the most likely interpretation of the gap in students’ performance on the state test and the Iowa test—and I freely admit that I have no direct evidence of this—is that the students at HCZ are very well-schooled in the specific content and testing format represented on the New York state tests. I find the predictability of item content and format on the state assessments from year to year to be worrisome. It creates the possibility that students can do well on some types of items, but perform poorly on items with unfamiliar content and formats that nevertheless represents the domain of skills that the test is supposed to represent.
-
May 20th, 2009 9:13 am
1. Is it true that Fryer/Dobbie control for income (as Quick and Ed notes) while your little tables don’t? If so, apples and oranges.
2. Aren’t you too dismissive of the Iowa test results? You say, “Scoring at the 33rd percentile is no great success story.” But given that the national achievement gap is about one standard deviation, the average black student in America scores at about the 16th percentile. Why sneer at raising that to the 33rd percentile?
-
May 20th, 2009 9:41 am
Stuart,
I’ve responded to Chad Alderman on Quick and the Ed. As you’ll see from my discussion there, my “little tables” are addressing a different question than the one that is the primary focus of the Fryer/Dobbie report. I encourage you to read the report for yourself. I’ve tried to be pretty careful in not impugning the internal validity of the lottery winner/loser analysis reported in Fryer/Dobbie. What I am criticizing is the claim that HCZ Promise Academy has eliminated the achievement gap. “No great success story” may not be the best choice of words to reiterate that there is still a large gap in performance on the ITBS between HCZ Promise Academy students and the typical white student, as an NCE of 41 may be higher than what is observed in the national norm sample for Black students on the ITBS. But it’s obviously not evidence that the Black-white achievement gap has been eliminated. Keep in mind, too, that it’s likely that the reference sample for the ITBS NCE value is not all white students, but a nationally-representative sample that includes racial and ethnic minorities who typically score lower on such standardized tests. If I can find ITBS norms that pertain specifically to white students, I’ll be sure to post them. -
May 20th, 2009 9:04 pm
[...] in Gotham Schools, has serious problems with the research claims, and, in a response, “How Gullible is David Brooks,” picks at Brook’s claims. Pallas asks, But here’s the kicker. In the HCZ [...]
-
May 21st, 2009 12:11 pm
Sorry for the phrasing “little” there . . . not sure why that came out, but it definitely was more snarky than I intended.
-
May 21st, 2009 4:03 pm
Here’s something no one seems to talk about when examining state test scores: with the complete lack of oversight of the testing conditions at schools, are these scores even reliable? Any teacher, at any time, not to mention any administrator, can feed or fudge answers to boost scores.
I would say that Geoff Canada’s integrity is unassailable, moreso than Roland Fryer’s fact-checking ability, but it doesn’t mean that some of the people he’s hired haven’t buckled under the test score pressure.
Oh, and by the way, district schools may be cheating. It’s human nature to cheat especially when perverse incentives are introduced. Why isn’t anyone concerned about this factor?
-
May 21st, 2009 4:07 pm
I hadn’t read that post about the possibility of cheating….and I certainly wouldn’t accuse anyone without evidence. But I thought that this discussion, so hyper-focused on one single set of data points, would be a good place to raise this issue.
It would be costly, but without proper oversight I find the entire data set of state test scores to be potentially dirty.
-
May 23rd, 2009 9:01 pm
Like Kitchen Sink, I am perplexed as to why these tests, administered without oversight, are taken so seriously. In my many years as a teacher at a low-achieving school, I witnessed frenzied test-prep from September to May. Most teachers “just” taught to the test, but others taught the specific test items. Oftentimes the principal would deliver the tests to the classrooms several days before testing so the teachers could “familiarize” themselves with the “format.” So I never considered these scores valid. I have a question: When researchers from prestigious universities study the progress of schools, do they do their own testing, or do they rely on tests administered by school personnel? If it’s the latter, then we are all dealing with a lot of misinformation.
-
May 25th, 2009 7:54 am
KitchenSink and Linda, there are a couple of issues here: the validity of the state test scores, and the validity of test scores used for research purposes. In my view, we don’t pay enough attention to the social processes that generates test scores. The consequences associated with test performance create incentives for manipulation; high-stakes tests have such incentives, whereas low-stakes tests do not. Some tests (e.g, the SAT) are administered and scored under very controlled conditions that minimize opportunities for manipulation; others are not. There’s evidence of occasional overt cheating in the administration and scoring of tests, and having tests administered and scored by individuals and organizations that have no stake in the test outcomes limits the risk of cheating. But it’s expensive. Researchers studying schools will sometimes administer tests themselves, and in other instances rely on school personnel. Few researchers examine the responses to individual test items, which is generally the best way to judge if students are taking a test seriously or if there are inconsistencies that suggest cheating.
-
May 25th, 2009 12:09 pm
The old Board of Ed used software to: spot check “erasure analysis” and “patterns of answers,” and in a number of cases actually charged school personnel of participating in cheating; to the best of my knowledge the current administration has abandoned the process … how effective is this software?
-
May 25th, 2009 12:15 pm
Aaron, thank you for your answer. It is difficult for me to believe that institutions like Harvard and Columbia would rely on school personnel for test data. In my opinion doing so would completely invalidate the results of any research study.
-
May 25th, 2009 6:43 pm
Peter, I don’t know specifically what software the old Board of Ed used, and it’s difficult to judge whether these kinds of checks are able to ferret out all kinds of cheating. Scores that are unusually high relative to the year before and the year after; patterns in which students in the same classroom mark the same answer (whether or not it’s correct) for a cluster of questions; and patterns in which students are more likely to get difficult questions correct than easier ones are all potential evidence of cheating. Jacob and Levitt applied an algorithm to pick up these forms of cheating to test scores in Chicago, and estimated that there was cheating in at least 4-5% of elementary school classrooms from 1993 to 2000.
-
June 17th, 2009 10:34 am
Arron, I think you’re just as guilty as Brooks (although, Brooks is a reporter), for taking an isolated view of these results, and an isolated view of the HCZ. Clearly, you recognize that “Coming on the heels of yesterday’s release of the 2009 New York State English Language Arts (ELA) results, in which the HCZ schools outperformed the citywide white average in “GRADE 3″, but were well behind the white average in grades 4, 5 and 8, skoolboy decided to drink a bit more deeply from the datastream” Aaron, Why not look at the third grade class??
Take a closer look at the entire HCZ program, and give your readers the full facts. Or, go and read Paul Tough’s Book, because i believe that he says,
“I also think Brooks is right that the Promise Academy’s impressive results in middle-school math are as much a demonstration of the power of high-quality schooling as of the impact of the full conveyor-belt model, because the students who got those scores only arrived at the Promise Academy in 6th grade and did not receive HCZ’s full wraparound of cradle-to-college social services.
Personally, though, I’m more excited by the achievements and the future prospects of the students in Promise Academy elementary school. They are, increasingly, experiencing a more comprehensive approach to their education, one that starts earlier, involves their parents more, and integrates other social services better. New York State’s ELA results came out last week, and the Promise Academy elementary school, especially the 3rd grade, had very impressive scores — more impressive, to me, than those of the middle school. My guess is that as those students progress, and as the pipeline connecting Baby College, Harlem Gems and Promise Academy is made more and more seamless, we will see still more impressive numbers coming out of the Harlem Children’s Zone.”
So, please tell your readers that the 6th, 7th and 8th grade classes have not gone through the entire program, but still improved upon test scores, when others around the state said it could not be done past a certain age (and most often gave up on these students). The students that have been in the entire program at HCZ, which consists of more than just running a charter school, are those in the 3rd grade. The HCZ is not a 36 million dollar a year charter school, it provides many other proven services to the community and has been doing so well before they decided to get into charter schools. I assure you, that if you focus on the 3rd grade data, going forward, you will see the results, which you simply gloss over in your piece.
-
June 17th, 2009 9:08 pm
My post was not intended as an evaluation of the entirety of the Harlem Children Zone program. Rather, it was focused on a specific claim made by David Brooks based on his reading of Roland Fryer’s unpublished paper. I agree that there may be differences in the effects of the program on elementary students and middle school students, but I would like more specificity about the nature of the program at each level. You should be aware that the argument that Brooks made largely discounted the importance of the program elements that you view as especially influential, as he emphasized the “no excuses” model–which is far-removed from what you refer to as the “full wraparound of cradle-to-college social services.”
A comprehensive evaluation of the HCZ program remains to be written. But I would caution that even at the elementary-school level, the evidence is mixed. The 2007-08 annual report submitted by HCZ to the state showed substantial declines from June 2007 to June 2008 in the average NCE on the Iowa Test of Basic Skills Reading Total Score for students in first, second and third grades at HCZ Promise Academy in 2008. (In math, there was a sharp decline for first-graders, a slight increase for second-graders, and a slight decrease for third-graders.) The performance of these students in the long-run may be more promising, but the jury is definitely still out.
-
August 3rd, 2009 6:49 pm
[...] are trained to do well on a particular test doesn’t mean they’ve mastered certain skills. As the Columbia University professor Aaron Pallas pointed out, on a different assessment — the Iowa Test of Basic Skills — eighth-grade Promise [...]
-
August 23rd, 2009 12:29 pm
So, I heard the This American Life story (http://www.thislife.org/Radio_Episode.aspx?sched=1311); I read the David Brooks opinion; I found this blog; then I looked for the actual study (http://www.economics.harvard.edu/faculty/fryer/files/hcz%204.15.2009.pdf). The actual study told me more than most. I’d direct you to page 34 of the study (35 in Adobe Acrobat pages). There you’ll find two sets of quite interesting figures. They compare a select group of lottery ‘winners’ to a control group of lottery ‘losers.’ (The control group has the unfortunate label ‘losers.’) With the comparison to a control group, the results are even more startling. There is no significant difference in the ELA scores between the study group and the control group in the Fifth, Sixth, and Seventh grades. The Eighth grade group shows gains. In math, there are visible differences between the two groups, with the most notable gain coming in the Eighth grade. These results suggest to me that further study is warranted and that any conclusions as to the efficacy of the program are premature, at best.
-
August 23rd, 2009 2:51 pm
Michael: There are actually three categories … “winners,” “losers,” and non-applicants …
1. What are the characteristics of parents who apply, and don’t apply?
2. In addition to the lottery does the school require an interview? How many families are rejected as a result of the interview? and why?
3. How many students were admitted outside of the lottery process? The lottery selections are not monitored and anecdotally schools admit that “some” students are enrolled outside of the process.
4. Do charter schools require parent involvement, and, do they reject parents who are unable to commit to this requirement?
Regular public schools and charter schools with high levels of parent involvement, parents with “social capital,” are almost always characterized by high student success.
-
August 23rd, 2009 4:15 pm
Well, I think that emphasizes my point. These are motivated students with motivated families. This program isn’t doing anything special for these kids for ELA. For math, I suspect there’s either a very good teacher involved, or they’ve lucked into an exceptional co-hort. There’s no evidence that the program works better than any other.
-
August 23rd, 2009 5:04 pm
What I find peculiar about Fryer’s study, Brooks’ column, and Klein’s rehashing of the results, is that while they all mention the social and health services of HCZ, the longer school year and day, the supposedly “quality” teachers, etc. etc., as possible contributors to its success, not one of them mentions the fact that the class sizes at HCZ are “tiny”, according to the Washington Post and elsewhere.
More specifically, classes are capped at 20 with one teacher and a teaching asst. in each class; and one teacher and two teaching assts. in Kindergarten.
This is despite the fact that Fryer cites Krueger’s finding of a relatively large effect size in earlier class size experiments like STAR. Fryer also notes that boys seem to benefit the most in the HCZ sample, esp. in math, though in ” the vast majority of the program-evaluation literature, boys are much less likely than girls to absorb the benefits of treatment (e.g. Sanbonmatsu et al., 2006; Hastings et al., 2006; Anderson, 2008). One exception to this pattern is class size, which also appears to benefit boys more than girls (Krueger, 1999).”
Another strange thing about the Fryer study is his discussion of attrition levels — often quite high in charter schools:
“First- (and second-) year attrition from the Promise Academy is 8.5 (17.6) percent for the elementary school and 15.6 (26.4) percent for the middle school, compared to 9.7 (17.0) percent for KIPP Star – a popular charter middle school two blocks from the western boundary of the Zone – and 16.4 (30.3) for the middle schools most demographically similar to HCZ. Demographically similar elementary schools all experience first-year attrition of over 40 percent..”
Demographically similar public elementary schools in NYC experience first year attrition of over 40%? That means that 40% of Kindergarten students do not make it to 1st grade? I find that hard to believe. Perhaps instead they are being held over.
In any event, the results for middle schoolers are for students who entered through lottery in the sixth grade. They have now eliminated the sixth grade lottery and will only enroll kids who started in Kindergarten or before. So we may see more gains in the future — even on the Iowa tests.
-
September 30th, 2009 3:38 pm
[...] among other — for helping to reduce the achievement gap, although other studies, including an analysis by Aaron Pallas, have called those claims into question. And reviewing this year’s school [...]
-
January 27th, 2010 3:44 pm
I hate to break it to you, but your eyeball assessment of academic performance data is a joke. If that’s how we analyzed data, there would be no such thing as statisticians or applied economists. Just because you talk about standard deviations doesn’t mean you’re talking about regressions and standard errors. Stick with what you know.
Leave a Reply
Mapping the Budget Cuts
64 Comments
Subscribe to comments with RSS or TrackBack