Just How Gullible Is David Brooks?

Eye on Education

May 8, 2009

Just How Gullible Is David Brooks?

by Aaron Pallas, at 5:20 pm

Now that I have your attention … Today’s New York Times column by David Brooks touts a new study by Roland Fryer and Will Dobbie of the Harlem Children’s Zone (HCZ) Promise Academy charter schools, two celebrated schools in Harlem. Fryer and Dobbie’s finding that the typical eighth-grader was in the 74^th percentile among New York City students in mathematics leads Brooks to state that HCZ Promise Academy eliminated the black-white achievement gap. He’s so dumbstruck by this that he says it twice. Brooks takes this evidence as support for the “no excuses” model of charter schools, and, claiming that “the approach works,” challenges all cities to adopt this “remedy for the achievement gap.”

Coming on the heels of yesterday’s release of the 2009 New York State English Language Arts (ELA) results, in which the HCZ schools outperformed the citywide white average in grade 3, but were well behind the white average in grades 4, 5 and 8, skoolboy decided to drink a bit more deeply from the datastream. The figure below shows the gap between the average performance in HCZ Promise Academy and white students in New York City in ELA and math, expressed as a fraction of the standard deviation of overall performance in a given grade and year. The left side of the figure shows math performance, and the right side shows ELA performance.

It’s true that eighth-graders in 2008 scored .20 standard deviations above the citywide average for white students. But it may also be apparent that this is a very unusual pattern relative to the other data represented in this figure, all of which show continuing and sizeable advantages for white students in New York City over HCZ students. The fact that HCZ seventh-graders in 2008 were only .3 standard deviations behind white students citywide in math is a real accomplishment, and represents a shrinkage of the gap of .42 standard deviations for these students in the preceding year. However, Fryer and Dobbie, and Brooks in turn, are putting an awful lot of faith in a single data point — the remarkable increase in math scores between seventh and eighth grade for the students at HCZ who entered sixth grade in 2006. If what HCZ is doing can routinely produce a .67 standard deviation shift in math test scores in the eighth grade, that would be great. But we’re certainly not seeing an effect of that magnitude in the seventh grade. And, of course, none of this speaks to the continuing large gaps in English performance.

But here’s the kicker. In the HCZ Annual Report for the 2007-08 school year submitted to the State Education Department, data are presented on not just the state ELA and math assessments, but also the Iowa Test of Basic Skills. Those eighth-graders who kicked ass on the state math test? They didn’t do so well on the low-stakes Iowa Tests. Curiously, only 2 of the 77 eighth-graders were absent on the ITBS reading test day in June, 2008, but 20 of these 77 were absent for the ITBS math test. For the 57 students who did take the ITBS math test, HCZ reported an average Normal Curve Equivalent (NCE) score of 41, which failed to meet the school’s objective of an average NCE of 50 for a cohort of students who have completed at least two consecutive years at HCZ Promise Academy. In fact, this same cohort had a slightly higher average NCE of 42 in June, 2007.

Normal Curve Equivalents (NCE’s) range from 1 to 99, and are scaled to have a mean of 50 and a standard deviation of 21.06. An NCE of 41 corresponds to roughly the 33^rd percentile of the reference distribution, which for the ITBS would likely be a national sample of on-grade test-takers. Scoring at the 33^rd percentile is no great success story.

How are we to make sense of this? One possibility is that the HCZ students didn’t take the Iowa tests seriously, and that their performance on that test doesn’t reflect their true mastery of eighth-grade mathematics. The HCZ Annual Report doesn’t offer this as a possibility, perhaps because it would be embarrassing to admit that students didn’t take some aspect of their schoolwork and school accountability plan seriously. But the three explanations that are offered are not compelling: the Iowa test skills were not consistently aligned with the New York State Standards and the Harcourt Curriculum used in the school; the linkage of classroom instruction to the skills tested on the Iowa test wasn’t consistent across the school year, and Iowa test prep began in February, 2008; and school staff didn’t use 2007 Iowa test results to identify areas of weaknesses for individual students and design appropriate intervention.

If proficiency in English and math are to mean anything, these skills have to be able to generalize to contexts other than a particular high-stakes state test. No college or employer is ever going to look at the New York State ELA and math exams in making judgments about who has the skills to be successful in their school or workplace. I’m going to hold off labeling the HCZ schools as the “Harlem Miracle” until there’s some additional evidence supporting the claim that these schools have placed their students on a level academic playing field with white students in New York City.

http://ljohnson562@charter.net Linda Johnson

Like Kitchen Sink, I am perplexed as to why these tests, administered without oversight, are taken so seriously. In my many years as a teacher at a low-achieving school, I witnessed frenzied test-prep from September to May. Most teachers “just” taught to the test, but others taught the specific test items. Oftentimes the principal would deliver the tests to the classrooms several days before testing so the teachers could “familiarize” themselves with the “format.” So I never considered these scores valid. I have a question: When researchers from prestigious universities study the progress of schools, do they do their own testing, or do they rely on tests administered by school personnel? If it’s the latter, then we are all dealing with a lot of misinformation.
Aaron Pallas

KitchenSink and Linda, there are a couple of issues here: the validity of the state test scores, and the validity of test scores used for research purposes. In my view, we don’t pay enough attention to the social processes that generates test scores. The consequences associated with test performance create incentives for manipulation; high-stakes tests have such incentives, whereas low-stakes tests do not. Some tests (e.g, the SAT) are administered and scored under very controlled conditions that minimize opportunities for manipulation; others are not. There’s evidence of occasional overt cheating in the administration and scoring of tests, and having tests administered and scored by individuals and organizations that have no stake in the test outcomes limits the risk of cheating. But it’s expensive. Researchers studying schools will sometimes administer tests themselves, and in other instances rely on school personnel. Few researchers examine the responses to individual test items, which is generally the best way to judge if students are taking a test seriously or if there are inconsistencies that suggest cheating.
http://edintheapple peter

The old Board of Ed used software to: spot check “erasure analysis” and “patterns of answers,” and in a number of cases actually charged school personnel of participating in cheating; to the best of my knowledge the current administration has abandoned the process … how effective is this software?
http://ljohnson562@charter.net Linda Johnson

Aaron, thank you for your answer. It is difficult for me to believe that institutions like Harvard and Columbia would rely on school personnel for test data. In my opinion doing so would completely invalidate the results of any research study.
Aaron Pallas

Peter, I don’t know specifically what software the old Board of Ed used, and it’s difficult to judge whether these kinds of checks are able to ferret out all kinds of cheating. Scores that are unusually high relative to the year before and the year after; patterns in which students in the same classroom mark the same answer (whether or not it’s correct) for a cluster of questions; and patterns in which students are more likely to get difficult questions correct than easier ones are all potential evidence of cheating. Jacob and Levitt applied an algorithm to pick up these forms of cheating to test scores in Chicago, and estimated that there was cheating in at least 4-5% of elementary school classrooms from 1993 to 2000.
The Real Truth

Arron, I think you’re just as guilty as Brooks (although, Brooks is a reporter), for taking an isolated view of these results, and an isolated view of the HCZ. Clearly, you recognize that “Coming on the heels of yesterday’s release of the 2009 New York State English Language Arts (ELA) results, in which the HCZ schools outperformed the citywide white average in “GRADE 3″, but were well behind the white average in grades 4, 5 and 8, skoolboy decided to drink a bit more deeply from the datastream” Aaron, Why not look at the third grade class??

Take a closer look at the entire HCZ program, and give your readers the full facts. Or, go and read Paul Tough’s Book, because i believe that he says,

“I also think Brooks is right that the Promise Academy’s impressive results in middle-school math are as much a demonstration of the power of high-quality schooling as of the impact of the full conveyor-belt model, because the students who got those scores only arrived at the Promise Academy in 6th grade and did not receive HCZ’s full wraparound of cradle-to-college social services.

Personally, though, I’m more excited by the achievements and the future prospects of the students in Promise Academy elementary school. They are, increasingly, experiencing a more comprehensive approach to their education, one that starts earlier, involves their parents more, and integrates other social services better. New York State’s ELA results came out last week, and the Promise Academy elementary school, especially the 3rd grade, had very impressive scores — more impressive, to me, than those of the middle school. My guess is that as those students progress, and as the pipeline connecting Baby College, Harlem Gems and Promise Academy is made more and more seamless, we will see still more impressive numbers coming out of the Harlem Children’s Zone.”

So, please tell your readers that the 6th, 7th and 8th grade classes have not gone through the entire program, but still improved upon test scores, when others around the state said it could not be done past a certain age (and most often gave up on these students). The students that have been in the entire program at HCZ, which consists of more than just running a charter school, are those in the 3rd grade. The HCZ is not a 36 million dollar a year charter school, it provides many other proven services to the community and has been doing so well before they decided to get into charter schools. I assure you, that if you focus on the 3rd grade data, going forward, you will see the results, which you simply gloss over in your piece.
Aaron Pallas

My post was not intended as an evaluation of the entirety of the Harlem Children Zone program. Rather, it was focused on a specific claim made by David Brooks based on his reading of Roland Fryer’s unpublished paper. I agree that there may be differences in the effects of the program on elementary students and middle school students, but I would like more specificity about the nature of the program at each level. You should be aware that the argument that Brooks made largely discounted the importance of the program elements that you view as especially influential, as he emphasized the “no excuses” model–which is far-removed from what you refer to as the “full wraparound of cradle-to-college social services.”

A comprehensive evaluation of the HCZ program remains to be written. But I would caution that even at the elementary-school level, the evidence is mixed. The 2007-08 annual report submitted by HCZ to the state showed substantial declines from June 2007 to June 2008 in the average NCE on the Iowa Test of Basic Skills Reading Total Score for students in first, second and third grades at HCZ Promise Academy in 2008. (In math, there was a sharp decline for first-graders, a slight increase for second-graders, and a slight decrease for third-graders.) The performance of these students in the long-run may be more promising, but the jury is definitely still out.
Pingback: What Do School Tests Measure? - Room for Debate Blog - NYTimes.com
Michael Allard

So, I heard the This American Life story (http://www.thislife.org/Radio_Episode.aspx?sched=1311); I read the David Brooks opinion; I found this blog; then I looked for the actual study (http://www.economics.harvard.edu/faculty/fryer/files/hcz%204.15.2009.pdf). The actual study told me more than most. I’d direct you to page 34 of the study (35 in Adobe Acrobat pages). There you’ll find two sets of quite interesting figures. They compare a select group of lottery ‘winners’ to a control group of lottery ‘losers.’ (The control group has the unfortunate label ‘losers.’) With the comparison to a control group, the results are even more startling. There is no significant difference in the ELA scores between the study group and the control group in the Fifth, Sixth, and Seventh grades. The Eighth grade group shows gains. In math, there are visible differences between the two groups, with the most notable gain coming in the Eighth grade. These results suggest to me that further study is warranted and that any conclusions as to the efficacy of the program are premature, at best.
http://edintheapple peter

Michael: There are actually three categories … “winners,” “losers,” and non-applicants …

1. What are the characteristics of parents who apply, and don’t apply?

2. In addition to the lottery does the school require an interview? How many families are rejected as a result of the interview? and why?

3. How many students were admitted outside of the lottery process? The lottery selections are not monitored and anecdotally schools admit that “some” students are enrolled outside of the process.

4. Do charter schools require parent involvement, and, do they reject parents who are unable to commit to this requirement?

Regular public schools and charter schools with high levels of parent involvement, parents with “social capital,” are almost always characterized by high student success.
Michael Allard

Well, I think that emphasizes my point. These are motivated students with motivated families. This program isn’t doing anything special for these kids for ELA. For math, I suspect there’s either a very good teacher involved, or they’ve lucked into an exceptional co-hort. There’s no evidence that the program works better than any other.
http://www.classsizematters.org Leonie Haimson

What I find peculiar about Fryer’s study, Brooks’ column, and Klein’s rehashing of the results, is that while they all mention the social and health services of HCZ, the longer school year and day, the supposedly “quality” teachers, etc. etc., as possible contributors to its success, not one of them mentions the fact that the class sizes at HCZ are “tiny”, according to the Washington Post and elsewhere.

More specifically, classes are capped at 20 with one teacher and a teaching asst. in each class; and one teacher and two teaching assts. in Kindergarten.

This is despite the fact that Fryer cites Krueger’s finding of a relatively large effect size in earlier class size experiments like STAR. Fryer also notes that boys seem to benefit the most in the HCZ sample, esp. in math, though in ” the vast majority of the program-evaluation literature, boys are much less likely than girls to absorb the benefits of treatment (e.g. Sanbonmatsu et al., 2006; Hastings et al., 2006; Anderson, 2008). One exception to this pattern is class size, which also appears to benefit boys more than girls (Krueger, 1999).”

Another strange thing about the Fryer study is his discussion of attrition levels — often quite high in charter schools:

“First- (and second-) year attrition from the Promise Academy is 8.5 (17.6) percent for the elementary school and 15.6 (26.4) percent for the middle school, compared to 9.7 (17.0) percent for KIPP Star – a popular charter middle school two blocks from the western boundary of the Zone – and 16.4 (30.3) for the middle schools most demographically similar to HCZ. Demographically similar elementary schools all experience first-year attrition of over 40 percent..”

Demographically similar public elementary schools in NYC experience first year attrition of over 40%? That means that 40% of Kindergarten students do not make it to 1st grade? I find that hard to believe. Perhaps instead they are being held over.

In any event, the results for middle schoolers are for students who entered through lottery in the sixth grade. They have now eliminated the sixth grade lottery and will only enroll kids who started in Kindergarten or before. So we may see more gains in the future — even on the Iowa tests.
Pingback: Gotham Gazette - The Wonkster » Blog Archive » A Push for Charter Schools
Jessica the Economist

I hate to break it to you, but your eyeball assessment of academic performance data is a joke. If that’s how we analyzed data, there would be no such thing as statisticians or applied economists. Just because you talk about standard deviations doesn’t mean you’re talking about regressions and standard errors. Stick with what you know.
Richard G. Williams

Aaron,

I am pleased that you have challenged Brooks’ misunderstanding of the results of the Fryer-Dobbie study. While I pray for policies and programs that will truly close the achievement gap between White-Asian and minority students (i.e., record a disordinal interaction effect), it is misleading reports such as these that waste needed funds to achieve the “Dream.”

As noted, there was a one-time, one-grade, one-subject “promise” of academic improvement. However, the achievement gap was NEVER closed as per Figures 3A to 4B. HCZ students performed below White students in Mathematics and for all three years of the study the gap between Whites and Blacks in English language arts remained virtually the same.

None of the analyses employed a pre-test and post-test design with matched comparison groups therein making it difficult to attribute gains to the HCZ program. Further, despite manipulation of the subjects, student performance in both reading and mathematics failed to meet the HCZ charter school’s objective of an average NCE of 50.

Finally, there was no evidence presented in the study regarding fidelity of implementation and the econometric analyses attempted acted as a “snow job” to hide the bare facts. Although standard deviations were reported, effect sizes were not computed for any of the analyses, leaving doubt about the relative educational value of the HCZ program.

Please note that Regressions (non-experimental) analysis may address causality but cannot replace field/quasi-experimental designs, with Repeated Measures of ANOVA.
Jessica

Please note that econometrics is the best method to analyze social science data. I have never known an econometrician to use ANOVA, and “quasi experiments” with small sample sizes suffer from micronumerosity and are not accurate.
Richard G. Williams

Jessica,

The technique used is led by the research question. If you goal is to identify the factors/variables that impact an outcome measure (e.g., test scores), then you first choice is regression analysis or its variants. If you goal is to find out if a treatment/program group performed better than a control/comparison group, then your first choice is ANOVA or its variants.

The technique used is also led by the data/variables. If the independent and dependent variables are all interval/continuous then OLS regression is best. If the dependent variable (e.g., group) and independent variables (e.g., ELL, ESE, FRL, Race, etc.) are nominal/discrete, therein violating Regression Assumptions, then ANOVA or its variants is best.

As economists typically seek to identify which exogenous/independent variable(s) have a significant impact on an endogenous/dependent variable, their training normally focuses on regression analysis. As educational researchers typically seek to evaluate the effectiveness programs or policies, their training normally focuses on ANOVA and its variants.

A good quantitative researcher will master both approaches and apply them appropriately. I have worked with and been trained by econometricians who use ANOVA. Because of ethics and laws that regulate US education, effective research and program evaluations in this field typically employ field/quasi-experiments with careful attention to internal and external validity.

In the case of the HCZ article and the Fryer-Dobbie study, knowledge of advanced research and statistics is not needed to realize that the program did not achieve its goals, was not effective, and cannot sustain any improvements, even with the power that charter school administrators have to terminate teachers/staff and remove whole student grade groups, at will.
Jessica

ANOVA is not more powerful than panel data techniques, and econometricians have already developed techniques to deal with discrete variables, and they also use matching. In addition, Fryer is not releasing his methods as far as I know, so neither of us can really analyze and say for certain, can we?
Richard G. Williams

Jessica,

Have you actually read the Dobbie-Fryer research study from which Brooks’ article was based? Do you know the Federal standards for the evaluation of educational programs? Do you know that ANOVA and its variants (e.g., ANCOVA, MANOVA, and GLM) are specialized forms of Regression Analysis? A skilled researcher chooses the procedure that fits the data so as to adequately answer the research/evaluation question(s) – not the other way around.
Jessica

Richard G. Williams,

Dobbie-Fryer have not released the data from which they base their study, so I am not able to replicate their results; therefore, I am skeptical. I am also skeptical of the “federal standards” that you seem so amazed by. I am a PhD education economist, and I was trying to let you talk like you knew what you were talking about. I work with leading econometricians in panel data, and they think these standards you speak of are a joke. The problem with social science is statisticians have been evaluating social science data, which is best for controlled experiments, such as those in the natural and physical sciences. Focusing on efficient estimators leads to the obvious point that statisticians are okay with biased estimators; econometricians are not. You are talking to me from a statistics background, and I am talking to you from an econometrics background, which I’m sure you know is a highly specialized form of applied statistics designed for dealing with the problems that come with imperfect data. I therefore am partial towards pure econometrics techniques; you are not. Hence, there is nothing I can say to make you think my techniques are superior and vice verse.
Richard G. Williams

Jessica,

Thanks for declaring your background – I was curious about the “bait & switch” responses. I was drafted into Education Research from the Ph.D. in Public Administration & Public Policy with the likes of Drs. Harvey Averch (Nobel Prize economist) and Anne Witte (NBER econometrician). Thus, I understand your background. However, even with the selection of the appropriate technique(s), one must always note the limitations. Below are links for you to review.

The Dobbie-Fryer HCZ study can be found at
http://www.economics.harvard.edu/faculty/fryer/files/hcz%204.15.2009.pdf

The HCZ Charter School Annual Report can be found at
http://www.economics.harvard.edu/faculty/fryer/files/hcz%204.15.2009.pdf

The Federal evidence standards I referred to can be found at http://ies.ed.gov/ncee/wwc/pdf/wwc_version1_standards.pdf
Jessica

“I was drafted into Education Research from the Ph.D. in Public Administration & Public Policy with the likes of Drs. Harvey Averch (Nobel Prize economist) and Anne Witte (NBER econometrician).”

I’m curious what work Dr. Averch did to earn his Nobel Prize in economics. I’m also curious if you would care to share what works you’ve done with these individuals, but I definitely understand if you use an alias and post anonymously, as I know most researchers do in online discussions.

In addition, I currently work on an $8 million grant funded by the IES. I am fully aware of the new econometric techniques being developed to study education issues specifically and how interested the government is in reforming the way we study education and replacing these outdated methods. Of course these new methods will still have their limitations, but the goal is to maximize the amount of information extracted from the data with the constraint of the problematic data itself.
Richard G. Williams

How about the Averch-Johnson Effect or the Nation Study of Daycare using a hedonic (econometric) model… you do the research… Based on the veracity of your comments I have to ask if you are the lead researcher on the $8M grant or freshly minted?

If you are well connected to the IES, why do you feel the need to personally attack Pallas on his review of Brooks’ article rather than present substantive arguments? Is that the new approach to research or do you have a connection to Brooks/Dobbie/Fryer/Canada?

Nothing you have written contributes any substance, pro or con, to the issues. I encourage you to read the HCZ study and annual report before regressing further. If it isn’t good research or an effective program, it is not – no amount of drilling down can change that.

M	T	W	T	F	S	S
« Apr
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31

GothamSchools — daily independent reporting on NYC public schools

Eye on Education

May 8, 2009

Just How Gullible Is David Brooks?

Tips, questions, feedback?

Word from Our Sponsor

Follow GothamSchools

From Our Jobs Board

Featured Employers

Recent Jobs

Chalk It Up

Recent Comments

Events Calendar

RSS Feeds

Archives

Copyright Notice