If you are well connected to the IES, why do you feel the need to personally attack Pallas on his review of Brooks’ article rather than present substantive arguments? Is that the new approach to research or do you have a connection to Brooks/Dobbie/Fryer/Canada?

Nothing you have written contributes any substance, pro or con, to the issues. I encourage you to read the HCZ study and annual report before regressing further. If it isn’t good research or an effective program, it is not – no amount of drilling down can change that.

I’m curious what work Dr. Averch did to earn his Nobel Prize in economics. I’m also curious if you would care to share what works you’ve done with these individuals, but I definitely understand if you use an alias and post anonymously, as I know most researchers do in online discussions.

In addition, I currently work on an $8 million grant funded by the IES. I am fully aware of the new econometric techniques being developed to study education issues specifically and how interested the government is in reforming the way we study education and replacing these outdated methods. Of course these new methods will still have their limitations, but the goal is to maximize the amount of information extracted from the data with the constraint of the problematic data itself.

Thanks for declaring your background – I was curious about the “bait & switch” responses. I was drafted into Education Research from the Ph.D. in Public Administration & Public Policy with the likes of Drs. Harvey Averch (Nobel Prize economist) and Anne Witte (NBER econometrician). Thus, I understand your background. However, even with the selection of the appropriate technique(s), one must always note the limitations. Below are links for you to review.

The Dobbie-Fryer HCZ study can be found at
http://www.economics.harvard.edu/faculty/fryer/files/hcz%204.15.2009.pdf

The HCZ Charter School Annual Report can be found at
http://www.economics.harvard.edu/faculty/fryer/files/hcz%204.15.2009.pdf

The Federal evidence standards I referred to can be found at http://ies.ed.gov/ncee/wwc/pdf/wwc_version1_standards.pdf

Dobbie-Fryer have not released the data from which they base their study, so I am not able to replicate their results; therefore, I am skeptical. I am also skeptical of the “federal standards” that you seem so amazed by. I am a PhD education economist, and I was trying to let you talk like you knew what you were talking about. I work with leading econometricians in panel data, and they think these standards you speak of are a joke. The problem with social science is statisticians have been evaluating social science data, which is best for controlled experiments, such as those in the natural and physical sciences. Focusing on efficient estimators leads to the obvious point that statisticians are okay with biased estimators; econometricians are not. You are talking to me from a statistics background, and I am talking to you from an econometrics background, which I’m sure you know is a highly specialized form of applied statistics designed for dealing with the problems that come with imperfect data. I therefore am partial towards pure econometrics techniques; you are not. Hence, there is nothing I can say to make you think my techniques are superior and vice verse.

Have you actually read the Dobbie-Fryer research study from which Brooks’ article was based? Do you know the Federal standards for the evaluation of educational programs? Do you know that ANOVA and its variants (e.g., ANCOVA, MANOVA, and GLM) are specialized forms of Regression Analysis? A skilled researcher chooses the procedure that fits the data so as to adequately answer the research/evaluation question(s) – not the other way around.

The technique used is led by the research question. If you goal is to identify the factors/variables that impact an outcome measure (e.g., test scores), then you first choice is regression analysis or its variants. If you goal is to find out if a treatment/program group performed better than a control/comparison group, then your first choice is ANOVA or its variants.

The technique used is also led by the data/variables. If the independent and dependent variables are all interval/continuous then OLS regression is best. If the dependent variable (e.g., group) and independent variables (e.g., ELL, ESE, FRL, Race, etc.) are nominal/discrete, therein violating Regression Assumptions, then ANOVA or its variants is best.

As economists typically seek to identify which exogenous/independent variable(s) have a significant impact on an endogenous/dependent variable, their training normally focuses on regression analysis. As educational researchers typically seek to evaluate the effectiveness programs or policies, their training normally focuses on ANOVA and its variants.

A good quantitative researcher will master both approaches and apply them appropriately. I have worked with and been trained by econometricians who use ANOVA. Because of ethics and laws that regulate US education, effective research and program evaluations in this field typically employ field/quasi-experiments with careful attention to internal and external validity.

In the case of the HCZ article and the Fryer-Dobbie study, knowledge of advanced research and statistics is not needed to realize that the program did not achieve its goals, was not effective, and cannot sustain any improvements, even with the power that charter school administrators have to terminate teachers/staff and remove whole student grade groups, at will.

I am pleased that you have challenged Brooks’ misunderstanding of the results of the Fryer-Dobbie study. While I pray for policies and programs that will truly close the achievement gap between White-Asian and minority students (i.e., record a disordinal interaction effect), it is misleading reports such as these that waste needed funds to achieve the “Dream.”

As noted, there was a one-time, one-grade, one-subject “promise” of academic improvement. However, the achievement gap was NEVER closed as per Figures 3A to 4B. HCZ students performed below White students in Mathematics and for all three years of the study the gap between Whites and Blacks in English language arts remained virtually the same.

None of the analyses employed a pre-test and post-test design with matched comparison groups therein making it difficult to attribute gains to the HCZ program. Further, despite manipulation of the subjects, student performance in both reading and mathematics failed to meet the HCZ charter school’s objective of an average NCE of 50.

Finally, there was no evidence presented in the study regarding fidelity of implementation and the econometric analyses attempted acted as a “snow job” to hide the bare facts. Although standard deviations were reported, effect sizes were not computed for any of the analyses, leaving doubt about the relative educational value of the HCZ program.

Please note that Regressions (non-experimental) analysis may address causality but cannot replace field/quasi-experimental designs, with Repeated Measures of ANOVA.

More specifically, classes are capped at 20 with one teacher and a teaching asst. in each class; and one teacher and two teaching assts. in Kindergarten.

This is despite the fact that Fryer cites Krueger’s finding of a relatively large effect size in earlier class size experiments like STAR. Fryer also notes that boys seem to benefit the most in the HCZ sample, esp. in math, though in ” the vast majority of the program-evaluation literature, boys are much less likely than girls to absorb the benefits of treatment (e.g. Sanbonmatsu et al., 2006; Hastings et al., 2006; Anderson, 2008). One exception to this pattern is class size, which also appears to benefit boys more than girls (Krueger, 1999).”

Another strange thing about the Fryer study is his discussion of attrition levels — often quite high in charter schools:

“First- (and second-) year attrition from the Promise Academy is 8.5 (17.6) percent for the elementary school and 15.6 (26.4) percent for the middle school, compared to 9.7 (17.0) percent for KIPP Star – a popular charter middle school two blocks from the western boundary of the Zone – and 16.4 (30.3) for the middle schools most demographically similar to HCZ. Demographically similar elementary schools all experience first-year attrition of over 40 percent..”

Demographically similar public elementary schools in NYC experience first year attrition of over 40%? That means that 40% of Kindergarten students do not make it to 1st grade? I find that hard to believe. Perhaps instead they are being held over.

In any event, the results for middle schoolers are for students who entered through lottery in the sixth grade. They have now eliminated the sixth grade lottery and will only enroll kids who started in Kindergarten or before. So we may see more gains in the future — even on the Iowa tests.

1. What are the characteristics of parents who apply, and don’t apply?

2. In addition to the lottery does the school require an interview? How many families are rejected as a result of the interview? and why?

3. How many students were admitted outside of the lottery process? The lottery selections are not monitored and anecdotally schools admit that “some” students are enrolled outside of the process.

4. Do charter schools require parent involvement, and, do they reject parents who are unable to commit to this requirement?

Regular public schools and charter schools with high levels of parent involvement, parents with “social capital,” are almost always characterized by high student success.

A comprehensive evaluation of the HCZ program remains to be written. But I would caution that even at the elementary-school level, the evidence is mixed. The 2007-08 annual report submitted by HCZ to the state showed substantial declines from June 2007 to June 2008 in the average NCE on the Iowa Test of Basic Skills Reading Total Score for students in first, second and third grades at HCZ Promise Academy in 2008. (In math, there was a sharp decline for first-graders, a slight increase for second-graders, and a slight decrease for third-graders.) The performance of these students in the long-run may be more promising, but the jury is definitely still out.

Take a closer look at the entire HCZ program, and give your readers the full facts. Or, go and read Paul Tough’s Book, because i believe that he says,

“I also think Brooks is right that the Promise Academy’s impressive results in middle-school math are as much a demonstration of the power of high-quality schooling as of the impact of the full conveyor-belt model, because the students who got those scores only arrived at the Promise Academy in 6th grade and did not receive HCZ’s full wraparound of cradle-to-college social services.

Personally, though, I’m more excited by the achievements and the future prospects of the students in Promise Academy elementary school. They are, increasingly, experiencing a more comprehensive approach to their education, one that starts earlier, involves their parents more, and integrates other social services better. New York State’s ELA results came out last week, and the Promise Academy elementary school, especially the 3rd grade, had very impressive scores — more impressive, to me, than those of the middle school. My guess is that as those students progress, and as the pipeline connecting Baby College, Harlem Gems and Promise Academy is made more and more seamless, we will see still more impressive numbers coming out of the Harlem Children’s Zone.”

So, please tell your readers that the 6th, 7th and 8th grade classes have not gone through the entire program, but still improved upon test scores, when others around the state said it could not be done past a certain age (and most often gave up on these students). The students that have been in the entire program at HCZ, which consists of more than just running a charter school, are those in the 3rd grade. The HCZ is not a 36 million dollar a year charter school, it provides many other proven services to the community and has been doing so well before they decided to get into charter schools. I assure you, that if you focus on the 3rd grade data, going forward, you will see the results, which you simply gloss over in your piece.