GothamSchools — daily independent reporting on NYC public schools

Eye on Education

March 2, 2009

Fun with Words

by Aaron Pallas, at 7:00 am

Thursday’s Wall Street Journal has an impassioned editorial imploring President Barack Obama to block Congressional Democrats from killing the DC Opportunity Scholarship program, the nation’s only federally-funded program providing vouchers to enable poor children to attend private schools.  Former Secretary of Education Margaret Spellings–gosh, skoolboy loves that phrase–took to the pages of the Washington Post last summer to champion the program, shortly after the release of a Congressionally-mandated evaluation sponsored by the Institute of Education Sciences in the U.S. Department of Education.  I called out Madame Secretary for distorting the results of the evaluation in a post here.

Lies, misstatements, and distortions can take on lives of their own, especially when people have strong opinions about what is at stake–which is certainly true of the DC Opportunity Scholarship program.  But that’s no excuse for the WSJ to get the basic facts wrong.  Here’s what they wrote in their editorial:  “A 2008 Department of Education evaluation found that participants had higher reading scores than their peers who didn’t receive a scholarship, and there are four applicants for each voucher.”  Here’s some text from the executive summary of that evaluation:  “Across the full sample, there were no statistically significant impacts on reading achievement (effect size (ES) = .09) or math achievement (ES = .01) from the offer of a scholarship (table 3) nor from the use of a scholarship.”

But the real fun comes in the wordplay.  “Without the vouchers, more than 80% of the 1,700 kids would have to attend public schools that haven’t made ‘adequate yearly progress’ under No Child Left Behind,” intones the WSJ editorial.  Well, that’s certainly disturbing.  How about this?

“Without the vouchers, 1,700 kids would have had to attend public schools in which they would have achieved at the same level in reading and math as they did with the vouchers.”

“Without the vouchers, 1,700 kids would have had to attend public schools that were reported to be just as safe as the private schools they attended with the vouchers.”

“Without the vouchers, 1,700 kids would have had to attend public schools with which they were just as satisfied as the private schools they attended with the vouchers.”

“Without the vouchers, 1,700 kids would have had similar aspirations for future schooling;  the same frequency of doing homework;  spent as much time reading for fun;  and the same attendance and patterns of tardiness as they did with the vouchers.”

As I wrote last summer:  skoolboy isn’t crazy about using public funds to support private schools, but he’s a big supporter of using public funds to support the education of children in D.C., who historically have been among the lowest performers in the nation. Congress authorized this program, it’s survived legal scrutiny, and it’s deserving of a fair shake. But distorting the results of an evaluation doesn’t serve the public good.

Citing some dodgy statistics doesn’t disguise the fact that this is a dispute over values, not research.  A Republican-led Congress and White House authorized the DC Opportunity Scholarship Program in the first place;  A Democratic-led Congress and White House may let the program elapse.  Sorry, WSJ:  this is democracy in action.  You’ll just have to wait for Michelle Rhee to work her magic in DC.

Print Print
  • Ken Hirsh

    Thanks, as always, for the interpretation of the research.

    You conclude: “You’ll just have to wait for Michelle Rhee to work her magic in DC.” That’s not entirely true. DC is #2 (after New Orleans) with respect to charter school market share. (See http://www.publiccharters.org/files/publications/2008%20Market%20Share%20Report.pdf.) 31% of the students in DC attend charter schools (and growing). These schools, of course, are independently managed. In fact, I think that the government entity that oversees charter schools doesn’t even report to Rhee. Luckily, charter school parents are less exposed (at the moment!) to the partisan battles of our two major tribes.

  • Aaron Pallas

    One has to wonder why, if there is a large supply of high-quality charter schools available to DC parents and their children, there is so much attention on a program serving 1,700 students — compared to the more than 26,000 DC students attending charter schools.

  • MK

    I think part of the issue is that there is not necessarily a “large supply of high-quality charter schools” in DC. While enrollment in DCPS has consistently declined (even with Rhee at the helm), the charter schools in DC are very much a mixed bag. I know the Washington Post recently had an article arguing that students in DC charter schools are doing better than their peers in DCPS but I think that is a contestable issue.

    As in other states, the charter schools in DC vary widely in terms of how well they serve students.

    On another note, I know in a number of articles I came across they mentioned that some of the students using the DC Opportunity Scholarship vouchers are in fact attending Sidwell Friends, the same school as the Obama girls. This seems to resonate with people.

  • Aaron Pallas

    MK,

    The “high-quality” line was skoolboy being playful, looking to see if anyone would bite.

    Today’s Washington Post editorial on the DC Opportunity Scholarship vouchers plays the Sidwell Friends card, in a really misleading way, in my opinion. Writes the Post:

    We would like Mr. Obey and his colleagues to talk about possible ‘disruption’ with Deborah Parker, mother of two children who attend Sidwell Friends School because of the D.C. Opportunity Scholarship Program. ‘The mere thought of returning to public school frightens me,’ Ms. Parker told us as she related the opportunities — such as a trip to China for her son — made possible by the program. Tell her, as critics claim, that vouchers don’t work, and she’ll list her children’s improved test scores, feeling of safety and improved motivation.

    Let’s not kid ourselves: Deborah Parker’s children are attending Sidwell Friends because of the school’s financial aid program, not because of the DC Opportunity Scholarship, which only covers $7,500 of the roughly $29,000 in tuition that Sidwell Friends charges. Only one in nine student receiving a voucher attended a school that charged above $7,500 in the second year of the evaluation, and the average tuition that the voucher was applied towards was $5,928. The evaluation report doesn’t indicate how many of the 1,700 students receiving vouchers attended schools such as Sidwell Friends, but it cannot have been very many.

    As for the Post‘s argument: It’s policy-making by anecdote. It’s lovely that Ms. Parker’s kids get to go to China–”made possible by the program,” we’re told–but the evaluation to date has not found effects of the voucher on students’ achievement, aspirations, attendance, or reports of school safety.

  • MK

    Aaron,

    I agree that the Sidwell Friends line is often used in a misleading way. Obviously a 7,500 voucher does not cover the cost of Sidwell. I think most of the Opportunity Scholarship propoganda is misleading, as is most information put forth by school choice advocates.

    And good call on the “policy-making by anecdote” displayed by the Post.

    I understand that evaluations to date have not found effects of the voucher on students’ achievement, aspirations, attendance, or reports of school safety. But haven’t some studies shown that students who attend Catholic School have higher rates of going to college even though they don’t show higher achievement scores? Isn’t it possible that the voucher students will be more likely to go to college then if they had attended DCPS schools? I know we don’t have any mechanism of measuring this currently, but isn’t it possible? I’m not saying this is a reason to continue the program, I am just curious how well the current evaluations of the program give a full picture of what these students are currently receiving. I imagine there are a lot of unobservable results.

  • Ken Hirsh

    Many voucher supporters view charter schools as an insufficient half-step towards “real” school choice, so it is not surprising that they fight to support vouchers even with the presence of a large charter school alternative. I haven’t studied the DC charter movement, so I can’t comment on the general quality of the schools. My brief experience with the pre-Rhee traditional system gave me the impression that is was a corrupt disaster, so I am happy that some parents are given an alternative. The competition from the charter movement will likely be a positive influence that helps to encourage the traditional schools to improve.

  • MK

    Ken,

    Very few studies have shown that the introduction of charter schools effectively leads nearby “public schools to improve.” I know Cecilia Rouse had a comprehensive examination of school choice that showed that there is no evidence to prove this is true. I think Caroline Hoxby is the only one who seems to continue to argue that this is the case in her examination of I believe Arizona and Michigan.

    Even under Rhee, enrollment at DCPS has continued to decrease (as parents choose charter schools). This seems more to do with people’s conception of the system than any thing else. And while DCPS is certainly trying to improve, and win back students, I often find the argument that charter schools leads to improvement of the general system to be a complete misnomer.

  • http://sinksalive.blogspot.com KitchenSink

    The identity of charter schools that share space with DOE schools should be readily available. I wonder if someone would be willing to put together a master list of all DOE schools that share space with charter schools, and some key indicators before and after sharing space?

    I would think test scores and attendance data would be no-brainers.

    I know Leonie Haimson has put together this data with class size (come to think of it, since she’s done that much work, I could probably complete her table with test scores and find out), and erroneously concluded that charter schools on balance raise class size in DOE schools when they share space with them. The data show otherwise; the average class size in the DOE schools decreased, and the number of grades in which average class size decreased outweighed the converse. (She misleadingly pointed out the sheer number of grades in which class size went up, without citing the larger number of grades in which is went down, in support her argument.)

    My point is going on about this is that while we have easily questionable truisms about charter schools improving DOE schools through competition, we can also gather some data in this area.

  • http://www.sinksalive.blogspot.com KitchenSink

    I did some research and realized (remembered) that there was not a Grades 3-8 state testing program in 2004-05.

  • ceolaf

    Getting to Aaron’s original post (and a dead horse I will continue to keep beating): If a program is evaluated and shown NOT to provide the promised benefits, shouldn’t it be ended?

    The DC voucer program was supposed to give these kids without access to decent schools a chance at a better education. But the research required by the authorization showed that they don’t provide additional benefits.

    Aaron rightly points out that the program was authorized for ideological reasons, and ended for idiological reason. I don’t question that for a second. But why don’t ANY of the voucher proponents now support ending the program? To me, this makes it hard to believe anyone calling for funding alternatives to the traditional public schools on the grounds of provding better schools for the kids. Sure, there’s some degree of guilt by association in my distrust, but these arguements appear increasingly disengenuous over time.

  • Christian

    Voucher proponents don’t support ending the program for a couple of reasons.

    Overall, the results of assessment are mixed. The qualitative data is largely pro-OSP; parents are more satisfied, they perceive their kids as safer, and they perceive them to be doing better in school. The quantitative stuff – test scores – is neutral. There’s no statistically significant differences in achievement in reading or math.

    This is only somewhat consistent with national data on Catholic schools – there’s 30 years worth of research, starting with James Coleman in the 1960s, Andrew Greeley in the 1970s, and Tony Bryk in the 1990s, and many others more recently, that demonstrates that kids, especially poor and minority kids, do slightly better in Catholic schools than in public schools – the researchers call it “the Catholic school advantage.”

    The bigger difference comes in long-term measures of achievement – while the test score differences btwn Catholic/public school are small, educational attainment outcomes are much larger. Kids who go to Catholic schools graduate from HS and go to college in much larger numbers, especially minority and poor kids. For example, Latinos who go to Catholic school are 250% more likely to graduate from HS and go to college. Neil’s 1997 study on the effects of Catholic schools on educational achievement and Jeynes 2007 study on religion and the achievement gap both suggest the long-term value of Catholic or other faith-based schooling.

    For me, the money is where things get interesting. The OSP provides for up to $7,500 per kid who leave the DCPS for a private school. Most of those kids are only claiming about $6,000. The DCPS per-pupil-expenditure is what, $13,500? So for every kid that takes a scholarship and goes to a private school, the DCPS saves $7,500. What’s going to happen when DCPS has to reabsorb these kids?

  • Aaron Pallas

    Christian: I’m not aware of any research by Jim Coleman in the 1960’s or Andy Greeley in the 1970’s, that bears on Catholic school effects on student achievement, and your estimates of the Catholic school advantage for urban minority youth are way off the mark. It’s true that Derek Neal found that urban black and Latino youth in the NLSY survey of 14- to 21-year-olds in 1978 who attended Catholic high schools had substantially higher high school and college graduation rates than their peers. He interpreted this as being due to the poor quality of the public schools that these youth might otherwise attend. Neal’s subsequent research on the NELS:88 cohort of eighth-graders in 1988 led to a similar conclusion. The effects of attending Catholic school on test scores are not as large, and are heterogeneous across the different kinds of students who attend Catholic schools. We do not have good explanations for these effects.

    These past findings would be most likely to hold for current cohorts of youth if the characteristics and quality of Catholic schools and their nearby public peers were the same today as they were a few decades ago, when the prior data were collected. As you know, there have been substantial shifts over time in the demographic composition of Catholic school attendees, with substantial declines in Catholic school attendance, a rise in diocesan school closings, and increased competition with public and charter schools in many communities. All of these changes make a straightforward extrapolation from the past difficult.

  • Ken Hirsh

    Hey Aaron,

    If and when you have time, could you tell us a bit more about the various DC voucher studies? Have any of them used random assignment? Separately, what do you think of Jay Greene’s posts on voucher effects on participants (http://jaypgreene.com/2008/08/21/voucher-effects-on-participants/) and voucher effects on incumbent school systems (http://jaypgreene.com/2008/08/25/systemic-effects-of-vouchers/)? He seems to list and review all of the “high-quality” studies and argues that they show:
    1. Vouchers have a positive effect on the outcomes for participants.
    2. Vouchers have a positive effect on incumbent school systems.

    I am not an academic, but I have always wanted to see a debate between Greene and someone that argues that vouchers have been ineffective. I am confused by the differing conclusions.

  • Ken Hirsh

    Hey ceolaf,

    I am not sure I understand your logic on the DC voucher situation.

    1. Aaron, in a post of his that he linked to, makes the good point:

    “If Ms. Spellings wants to argue that the program should be renewed by Congress because parents are more satisfied with their child’s school, or because they are less likely to report serious concerns about school danger, she’s welcome to make that argument. Those are good outcomes, and some might argue that they’re ample justification for renewing the program. (Others might point out that students who received scholarships did not report higher levels of satisfaction with their school, or better school safety.) Or, alternatively, one could argue that the program needs more time to mature in order to be successful.”

    I think these are good reasons to continue the program.

    2. Why do you think the incumbent school system deserves priority over programs that have not been shown to hurt student outcomes?

    3. The DC vouchers are $7500 maximum per student. I think this is a much smaller number than the amounts spent on traditional DCPS students. This is another argument in favor of the vouchers: they either save money for other programs or, perhaps, we should be increasing the voucher amount to match the traditional school cost.

  • ceolaf

    Ken,

    The goal, in my view, should not be to provide options. Choice is not the goal, because choice is not a panacea.

    The goal is a better set of schools for kids.

    So, we have a large traditional public system. Most of us believe that it is not providing the kind of educational outcomes that we want. All of this is about what we can to improve those educational outcomes.

    We could put a lot of time money and attention into any number of things that won’t improve educational outcomes. Or, we could try to put all that time, money and attention into thing that WILL improve educational outcomes.

    We could give each and every student an iPhone, or a cell phone, or a skateboard. We could supply every child with the wardrobe of his/choice. We could put a lot of time, money and attention into those things without improving educational outcomes. I would, therefore, say that education wonks and professional should devote that time, money and attention to something other than supplying ipods, cellphones, skateboard and wardrobes.

    Do charter schools hurt? The primary way they hurt, in my view, is that they distract from figuring out how actually to improve educational outcomes for the whole set of schools. There’s an opportunity cost there. And even if they did provide marginally better outcomes on average, I would still have a problem with them, as we need far more than marginal improvement.

    ***********************

    I think that there are a lot of pushy, obnoxious and collectively powerful parents out there. They have the ears of politicians and of people who have the ears of politicians. They can get access to decision-makers, or to people who can get access to decision-makers. We need these pushy and obnoxious parents to push for all the schools to produce significantly better outcomes.

    I don’t just want to tax the rich at a higher rate than the poor. I want everyone to contribute in the ways they can best towards a betterment of society. Some of that is entertainment. Some of that is medicine. A a broad cross section of that is pushing for better schools.

    Each time we give those pushy, obnoxious and maybe even powerful parents another escape route from the system, we make it less likely that the system will improve. We all drive the same roads, so the roads are pretty good. But we don’t all have the same health care coverage, so some of it is pretty bad. Which do we want our schools to be more like?

    Obviously, private schools are an escape route. Well, I think that private schools are bad for democracy and bad for America. I want the best possible common schools! I want every child to have a shot at the best schools. Charters are only a half-step if private schools are a valid choice, and I think that they are a bad choice (i.e. bad for democracy, bad for America).

    Charters are both better and worse than private schools in this regard. And I am in favor of neither, unless or until we find that either actually improves the whole system of schools, of which they are but a small fraction.

    ***********************

    Is that more clear, Ken?

  • Aaron Pallas

    Ken: The evaluation of the DC Opportunity Scholarship Program is a random-assignment study. But it also indicates how random-assignment is no panacea, as 26% of the students who were offered a voucher during the first two years of the program did not use it, and the characteristics of the non-users differed from those of the users. Therefore, the researchers distinguished between the effects of being offered a voucher on student outcomes and the effects of using a voucher. There are other limitations as well; it’s difficult to obtain data from a control group that is in the study because they lost the lottery, and so response rates for tests and other information are not as high as I’d like.

    And as an aside, in my view, the fact that more than one-quarter of the students offered vouchers did not use them raises questions about the claims made in the Washington Post and elsewhere that the demand for this program is overwhelmingly high.

    As for the literature on voucher effects in general, my major concern is that the results of many studies seem to be sensitive to the analytic decisions made by researchers, such that different researchers are drawing differing conclusions about the same program. This lack of robustness makes strong claims for effects in either direction somewhat suspect. In some cases, there may be analytic approaches that are clearly superior, but in others, the choices that researchers make are equally plausible. For this reason, I do not find claims about overwhelmingly positive effects of vouchers to be persuasive.

    For a counterpoint to the posts you list, I would recommend the review article “School Vouchers and Student Achievement: Recent Evidence, Remaining Questions” by Cecilia Rouse of Princeton University and Lisa Barrow of the Federal Reserve Bank of Chicago, which is to be published in Annual Review of Economics in 2009. I think you can find a copy at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1268316

  • http://www.republicofdave.com Dave Nalle

    What is demonstrated here is a gross lack of familiarity with conditions in the DC school system and the private school alternatives. That anyone could even suggest that a voucher which makes it possible to attend Sidwell Friends with the addition of scholarship money from their endowment doesn’t increase opportunities for a student who would otherwise be mired in the nightmare which is the DCPS system is just mind boggling.

    There is a long history of schools like Sidwell offering a limited number of scholarships to poor kids from DC’s ethnic communities. Overwhelmingly these kids go on to college and successful careers while their peers from the DC public school system find themselves without opportunities or hope, caught up in the worst public school system in the nation.

    Dave

  • Ken Hirsh

    Hey ceolaf,

    Thanks for the explanation. Obviously, we disagree about the relative effectiveness of parental choice as compared to public choice. I am sure we will both have more to say about this in the future!

Take the GothamSchools Survey

Tips, questions, feedback?

Contact us at .

Follow GothamSchools

RSS

Chalk It Up

Recent Comments

0 comments so far today
View all comments »

Events Calendar

RSS Feeds

Newsroom posts Community posts All GothamSchools content

Our Twitter Updates

Archives

May 2012
M T W T F S S
« Apr  
 123456
78910111213
14151617181920
21222324252627
28293031  

Copyright Notice