CREDO’s New Study Biased against Public Schools by Thomas Ultican
The Center for Research on Education Outcomes (CREDO) started releasing the results of its new Cities Study Project in mid-2019. It is not a coincidence that the cities chosen for the study have long been targeted for public school privatization. The ten cities selected are: Indianapolis; Baton Rouge; Camden; Kansas City; Memphis; New Orleans; Oakland; St. Louis; San Antonio; and Washington DC. This CREDO study is even more opaque and biased than its previous efforts.
Who is CREDO?
Husband and Wife Team Who Founded CREDO
In the early 1980s, Margaret (Macke) Raymond was completing a lengthy graduate school agenda at the University of Rochester, a relatively small private university in Rochester, New York. She garnered an MS of public policy in 1980, a community medicine MS in 1982, an MA of political science in 1983 and finished with a PhD in political science in 1985. From 1985 to 2000 she ran Raymond Consulting and worked a few years in the telecommunications industry.
At that same time, Eric Hanushek was Professor of Economics and Political Science at Rochester University. The former Air Force cadet had earned a doctorate of Economics from the Massachusetts Institute of Technology in 1968. It was Hanushek’s 1981 paper “Throwing Money at Schools” that put him on the watch list of right leaning philanthropies and institutions. In his notice making missive he stated,
“The conventional wisdom about public schools is that they face serious problems in terms of performance and that improving schools requires additional money. However, the available evidence suggests that there is no relationship between expenditures and the achievement of students and that such traditional remedies as reducing class sizes or hiring better trained teachers are unlikely to improve matters.”
In a 1981 Ed Week commentary referencing this paper, Hanushek points to SAT testing as the gold standard for judging school performance. In complete accordance with the old aphorism, “to a man with a hammer all problems look like a nail,” Hanushek brags, “Advanced statistical techniques are employed to disentangle the influences on achievement of schools and teachers from those of other factors such as family backgrounds and student abilities.”
A 1999 announcement from Rochester University said, “The Center for Research on Education Outcomes has been established at the University of Rochester’s Wallis Institute of Political Economy…” In the same posting, it was revealed, “Two foundations have committed $1.25 million to fund a three-and one-half year initiative to address the current shortage of evaluation research in education policy matters.” CREDO never made the name of the two foundations public, but a knowledgeable academic says one of them was the Walton Family Foundation.
The announcement listed two employees of the new center, Eric Hanushek and Margaret Raymond. Hanushek was listed first but Raymond was cited as the founding Director.
CREDO moved from the University of Rochester to Stanford University’s Hoover Institute in July, 2000 which made networking in conservative circles much easier. In CREDO’s 2nd year report, they stated that moving to the Hoover Institute brought them many new contacts including the New Schools Venture Fund, the District of Columbia Charter School Board, the Teacher Union Reform Network and others.
A description of the Hoover Institute from Source Watch says,
“The Hoover Institution is influential in the American conservative and libertarian movements, and the Institution has long been a place of scholarship for high profile conservatives with government experience. A number of fellows have connections to or positions in the Bush administration, and other Republican administrations. … Other fellows of the Institution include such high profile conservatives as Condoleezza Rice, George Shultz, Thomas Sowell, Shelby Steele, and Edwin Meese.”
Macke Raymond’s 2015 Hoover Institute Fellow’s profile says in part, “In partnership with the Walton Family Foundation and Pearson Learning Systems, Raymond is leading a national study of the effectiveness of public charter schools.”
Valerie Strauss of the Washington Post has pointed out that Eric Hanushek
“… a Hoover economist was a pioneer in creating systems that evaluate teachers by student standardized tests, a method that many assessment experts say should not be used in the high-stakes ways that school reformers are using them. He is often cited in CREDO studies as a ‘principal investigator.”’
Discredited and Biased
The Forbes commentator, Peter Greene, wrote about Eric Hanushek for his popular blog Curmudgucation:
“Now when Hanushek says that teachers make a huge difference, he is obliquely referencing his own crazy-pants assertion that having a good first grade teacher will make you almost a million bucks richer over your lifetime (you can also find the same baloney being sliced by Chetty, Friedman, and Rockoff). Both researchers demonstrate their complete lack of understanding of the difference between correlation and causation.”
Greene also shares the following graphic that clearly highlights the difference between correlation and causation.
Does Margarine Consumption Cause Divorce?
Business writer Andrea Gabor states that CREDO studies which compare charter schools with public schools start with two key assumptions “A) That standardized-test scores are an adequate measure of school quality and B) that creaming in charter schools does not exist.”
With regards to assumption ‘A’, using standardized testing for this purpose has been shown faulty from studies dating back to the eugenics movement (which originated high stakes standardized testing) to recent works debunking them for mistaking correlation versus causation and for not being able to compensate for the problem of error.
As for assumption B, there is no doubt that most charter schools push out and avoid students that are classified as special education, language learners or discipline problems. The data proving that is in state enrollment reports wherever charter schools exist.
In the new study, “The Cities Project”, the Technical Appendix says the reports uses growth models but doesn’t share which of the many growth models it uses. It also says,
“In our study, scores for all these separate tests are transformed to a common scale. All test scores have been converted to standardized scores to fit a ‘bell curve’, in order to allow for year-to-year computations of growth.”
The “Education Growth Model Handbook” lists seven types of growth models in general use and their requirements. Most growth models require vertical scales but that does not seem possible with CREDO’s use of multiple tests many of which are not vertically scaled. Their mathematical conversions add another locus of error. Growth models have proven to be unstable and have never been satisfactorily validated.
The research methodology used in “The Cities Project” appears to be the same as that used in CREDO’s 2015 “Urban Charter School Study Report on 41 Regions.” A particularly troubling practice employed then and apparently still being used is the “virtual twin” method which creates a pro-charter school bias.
Professor Andrew Maul of UC Santa-Barbara reviewed the 2015 study for the National Education Policy Center. The CREDO method does not compare charter school performance to actual public schools; rather it creates mathematical simulations. Maul described the “virtual twin” schema employed to develop a “virtual control record.” He reports,
“CREDO’s approach to this estimate is the construction of a ‘Virtual Control Record’ (VCR) for each student in a charter school, obtained by averaging together up to seven students in “feeder” public schools (i.e., those schools whose students transfer to charters) with the same gender, ethnicity, English proficiency status, eligibility for subsidized meals, special education status, grade level, and a similar score from a prior year’s standardized test (within a tenth of a standard deviation) as the specified charter student.”
Maul adds, “The study’s “virtual twin” technique is insufficiently documented, and it remains unclear and puzzling why the researchers use this approach rather than the more accepted approach of propensity score matching.”
The stipulation that “virtual twins” come from “feeder schools” biases the study in favor of charter schools. Andrea Gabor explained that in practice, CREDO used less than five students transferring to a charter school as the cutoff for using a particular public school’s data. She reports the, “study excludes public schools that do NOT send students to charters, thus introducing a bias against the best urban public schools, especially small public schools that may send few, if any, students to charters.” Gabor gave the example of two well regarded New York title-1 schools, Global Technology Preparatory and West Side Collaborative which were excluded. They did not meet the transfer criteria but easily matched the required demographics.
The CREDO study is singularly focused on test results as determinate of school quality and ignores other advantages of public schools. It is a well known fact that many charter school systems like IDEA and Success Academy spend an inordinate amount of time teaching to and preparing for standardized tests. To these criticisms, Professor Mark Weber of Rutgers University adds a few more observations:
“The matching variables that create the counterfactuals are far too crude to do the job properly.”
“The definition of the treatment — enrolling in a charter school — does not account for factors such as increased spending, peer effects, and other advantages which have nothing to do with ‘charteriness.’”
“The consistently small effect sizes have been pumped up by an unvalidated conversion into ‘days of learning’ which has never been properly justified by the authors.”
“The Cities Project” Technical Appendix also states,
“To assist the reader in interpreting the meaning of effect sizes, we include an estimate of the average number of days of learning required to achieve a particular effect size. This estimate was calculated by Dr. Eric Hanushek and Dr. Margaret Raymond based on the latest (2017) 4th and 8th grade test scores from the National Assessment of Educational Progress (NAEP).”
The CREDO Days of Learning Conversion Table from “The Cities Project”
This is a metric that seems to have been created with next to nothing validating it. Mark Webber quoted the psychometrician Michael T. Kane,
“The 2015 study (p. 5) cites a paper published in Education Next (Hanushek, Peterson & Woessmann, 2012) that asserts: “On most measures of student performance, student growth is typically about 1 full std. dev. on standardized tests between 4th and 8th grade, or about 25 percent of a std. dev. from one grade to the next.” (p. 3-4) No citation, however, is given to back up this claim: it is simply stated as a received truth.”
CREDO tells us that Hanushek and Raymond did something with NAEP data from 2017 but still do not offer any justification for the conversion. It appears at best to be sloppy science and the headlines engendered from it are nothing short of propaganda.
Using CREDO Claims to Sell Privatizing Public Schools
Neerav Kingsland the Managing Partner of the City Fund posted to his Blog last July when the first results from “The Cities Project” arrived explaining,
“Last year, Arnold Ventures commissioned CREDO (out of Stanford University) to study the effects of charter, innovation, and traditional schools in select cities across the country.”
“Most of the cities included in the study were cities where Arnold Ventures (and now The City Fund) have partnered with local leaders to expand high-quality schools.”
“Camden’s city level effects are large.”
“In just two years, scores are up ~.15 standard deviations in math and ~.05 standard deviations in reading (compared to similar schools across the state).”
The reality is those changes are very small. Noise in the data is a better explanation than awesome charter schools for these tiny differences.
In Indianapolis, the CEO of The Mind Trust, Brandon Brown, just wrote an Indy Star opinion piece stating,
“A 2019 study from Stanford University found that students who attend Innovation Network Schools achieve the equivalent of 53 additional days of learning in English and 89 additional days of learning in math each year when compared to their traditional public school peers. This equates to several years of additional learning during the span of a K-12 academic career, and the gains are largest for students of color and students from low-income backgrounds.”
In the billionaire financed effort to privatize public education, CREDO has become their source for data proving things like smaller class sizes and teacher professionalism are not important. The “Cities Project” commissioned by an organization intent on privatizing public schools through promoting the portfolio management scheme – The City Fund – is biased toward the privatization agenda. Rather than shining the light of scholarly work on education policy, it obscures reality with obfuscation.