x

We are happy to announce that all trial registrations will now be issued DOIs (digital object identifiers). For more information, see here.
Are High-Quality Schools Enough to Increase Achievement Among the Poor? Evidence from the Harlem Children’s Zone
Last registered on February 05, 2017

Pre-Trial

Trial Information
General Information
Title
Are High-Quality Schools Enough to Increase Achievement Among the Poor? Evidence from the Harlem Children’s Zone
RCT ID
AEARCTR-0001984
Initial registration date
February 05, 2017
Last updated
February 05, 2017 8:56 PM EST
Location(s)
Primary Investigator
Affiliation
Harvard University
Other Primary Investigator(s)
PI Affiliation
Public Policy Doctoral Program, Harvard University Kennedy School of Government
Additional Trial Information
Status
Completed
Start date
2004-01-01
End date
2011-07-31
Secondary IDs
Abstract
In the United States, a black-white achievement gap in education outcomes begins to manifest itself as early as two years of age. In fact, the typical black 17-year-old reads at the proficiency level of the typical white 13-year-old. Although policymakers have attempted to close this achievement gap through a variety of measures there has been mixed success rates. In particular, debate exists around a school-centered approach (in which strong schools alone can reduce the achievement gap) versus a community-based approach (in which community programs can close the gap). Using lottery and instrumental variable identification strategies, researchers analyze the causal impact on test scores of attending the Promise Academy charter schools within the Harlem Children's Zone, a non-profit organization that offers community programs and other comprehensive services to a 97-block area in Harlem, New York City in which 72.8% of the youth are black. The main result suggests that the effects of attending an HCZ middle school are enough to close the black-white achievement gap in math. In elementary schools, the effects can close the racial achievement gap in both mathematics and English Language Arts. The evidence suggests that the Promise Academy student performance was driven by the school itself as opposed to other HCZ community programs.
External Link(s)
Registration Citation
Citation
Dobbie, Will and Roland Fryer. 2017. "Are High-Quality Schools Enough to Increase Achievement Among the Poor? Evidence from the Harlem Children’s Zone." AEA RCT Registry. February 05. https://doi.org/10.1257/rct.1984-1.0.
Former Citation
Dobbie, Will and Roland Fryer. 2017. "Are High-Quality Schools Enough to Increase Achievement Among the Poor? Evidence from the Harlem Children’s Zone." AEA RCT Registry. February 05. https://www.socialscienceregistry.org/trials/1984/history/13754.
Experimental Details
Interventions
Intervention(s)
Researchers study the effects of attending high-quality charter schools in the Harlem Children's Zone, exploiting the fact that the Promise Academy charter schools are required to select students by lottery when the number of applicants exceeds the number of slots for admission. As a result, the control group includes lottery losers and the treatment group contains lottery winners. The experiment also created an instrumental variable out of the interaction between the student's cohort year and whether he or she lives within the HCZ itself. The focus on these two criteria is relevant because only students living within HCZ are recruited by HCZ staff and some cohorts of children may be ineligible.

The HCZ is a non-profit that provides community and school inputs (i.e. college success office, family and health programs, foster care prevention services, tutors on Saturday, etc) to individuals in a 97-block area in Harlem. The Promise Academy caters to elementary and middle schools, including approximately 1,300 students.
Intervention Start Date
2004-01-01
Intervention End Date
2010-12-31
Primary Outcomes
Primary Outcomes (end points)
Math and English Language Arts (ELA) scores
Primary Outcomes (explanation)
State math and English Language Arts tests
Secondary Outcomes
Secondary Outcomes (end points)
Secondary Outcomes (explanation)
Experimental Design
Experimental Design
The data comes from two sources: 1) information from files at HCZ and 2) administrative data on student demographics and outcomes from the New York City Department of Education (NYCDOE). The experiment is to analyze the causal impact of attending the Promise Academy charter schools in HCZ on academic outcomes.

To ensure proper randomization, researchers excluded students with siblings who were lottery winners (since then siblings are automatically enrolled). The HCZ data were then matched to the New York City administrative data using last name, first name, date of birth, prior grade, etc. 69.6% of middle school and 59.9% of elementary school lottery winners enrolled in the Promise Academy for at least one year.

The results indicate that the Promise Academy charter schools are effective in increasing achievement as students in the middle school increased their math and ELA scores by 0.229 and 0.047 standard deviations respectively, using the 2SLS estimates, which is enough to close the black-white achievement gap in mathematics by ninth grade. In the elementary school, students increased their scores in math by 0.191 standard deviations and in ELA by 0.114 standard deviations, which is enough to close the achievement gap by third grade.

According to the second empirical test, the distance x cohort instrumental variable strategy for middle school indicates that the effect of attending the Promise Academy is 0.206 standard deviations in math and -0.054 in ELA; therefore, the estimate suggests that math scores will increase 0.618 standard deviations - having a significant impact on math scores but not on ELA scores. For elementary schools, the distance x cohort estimates that students at the Promise Academy gained 0.324 standard deviations in math and 0.420 in ELA for each year. Therefore, the increase in scores was noticeable even if students were not officially in the HCZ zone. And finally, siblings of enrolled lottery winners gained approximately 0.051 standard deviations in math and 0.087 standard deviations in ELA, which is smaller than those who were enrolled in the Promise Academy. Therefore, the experiment suggests that high-quality schools can be more effective than community programs alone.
Experimental Design Details
Randomization Method
Admissions lottery by the Promise Academy charter schools in HCZ
Randomization Unit
individual
Was the treatment clustered?
No
Experiment Characteristics
Sample size: planned number of clusters
n/a
Sample size: planned number of observations
Kindergarten lottery: 347 individuals Middle school lottery: 495 individuals
Sample size (or number of clusters) by treatment arms
Kindergarten lottery: 177 individuals treatment group (lottery winners), 170 individuals control group (lottery losers)

Middle school lottery: 176 individuals treatment group (lottery winners), 319 individuals control group (lottery losers)
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB
INSTITUTIONAL REVIEW BOARDS (IRBs)
IRB Name
Study has received IRB approval. Details not available.
IRB Approval Date
Details not available
IRB Approval Number
Details not available
Post-Trial
Post Trial Information
Study Withdrawal
Intervention
Is the intervention completed?
Yes
Intervention Completion Date
December 31, 2010, 12:00 AM +00:00
Is data collection complete?
Yes
Data Collection Completion Date
December 31, 2010, 12:00 AM +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
n/a
Was attrition correlated with treatment status?
No
Final Sample Size: Total Number of Observations
Kindergarten lottery: 347 individuals
Middle school lottery: 495 individuals
Final Sample Size (or Number of Clusters) by Treatment Arms
Kindergarten lottery: 177 individuals treatment group (lottery winners), 170 individuals control group (lottery losers) Middle school lottery: 176 individuals treatment group (lottery winners), 319 individuals control group (lottery losers)
Data Publication
Data Publication
Is public data available?
Yes
Program Files
Program Files
Yes
Reports and Papers
Preliminary Reports
Relevant Papers
Abstract
Harlem Children’s Zone (HCZ), an ambitious social experiment, combines community programs with charter schools. We provide the
first empirical test of the causal impact of HCZ charters on educational outcomes. Both lottery and instrumental variable identification
strategies suggest that the effects of attending an HCZ middle school are enough to close the black-white achievement gap in mathematics. The effects in elementary school are large enough to close the racial achievement gap in both mathematics and ELA. We conclude with evidence that suggests high-quality schools are enough to significantly increase academic achievement among the poor.
Community programs appear neither necessary nor sufficient.
Citation
Dobbie, Will and Roland G. Fryer Jr. 2011. "Are High-Quality Schools Enough to Increase Achievement among the Poor? Evidence from the Harlem Children's Zone." American Economic Journal: Applied Economics, 3(3): 158-87.