NEW UPDATE: Completed trials may now upload and register supplementary documents (e.g. null results reports, populated pre-analysis plans, or post-trial results reports) in the Post Trial section under Reports, Papers, & Other Materials.
Leveling the Playing Field for High School Choice: A Field Experiment of Informational Interventions
Initial registration date
May 03, 2018
January 04, 2019 10:23 AM EST
New York University
Other Primary Investigator(s)
Teachers College Columbia University
Seton Hall University
Additional Trial Information
We conducted a school-level randomized controlled trial in 165 high-poverty New York City middle schools to help 8th grade students navigate a complex high school choice process and access higher-performing schools. The trial included 19,109 students across all study schools. Students in treatment schools were given a customized one-page list of proximate high schools with a graduation rate at or above the city median (70%). Some also received a supplemental list highlighting academically non-selective schools or high schools organized by academic interest area. The interventions changed student application behavior in ways that led to more matches to higher-performing schools. While treatment students did not apply to higher graduation rate schools, they applied to schools where their odds of admission were higher, were more likely to receive their first-choice high school, and were less likely to match to a school with a low graduation rate. Our findings also suggest that informational interventions may not reduce inequality, since both disadvantaged and comparatively advantaged students used our materials, and in some cases the latter benefited more from them by applying and matching to more schools on our lists. Future work will continue to follow these students and track their academic outcomes.
Cohodes, Sarah et al. 2019. "Leveling the Playing Field for High School Choice: A Field Experiment of Informational Interventions." AEA RCT Registry. January 04.
Informed by the existing literature and our understanding of the challenges faced by NYC students choosing high schools, we developed a one-page informational tool called “FastFacts,” a customized list of 30 high schools for each middle school in our study. We designed Fast Facts to be an accessible starting point for students, and a useful reference for school performance information and admissions requirements. The intent was not for students to limit their search to these schools, but rather to begin with a initially smaller set of choices, and to be more aware of higher-performing schools in their proximity.
Schools recruited to the study were randomized into three treatment groups and a control group. Eighth grade students in the first treatment arm (FF1) received the basic Fast Facts school list described above. Students in the second treatment arm (FF2) received Fast Facts and a supplementary list of academically non-selective schools that give priority admission to students who attend an open house or information session and sign in. This group could also opt in to receive weekly text message reminders about open house dates, time, and locations. Schools in the third treatment arm (FF3) received Fast Facts and a supplementary list of high school programs organized by academic interest area. Both supplemental lists were attached to Fast Facts, on the inside of a bi-folded sheet. All treatment schools received a separate one-page insert of “screened language” programs citywide that serve recent immigrants and students learning English.
Materials specific to each intervention were delivered direct to students by trained research assistants via a 40-minute standardized lesson in a group setting (often but not always in a classroom). This lesson explained how to use the tools and emphasized the importance of graduation rates, admissions methods, and location when making school choices. All materials--including text messages--were available in English and Spanish, and lessons were delivered in Spanish when requested by the school guidance counselor. Control schools did not receive any materials until after the study was complete. (These schools were later provided Fast Facts lists that they could used with their 7th grade students).
Intervention Start Date
Intervention End Date
Primary Outcomes (end points)
a. High school choice outcomes (for 1st, top 3, and all choices; matched school; and enrolled school)
i. School presence on the Fast Facts and/or supplementary lists
1. presence on the list
2. presence in the top or bottom half of the list
ii. High school characteristics
1. 4-year graduation rate
2. whether the graduation rate was below the Fast Facts threshold of 70%
3. travel time from the middle school to the high school
4. location in the same borough as the student’s residence
5. applications per seat in the prior year (a measure of demand)
6. admissions method (e.g. screened or limited unscreened, different types of high schools in the NYC system)
7. variability in graduation rates (the difference between the highest and lowest graduation rate of schools appearing on a student's application)
8. academic or career interest area (e.g. STEM)
9. within-application consistency in interest area, calculated as the highest percentage of choices from the same interest area
i. Other admissions process and enrollment outcomes
1. number of choices submitted (up to12)
2. open house priority status for limited unscreened programs (a lottery preference given to students who attend an event for certain schools)
3. ranking of the student by screened programs
4. whether or not the student was matched to his or her 1st choice, 1st-3rd choice, or any choice in the first round
5. participation in the second round after a successful match
6. matriculation to the matched school in 9th grade. Other outcome variables
b. Students’ academic progress and other outcomes in high school
i. Credit accumulation in 9th-12th grade, and GPA;
ii. New York State Regents exam scores and passing rates;
iii. On-time graduation;
iv. Engagement and perceptions of school (e.g. attendance rate and responses to the NYC DOE’s Learning Environment Survey)
C. Post-high school outcomes (pending data availability)
i. College enrollment, persistence, and graduation
iii. Arrest records
iv. Voter registration, voting
Primary Outcomes (explanation)
Secondary Outcomes (end points)
Secondary Outcomes (explanation)
Schools were assigned to one of the three “Fast Facts” interventions described in the “interventions” section, or a control group. We recruited 165 schools from the more than 500 schools serving 8th grade students in NYC, focusing on some of the highest-poverty schools in the city. To increase power over a simple cluster randomized trial in which schools are randomly assigned to treatment conditions, we randomized schools within 39 blocks of similar schools located in the same borough. Blocks were matched quadruplets of schools selected using a Mahalanobis distance measure of difference between schools (see Bruhn & McKenzie 2009; King et al. 2007). School variables used in the matching procedure included prior choice outcomes (e.g., the mean graduation rate of first round matches in 2013-14), prior achievement (mean ELA and math scores in 2013-14), economic disadvantage (the percent eligible for free or reduced price meals), and school size. The 23 schools in our pilot study were blocked separately. For details on randomization please see the online appendix to our NBER Working Paper available here: http://www.nber.org/data-appendix/w24471/w24471.appendix.pdf
Experimental Design Details
Was the treatment clustered?
Sample size: planned number of clusters
Sample size: planned number of observations
Sample size (or number of clusters) by treatment arms
39 schools to Fast Facts 1, 39 schools to Fast Facts 2, 40 schools to Fast Facts 3, and 47 schools to control
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Our power calculations were done before conducting the experiment. For estimating MDES we made standard assumptions regarding power (.80) and significance (α=.05). We assumed 150 eighth graders per cluster (school), the NYC average, and 2 clusters per block. (In practice there will be 4 schools per block, representing our 3 treatments and one control. For illustrative purposes we provide the MDES for only one treatment-control comparison). We assumed two inter-class (within-middle school) correlations: a relatively low ICC (ρ=0.08) and a modest ICC (ρ=0.20). These correspond to the range of ICCs observed for several outcomes in existing administrative data, after removing the effect of pre-treatment covariates. We assumed a conservative R2 from regression on pre-treatment covariates of 0.1, and that the blocks account for 0.1 to 0.4 of the variance.
For example, with 60 schools (30 blocks), we have sufficient power to detect effect sizes ranging from 0.20 under the low ICC, high between-block variation assumption, to 0.35 under the high ICC, low between-block variation assumption. These translate into an increase of 2.3 to 4.1 percentage points in the four-year graduation rate of the first choice high school (or average of the top three choices), 1.1 to 1.7 points in high school value-added to graduation rates, and 2.4 to 3.5 percentage points in the percent of students accumulating 10 or more credits in year one of high school at the first choice (or average of the top three). This exercise offers some confidence in our decision to recruit at least 30 schools per treatment group, and control group.
INSTITUTIONAL REVIEW BOARDS (IRBs)
NYC Department of Education
IRB Approval Date
IRB Approval Number
Study number 736
New York University
IRB Approval Date
IRB Approval Number
Post Trial Information
Is the intervention completed?
Is data collection complete?