Leveling the Playing Field for High School Choice: A Field Experiment of Informational Interventions

Last registered on August 27, 2020

Pre-Trial

Trial Information

General Information

Title
Leveling the Playing Field for High School Choice: A Field Experiment of Informational Interventions
RCT ID
AEARCTR-0002951
Initial registration date
May 03, 2018

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
May 04, 2018, 10:49 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
August 27, 2020, 3:37 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Primary Investigator

Affiliation
Vanderbilt University

Other Primary Investigator(s)

PI Affiliation
Princeton University
PI Affiliation
UCSB
PI Affiliation
Teachers College Columbia University

Additional Trial Information

Status
Completed
Start date
2014-08-04
End date
2016-08-09
Secondary IDs
Abstract
We conducted a school-level randomized controlled trial in 165 high-poverty New York City middle schools to help 8th grade students navigate a complex high school choice process and access higher-performing schools. The trial included 19,109 students across all study schools. Students in treatment schools were given a customized one-page list of proximate high schools with a graduation rate at or above the city median (70%). Some also received a supplemental list highlighting academically non-selective schools or high schools organized by academic interest area. The interventions changed student application behavior in ways that led to more matches to higher-performing schools. While treatment students did not apply to higher graduation rate schools, they applied to schools where their odds of admission were higher, were more likely to receive their first-choice high school, and were less likely to match to a school with a low graduation rate. Our findings also suggest that informational interventions may not reduce inequality, since both disadvantaged and comparatively advantaged students used our materials, and in some cases the latter benefited more from them by applying and matching to more schools on our lists. Future work will continue to follow these students and track their academic outcomes.
External Link(s)

Registration Citation

Citation
Cohodes, Sarah et al. 2020. "Leveling the Playing Field for High School Choice: A Field Experiment of Informational Interventions." AEA RCT Registry. August 27. https://doi.org/10.1257/rct.2951-5.199999999999999
Former Citation
Cohodes, Sarah et al. 2020. "Leveling the Playing Field for High School Choice: A Field Experiment of Informational Interventions." AEA RCT Registry. August 27. https://www.socialscienceregistry.org/trials/2951/history/74824
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Informed by the existing literature and our understanding of the challenges faced by NYC students choosing high schools, we developed a one-page informational tool called “FastFacts,” a customized list of 30 high schools for each middle school in our study. We designed Fast Facts to be an accessible starting point for students, and a useful reference for school performance information and admissions requirements. The intent was not for students to limit their search to these schools, but rather to begin with a initially smaller set of choices, and to be more aware of higher-performing schools in their proximity.

Schools recruited to the study were randomized into three treatment groups and a control group. Eighth grade students in the first treatment arm (FF1) received the basic Fast Facts school list described above. Students in the second treatment arm (FF2) received Fast Facts and a supplementary list of academically non-selective schools that give priority admission to students who attend an open house or information session and sign in. This group could also opt in to receive weekly text message reminders about open house dates, time, and locations. Schools in the third treatment arm (FF3) received Fast Facts and a supplementary list of high school programs organized by academic interest area. Both supplemental lists were attached to Fast Facts, on the inside of a bi-folded sheet. All treatment schools received a separate one-page insert of “screened language” programs citywide that serve recent immigrants and students learning English.

Materials specific to each intervention were delivered direct to students by trained research assistants via a 40-minute standardized lesson in a group setting (often but not always in a classroom). This lesson explained how to use the tools and emphasized the importance of graduation rates, admissions methods, and location when making school choices. All materials--including text messages--were available in English and Spanish, and lessons were delivered in Spanish when requested by the school guidance counselor. Control schools did not receive any materials until after the study was complete. (These schools were later provided Fast Facts lists that they could used with their 7th grade students).
Intervention Start Date
2015-09-08
Intervention End Date
2016-05-23

Primary Outcomes

Primary Outcomes (end points)
a. High school choice outcomes (for 1st, top 3, and all choices; matched school; and enrolled school)
i. School presence on the Fast Facts and/or supplementary lists
1. presence on the list
2. presence in the top or bottom half of the list
ii. High school characteristics
1. 4-year graduation rate
2. whether the graduation rate was below the Fast Facts threshold of 70%
3. travel time from the middle school to the high school
4. location in the same borough as the student’s residence
5. applications per seat in the prior year (a measure of demand)
6. admissions method (e.g. screened or limited unscreened, different types of high schools in the NYC system)
7. variability in graduation rates (the difference between the highest and lowest graduation rate of schools appearing on a student's application)
8. academic or career interest area (e.g. STEM)
9. within-application consistency in interest area, calculated as the highest percentage of choices from the same interest area
i. Other admissions process and enrollment outcomes
1. number of choices submitted (up to12)
2. open house priority status for limited unscreened programs (a lottery preference given to students who attend an event for certain schools)
3. ranking of the student by screened programs
4. whether or not the student was matched to his or her 1st choice, 1st-3rd choice, or any choice in the first round
5. participation in the second round after a successful match
6. matriculation to the matched school in 9th grade. Other outcome variables
b. Students’ academic progress and other outcomes in high school
i. Credit accumulation in 9th-12th grade, and GPA;
ii. New York State Regents exam scores and passing rates;
iii. On-time graduation;
iv. Engagement and perceptions of school (e.g. attendance rate and responses to the NYC DOE’s Learning Environment Survey)
C. Post-high school outcomes (pending data availability)
i. College enrollment, persistence, and graduation
ii. Earnings
iii. Arrest records
iv. Voter registration, voting
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Schools were assigned to one of the three “Fast Facts” interventions described in the “interventions” section, or a control group. We recruited 165 schools from the more than 500 schools serving 8th grade students in NYC, focusing on some of the highest-poverty schools in the city. To increase power over a simple cluster randomized trial in which schools are randomly assigned to treatment conditions, we randomized schools within 39 blocks of similar schools located in the same borough. Blocks were matched quadruplets of schools selected using a Mahalanobis distance measure of difference between schools (see Bruhn & McKenzie 2009; King et al. 2007). School variables used in the matching procedure included prior choice outcomes (e.g., the mean graduation rate of first round matches in 2013-14), prior achievement (mean ELA and math scores in 2013-14), economic disadvantage (the percent eligible for free or reduced price meals), and school size. The 23 schools in our pilot study were blocked separately. For details on randomization please see the online appendix to our NBER Working Paper available here: http://www.nber.org/data-appendix/w24471/w24471.appendix.pdf
Experimental Design Details
Randomization Method
computer program
Randomization Unit
school
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
165 schools
Sample size: planned number of observations
19,109 students
Sample size (or number of clusters) by treatment arms
39 schools to Fast Facts 1, 39 schools to Fast Facts 2, 40 schools to Fast Facts 3, and 47 schools to control

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Our power calculations were done before conducting the experiment. For estimating MDES we made standard assumptions regarding power (.80) and significance (α=.05). We assumed 150 eighth graders per cluster (school), the NYC average, and 2 clusters per block. (In practice there will be 4 schools per block, representing our 3 treatments and one control. For illustrative purposes we provide the MDES for only one treatment-control comparison). We assumed two inter-class (within-middle school) correlations: a relatively low ICC (ρ=0.08) and a modest ICC (ρ=0.20). These correspond to the range of ICCs observed for several outcomes in existing administrative data, after removing the effect of pre-treatment covariates. We assumed a conservative R2 from regression on pre-treatment covariates of 0.1, and that the blocks account for 0.1 to 0.4 of the variance. For example, with 60 schools (30 blocks), we have sufficient power to detect effect sizes ranging from 0.20 under the low ICC, high between-block variation assumption, to 0.35 under the high ICC, low between-block variation assumption. These translate into an increase of 2.3 to 4.1 percentage points in the four-year graduation rate of the first choice high school (or average of the top three choices), 1.1 to 1.7 points in high school value-added to graduation rates, and 2.4 to 3.5 percentage points in the percent of students accumulating 10 or more credits in year one of high school at the first choice (or average of the top three). This exercise offers some confidence in our decision to recruit at least 30 schools per treatment group, and control group.
Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
New York University
IRB Approval Date
2015-08-10
IRB Approval Number
IRB#15-10785
IRB Name
NYC Department of Education
IRB Approval Date
2015-09-02
IRB Approval Number
Study number 736

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
May 23, 2016, 12:00 +00:00
Data Collection Complete
No
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Abstract
We conducted a field experiment in 165 high-poverty New York City middle schools to help students navigate a complex high school choice process and access higher-performing schools. Students in treatment schools were given a customized one-page list of proximate high schools with a graduation rate at or above the city median (70%). Some also received a supplemental list highlighting academically non-selective schools or high schools organized by academic interest area. The interventions changed student application behavior in ways that led to more matches to higher-performing schools. While treatment students did not apply to higher graduation rate schools, they applied to schools where their odds of admission were higher, were more likely to receive their first-choice high school, and were less likely to match to a school with a low graduation rate. Our findings also suggest that informational interventions may not reduce inequality, since both disadvantaged and comparatively advantaged students used our materials, and in some cases the latter benefited more from them by applying and matching to more schools on our lists. Students in non-English speaking households, who were particularly responsive to the intervention and were much less likely to match to a low-performing school, were one notable exception to this pattern.
Citation
Sean P. Corcoran, Jennifer L. Jennings, Sarah R. Cohodes, and Carolyn Sattin-Bajaj. 2018. Leveling the Playing Field for High School Choice: Results from a Field Experiment of Informational Interventions. NBER Working Paper No. 24471.

Reports & Other Materials