Nudges for high school applications

We send emails to applicants with lists of programs to consider applying to.

The intervention consists of a “nudge” email, with applicants within the experimental group randomly assigned either to treatment (receiving emails) or control (not receiving emails). Initial emails to treatment students will be sent in two batches, one in late October and the other in mid November. A follow-up email with feedback on their application will be sent a few weeks after the initial email for each batch. Applicants in the experimental group will also receive a text message from NYCPS on the date of the mid November emails being out.

2025-10-30

2025-12-13

Our primary hypothesis is that the nudges will increase the rate at which applicants apply to (list in their ranked list) programs that they are nudged to, compared to the control group. In particular:
For each treatment applicant, we have the set of programs to which they were nudged.
For each control applicant, we have the set of programs to which they would have been nudged had they been in the treatment group.

Then, our primary outcome is:
Y = Whether an applicant applied (listed somewhere in their list) to at least 1 program to which they were nudged.

We will measure this through logistic regression in which the treatment assignment is an indicator variable. We will cluster standard errors by middle school, and include analyses with controls corresponding to the stratification variables.

The co-primary outcome is:
Y = Whether an applicant matched (listed somewhere in their list) to at least 1 program to which they were nudged.

Alternate specifications to primary outcome
Alternate outcomes
Y = whether an applicant applied to a nudged program first.
Y = rank position of the first nudged program listed in profile

Scale-up analyses for match outcomes
Given the outcomes in application behavior (do treatment applicants list, and where do they list nudged programs, compared to the treatment group), we will simulate:
Match outcomes under scaleups of the treatment group, for example, if everyone in the target group was in control, or everyone in treatment.
Match outcomes under stronger interventions, that increase application behavior takeup of applications to nudge programs.
Relatedly, In all cases, congestion: rate of acceptance among applicants with high offer likelihood estimates, at each high school. Estimating congestion at scale-up analyses, including with corresponding alternative choices of program nudge caps.
This will help estimate match outcomes under worlds where the intervention is scaled up.

Email click response outcomes
We have individual applicant click data on the emails (separate url for each treatment applicant-nudged school pair). We will analyze this click data to measure engagement with the nudges. An applicant click on a specific link is defined as whether the link is clicked at least once before the application deadline.

We will use the following measurements on click behavior, all measured as clicks before the application deadline:
Did the applicant click on at least one link
Number of clicks by the applicant
Fraction of nudged programs on which the applicant clicked.

For each measurement, we will report:
Overall click rate
Heterogeneous click behavior by free/reduced lunch status, middle school grades/application competitiveness, other demographic characteristics; specific nudged high school
While heterogeneous application and match outcomes are underpowered for the expected engagement levels, these initial engagement outcomes may provide information toward heterogeneous behavior and engagement.

---------------
Survey response outcomes [Added on 3/5/2026, after survey closed but before responses were analyzed]
For example, satisfaction with match outcome; knowledge/awareness of nudged programs

For each applicant, we sample one high-performing program within 45 minutes of their home where they have high offer likelihood. If the applicant applied to at least one such program, the sampled program is randomly drawn from that set. If they did not, the sampled program is randomly drawn from all such programs. The survey is individualized to ask each applicant about their individual sampled program, and we analyze outcomes on their pre-determined sampled program.

Primary survey outcomes
Our primary hypothesis is that the nudges increase applicants’ familiarity with high-performing high school programs near their home. We will measure both of the following through regressions in which the treatment assignment is an indicator variable. We will cluster standard errors by middle school, and include stratum fixed effects. We will also look at descriptive statistics like mean, median, and variance of these outcomes by group.
Self-reported familiarity:
Y = How familiar an applicant says they are with high-performing programs near their home.
Applicants use a 4-point scale to report their perceived level of familiarity with high-performing programs near their home (1 = Not familiar at all; 4 = Very familiar). The outcome will be analyzed using an ordered logit model (and OLS using the outcome as a continuous variable, for robustness).
Familiarity with specific high-performing program:
Y = Whether an applicant is familiar with the specific sampled high-performing program near their home, where they have high offer likelihood.
We will measure this through logistic regression (and OLS for robustness).

Secondary and exploratory survey analyses
Responses to emails (treatment group only)
We will look at rates of each of these outcomes.
Y = Whether an applicant reports reading the email
Y = Whether an applicant reports that the email affected their application decisions
Y = Whether an applicant reports that their perception of the sampled nudged program improved upon reading the email
Y = Whether an applicant who applied to sampled nudged program reports they would not have applied if they had not received the email
Y = Whether an applicant reports the email was helpful

Other perceptions
We will look at rates of each of these outcomes.
Y = Whether an applicant reports knowing the program was at a high-performing school (before / in the absence of nudge email)
Y = Whether an applicant reports knowing the program was within 45 minutes of their home (before / in the absence of nudge email)
Y = Whether an applicant reports knowing they had high offer likelihood at the program (before / in the absence of nudge email)
Y = Whether an applicant reports they did not apply to the sampled high-performing program because of given reason [I did not know enough about it; I did not think my child would get in; It’s too far from home; It does not match my child’s interests]
Y = Whether an applicant reports they are interested in receiving recommendations (control and non-target groups only)

We will look at descriptive statistics of feature importance ratings.
Y = Importance weights of various features [Program theme or Interest Area; School academic performance; Extracurricular offerings; High chance of offer; Distance from home; Safety] when receiving recommendations

Analysis on self-reported compilers (only treatment students who say they read the email)
We will look at rates of each of these outcomes. These analyses condition on self-reported email reading and are exploratory.
Y = How familiar an applicant says they are with high-performing programs near their home
Y = Whether an applicant is familiar with the specific sampled high-performing program near their home, where they have high offer likelihood
Y = Whether an applicant says the email affected their application decisions
Y = Whether an applicant applied (listed somewhere in their list) to at least 1 program to which they were nudged

Other outcomes given applicant’s primary information sources
We will measure these through regressions in which there is an indicator variable for each potential information source. These analyses are based on self-reported information sources and are exploratory.
Y = How familiar an applicant says they are with high-performing programs near their home
Y = Whether an applicant applied (listed somewhere in their list) to at least 1 program to which they were nudged
Y = Whether an applicant read the email (treatment group only)
Y = Whether an applicant says they are interested in receiving recommendations (control and non-target groups only)

Qualitative outcomes
We will do qualitative coding to analyze free-response answers on the following. These findings are descriptive and exploratory.
Reasons for not applying to specific program
Reasons for not considering recommendations
Perceptions of email
What would have helped them navigate the process better
---------------

We will not adjust for multiple comparisons across secondary outcomes. Instead, we will clearly label primary (confirmatory) vs. secondary (pre-specified) vs. exploratory analyses.

Exploratory analyses
The same as our primary outcomes, but calculating the Local Average Treatment Effect (LATE/CACE): where a complier is defined as someone who clicks on at least one school link in their given email.
Heterogeneous treatment effects by: FRL status, geography, middle school grades/application competitiveness, other demographic characteristics.
Quantitative match/enrollment metrics: Graduation rate/performance/impact/safety of assigned/enrolled school
Within-applicant comparing applicant saved school lists over time (from applicant data dumps, before and after nudges)
Outcomes at [high school or middle school] school level
Increase in matches/enrollment from target middle schools to nudged high schools, as fraction of applicants eligible for nudges.
Comparing middle school outcomes from previous year(s)
Long-term metrics [in years after experiment]
9th, 10th, 11th, 12th grade State test scores
Graduation rate
College enrollment rate

We identify a set of target middle schools, and nudge applicants from those middle schools to a set of high-performing high schools nearby.

We identify a set of high-performing High Schools, based on criteria with NYCPS. From these, a subset of programs at these high schools belongs to the set of “nudged” programs, based on various internal criteria and if they have offer likelihoods for admissions available.


Target Middle Schools
A public middle school X is selected to be a “target middle school” for the experiment based on various criteria, including whether the middle school historically did not send many students to high performing high schools. Furthermore, applicants from a target middle school will NOT be sent nudges to a specific nudged high school if a relatively high fraction of its applicants already attend that high school.

Target middle school statistics:
218 target middle schools have at least 1 nudge-eligible student in 2025; these 218 schools (with ~20,000 total students in 2025 data) become our experimental middle schools.
Students from the experimental middle schools account for 8.5% of seats at high-performing high school programs and 10% of seats at nudged high schools, despite accounting for 29% of middle school seats overall.

Nudge-eligible applicants
Applicants are eligible to receive a nudge to a certain nudged high school if all of the following criteria hold:
Their current school is a target middle school for the high school
Their home address census tract is within 45 minutes of the high school (by OpenTripPlanner’s estimates of travel distance using transit data)
They have a high estimated offer likelihood at the high school

Nudge-eligible applicant statistics, based on 2025 cycle data:
~4,000 applicants eligible for nudges (of ~72,000 total)

Experimental population
Finally, an applicant is considered in the experimental population if:
The applicant is “nudge-eligible”
Their application is “in progress” for the 2026 application cycle during the dates that the nudge emails are sent (sent in two batches): the application has been started but not submitted when the applicant data is pulled for each email batch (0-2 days before each email batch is sent out).

Based on previous years, we expect ~1300 applicants to be in the experimental population. The drop from nudge-eligible to experimental population is due to the number of applicants expected to have in progress applications during the two nudge email batch periods.

Treatment group size
About 500 applicants will be in the treatment group, as described below (randomization occurs at the middle-school level).

Allocating nudges among treatment applicants
To determine which programs we actually recommend to which applicants (among applicants in the treatment group), we model the problem as an integer optimization problem.

For each (applicant, program) pair where the applicant is eligible to be nudged toward the program, a binary decision variable indicates whether the applicant will actually be nudged toward that program. At a high level, we want to give many nudges to each applicant, but cannot give too many nudges to any particular school, due to induced congestion.

Each program P can be recommended to at most x_P applicants, where x_P is approximately the number of nudges such that the program would still accept at least 95% of applicants who saw a high offer likelihood estimate if up to 40% of nudged applicants actually list their nudged programs at the top of their list (simulated using 2025 applicant application data). Each applicant can be recommended at most 10 programs. These nudge caps {x_P} are determined before the experiment begins.

We use Gurobi to find a solution to the optimization problem that maximizes the sum across applicants of the square root of the number of programs recommended to them, so as to reward balanced allocations among applicants (i.e., all else equal, the model prioritizes an applicant receiving a 2nd nudge over another applicant receiving a 3rd nudge).

We will run a separate optimization for each “batch” (where an applicant’s batch is determined solely by when they open their application, since applicants must open their application in order for us to compute their offer likelihood at each program), scaling all program nudge caps based on the size of the control group relative to the treatment group and the batch size.

We will also send follow-up emails to treatment-group applicants according to the following rules, with text that depends on their application status.

Randomization will be done privately in the NYCPS office on a computer using Python code

Among target middle schools, the share of applicants receiving treatment will vary, with 0%, 50%, and 100% of applicants treated. Middle schools will be assigned these values at random, with stratification based on geography (borough) and number of nudge-eligible applicants in the application cycle for fall 2025 (2 size categories). The ratios of these will be approximately 50% at 0% treatment intensity, and 25% at 50% and 100% treatment intensity, so as to achieve a treatment group size of ~500 from a set of 1300 students in the overall experimental population.

Yes

Approximately 220 middle schools

Approximately 1300 applicants

~105 middle schools as control, ~55 at 50% treatment, ~55 at 100% treatment

For primary outcome of application behavior change (takeup): 80% power for about 7% takeup compared to control.