Secondary Outcomes (end points)
Alternate specifications to primary outcome
Alternate outcomes
Y = whether an applicant applied to a nudged program first.
Y = rank position of the first nudged program listed in profile
Scale-up analyses for match outcomes
Given the outcomes in application behavior (do treatment applicants list, and where do they list nudged programs, compared to the treatment group), we will simulate:
Match outcomes under scaleups of the treatment group, for example, if everyone in the target group was in control, or everyone in treatment.
Match outcomes under stronger interventions, that increase application behavior takeup of applications to nudge programs.
Relatedly, In all cases, congestion: rate of acceptance among applicants with high offer likelihood estimates, at each high school. Estimating congestion at scale-up analyses, including with corresponding alternative choices of program nudge caps.
This will help estimate match outcomes under worlds where the intervention is scaled up.
Email click response outcomes
We have individual applicant click data on the emails (separate url for each treatment applicant-nudged school pair). We will analyze this click data to measure engagement with the nudges. An applicant click on a specific link is defined as whether the link is clicked at least once before the application deadline.
We will use the following measurements on click behavior, all measured as clicks before the application deadline:
Did the applicant click on at least one link
Number of clicks by the applicant
Fraction of nudged programs on which the applicant clicked.
For each measurement, we will report:
Overall click rate
Heterogeneous click behavior by free/reduced lunch status, middle school grades/application competitiveness, other demographic characteristics; specific nudged high school
While heterogeneous application and match outcomes are underpowered for the expected engagement levels, these initial engagement outcomes may provide information toward heterogeneous behavior and engagement.
---------------
Survey response outcomes [Added on 3/5/2026, after survey closed but before responses were analyzed]
For example, satisfaction with match outcome; knowledge/awareness of nudged programs
For each applicant, we sample one high-performing program within 45 minutes of their home where they have high offer likelihood. If the applicant applied to at least one such program, the sampled program is randomly drawn from that set. If they did not, the sampled program is randomly drawn from all such programs. The survey is individualized to ask each applicant about their individual sampled program, and we analyze outcomes on their pre-determined sampled program.
Primary survey outcomes
Our primary hypothesis is that the nudges increase applicants’ familiarity with high-performing high school programs near their home. We will measure both of the following through regressions in which the treatment assignment is an indicator variable. We will cluster standard errors by middle school, and include stratum fixed effects. We will also look at descriptive statistics like mean, median, and variance of these outcomes by group.
Self-reported familiarity:
Y = How familiar an applicant says they are with high-performing programs near their home.
Applicants use a 4-point scale to report their perceived level of familiarity with high-performing programs near their home (1 = Not familiar at all; 4 = Very familiar). The outcome will be analyzed using an ordered logit model (and OLS using the outcome as a continuous variable, for robustness).
Familiarity with specific high-performing program:
Y = Whether an applicant is familiar with the specific sampled high-performing program near their home, where they have high offer likelihood.
We will measure this through logistic regression (and OLS for robustness).
Secondary and exploratory survey analyses
Responses to emails (treatment group only)
We will look at rates of each of these outcomes.
Y = Whether an applicant reports reading the email
Y = Whether an applicant reports that the email affected their application decisions
Y = Whether an applicant reports that their perception of the sampled nudged program improved upon reading the email
Y = Whether an applicant who applied to sampled nudged program reports they would not have applied if they had not received the email
Y = Whether an applicant reports the email was helpful
Other perceptions
We will look at rates of each of these outcomes.
Y = Whether an applicant reports knowing the program was at a high-performing school (before / in the absence of nudge email)
Y = Whether an applicant reports knowing the program was within 45 minutes of their home (before / in the absence of nudge email)
Y = Whether an applicant reports knowing they had high offer likelihood at the program (before / in the absence of nudge email)
Y = Whether an applicant reports they did not apply to the sampled high-performing program because of given reason [I did not know enough about it; I did not think my child would get in; It’s too far from home; It does not match my child’s interests]
Y = Whether an applicant reports they are interested in receiving recommendations (control and non-target groups only)
We will look at descriptive statistics of feature importance ratings.
Y = Importance weights of various features [Program theme or Interest Area; School academic performance; Extracurricular offerings; High chance of offer; Distance from home; Safety] when receiving recommendations
Analysis on self-reported compilers (only treatment students who say they read the email)
We will look at rates of each of these outcomes. These analyses condition on self-reported email reading and are exploratory.
Y = How familiar an applicant says they are with high-performing programs near their home
Y = Whether an applicant is familiar with the specific sampled high-performing program near their home, where they have high offer likelihood
Y = Whether an applicant says the email affected their application decisions
Y = Whether an applicant applied (listed somewhere in their list) to at least 1 program to which they were nudged
Other outcomes given applicant’s primary information sources
We will measure these through regressions in which there is an indicator variable for each potential information source. These analyses are based on self-reported information sources and are exploratory.
Y = How familiar an applicant says they are with high-performing programs near their home
Y = Whether an applicant applied (listed somewhere in their list) to at least 1 program to which they were nudged
Y = Whether an applicant read the email (treatment group only)
Y = Whether an applicant says they are interested in receiving recommendations (control and non-target groups only)
Qualitative outcomes
We will do qualitative coding to analyze free-response answers on the following. These findings are descriptive and exploratory.
Reasons for not applying to specific program
Reasons for not considering recommendations
Perceptions of email
What would have helped them navigate the process better
---------------
We will not adjust for multiple comparisons across secondary outcomes. Instead, we will clearly label primary (confirmatory) vs. secondary (pre-specified) vs. exploratory analyses.
Exploratory analyses
The same as our primary outcomes, but calculating the Local Average Treatment Effect (LATE/CACE): where a complier is defined as someone who clicks on at least one school link in their given email.
Heterogeneous treatment effects by: FRL status, geography, middle school grades/application competitiveness, other demographic characteristics.
Quantitative match/enrollment metrics: Graduation rate/performance/impact/safety of assigned/enrolled school
Within-applicant comparing applicant saved school lists over time (from applicant data dumps, before and after nudges)
Outcomes at [high school or middle school] school level
Increase in matches/enrollment from target middle schools to nudged high schools, as fraction of applicants eligible for nudges.
Comparing middle school outcomes from previous year(s)
Long-term metrics [in years after experiment]
9th, 10th, 11th, 12th grade State test scores
Graduation rate
College enrollment rate