NEW UPDATE: Completed trials may now upload and register supplementary documents (e.g. null results reports, populated pre-analysis plans, or post-trial results reports) in the Post Trial section under Reports, Papers, & Other Materials.
The Effects of Youth Letters of Recommendation
Initial registration date
March 21, 2018
October 10, 2019 5:18 PM EDT
This section is unavailable to the public. Use the button below
to request access to this information.
Wharton School - University of Pennsylvania
Other Primary Investigator(s)
University of Michigan
Additional Trial Information
Summer Youth Employment Programs (SYEPs) are city-run programs that provide youth with paid work during the summer. These programs have been shown to significantly improve important youth outcomes including criminality, incarceration, and mortality. However, researchers have failed to find positive average effects on other important outcomes, such as future employment or college attendance. We aim to evaluate whether a minor program change, offering SYEP participants with personalized "letters of recommendation" (i.e., letters than can be shown to potential future employers, teachers, or guidance counselors), can improve outcomes for participating youth.
This trial studies the impact of providing personalized letters of recommendation to youth participants of the New York City Summer Youth Employment Program (SYEP). SYEP provides youth with up to six weeks of employment at public and private organizations throughout New York City. We survey summer supervisors about youth performance. Each youth in the treatment group who is rated highly enough by their supervisor is sent a letter with personalized information from their supervisor about the youth's strengths. Youth in the control group are excluded from receiving a letter of recommendation through our program but are not prevented from requesting letters of recommendation from their employers directly.
Intervention Start Date
Intervention End Date
Primary Outcomes (end points)
Employment and earnings in the formal sector
Primary Outcomes (explanation)
Our main outcomes of interest are measures of success in the formal labor market: employment and earnings. We plan to measure these through whichever data source we can obtain permission to use, likely the NYS Department of Labor Unemployment Insurance records.
Secondary Outcomes (end points)
Pending data availability: college enrollment and attainment; high school enrollment, performance, and attainment; crime outcomes, such as arrests and convictions; job application and letter use rates among a subset of study youth
Secondary Outcomes (explanation)
Since letters may also be used in college applications, we aim to match youth to National Student Clearinghouse college matriculation records. Given that letters may also affect youths' beliefs about themselves and their labor market prospects, we may also seek NYC Department of Education records and NYS arrest/incarceration records, but our use of these outcomes depends on whether we find initial effects on our main outcomes and whether we are able to obtain the data.
We are also aiming to measure an intermediate outcome: how letter receipt changes youth job seeking and job application behavior. A subset of youth have been invited to apply to a job that we are offering, so we can measure application rates and the use of letters of recommendation in job applications.
The administrators of New York City's Summer Youth Employment Program (SYEP) select which summer employers (i.e., supervisors) will be part of the study. With the technical assistance of the research team, those supervisors will be sent a link to a survey instrument in which they will be shown the names of randomly selected youth who worked for them during the summer. After supervisors complete the survey, the research team will generate and send letters of recommendation to all youth who are rated highly enough by their supervisors to receive positive letters. We will subsequently link youth to existing administrative data on employment, and perhaps education and crime outcomes, in order to test whether the offer of a letter improves outcomes over a 3-year follow-up period. Data from a pilot of these measures conducted with 2016 SYEP participants will be combined with data from the full-scale trial of 2017 SYEP participants. Additionally, participants of the 2017 SYEP program will be invited to apply for a job, which will allow us to evaluate how the letter of recommendation affects the job application process directly.
Experimental Design Details
Participants of the 2016 and 2017 NYC Summer Youth Employment Programs were randomly assigned to treatment or control groups. The research team conducted the lottery using Stata.
The randomization was conducted at the individual youth level.
Was the treatment clustered?
Sample size: planned number of clusters
No clustering, see planned number of observations response below.
Sample size: planned number of observations
70,000-80,000 individual youth who were SYEP participants, subject to data cleaning and de-duplication across study years. Additionally, in order to compare our study population to the broader SYEP applicant pool, we may collect data on the broader population of all youth who applied to SYEP in 2016 and 2017, even if they did not participate in SYEP. This broader group would be on the order of 200,000 individual youth. Additionally, we expect some non-compliance driven by factors such as whether supervisors received our survey recruitment emails and clicked on the survey invitation link. Whether a supervisor clicks on the survey invitation link should not differ between treatment and control youth on average, since survey recruitment emails give no indication of which youth were randomized to treatment or control. As such, we may separately analyze subsets where compliance is likely to be higher, but treatment is still random, such as youth who had a supervisor who clicked on a survey invitation link.
Sample size (or number of clusters) by treatment arms
There is only one treatment arm and one control arm. The treatment and control arms each have 35,000-40,000 individual SYEP participants, subject to data cleaning and de-duplication across study years. Additionally, in order to compare our study population to the broader SYEP applicant pool, we may collect data on the broader population of all youth who applied to SYEP in 2016 and 2017, even if they did not participate in SYEP.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
INSTITUTIONAL REVIEW BOARDS (IRBs)
University of Pennsylvania Institutional Review Board
IRB Approval Date
IRB Approval Number