The Effects of Youth Letters of Recommendation

Last registered on August 25, 2022

Pre-Trial

Trial Information

General Information

Title
The Effects of Youth Letters of Recommendation
RCT ID
AEARCTR-0002771
Initial registration date
March 21, 2018

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
March 22, 2018, 3:34 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
August 25, 2022, 4:21 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
Wharton School - University of Pennsylvania

Other Primary Investigator(s)

PI Affiliation
University of Michigan

Additional Trial Information

Status
On going
Start date
2016-09-01
End date
2023-09-01
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Summer Youth Employment Programs (SYEPs) are city-run programs that provide youth with paid work during the summer. These programs have been shown to significantly improve important youth outcomes including criminality, incarceration, and mortality. However, researchers have failed to find positive average effects on other important outcomes, such as future employment or college attendance. We aim to evaluate whether a minor program change, offering SYEP participants with personalized "letters of recommendation" (i.e., letters than can be shown to potential future employers, teachers, or guidance counselors), can improve outcomes for participating youth.
External Link(s)

Registration Citation

Citation
Heller, Sara and Judd Kessler. 2022. "The Effects of Youth Letters of Recommendation." AEA RCT Registry. August 25. https://doi.org/10.1257/rct.2771-2.3
Former Citation
Heller, Sara and Judd Kessler. 2022. "The Effects of Youth Letters of Recommendation." AEA RCT Registry. August 25. https://www.socialscienceregistry.org/trials/2771/history/154208
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
This trial studies the impact of providing personalized letters of recommendation to youth participants of the New York City Summer Youth Employment Program (SYEP). SYEP provides youth with up to six weeks of employment at public and private organizations throughout New York City. We survey summer supervisors about youth performance. Each youth in the treatment group who is rated highly enough by their supervisor is sent a letter with personalized information from their supervisor about the youth's strengths. Youth in the control group are excluded from receiving a letter of recommendation through our program but are not prevented from requesting letters of recommendation from their employers directly.
Intervention Start Date
2016-09-01
Intervention End Date
2018-09-01

Primary Outcomes

Primary Outcomes (end points)
Employment and earnings in the formal sector
Primary Outcomes (explanation)
Our main outcomes of interest are measures of success in the formal labor market: employment and earnings. We plan to measure these through whichever data source we can obtain permission to use, likely the NYS Department of Labor Unemployment Insurance records.

Secondary Outcomes

Secondary Outcomes (end points)
Pending data availability: college enrollment and attainment; high school enrollment, performance, and attainment; crime outcomes, such as arrests and convictions; job application and letter use rates among a subset of study youth
Secondary Outcomes (explanation)
Since letters may also be used in college applications, we aim to match youth to National Student Clearinghouse college matriculation records. Given that letters may also affect youths' beliefs about themselves and their labor market prospects, we may also seek NYC Department of Education records and NYS arrest/incarceration records, but our use of these outcomes depends on whether we find initial effects on our main outcomes and whether we are able to obtain the data.
We are also aiming to measure an intermediate outcome: how letter receipt changes youth job seeking and job application behavior. A subset of youth have been invited to apply to a job that we are offering, so we can measure application rates and the use of letters of recommendation in job applications.

Experimental Design

Experimental Design
The administrators of New York City's Summer Youth Employment Program (SYEP) select which summer employers (i.e., supervisors) will be part of the study. With the technical assistance of the research team, those supervisors will be sent a link to a survey instrument in which they will be shown the names of randomly selected youth who worked for them during the summer. After supervisors complete the survey, the research team will generate and send letters of recommendation to all youth who are rated highly enough by their supervisors to receive positive letters. We will subsequently link youth to existing administrative data on employment, and perhaps education and crime outcomes, in order to test whether the offer of a letter improves outcomes over a 3-year follow-up period. Data from a pilot of these measures conducted with 2016 SYEP participants will be combined with data from the full-scale trial of 2017 SYEP participants. Additionally, participants of the 2017 SYEP program will be invited to apply for a job, which will allow us to evaluate how the letter of recommendation affects the job application process directly.
Experimental Design Details
Randomization Method
Participants of the 2016 and 2017 NYC Summer Youth Employment Programs were randomly assigned to treatment or control groups. The research team conducted the lottery using Stata.
Randomization Unit
The randomization was conducted at the individual youth level.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
No clustering, see planned number of observations response below.
Sample size: planned number of observations
70,000-80,000 individual youth who were SYEP participants, subject to data cleaning and de-duplication across study years. Additionally, in order to compare our study population to the broader SYEP applicant pool, we may collect data on the broader population of all youth who applied to SYEP in 2016 and 2017, even if they did not participate in SYEP. This broader group would be on the order of 200,000 individual youth. Additionally, we expect some non-compliance driven by factors such as whether supervisors received our survey recruitment emails and clicked on the survey invitation link. Whether a supervisor clicks on the survey invitation link should not differ between treatment and control youth on average, since survey recruitment emails give no indication of which youth were randomized to treatment or control. As such, we may separately analyze subsets where compliance is likely to be higher, but treatment is still random, such as youth who had a supervisor who clicked on a survey invitation link.
Sample size (or number of clusters) by treatment arms
There is only one treatment arm and one control arm. The treatment and control arms each have 35,000-40,000 individual SYEP participants, subject to data cleaning and de-duplication across study years. Additionally, in order to compare our study population to the broader SYEP applicant pool, we may collect data on the broader population of all youth who applied to SYEP in 2016 and 2017, even if they did not participate in SYEP.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Pennsylvania Institutional Review Board
IRB Approval Date
2016-08-05
IRB Approval Number
825434
Analysis Plan

Analysis Plan Documents

Heller_Kessler_PAP_10-10-2019

MD5: 0ff8ed7ae99f348243151dc84f01af78

SHA1: bf3e31f8338e31fc2b5b42d94c422ed433ac1427

Uploaded At: October 10, 2019

Heller_Kessler_PAP

MD5: 4921740544aa8d10acc6ab57d878f6b3

SHA1: 9947f94645904897670c47701eec32d9c0dc8a3c

Uploaded At: March 01, 2019

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials