The Effects of Youth Letters of Recommendation

This trial studies the impact of providing personalized letters of recommendation to youth participants of the New York City Summer Youth Employment Program (SYEP). SYEP provides youth with up to six weeks of employment at public and private organizations throughout New York City. We survey summer supervisors about youth performance. Each youth in the treatment group who is rated highly enough by their supervisor is sent a letter with personalized information from their supervisor about the youth's strengths. Youth in the control group are excluded from receiving a letter of recommendation through our program but are not prevented from requesting letters of recommendation from their employers directly.

2016-09-01

2018-09-01

Employment and earnings in the formal sector

Our main outcomes of interest are measures of success in the formal labor market: employment and earnings. We plan to measure these through whichever data source we can obtain permission to use, likely the NYS Department of Labor Unemployment Insurance records.

Pending data availability: college enrollment and attainment; high school enrollment, performance, and attainment; crime outcomes, such as arrests and convictions; job application and letter use rates among a subset of study youth

Since letters may also be used in college applications, we aim to match youth to National Student Clearinghouse college matriculation records. Given that letters may also affect youths' beliefs about themselves and their labor market prospects, we may also seek NYC Department of Education records and NYS arrest/incarceration records, but our use of these outcomes depends on whether we find initial effects on our main outcomes and whether we are able to obtain the data.
We are also aiming to measure an intermediate outcome: how letter receipt changes youth job seeking and job application behavior. A subset of youth have been invited to apply to a job that we are offering, so we can measure application rates and the use of letters of recommendation in job applications.

The administrators of New York City's Summer Youth Employment Program (SYEP) select which summer employers (i.e., supervisors) will be part of the study. With the technical assistance of the research team, those supervisors will be sent a link to a survey instrument in which they will be shown the names of randomly selected youth who worked for them during the summer. After supervisors complete the survey, the research team will generate and send letters of recommendation to all youth who are rated highly enough by their supervisors to receive positive letters. We will subsequently link youth to existing administrative data on employment, and perhaps education and crime outcomes, in order to test whether the offer of a letter improves outcomes over a 3-year follow-up period. Data from a pilot of these measures conducted with 2016 SYEP participants will be combined with data from the full-scale trial of 2017 SYEP participants. Additionally, participants of the 2017 SYEP program will be invited to apply for a job, which will allow us to evaluate how the letter of recommendation affects the job application process directly.

Participants of the 2016 and 2017 NYC Summer Youth Employment Programs were randomly assigned to treatment or control groups. The research team conducted the lottery using Stata.

The randomization was conducted at the individual youth level.

No

No clustering, see planned number of observations response below.

70,000-80,000 individual youth who were SYEP participants, subject to data cleaning and de-duplication across study years. Additionally, in order to compare our study population to the broader SYEP applicant pool, we may collect data on the broader population of all youth who applied to SYEP in 2016 and 2017, even if they did not participate in SYEP. This broader group would be on the order of 200,000 individual youth. Additionally, we expect some non-compliance driven by factors such as whether supervisors received our survey recruitment emails and clicked on the survey invitation link. Whether a supervisor clicks on the survey invitation link should not differ between treatment and control youth on average, since survey recruitment emails give no indication of which youth were randomized to treatment or control. As such, we may separately analyze subsets where compliance is likely to be higher, but treatment is still random, such as youth who had a supervisor who clicked on a survey invitation link.

There is only one treatment arm and one control arm. The treatment and control arms each have 35,000-40,000 individual SYEP participants, subject to data cleaning and de-duplication across study years. Additionally, in order to compare our study population to the broader SYEP applicant pool, we may collect data on the broader population of all youth who applied to SYEP in 2016 and 2017, even if they did not participate in SYEP.

Heller_Kessler_PAP_10-10-2019

Heller_Kessler_PAP