|
Field
Primary Outcomes (End Points)
|
Before
The primary outcomes from administrative data are indicators for whether an individual is employed in each of the four quarters after random assignment; whether an individual has quarterly earnings above a certain threshold; and the level of quarterly earnings (with and without adjustments for outliers). The primary outcomes from the endline survey include measures of whether individuals believe they have a good idea of the types of jobs that are good matches for them, subjective well-being, financial security, depression, and anxiety. We also will construct measures of the expected utility from the occupation in which individuals are employed. Some individuals might not receive incentives to complete the endline survey, which could lower the reliability of these results. We will analyze these results using intent-to-treat analyses as well as treatment-on-the-treated analyses (where treated is defined as having spent a minimal amount of time on the site). We will use a data-driven approach to select control variables for some specifications. We also will address any imbalance in survey response rates between the treatment and control groups through a bounding exercise and randomized survey encouragement messages.
|
After
The primary outcomes from administrative data are indicators for whether an individual is employed in each of the four quarters after random assignment; whether an individual has quarterly earnings above a certain threshold; and the level of quarterly earnings (with and without adjustments for outliers). The primary outcomes from the endline survey include measures of whether individuals believe they have a good idea of the types of jobs that are good matches for them, subjective well-being, financial security, depression, and anxiety. We also will construct measures of the expected utility from the occupation in which individuals are employed and look at whether people are employed in high wage occupations more generally. Some individuals might not receive incentives to complete the endline survey, which could lower the reliability of these results. We will analyze these results using intent-to-treat analyses as well as treatment-on-the-treated analyses (where treated is defined as having spent a minimal amount of time on the site). We will use a data-driven approach to select control variables for some specifications. We also will address any imbalance in survey response rates between the treatment and control groups through a bounding exercise.
|
|
Field
Intervention (Hidden)
|
Before
The intervention randomly provides some individuals with access to NextUp Jobs, a website developed by the Allegheny County Department of Human Services (ACDHS) to help job seekers. The website contains customized information about occupations that are likely to be feasible and attractive to each person.
Individuals will be recruited to the study through two key channels. First, ACDHS will randomly select some individuals from the data warehouse to be contacted each month. These individuals will receive a text message that briefly describes the website and directs them to a website where they will complete the baseline survey. Second, individuals will be able to indicate their interest in the study by completing the baseline survey through a publicly available URL on the ACDHS website. For both recruitment channels, individuals who complete the baseline survey will be randomly assigned to the treatment or control groups (with 50 percent probability of each group).
The treatment group will receive text messages, emails, and letters that contain a personalized URL that allows them to access the job search website. The survey control group will not receive information from DHS about how to access the website during the study period, but they will receive the same surveys as the treatment group. The study also will feature a “pure control” group who will not receive any communications about the study or survey invitations, but their outcomes will be tracked in administrative data.
Individuals in the treatment group will receive text messages, emails, letters, and/or telephone calls that encourage them to use the job search tool. Individuals in the treatment group and control group will receive weekly text message surveys, followed by an endline survey.
The job search website displays three main types of information. First, it provides a user-friendly
list of 10-40 occupations that are estimated by us to offer the highest levels of expected
utility. This list contains the occupation title, a brief description of the occupation, the expected
hourly wage for the individual in this occupation, a description of the occupation’s predicted job
growth, and whether the occupation typically offers work from home opportunities or employer-sponsored health insurance. Second, individuals can click on an occupation to see more detailed information: the five most common tasks that are done on the job and the 25th and 75th percentiles of the predicted wage distribution for each individual. Finally, the website provides occupation-specific job postings from Indeed within a 25-mile commuting radius of the person’s ZIP code of residence.
We use individuals’ age, education level, and current or prior occupation to construct predicted
wages in each potential target occupation. In particular, we estimate generalized random forest
models using workers observed in the 2002-2019 and 2021-2022 Current Population Survey (CPS) who make occupational transitions during the eight CPS survey months (covering 16 calendar months). The dependent variable of the generalized random forest models is the log hourly wage. The potential explanatory variables are individuals’ education level, years of potential labor market experience (equal to age minus years of schooling minus six), the month in which they are searching for a job (to capture seasonal patterns), the state in which they are searching for a job, recent employment growth in their state of residence, recent employment growth in the target occupation, recent employment growth in the target occupation in their state, and the skill distance between individuals’ current/prior occupation and the target occupation. We construct skill distance using 35 different measures of skills—such as the level of reading comprehension, mathematics, or equipment maintenance used on the job—from O*NET. We estimate a separate random forest model for each target occupation. The outputs of the random forest model are the conditional mean and conditional quantiles of the predicted hourly wage for each person (as defined by their observed characteristics) in each occupation.
We restrict the potential set of occupations considered for each individual to a set of feasible
occupations using information on education and training requirements, observed occupation-to-occupation transitions, and local job postings. Specifically, we only consider occupations for an individual if no more than 90 percent of individuals in that occupation have more education than them, there are at least 100 people employed in the job in Pittsburgh in 2020, and there was at least one job posting on Indeed in Pittsburgh for the occupation in both March and May 2023, and the occupational transition would not require infeasible amounts of retraining given the O*NET occupational training requirements categorization. This procedure prevents us from recommending occupations in which it would be extremely difficult or infeasible to find a job given the individual’s background.
Finally, we incorporate information on individuals’ stated preferences over job characteristics to calculate the expected utility from each occupation. On the job search website, individuals answer questions about how strongly they value high wages, working similar hours each week, being able to work from home, and exerting no more than moderate physical activity on the job. We combine these preferences with our customized wage prediction, occupational characteristics from the CPS, and estimates of the average willingness to pay for occupational characteristics from Mas and Pallais (2017) and Maestas et al. (2022). The result is a customized ranking of occupations that reflects both individual-specific wage estimates and individual-specific preferences. Incorporating individual preferences further increases the relevance of the recommended occupations.
|
After
The intervention randomly provides some individuals with access to NextUp Jobs, a website developed by the Allegheny County Department of Human Services (ACDHS) to help job seekers. The website contains customized information about occupations that are likely to be feasible and attractive to each person.
Individuals will be recruited to the study through two key channels. First, ACDHS will randomly select some individuals from the data warehouse to be contacted each month. These individuals will receive a text message that briefly describes the website and directs them to a website where they will complete the baseline survey. Second, individuals will be able to indicate their interest in the study by completing the baseline survey through a publicly available URL on the ACDHS website. For both recruitment channels, individuals who complete the baseline survey will be randomly assigned to the treatment or control groups (with 50 percent probability of each group).
The treatment group will receive text messages, emails, and letters that contain a personalized URL that allows them to access the job search website. The survey control group will not receive information from DHS about how to access the website during the study period, but they will receive the same surveys as the treatment group. The study also will feature a "pure control" group who will not receive the weekly job search survey. This group will receive the baseline and endline survey, and their outcomes will be measurable in administrative data.
Individuals in the treatment group will receive text messages, emails, letters, and/or telephone calls that encourage them to use the job search tool. Individuals in the treatment group and control group will receive weekly text message surveys, followed by an endline survey.
The job search website displays three main types of information. First, it provides a user-friendly list of 10-40 occupations that are estimated by us to offer the highest levels of expected utility. This list contains the occupation title, a brief description of the occupation, the expected hourly wage for the individual in this occupation, a description of the occupation’s predicted job growth, and whether the occupation typically offers work from home opportunities or employer-sponsored health insurance. Second, individuals can click on an occupation to see more detailed information: the five most common tasks that are done on the job and the 25th and 75th percentiles of the predicted wage distribution for each individual. Finally, the website provides occupation-specific job postings from Indeed within a 25-mile commuting radius of the person’s ZIP code of residence. Note: This feature was not available throughout the entire study because of a technical change made by Indeed to job postings.
We use individuals’ age, education level, and current or prior occupation to construct predicted wages in each potential target occupation. In particular, we estimate generalized random forest models using workers observed in the 2002-2019 and 2021-2022 Current Population Survey (CPS) who make occupational transitions during the eight CPS survey months (covering 16 calendar months). The dependent variable of the generalized random forest models is the log hourly wage. The potential explanatory variables are individuals’ education level, years of potential labor market experience (equal to age minus years of schooling minus six), the month in which they are searching for a job (to capture seasonal patterns), the state in which they are searching for a job, recent employment growth in their state of residence, recent employment growth in the target occupation, recent employment growth in the target occupation in their state, and the skill distance between individuals’ current/prior occupation and the target occupation. We construct skill distance using 35 different measures of skills—such as the level of reading comprehension, mathematics, or equipment maintenance used on the job—from O*NET. We estimate a separate random forest model for each target occupation. The outputs of the random forest model are the conditional mean and conditional quantiles of the predicted hourly wage for each person (as defined by their observed characteristics) in each occupation.
We restrict the potential set of occupations considered for each individual to a set of feasible occupations using information on education and training requirements, observed occupation-to-occupation transitions, and local job postings. Specifically, we only consider occupations for an individual if no more than 90 percent of individuals in that occupation have more education than them, there are at least 100 people employed in the job in Pittsburgh in 2020, and there was at least one job posting on Indeed in Pittsburgh for the occupation in both March and May 2023, and the occupational transition would not require infeasible amounts of retraining given the O*NET occupational training requirements categorization. This procedure prevents us from recommending occupations in which it would be extremely difficult or infeasible to find a job given the individual’s background.
Finally, we incorporate information on individuals’ stated preferences over job characteristics to calculate the expected utility from each occupation. On the job search website, individuals answer questions about how strongly they value high wages, working similar hours each week, being able to work from home, and exerting no more than moderate physical activity on the job. We combine these preferences with our customized wage prediction, occupational characteristics from the CPS, and estimates of the average willingness to pay for occupational characteristics from Mas and Pallais (2017) and Maestas et al. (2022). The result is a customized ranking of occupations that reflects both individual-specific wage estimates and individual-specific preferences. Incorporating individual preferences further increases the relevance of the recommended occupations.
|