|
Field
Trial Start Date
|
Before
July 01, 2019
|
After
June 12, 2024
|
|
Field
Trial End Date
|
Before
January 27, 2020
|
After
June 30, 2025
|
|
Field
Last Published
|
Before
March 02, 2019 01:03 PM
|
After
June 13, 2024 12:20 AM
|
|
Field
Intervention (Public)
|
Before
Job seekers are assigned to Treatment Group A, Treatment Group B, or the control group. Treatment Grop A provides job-seekers with access to a website and smartphone app that provides tailored information about occupations providing relatively high wages and strong employment growth, along with nudges to search and apply for jobs. Treatment Group B receives access to a website and smartphone app that provides non-tailored information about occupations, along with nudges to search and apply for jobs. The control group receives general information.
|
After
Job seekers are assigned to Group A, Group B, or Group C. These users are invited to three different versions of a website that provides job-search information with each version providing different types or amounts of information.
|
|
Field
Intervention Start Date
|
Before
July 01, 2019
|
After
June 12, 2024
|
|
Field
Intervention End Date
|
Before
October 21, 2019
|
After
June 30, 2025
|
|
Field
Primary Outcomes (End Points)
|
Before
The four key outcome variables are:
i) earnings information bias (i.e. the bias in beliefs about how much different occupations pay)
ii) whether or not individuals switch two-digit occupation categories
iii) weekly earnings 12 weeks after filling out the baseline survey and starting the intervention
iv) days nonemployed within first 12 weeks after filling out the baseline survey and starting the intervention
|
After
The primary outcomes from administrative data are indicators for whether an individual is employed in each of the four quarters after random assignment; whether an individual has quarterly earnings above a certain threshold; and the level of quarterly earnings (with and without adjustments for outliers). The primary outcomes from the endline survey include measures of whether individuals believe they have a good idea of the types of jobs that are good matches for them, subjective well-being, financial security, depression, and anxiety. We also will construct measures of the expected utility from the occupation in which individuals are employed. We will analyze these results using intent-to-treat analyses as well as treatment-on-the-treated analyses (where treated is defined as having spent a minimal amount of time on the site). We will use a data-driven approach to select control variables for some specifications. We also will address any imbalance in survey response rates between the treatment and control groups through a bounding exercise and randomized survey completion incentives.
|
|
Field
Primary Outcomes (Explanation)
|
Before
i) will be measured based on surveys asking people to rank 6 randomly chosen occupations form 1-6, with 1 offering the lowest wages for the individual and 6 offering the highest. We will answer this both before and 12 weeks after the information intervention. We will measure information quality as the within person correlation of the responded occupation ranking with the true occupation ranking.
|
After
|
|
Field
Experimental Design (Public)
|
Before
Job-seekers coming into four Michigan one-stop job centers will fill out an intake survey the first time they come in, as part of the normal process of beginning to receive services at the one-stop job center. Individuals completing this intake survey will be asked if they want to participate in a research study about job-search tools. If they provide their informed consent, they will be invited to go to the app/website to fill out a baseline survey on their labor market beliefs. After completing this baseline survey, job-seekers will be randomly assigned to be offered website and app Treatment A, Treatment B, or the control treatment, all described in more detail above.
|
After
Job-seekers coming into four Michigan one-stop job centers will fill out an intake survey the first time they come in, as part of the normal process of beginning to receive services at the one-stop job center. Individuals completing this intake survey will be asked if they want to participate in a research study about job-search tools. After completing this intake survey, job-seekers consenting to be in the study will be randomly assigned to be offered website and app version A, B, or C with different types of job-search information. 50% of people will be assigned to website B, 25% to website A and 25% to website C.
|
|
Field
Randomization Method
|
Before
Randomization will be done in the job-search app/website by computer. Individuals are randomized individually when they fill out the baseline survey on the job-search app/website.
|
After
Randomization will be done in the job-search app/website by computer.
|
|
Field
Randomization Unit
|
Before
Individual job-seeker randomization into Treatment A, treatment group B, and the control group.
|
After
Individual individuals are randomized into group A, B, or C.
|
|
Field
Planned Number of Clusters
|
Before
4 Michigan Works Southwest Ofices
|
After
Treatment is not clustered.
|
|
Field
Planned Number of Observations
|
Before
2500 job-seekers
|
After
5000 individuals
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
834: Treatment Group A (tailored job information)
833: Treatment Group B (general job information)
833: Control Group
|
After
1250 individuals: Group A (tailored job information)
2500 individuals: Group B (general job information)
1250 individuals: Group C (tailored job information with training information)
|
|
Field
Power calculation: Minimum Detectable Effect Size for Main Outcomes
|
Before
i) earnings information bias: unit is correlation, baseline mean is unknown, so we can only say that we estimate that the minimum detectable effect size will be .17 standard deviations
ii) whether or not individuals switch two-digit occupation categories: unit is share switching two digit occupations, the minimum detectable effect size will be .08 percentage points, or .15 standard deviations, which is 33% of the expected baseline mean.
iii) weekly earnings 12 weeks after filling out the baseline survey and starting the intervention: the minimum detectable effect size is $61 per week, which is .12 standard deviations and 11% of the baseline mean
iv) days nonemployed within first 12 weeks after filling out the baseline survey and starting the intervention: the minimum detectable effect size will be 2.4 days nonemployed within the first 12 weeks of filling out the baseline survey, which is .12 standard deviations or 4.7% of the baseline mean.
|
After
|
|
Field
Keyword(s)
|
Before
Labor
|
After
Labor
|
|
Field
Intervention (Hidden)
|
Before
Setting and Sample
Our partner is Michigan Works! Southwest (MWS), which operates federally-funded American Job Centers in Branch, Calhoun, Kalamazoo, and St. Joseph counties. MWS provides a variety of services under the Wagner-Peyser Act Employment Services, Workforce Innovation and Opportunity Act (WIOA), Trade Adjustment Assistance (TAA), and cash assistance programs. Only the five percent of clients who qualify for WIOA, TAA, or cash assistance programs receive intensive, one-on-one job search assistance. The remaining 95 percent, who qualify only for Employment Services, receive little tailored support. MWS staff believe that these clients' limited information about job opportunities hampers their job search. We focus on individuals eligible only for Employment Services, as the intervention is likely to have the largest impact and interpreting the results is more straightforward for this group.
Intervention
The RCT randomly provides some MWS clients with access to a website and smartphone app that provides tailored information about occupations providing relatively high wages and strong employment growth, along with nudges to search and apply for jobs. The control group receives general information. The main steps of the intervention are:
1) Job seekers who visit MWS complete an intake survey as part of the standard operating procedure. (Details on all surveys are provided below.) All individuals receive a username and link to the website/app via email and text message.
2) Individuals sign into the website/app using a MWS computer or their own computer or smartphone. Individuals complete a short baseline survey that measures their perceptions of labor market opportunities.
3) After individuals complete the baseline survey, we randomly assign them into one of two treatment groups or the control group. For treatment group A, the website/app provides tailored information, based on answers to the intake survey, about the occupations estimated to offer relatively high wages and strong employment growth. (We describe the statistical model that produces these estimates below.) We provide a range of predicted wages that individuals might earn in each occupation, plus links to current job postings from Indeed, one of the largest job sites in the US. For treatment group B, we provide information about the same occupations as in treatment group A, but we do not provide predicted wages in each occupation. For both treatment groups, we only provide information on \textit{relevant} occupations, defined as those in which a sizable number of workers with the same background characteristics are employed (to take an extreme example, we will not provide information about physician jobs to individuals with only a high school degree). Having two treatment groups allows us to identify whether the effects stem from narrowing down the set of all possible occupations to this relevant set (as is done for both treatment groups) or from providing quantitative predictions about wages in each job (done only for treatment group A). Individuals in both treatment groups can use the website/app whenever they want. The website/app sends regular email and text messages that encourage individuals to search for a job.
For the control group, the website/app provides general information -- a list of websites where jobs are posted, such as Indeed and Pure Michigan Talent Connect -- currently provided to MWS clients.
4) Short weekly text message surveys ask individuals about job search activity. An endline survey asks more questions about their employment situation.
|
After
Setting and Sample
Our partner is Michigan Works! Southwest (MWS), which operates federally-funded American Job Centers in Branch, Calhoun, Kalamazoo, and St. Joseph counties.
Intervention
The RCT randomly provides some MWS clients with access to a website and smartphone app that provides tailored information about occupations providing relatively high wages and strong employment growth. The control group receives general information. The main steps of the intervention are:
1) Job seekers who visit MWS complete an intake survey as part of the standard operating procedure. All individuals receive a link to the website/app via email. At this time, they are also randomly assigned to study group A, B, or C.
2) Individuals sign into the website/app using a MWS computer or their own computer or smartphone.
3) Individuals sign into the website/app using a MWS computer or their own computer or smartphone. The website/app is called NextUp Jobs. When they sign in, they are directed to their assigned website type (i.e. group A, group B, or group C).
For the first treatment group (group A), the website/app provides tailored information, based on self-reported information from the MWS intake, about the occupations estimated to offer relatively high wages and high amenities. (We describe the statistical model that produces these estimates below.) For each occupation, we provide a range of possible wages and other information on job characteristics. Individuals in the treatment group can use the website/app whenever they want.
For the second treatment group (group C), the website/app provides the same information on occupations estimated to offer relatively high wages and rapid re-employment. This treatment then provides additional information on whether there is additional training required to get into the given jobs and, if so, what the training path looks like (time period, costs, components, etc…) and what the specific steps are to enroll in such a training program in the Southwest Michigan Area.
The job search website displays three main types of information. First, it provides a user-friendly list of 10-40 occupations that are estimated by us to offer the highest levels of expected utility. This list contains the occupation title, a brief description of the occupation, the expected hourly wage for the individual in this occupation, a description of the occupation’s predicted job growth, and whether the occupation typically offers work from home opportunities or employer-sponsored health insurance. Second, individuals can click on an occupation to see more detailed information: the five most common tasks that are done on the job and the 25th and 75th percentiles of the predicted wage distribution for each individual.
We use individuals’ age, education level, and current or prior occupation to construct predicted wages in each potential target occupation. In particular, we estimate generalized random forest models using workers observed in the 2002-2019 and 2021-2024 Current Population Survey (CPS) who make occupational transitions during the eight CPS survey months (covering 16 calendar months). The dependent variable of the generalized random forest models is the log hourly wage. The potential explanatory variables are individuals’ education level, years of potential labor market experience (equal to age minus years of schooling minus six), the month in which they are searching for a job (to capture seasonal patterns), the state in which they are searching for a job, recent employment growth in their state of residence, recent employment growth in the target occupation, recent employment growth in the target occupation in their state, and the skill distance between individuals’ current/prior occupation and the target occupation. We construct skill distance using 35 different measures of skills— such as the level of reading comprehension, mathematics, or equipment maintenance used on the job—from O*NET. We estimate a separate random forest model for each target occupation. The outputs of the random forest model are the conditional mean and conditional quantiles of the predicted hourly wage for each person (as defined by their observed characteristics) in each occupation.
We restrict the potential set of occupations considered for each individual to a set of feasible occupations using information on education and training requirements, observed occupation-to-occupation transitions, and local job postings. Specifically, we only consider occupations for an individual if no more than 90 percent of individuals in that occupation have more education than them, there are at least 150 people employed in the job in Southwest Michigan in 2016, and there was at least 5 job posting on Indeed in Southwest Michigan for the occupation in May, July, and November 2019, and the occupational transition would not require infeasible amounts of retraining given the O*NET occupational training requirements categorization. This procedure prevents us from recommending occupations in which it would be extremely difficult or infeasible to find a job given the individual’s background.
Finally, we incorporate information on individuals’ stated preferences over job characteristics to calculate the expected utility from each occupation. On the intake survey, individuals answer questions about how strongly they value high wages, working similar hours each week, being able to work from home, and exerting no more than moderate physical activity on the job. We combine these preferences with our customized wage prediction, occupational characteristics from the CPS, and estimates of the average willingness to pay for occupational characteristics from Mas and Pallais (2017) and Maestas et al. (2022). The result is a customized ranking of occupations that reflects both individual-specific wage estimates and individual-specific preferences. Incorporating individual preferences further increases the relevance of the recommended occupations.
For the control group (group B), the website/app provides general information - a list of websites where jobs are posted, such as Indeed, Monster, and Pure Michigan Talent Connect - currently provided to MWS clients.
4) Short weekly text message surveys ask individuals about job search activity. An endline text message survey asks about their current employment situation and occupation. An endline online survey asks more questions about their employment situation, their information and beliefs about different job types, their subjective well-being, and other questions. Survey incentives will be provided for both the text message and qualtrics survey. The survey incentives amounts may be randomized.
|
|
Field
Secondary Outcomes (End Points)
|
Before
i) Number of jobs-applied for per week
ii) Number of jobs-applied for in new occupations (i.e. different than previous occupation) per week
|
After
The secondary outcomes from weekly surveys include job search intensity and the direction of job search. The weekly surveys are not incentivized, so the reliability of these measures is uncertain. The intake survey also allows us to examine heterogeneity in the effects of the intervention.
|
|
Field
Building on Existing Work
|
Before
|
After
No
|