|
Field
Abstract
|
Before
Digital job tools promise to reduce frictions on the labor market. In this randomized controlled trial, we study an intervention seeking to increase the usage of a public online platform in France. The intervention, implemented with an adaptive design, consists in sending mails to job seekers, providing information, help and motivation to register or update their profiles on the platform. We focus primarily on discovering which treatments maximize uptake of the tool among the several type of incentives considered. We then analyze the impact of the platform's use on labor market outcomes.
|
After
Digital job tools promise to reduce frictions on the labor market. In this randomized controlled trial, we study an intervention seeking to increase the usage of a public online platform in France. The intervention, implemented with an adaptive design, consists in sending mails to job seekers, providing information, help and motivation to register or update their profiles on the platform. We seek to discover which treatments maximize uptake of the tool among the several types of incentives considered. We then analyze the impact of the platform's use on labor market outcomes.
|
|
Field
Trial Start Date
|
Before
September 26, 2022
|
After
January 09, 2023
|
|
Field
Trial End Date
|
Before
December 31, 2022
|
After
December 31, 2023
|
|
Field
JEL Code(s)
|
Before
J23, J24, C93
|
After
J23, J24, C93, C80, C81, C90
|
|
Field
Last Published
|
Before
September 27, 2022 12:08 PM
|
After
January 19, 2023 11:43 AM
|
|
Field
Intervention (Public)
|
Before
The intervention aims at sending emails to registered job seekers in order to encourage them to engage with their profile on the public employment service's platform, i.e to fill the profile and potentially publish it to become visible to recruiters. We test several treatments that either provide information or reduce the cost of the tool's use, in combination with behavioral levers.
|
After
We send emails to registered job seekers in order to encourage them to use their profile on the public employment service's platform, i.e to fill the profile and potentially publish it to become visible to recruiters. We test several emails to maximize take-up by either providing information or reducing the cost of the tool's use, in combination with behavioral levers. We also test two different sending hours.
- Informational treatments
(Info1) Information about the use by recruiters
In order to emphasize the potential gains of filling the profile, we highlight its use by other agents. We show information about the number of recruiters looking for candidates on the platform each month. We vary whether this information appears or not.
(Info2) Information about the gains in terms of service quality
Another information shared is related to the effect of providing information through the profile on the quality of the service given by the institution. Counselors rely on the profile to recommend job ads to job seekers and so do automatic recommender systems built by the institution. We highlight how a well-filled skill profile can enable Pôle emploi's counselors to better assist individuals in finding a job, a training program and more broadly communicate with them. We vary whether this information is present or not.
- Help provision treatments
(Help1) Including a list of steps
In order to reduce the cost of completion we write the various steps to follow in order to fully mobilize the tool directly in the body of the email. We vary whether this tutorial is proposed or not.
- Including an intensive help
Some job seekers may find it difficult to complete their profile independently. Some may have difficulties with the French language or with digital tools. Others may want to be accompanied in the process because of a lack of self-confidence. We propose an intensive help that can take 2 forms:
o (Help2workshop) Workshop: We propose job seekers to self-register to a half-day workshop designed to help them fill-in their profile at the PES.
o (Help2counselor) Counselor: We propose jobseekers to book an appointment with a counselor via a call-to-action button which redirects them to the appropriate webpage.
We vary whether the intensive help is proposed or not and whether it takes the form of a workshop or a counselor appointment.
- Scheduled time to send the email
(Schedule) Sending hours:
We vary whether the email is sent at 9am or at 3pm.
Possible combinations
We aim to test the interaction of some of these email contents. We selected a subset of interactions between email contents that we considered to have a chance of being relevant, leading to a total of 16 different emails summarized.
Since emails can be sent at either 9am or 3pm, this experiment includes a total of 32 treatment arms.
|
|
Field
Intervention Start Date
|
Before
September 26, 2022
|
After
January 10, 2023
|
|
Field
Intervention End Date
|
Before
December 31, 2022
|
After
April 10, 2023
|
|
Field
Primary Outcomes (Explanation)
|
Before
- Online profile related outcomes: (1) Whether the job seeker connected to his account, went to the profile page within a 2 days time window after sending emails and modified his/her profile (2) Whether the job seeker published his profile on the public employment service's website (3) Number of visits on the job seeker's profile by recruiters and or by caseworkers (4) An indicator of the degree of completion of the profile in %
- Y = 1 if the job seeker visits relevant pages on Pôle emploi's website after 2 days, 0 otherwise
- Labor market outcomes: (1) number of recruiter clicks on the individual's profile, (2) number and type of job ads recommended to the job seeker by the caseworker and by automatic suggestions (3) number and type of ads the job seeker clicked on or applied to (4) return to employment
|
After
- Online profile related outcomes: (1) Whether the job seeker connected to his account, went to the profile page for different time windows (3, 7 and 30 days) after sending emails and modified his/her profile (2) Whether the job seeker published his profile on the public employment service's website (3) Number of visits on the job seeker's profile by recruiters and or by caseworkers (4) An indicator of the degree of completion and quality of the profile in percentages.
- Usage of Pôle emploi's website in general: Whether the job seeker visits pages related to job search on Pôle emploi's website for different time windows (3, 7 and 30 days).
- Labor market outcomes: (1) number and type of job ads recommended to the job seeker by the caseworker and by automatic suggestions (2) number of times the jobseeker was contacted by recruiters on the website (3) number and type of ads the job seeker clicked on or applied to (4) return to employment (5) characterization of the job when re-employed.
|
|
Field
Experimental Design (Public)
|
Before
We repeatedly follow these steps over the 300,000 individuals:
1. We sample 20 000 job seekers from the eligible population at time t.
2. We split them in two equal-sized samples: a control (who does not receive any incentive) and a treated group (who receives one of the incentives according to estimated individual probability of allocation).
3. For the treated group, an algorithm determines allocation probabilities for each individual; treatments are then assigned accordingly.
4. Individuals in the treated group are sent a message shortly after the assignment.
5. We observe whether individuals in the treated group clicked on relevant pages on Pôle emploi's website and this information is used by the algorithm to update the allocation probabilities.
|
After
Each week, we follow these steps:
1. We sample 36 000 job seekers from the eligible population at week t.
2. We split them in two samples: 2/3 of the sample is allocated to a control group (who does not receive any incentive) and the other 1/3 is allocated to a treated group (who receives one of the incentives according to estimated individual probability of allocation).
3. For the treated group, an algorithm* determines allocation probabilities for each individual; treatments are then assigned accordingly.
4. Individuals in the treated group are sent a message shortly after the assignment.
5. We observe whether individuals in the treated group clicked on the pages of the online profile. This information is used by the algorithm to update the allocation probabilities.
After at most 10 weeks, we stop the adaptive assignment and keep some observations for the evaluation of the assignment policy learnt.
*Description of the algorithm: we use a honest generalized random forest to predict the impact of each treatment based on data collected the previous weeks. Assignments are generated using a Thompson sampling procedure based on the mean and variances of the indiviudal predictions from the forest (the first week, interventions are assigned at random).
|
|
Field
Randomization Method
|
Before
The allocation to the control or treated groups is done by a computer with uniform probabilities. Within the treated group, the kind of intervention is chosen using a contextual bandits algorithm (i.e, randomization done by a machine learning algorithm with personalized sampling weights).
|
After
The allocation to the control or treated groups is done by a computer with uniform probabilities (with mean 1/3). Within the treated group, the kind of intervention is chosen using a contextual bandits algorithm (i.e, randomization done by a machine learning algorithm with personalized sampling weights).
|
|
Field
Planned Number of Observations
|
Before
~ 300,000 observations
|
After
~ 450,000 observations, one third in the treated group.
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
The treated group of size 150,000 is split between 16 treatment arms; the sample size for each arm is not known in advance (it is progressively adapted so that people are sent to the best treatments).
|
After
The treated group of size 150,000 is split between 32 treatment arms; the sample size for each arm is not known in advance (it is progressively adapted so that people are sent to the best treatments).
|
|
Field
Additional Keyword(s)
|
Before
Adaptive experiment, contextual bandits, online matching, employment platform
|
After
Adaptive experiment, contextual bandits, online matching, job platform, job search, take-up, nudges
|