Using stated individual preferences for job ads’ attributes to design a better matching algorithm between job ads and job-seekers

Last registered on July 29, 2022

Pre-Trial

Trial Information

General Information

Title
Using stated individual preferences for job ads’ attributes to design a better matching algorithm between job ads and job-seekers
RCT ID
AEARCTR-0008719
Initial registration date
December 17, 2021

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
December 19, 2021, 1:25 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
July 29, 2022, 3:26 AM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
CREST

Other Primary Investigator(s)

PI Affiliation
MIT / J-PAL
PI Affiliation
CREST / J-PAL
PI Affiliation
CREST

Additional Trial Information

Status
In development
Start date
2022-01-17
End date
2022-12-31
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
The determinants of unemployment in France are numerous, but a well-known cause is the presence of frictions in the labor market. Many frictions are related to informational imperfections, due to the cost of collecting, processing and disseminating information. These imperfections are one of the reasons why job seekers are offered job vacancies which are not entirely suitable for them and may remain unemployed for a long time whereas there are vacancies they would have liked more and to which they could apply.

This experiment consists in asking job seekers to assign a weight to each of the job search criteria (occupation targeted, distance from home, salary, type of contract and working hours) that reflects their preferences, in order to make them more accurate vacancies recommendations regarding their own concerns, through a personalized version of the Pôle emploi (the French public employment service) recommendation algorithm. We will then study extensively their job search behaviour and their return to employment in order to measure the benefits of such intervention.
External Link(s)

Registration Citation

Citation
BANERJEE, Abhijit et al. 2022. "Using stated individual preferences for job ads’ attributes to design a better matching algorithm between job ads and job-seekers." AEA RCT Registry. July 29. https://doi.org/10.1257/rct.8719-3.0
Experimental Details

Interventions

Intervention(s)
Intervention Start Date
2022-01-17
Intervention End Date
2022-12-31

Primary Outcomes

Primary Outcomes (end points)
- Job search behavior: quantity and timing of search effort and the characteristics of jobs applied to.
- General labor market outcomes: unemployment duration and hiring: contract types (temporary or long term), geographic and sectoral mobility.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
This experiment will take place online, a website has been designed for this purpose. It concerns job seekers over the age of majority, registered at Pôle emploi, accepting email communications with Pôle emploi, who agreed to respond to surveys emanating from Pôle emploi, and having a valid Reasonable Employment Offer (contracted research with their advisor, Offre Raisonnable d’Emploi [ORE]).

(1) email invitation :
All job seekers concerned by this experiment will receive by email (using a functional email address) an invitation to participate in the experiment, containing a secured link to an ad hoc and secured website designed for the experiment. This email will also contain a short description of the experiment. If the job seeker clicks on the link, he’ll access to the website landing page.
The only exception are job seekers from the pure control group (see below): they will not receive any invitation to participate in the experiment and only information already collected by Pôle emploi in the usual processes will be used.

(2) Access to the online interface/surveys :
The website landing page contains: (1) a short description of the purpose of the study, (2) what use of their data will be done [they will be fully anonymized before being studied for statistics and research purposes only], (3) a request of informed consent clearly stated and (4) a button to notify consent to participate in the experiment and to access the questionnaire for collecting preferences.

(a) Individual preferences data collection :
The job seekers who notify their consent to participate will access to the survey designed to collect individual preferences. This survey is designed in order to be intuitive. Job seekers have to allocate 15 weights to the five criteria (occupation targeted, distance from home, salary, type of contract and working hours) according to their importance. The whole 15 weights must be allocated, the more important the criterion, the more weight the job seeker allocates to it. If a criterion does not matter, the job seeker can allocate zero weight to it, and if only one criterion is important, he can allocate the whole weights to this single criterion. The button to validate is only accessible when the whole weights are attributed.

(b) Internal validity (PILOT STUDY ONLY) :
One way to ensure that job seekers have understood the first survey, is to ask them to confirm what the algorithm suggests using their individual preferences. Using the individual preferences collected through the questionnaire described above as input of the recommendation algorithm, we select 8 of the personalized vacancies and show them reshuffled to the user. He will be asked to sort them from the one he prefers to the one he likes least, giving him the possibility to put aside 2 offers maximum. We then will be able to validate (or not) the way to collect preferences in the first place by comparing their order to this we found using personalized weighted algorithm.
Job seekers who are not concerned by the pilot study will go directly to the next step (c), without going through this step (b).

(c) personalized job offers and link to the Pôle emploi portal :
Then, simplified job offers will be displayed to the user; according to his treatment group, he will have personalized recommendation using the current algorithm based on his ORE’s criteria (Partial treatment group and Control group, see below) or fully personalized recommendation using his ORE’s criteria and his associated personalized weights (Full treatment group, see below). Clicking on simplified job offers redirects the user to the offer published online on the Pôle Emploi website, and exits the interface.

We distinguish three groups of “treated” job seekers (see figure below):
(1) The individualized preferences will be used on a random subsample of fully treated job seekers, to whom job offers recommendations will be made thanks to the matching system using the weighting corresponding to their own preferences. (2) Even if they have filled in the survey to collect their preferences, a partially treated group of job seekers will get recommendations from the initial and fixed weighting. (3) A control group of remaining job seekers will get recommendations from the initial and fixed weighting, without preferences data collection, (4) and finally, we will also consider another subsample of job seekers we will not directly solicit: the pure control group.

- a first pilot study: 10 000 job seekers will be randomly selected for this stage.
This pilot study aims at: (1) testing the job seekers’ understanding of the questionnaire developed to collect their preferences, (2) testing internal validity; and it also aims at (3) identifying if there are some frequent sets of preferences across job seekers, which could help to identify some job seekers' average profiles and also highlight the limits of the current fixed weighting used in the matching system. Depending on the understanding/internal validity of the questionnaire, the researchers may adapt the preference data collection survey.

- the main study: 250 000 job seekers will be selected for this stage (and will be different from the 10 000 of the pilot study).
Experimental Design Details
Not available
Randomization Method
Clustered randomization: micromarkets (occupation x employment area) will be randomly selected, and then 10 % of each selected micromarket will be randomly selected.
Randomization will be done on a computer in the offices of the French Public Employment Service.
Randomization Unit
Job seeker
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
250 000 job seekers + 10 000 for the pilot study
Sample size: planned number of observations
250 000 job seekers + 10 000 for the pilot study
Sample size (or number of clusters) by treatment arms
83 300 job seekers (250 000/3 approx)
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
MIT COUHES
IRB Approval Date
2022-01-28
IRB Approval Number
2112000531
IRB Name
PSE IRB
IRB Approval Date
2022-02-02
IRB Approval Number
2021-027