Designing job recommendation algorithms for job seekers: trading-off suitability to search criteria and hiring chances

Last registered on March 23, 2022
View Trial History

Pre-Trial

Trial Information

General Information

Title
Designing job recommendation algorithms for job seekers: trading-off suitability to search criteria and hiring chances
RCT ID
AEARCTR-0008998
Initial registration date
February 21, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
February 24, 2022, 1:03 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
March 23, 2022, 2:35 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country
France
Region

Primary Investigator

Name
Elia Perennes
Affiliation
CREST
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Guillaume Bied
PI Affiliation
CREST-LISN
Contact Investigator
PI Name
Philippe Caillou
PI Affiliation
LISN
Contact Investigator
PI Name
Christophe Gaillac
PI Affiliation
University of Oxford
Contact Investigator
PI Name
Michèle Sebag
PI Affiliation
LISN
Contact Investigator
PI Name
Bruno Crépon
PI Affiliation
CREST
Contact Investigator

Additional Trial Information

Status
In development
Start date
2022-02-28
End date
2022-08-31
Keywords
Labor
Additional Keywords
Artificial intelligence, recommender systems, matching algorithms, job recommendation, online labor markets, labor economics, online matching, job search
JEL code(s)
L86, C88, J64, J68
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
We study the implementation of a job recommender system dedicated to the jobseekers at the French Public Employment Service (PES). For each job seeker, the system ranks vacancies according to the job seeker's chances of being hired on them. It is based on a state of the art machine learning (ML) model, based on the very rich data available at the French PES (including textual data and past hires).

The goal of this study is to implement a large scale beta test of this recommender system. We will set up a randomized controlled trial in which we will expose a set of job seekers to algorithmic recommendations of vacancies and ask them to rate the recommended job ads.

This experiment will allow us to identify, thanks to job seekers' ratings and applications, the best combination of two vacancy rankings : one of the rankings is based on an expert system that recommends matches based on the fit between the job seekers' search criteria and the characteristics of job postings ; the other one is obtained from the ML model mentionned above. We will also learn about the impact of providing job seekers with information about the ranks of recommended job ads within these two rankings.
External Link(s)

Registration Citation

Citation
Bied, Guillaume et al. 2022. "Designing job recommendation algorithms for job seekers: trading-off suitability to search criteria and hiring chances." AEA RCT Registry. March 23. https://doi.org/10.1257/rct.8998-1.3
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
We study the implementation of a job recommender system dedicated to the jobseekers at the French Public Employment Service (PES). For each job seeker, this algorithm ranks the available job ads according to the job seeker's chances of being hired on each of them, using a state of the art maching learning algorithm trained on data on past hires recorded by the PES.

The purpose of this experiment is to conduct a large scale beta test of this recommender system. The goal of our intervention is to find an optimal combination between two job vacancy rankings: a ranking based on the chances of a hire taking place and another based on the suitability of the job to the jobseeker's explicit search criteria. The first ranking, called Ranking_ML (ML for machine learning), is obtained from the aforementioned recommender system. The latter is inspired by the matching score currently used at the French PES, which is computed as the weighted sum of the intensities of alignment between the job seeker's search criteria and the job posting's characteristics and requirements (e.g. job seeker's experience vs. required experience, salary sought vs. proposed salary, etc.). We call it Ranking_S (S for matching Score). This intervention will also allow us to investigate the impact of providing jobseekers with information about the ranks of recommended job ads within these two rankings.

This intervention proceeds as follows:
- Jobseekers receive an invitation email
- By clicking on a link in the email, jobseekers access an online survey
- Jobseekers are first exposed to two recommended vacancies. They are asked to evaluate: the overall relevance of the recommendation, their perception of their own chances of recruitment and their assessment of the fit with their search criteria.
- After having rated these two recommended vacancies, the jobseekers access a final page including the 2 previous ads, as well as 8 additional recommendations. They do not have to rate these 8 additional vacancies. On this final page, they have the possibility to click on recommended job ads in order to view further details and apply.
Intervention Start Date
2022-02-28
Intervention End Date
2022-03-31

Primary Outcomes

Primary Outcomes (end points)
Collected during and following the beta test:
- ratings given by jobseekers to recommended job ads.
- clicks of jobseekers on recommended job ads.
- applications of jobseekers to recommended job ads.

Impacts on:
- job search behavior: quantity and timing of search effort and the characteristics of jobs applied to.
- general labor market outcomes : unemployment duration and hiring (contract types -temporary or long term-, geographic and sectoral mobility)
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Job seekers are divided into 10 groups of equal size. How recommendations are computed as well as whether or not additional information is displayed will vary between these groups.
* For all of the 10 groups, recommendations are calculated by mixing the Ranking_ML and Ranking_S. These 10 groups are obtained by breaking each of 5 base groups into 2 further subgroups, as follows:
** The five base groups are recommended vacancies using a ranking algorithm based on a mixture of the Ranking_ML and Ranking_S rankings. Each group is characterised by different mixture weights.
** Each of these 5 groups is further split into two groups, which differ in terms of displayed information about the rankings of the recommended vacancies. One group is given information about the vacancies' ranks in both the Ranking_ML and Ranking_S rankings; the other group does not receive this information.
Experimental Design Details
Randomization Method
Randomization done in office by a computer
Randomization Unit
Individual (jobseeker)
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
100 000 jobseekers
Sample size: planned number of observations
100 000 jobseekers
Sample size (or number of clusters) by treatment arms
10 000 jobseekers per treatment group
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
Paris School of Economics
IRB Approval Date
2021-12-08
IRB Approval Number
2021-026
Analysis Plan

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials