Aversion to job recommendation algorithms among job seekers: a field experiment to measure it and understand what cause it

Last registered on April 13, 2023
View Trial History

Pre-Trial

Trial Information

General Information

Title
Aversion to job recommendation algorithms among job seekers: a field experiment to measure it and understand what cause it
RCT ID
AEARCTR-0011157
Initial registration date
March 28, 2023

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
March 30, 2023, 3:55 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
April 13, 2023, 8:10 AM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country
France
Region

Primary Investigator

Name
Elia Perennes
Affiliation
CREST
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Guillaume Bied
PI Affiliation
LISN / CREST
Contact Investigator

Additional Trial Information

Status
In development
Start date
2023-01-01
End date
2023-12-31
Keywords
Behavior, Labor
Additional Keywords
Artificial intelligence, recommender systems, matching algorithms, job recommendation, online labor markets, labor economics, online matching, job search, algorithm aversion
JEL code(s)
L86, C88, J64, J68
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
We study the implementation of a job recommender system dedicated to the jobseekers at the French Public Employment Service (PES). The recommender system is a combination of two vacancy rankings : the first ranking is based on an expert system that recommends matches based on the fit between the job seekers' search criteria and the characteristics of job postings ; the other one is obtained from a state of the art machine learning (ML) model, based on the very rich data available at the French PES (including textual data and past hires).

The objectives of this study are threefold: first, it will measure the degree of aversion (and/or acceptance) of job seekers for algorithmic recommendations ; second, it will determine the mechanisms that cause aversion ; and third, it will investigate how the framing of recommendations can minimize the risk of aversion.

We set up a randomized controlled trial in which a set of job seekers is exposed to algorithmic recommendations of vacancies. Several framing alternatives are tested: the first explains to job seekers the global functioning of the algorithm, the second emphasizes the level of performance of the algorithm and the third emphasizes the fact that the algorithm has been co-constructed with job seekers. During the experiment, job seekers are explicitly asked about how they perceive the recommendations. I also use implicit satisfaction measures, such as the click rate on the recommended ads.
External Link(s)

Registration Citation

Citation
Perennes, Elia and Guillaume Bied. 2023. "Aversion to job recommendation algorithms among job seekers: a field experiment to measure it and understand what cause it." AEA RCT Registry. April 13. https://doi.org/10.1257/rct.11157-1.2
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
The intervention consists in exposing a set of job seekers to algorithmic recommendations of vacancies. The recommender system used in the intervention is a combination of two vacancy rankings : a ranking based on the chances of a hire taking place and another based on the suitability of the job to the jobseeker's explicit search criteria. The first ranking is obtained from a state of the art maching learning algorithm trained on data on past hires recorded by the PES. The latter is inspired by the matching score currently used at the French PES, which is computed as the weighted sum of the degrees of adequacy between the job seeker's search criteria and the job posting's characteristics and requirements (e.g. job seeker's experience vs. required experience, salary sought vs. proposed salary, etc.).

The intervention proceeds as follows:
- Selected jobseekers receive an invitation email.
- By clicking on a link in the email, jobseekers access an online survey.
- Job seekers are first exposed (or not depending on the treatment group, see below) to an infographic giving details on the algorithm that generated the recommendations.
- Job seekers see 5 vacancies, which are recommended to them by the algorithm.
- Job seekers can (if they want to) give their opinion on each recommendation by indicating whether or not the vacancy interests them.
- Job seekers are then asked three questions regarding: their level of understanding of the process behind the recommendations, their chances of being hired on recommendations, and their perception that the recommendations are based on job search experience of previous job seekers.
- Once the job seekers have answered these three questions, a new page with the 5 recommended ads (those viewed just before) is displayed. On this page, they have the possibility to click on recommended job ads in order to view further details and apply.
Intervention (Hidden)
Intervention Start Date
2023-05-10
Intervention End Date
2023-07-01

Primary Outcomes

Primary Outcomes (end points)
Collected during and following the survey:
- responses to questions asked during the survey.
- clicks of jobseekers on recommended job ads.
- applications of jobseekers to recommended job ads.

Collected outside the survey:
- web behavior: visits to the PES job board (quantity, timing).
- job search behavior: quantity and timing of search effort and the characteristics of jobs applied to.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Job seekers are divided into 6 groups of equal size. Two elements vary between these groups: (1) the perceived source of the recommendations (3 possible conditions) and (2) the display of additional information accompanying the recommendations (4 possible conditions).

The three alternatives regarding the perceived source of recommendations are:
- "Machine source": Recommendations are framed as coming from an artificial intelligence algorithm
- "Human source": Recommendations are framed as coming from a (human) labor market expert
- "Neutral source": Recommendations are not framed as coming from a particular source

The four alternatives regarding the recommendations framing are:
- "Global explanation": explains to job seekers the overall functioning of the algorithm
- "Performance": emphasizes the performance level of the algorithm
- "Human in the loop" : highlights the fact that the algorithm was co-constructed with job seekers
- "Control": no particular framing

The 6 treatment groups are (Source x Framing) :
* 1: Machine source x Control
* 2: Machine source x Global explanation
* 3: Machine source x Performance
* 4: Machine source x Human in the loop
* 5: Human source x Control
* 6: Neutral source x Control

*** Preview links:
Treatment 2: https://www.dropbox.com/s/5ut0uh0gwuf9jf7/Projet%20aversion%20-%20version%20explication%20globale.mov?dl=0
Treatment 3: https://www.dropbox.com/s/uemfvw4dm37lrng/Projet%20aversion%20-%20version%20performance.mov?dl=0
Treatment 4: https://www.dropbox.com/s/j2rro7efgk0kmte/Projet%20aversion%20-%20version%20feedback.mov?dl=0
Treatment 5: https://www.dropbox.com/s/3qfwaqfmpazits1/Projet%20aversion%20-%20version%20expert.mov?dl=0
Treatment 6: https://www.dropbox.com/s/8lchw0478j1rkl3/Projet%20aversion%20-%20version%20neutre.mov?dl=0
(Treatment 1: no preview available, but this treatment corresponds to treatment 2 (or 3 or 4) without the "framing" part).
Experimental Design Details
Randomization Method
Randomization done in office by a computer
Randomization Unit
Individual (jobseeker)
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
60 000 job seekers
Sample size: planned number of observations
60 000 job seekers
Sample size (or number of clusters) by treatment arms
10 000 jobseekers per treatment group
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials

Documents

Document Name
Infographic 1: Global explanation of the algorithm (French version)
Document Type
survey_instrument
Document Description
File
Infographic 1: Global explanation of the algorithm (French version)

MD5: ded154bea05d6d77b207205f621b39b8

SHA1: 60e1b18128cff013e450d6731400e971d03a22e1

Uploaded At: March 28, 2023

Document Name
Infographic 2: Performance of the algorithm (French version)
Document Type
survey_instrument
Document Description
File
Infographic 2: Performance of the algorithm (French version)

MD5: da17bf578c7b32f70765eeb28f59a65d

SHA1: 435289544b9d5b1efdefb7247a13aca11c75c200

Uploaded At: March 28, 2023

Document Name
Infographic 3: Human in the loop (French version)
Document Type
survey_instrument
Document Description
File
Infographic 3: Human in the loop (French version)

MD5: a420e9d059a6f12ee84e3eb34f46ccb0

SHA1: 0ac7d7fb178349188de176fc119b4edae697abaa

Uploaded At: March 28, 2023

Document Name
Survey preview (French version)
Document Type
survey_instrument
Document Description
This document provides screenshots of each page of the survey.
File
Survey preview (French version)

MD5: 37184c1de24e018a570ae90da53437dd

SHA1: ae909e30f60ab8d81c6c72102d12dfeb1a9152c1

Uploaded At: March 28, 2023

IRB

Institutional Review Boards (IRBs)

IRB Name
Paris School of Economics
IRB Approval Date
2023-01-13
IRB Approval Number
2021-026
Analysis Plan

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials