Back to History Current Version

Trust in hiring algorithms: causal effect of beliefs about humans’ discrimination for workers, and self-confidence for managers

Last registered on March 21, 2022

Pre-Trial

Trial Information

General Information

Title
Trust in hiring algorithms: causal effect of beliefs about humans’ discrimination for workers, and self-confidence for managers
RCT ID
AEARCTR-0009068
Initial registration date
March 17, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
March 21, 2022, 1:23 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
Paris Dauphine University

Other Primary Investigator(s)

PI Affiliation
University of Lausanne & WZB Berlin
PI Affiliation
WZB Berlin, TU Berlin

Additional Trial Information

Status
In development
Start date
2022-03-21
End date
2022-05-15
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
We run an online experiment to study ways to overcome algorithm aversion. We use a 2*2 design. Participants are either in the role of workers or of managers. They will furthermore either be assigned to the baseline or the treatment. Workers perform three real-effort tasks: task 1, task 2 and the job task. They must choose whether they prefer hiring decisions between themsef and another worker to be made by a participant in the role of a manager or an algorithm. They will get paid if they are hired in one random pair of workers they belong to. They know that the manager will get paid if they hire the best worker at the job task out of a random pair of workers they have made a hiring decision for. The managers and the algorithm will base their hiring decision on the genders of the workers and their task 1 and task 2 perfromances. In the workers' treatment, before deciding between the manager and the algorithm, workers are informed of the proportion of men and women who were chosen to be hired by the managers in a managers' session. Managers must make 20 hiring decisions between pairs of workers. They can see the gender of the workers and their task 1 and task 2 performances. We elicit for how many of the 20 decisions they believe they have hired the best worker of the pair. They must then decide whether they want to delegate the hiring decisions to the algorithm. If they do, their payoff will depend on the algorithm's hiring decision, not theirs. In the managers' treatment, managers will get a feedback about their over/under or well-calibrated confidence before the decision whether to delegate the hiring decisions to the algorithm
External Link(s)

Registration Citation

Citation
Dargnies, Marie-Pierre, Rustamdjan Hakimov and Dorothea Kübler. 2022. "Trust in hiring algorithms: causal effect of beliefs about humans’ discrimination for workers, and self-confidence for managers." AEA RCT Registry. March 21. https://doi.org/10.1257/rct.9068-1.0
Experimental Details

Interventions

Intervention(s)
Intervention Start Date
2022-03-21
Intervention End Date
2022-05-15

Primary Outcomes

Primary Outcomes (end points)
For both workers and managers, we are interested in whether the treatments decrease algorithm aversion.
For workers, a decreased algorithm aversion would mean a higher proportion of workers would choose that the hiring decisions are made by the algorithm in the workers' treatments than in the workers' baseline.
For managers, a decreased algorithm aversion would mean a higher proportion of managers would choose to delegate the hiring decisions to the algorithm in the managers' treatment than in the managers' baseline.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
We are interested in finding additional evidence of algorithm aversion and investigating whether it depends on the confidence of workers in their relative performance and the confidence of managers in the quality of their hiring decisions.

We are also interested in potentially different effects of the treatment depending on the gender of the participant, their belief about the discriminative behavior of the algorithm compared to that of the manager and , the confidence of workers in their relative performance, and the confidence of managers in the quality of their hiring decisions.

Finally, we are interested in whether workers' beliefs about how discriminatory the algorithm is changes with the treatments.
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We run an online experiment to study ways to overcome algorithm aversion. Participants are either in the role of workers or of managers. Workers will either be assigned to the baseline, treatment 1 or treatment 2. Managers will either be assigned to the baseline or the treatment. Workers perform three real-effort tasks: task 1, task 2 and the job task. They must choose whether they prefer hiring decisions between themself and another worker to be made by a participant in the role of a manager or an algorithm. They will get paid if they are hired in one random pair of workers they belong to. They know that the manager will get paid if they hire the best worker at the job task out of a random pair of workers they have made a hiring decision for. In the workers baseline and treatment 1, the managers and the algorithm will base their hiring decision on the genders of the workers and their task 1 and task 2 performances. In the workers' treatment 1, before deciding between the manager and the algorithm, workers are informed of the proportion of men and women who were chosen to be hired by the managers in a managers' session. In the workers' treatment 2, the algorithm will base its hiring decisions on the task 1 and task 2 performances of the workers but not on their gender. Managers must always make 20 hiring decisions between pairs of workers. They can see the gender of the workers and their task 1 and task 2 performances. We elicit for how many of the 20 decisions they believe they have hired the best worker of the pair. They must then decide whether they want to delegate the hiring decisions to the algorithm. If they do, their payoff will depend on the algorithm's hiring decision, not theirs. In the managers' treatment, managers will get a feedback about their over/under or well-calibrated confidence before the decision whether to delegate the hiring decisions to the algorithm.
Experimental Design Details
We will run an online experiment. Participants will be men and women in equal numbers. There will be 250 participants in each treatment, for a total of 250*5=1250 participants.
The experiment will consist of a baseline and two treatments for participants in the role of workers and a baseline and one treatment for participants in the role of managers.

Baseline workers:
Workers first have 2 minutes to solve 12 type-1 real-effort exercises (task 1), then 2 minutes to solve 12 type-2 real-effort exercises (task 2). They then solve what we call the job task for 2 minutes which consists in 7 type-1 exercises and 5 type-2 exercises. They will be paid according to their performance in one randomly chosen task among task 1, task 2 and the job task. Task 1 is the standard Raven Matrices test. Task 2 consists in counting zeros in a 6x6 matrix. Workers are paid 0.15 pounds for each correct answer, but only one of the tasks is payoff relevant.
Workers are then told that an algorithm and participants in the role of managers will have to make hiring decisions between pairs of workers. The algorithm and managers choose which of the two workers to hire based on the workers’ genders and task 1 and task 2 performances. The algorithm is trained to give the best prediction of the highest performer in task 3 based on the data from 200 workers, and will hire the worker with the best predicted performance in task 3. The managers will get 2 pounds if the job task performance of the worker they chose to hire in one randomly chosen pair is higher than that of the other worker.
Workers must choose whether they prefer the hiring decision to be made by the algorithm or a participant in the role of a manager. A worker will get an additional payment of 0.50 pounds if (and only if) they were chosen to be hired in one random pair of workers they belong to. Note that payments to workers will be implemented only after the sessions with managers are run.
We elicit participants’ confidence in their relative performance for the job task. A participant can get additionally 0.25 pounds if they guess within 5 percentage points what percentage of workers have a lower performance than themselves.
Lastly, we elicit participants' beliefs about the gender composition of workers hired by the managers and the algorithm. Given the equal number of men and women in the candidates' pool, we ask how many of 100 hired workers will be men. Participants can get additional 0.25 pounds if they guess within five from the correct answer for managers and the algorithm.

Treatment 1 workers:
All as in Baseline workers, but before choosing between the algorithm and the manager, workers learn the proportion of men and women who were chosen to be hired by the managers in the managers’ baseline treatment.

Treatment 2 workers:
All as in Baseline workers, but the algorithm will base its hiring decisions on the task 1 and task 2 performances of the workers but not on their gender.

Baseline Managers:
Managers observe all questions in three tasks that workers had to solve. They are not paid to do them.
They are asked to make 20 hiring decisions among pairs of workers from the baseline condition based on gender, tasks 1 and 2 performance of each worker.
For one randomly chosen hiring decision, each manager earns 2 pounds if the decision is correct meaning the worker they chose to hire has a better performance in the job task than the other worker in the pair.
After the hiring decisions are made, we elicit participants’ belief in how many out of 20 pairs they hired the best worker (the one with the highest task 3 performance). Participants can get an additional 0.25 pounds if they guess within 1 rank from the correct answer.
Finally, managers are asked whether they want to delegate the hiring decisions to an algorithm. The algorithm is a computer program that chooses which of the two workers to hire based on the workers’ genders, and their performance in task 1 and task 2. The algorithm is trained to give the best prediction of the best performer in task 3 based on the data from 200 workers. If the manager decides to delegate their hiring decisions to the algorithm then their payoff will depend on one randomly chosen hiring decision made by the algorithm, and not the manager.

Treatment Managers:
Unlike managers in the Baseline, managers in the treatment will receive feedback after the belief elicitation about the number of correct hires out of 20. The feedback will inform them whether they are overconfident (guessed at least 2 more correct hires than what they did), underconfident (guessed at least 2 less correct hires than what they did) or well calibrated (correct hires within interval of +/-1 from stated).
Randomization Method
The participants will select into one of the treatments (Baseline workers, Treatment 1 workers, Treatment 2 workers, Baseline managers, Treatment managers) randomly.
Randomization Unit
Individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
1250 individuals
Sample size: planned number of observations
1250 individuals
Sample size (or number of clusters) by treatment arms
250 in each of the treatments : Baseline workers, Treatment 1 workers, Treatment 2 workers, Baseline managers, Treatment managers
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
TRIAL University Lausanne HEC IRB Board
IRB Approval Date
2022-03-09
IRB Approval Number
N/A
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials