The Impact of Algorithmic Decision-Making in Labor Market

Last registered on June 17, 2024

View Trial History

Pre-Trial

Trial Information

General Information

Title

The Impact of Algorithmic Decision-Making in Labor Market

RCT ID

AEARCTR-0013580

Initial registration date

June 08, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

June 17, 2024, 2:43 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

Netherlands

Region

Utrecht

Primary Investigator

Name

Cielo Cai

Affiliation

Utrecht Universiteit

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Sarah Rezaei Khavas

PI Affiliation

Supervisor

Contact Investigator

PI Name

Elena Fumagalli

PI Affiliation

Supervisor

Contact Investigator

Additional Trial Information

Status

On going

Start date

2024-04-27

End date

2024-08-31

Keywords

Behavior, Labor

Additional Keywords

Willful ignorance, algorithmic recommendations, Algorithmic recruitment, Human-AI interactions

JEL code(s)

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

The adoption of AI in decision-making has seen rapid advances, especially in the context of the hiring process (Chalfin et al., 2016; Raghavan et al., 2020). Algorithms can accelerate the hiring process as well as improve the efficiency. However, ethical concerns have emerged due to potential bias and discrimination in hiring, posing more risks to vulnerable groups (Obermeyer et al., 2019; Giermindl et al., 2022). This project aims to comprehend how individuals adapt to and interact with algorithms in the labor market, focusing on how workers and HR managers interact with hiring algorithms. To achieve this goal, we will conduct several online, and lab experiments and collect data from the students population in the Netherlands as well as workers in online labor markets (i.e. Mturk, Prolifics) from the US and UK. In particular, this study serves as a pilot to initiate the previously mentioned project.

External Link(s)

Registration Citation

Citation

Cai, Cielo, Elena Fumagalli and Sarah Rezaei Khavas. 2024. "The Impact of Algorithmic Decision-Making in Labor Market ." AEA RCT Registry. June 17. https://doi.org/10.1257/rct.13580-1.0

Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Experimental Details

Interventions

Intervention(s)

We want to explore the impact of willful ignorance on HR managers' decisions when they receive recommendations from hiring algorithms. In the experiment, participants need to select high-potential workers for a particular task, and they will be provided with a series of information, includes the workers' basic performance information and the recommendation from an algorithm. They are informed that the algorithm might be biased, and they can acquire more information about each worker's performance and overrule or accept the algorithmic decision without further examination. HR managers in the control group have free access to the detailed information about each worker's performance, while in the treatment group, accessing this information is costly.

Intervention Start Date

2024-06-10

Intervention End Date

2024-06-24

Primary Outcomes

Primary Outcomes (end points)

1. The time each HR manager spent on the hiring task
2. The number of times each HR manager clicked to obtain additional information
3. The share of women of the workers selected by each HR manager
4. The comparison of the average real Raven performance for workers selected by humans and algorithms respectively
5. The role of social preference in the worker selection process

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

This pilot includes two phases of experiments: the Workers' experiment and the HR Managers' experiment.

In the Workers' experiment, workers will perform a few real-effort tasks and we will generate workers' performance data based on their performance and their demographic data.

In the HR managers' experiment, HR managers need to select high-potential workers. They will receive the hiring table, includes the workers' demographic and task performance, the recommendation from the hiring algorithm, and information regarding the potential biases of the algorithm. The HR manager can acquire more information about each worker's performance and overrule or accept the algorithmic decision without further examination.

Experimental Design Details

The experiment will involve 250 participants from a representative sample and will simulate a hiring setting. Participants will be randomly assigned to one of two roles: workers or HR managers.

Workers. Workers will perform a few real-effort tasks and will be informed that their performance in each task and their demographic information will be transferred to HR managers for evaluation. Workers will be notified that, in case of being selected by the HR managers, they will perform a second set of tasks with improved pay compared to the first round.

HR managers. HR managers must select high-potential workers to perform the second set of tasks. Depending on the type of treatment, the HR manager will be provided with one or more of the following pieces of information: (i) for every worker, a recommendation from an algorithm that was trained based on data from a similar population of workers, (ii) algorithm accuracy, meaning that they know that the algorithm might be biased, (iii) workers' information, including their average performances in the first set of real-effort tasks and their demographics. The HR manager can acquire information which is costly about each worker's performance to make more informed decisions, and overrule or accept the algorithmic decision without further examination.

Both workers' and HR managers' choices are incentivized. Workers' remunerations increase in their task-specific performance, increasing the probability of being admitted to a better-paid second round. Similarly, HR managers' remunerations increase in the performance of workers they have recruited.

Randomization Method

We will program on Qualtrics and randomization will therefore be done by Qualtrics

Randomization Unit

We will randomize in the individual level, each participant in the HR managers' experiment will be randomly assigned to either the control group or the treatment group.

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

We will have two groups, the control group and the treatment group.

Sample size: planned number of observations

We plan to recruit 250 participants for this pilot, 100 for workers' experiment, 75 each for the control group and the treatment group for the HR managers' experiment.

Sample size (or number of clusters) by treatment arms

In Workers' experiment, we collect participants' performance to generate the performance data, therefore we will recruit 100 participants to act as workers.
In HR managers' experiment, we will have two groups, one control group and one treatment group, and each group will have 75 participants.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Supporting Documents and Materials

IRB