Measuring Algorithmic Bias in Online Job Recommendation Systems

Last registered on June 19, 2023

View Trial History

Pre-Trial

Trial Information

General Information

Title

Measuring Algorithmic Bias in Online Job Recommendation Systems

RCT ID

AEARCTR-0006101

Initial registration date

June 30, 2020

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

July 21, 2020, 12:03 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated

June 19, 2023, 12:23 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country

China

Region

Primary Investigator

Name

Shuo Zhang

Affiliation

University of California, Santa Barbara

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Peter Kuhn

PI Affiliation

University of California, Santa Barbara

Contact Investigator

Additional Trial Information

Status

In development

Start date

2020-07-01

End date

2024-06-30

Keywords

Gender, Labor

Additional Keywords

Algorithmic Fairness, Gender Discrimination, Recommender System

JEL code(s)

J16, J71, C93

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

Virtually all internet job boards now recommend jobs to the workers who use their platforms. These job recommendations are generated by algorithms, using criteria that include the worker’s characteristics and previous search behaviors, and the match between the worker’s characteristics and the job’s requirements. While job recommendation algorithms have the potential to help workers find their preferred jobs faster and to improve worker-job match quality, they also have sparked deep concern about the fairness of the job recommendation algorithms: even when this is not intended by the designers, the recommended jobs may reinforce gender and other stereotypes. We plan to conduct a resume audit study on several leading online job boards in China to learn whether, to what extent, and how job board algorithms systematically treat male and female job seekers differently. Specifically, we will collect the jobs that are recommended to fictitious workers who have identical backgrounds except for gender, compare the job recommendation outcomes and measure the difference of outcomes between two gender groups.

External Link(s)

Registration Citation

Citation

Kuhn, Peter and Shuo Zhang. 2023. "Measuring Algorithmic Bias in Online Job Recommendation Systems." AEA RCT Registry. June 19. https://doi.org/10.1257/rct.6101-1.2

Sponsors & Partners

Experimental Details

Interventions

Intervention(s)

We will set up fictitious profiles and send fictitious applicants to jobs posted on online job boards.

Intervention (Hidden)

Intervention Start Date

2020-07-01

Intervention End Date

2020-12-31

Primary Outcomes

Primary Outcomes (end points)

Characteristics of jobs recommended by online job boards.

Primary Outcomes (explanation)

The primary outcome that we are interested in is the job ads that are recommended and displayed to job seekers. We will test whether the recommender systems used by these job boards recommend different jobs to male and female job seekers with identical resumes.

The outcomes we will study include (but are not restricted to):
•Advertised salary and benefits (total pay, base pay, performance pay, annual bonus, insurance, vacation days, paid leave days, compensation/subsidy on travel and meal, staff dormitory, shuttle service, hukou, professional training, flexible working hours, holiday benefits, etc.)
•Job Title: (we can use natural language processing tools to compare the extracted words)
•Job description (especially words like manage, coordinate, independently design, business travel, etc.)
•Job requirements (years of working experience, education, requested age and skills)
•Company characteristics (number of employees, ownership such as foreign, public, private, etc.)
•Job’s industry, occupation, and location.

Secondary Outcomes

Secondary Outcomes (end points)

Reactions received by the fictitious resumes.

Secondary Outcomes (explanation)

These will consist mostly of automated messages generated by the platform when the HR agent takes particular actions, such as reading or saving (downloading) a worker's resume. Our profiles might also receive ‘call-back’ emails from human recruiters, though we expect this to be rare due to the sparse online worker profiles we will use. As a secondary aspect of our study, we will measure whether there are any gender differences in these reactions from the job boards.

Experimental Design

Our research design is a type of resume audit study, which begins by creating otherwise identical male and female worker profiles on Chinese job boards, then observing which jobs are recommended to the profiles.

Experimental Design Details

Our experiment is designed to simulate a worker’s job search process on the job boards and consists of three parts: (1) Job Selection, (2) Resume Setup, and (3) Applications. See Pre-Analysis Plan for more details.

Randomization Method

In our audit study, the resumes come in pairs, in which the two workers in each pair have identical backgrounds except for gender.

Randomization Unit

Individual fictitious profile.

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

No cluster

Sample size: planned number of observations

We plan to create about 140 fictitious profiles on each job board and replicate these profiles and application processes across a few big cities. Each profile will apply to a substantial number of jobs, which are recommended to profile by the applicant. .

Sample size (or number of clusters) by treatment arms

Half of the fictitious profiles are females, and half of the fictitious profiles are males.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Supporting Documents and Materials

IRB