AEA RCT Registry

Unconscious Bias in Hiring in the Australian Public Service: Evidence from a Framed Field Experiment in Shortlisting of Job Candidates

Last registered on December 20, 2017

Pre-Trial

General Information

Title

Unconscious Bias in Hiring in the Australian Public Service: Evidence from a Framed Field Experiment in Shortlisting of Job Candidates

RCT ID

AEARCTR-0001783

Initial registration date

November 22, 2016

Last updated

December 20, 2017 10:48 AM EST

Location(s)

Country

Australia

Region

Primary Investigator

Name

Michael Hiscox

Affiliation

Harvard University

Contact Primary Investigator

Other Primary Investigator(s)

Additional Trial Information

Status

Completed

Start date

2016-10-24

End date

2016-12-09

Keywords

Labor

Additional Keywords

recruitment, bias, gender, discrimination, senior, manager, equity, government

JEL code(s)

D80 (Information, Knowledge, and Uncertainty—general); D03 (Behavioural Microeconomics: underlying principles)

Secondary IDs

Abstract

Women are under-represented in management and executive-level positions in the private sector and in many areas in the Australian government. This may reflect gender discrimination in hiring and promotion decisions. Such discrimination is generally difficult to overcome because cognitive biases can be so internalized that individuals are unaware that their decision making processes are affected. Creating gender-blind processes for reviewing job applications may provide an effective way to mitigate unconscious bias. The proposed study aims to assess the magnitude of gender bias in standard Australian government processes for selecting job candidates and whether the introduction of a gender-blind approach to reviewing job applications can help eliminate bias. The primary outcomes we will measure are reviewers’ initial assessments of whether applicants are ‘potentially suitable’ to the job or not, the rating scores (0-10 cardinal scale) reviewers give to applicants they initially select as ‘potentially suitable’ for the job, and whether any of the ‘potentially suitable’ applicants are recommended among the top 5 applicants to be shortlisted for further consideration for appointment to a senior position; in each case we will compare average outcomes when the applicants are identified in review materials as female or male, and when they are not identified by gender.

External Link(s)

Registration Citation

Citation

Hiscox, Michael. 2017. "Unconscious Bias in Hiring in the Australian Public Service: Evidence from a Framed Field Experiment in Shortlisting of Job Candidates." AEA RCT Registry. December 20. https://doi.org/10.1257/rct.1783-3.0.

Former Citation

Sponsors & Partners

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information

Experimental Details

Interventions

Intervention(s)

The trial is an individually randomized controlled trial. Specifically it is a “framed field experiment” in which subjects are drawn from the population of interest and complete a familiar task, something they would normally perform in a naturally-occurring context, but understand that in this instance they are completing the task as part of an experiment and their behaviour is being studied.

Individuals who agree to participate in the study and register online will be randomly assigned to review materials in either the de-identified (treatment) or usual (control) condition, with pre-randomization stratification based upon agency, age, gender, and job level. If our sample of participants is large enough, we will have two control groups and one treatment group such that the only difference between the two control groups is that each of the applicant CVs will have the first name swapped to a similar first name of the opposite gender (e.g. Jane Smith in control group 1 would be John Smith in control group 2). Having these two control groups would allow us to fully control for all other characteristics of the CVs while experimentally altering only gender (i.e. the first name in the CVs).

Intervention Start Date

2016-11-14

Intervention End Date

2016-11-22

Primary Outcomes

Primary Outcomes (end points)

The key outcome variables are the percentage of women initially recommended as ‘potentially suitable’ for the job, the percentage of women shortlisted for the job, and the average suitability score (on a 0-10 scale) of women among the ‘potentially suitable’ applicants.

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

The trial is an individually randomized controlled trial, conducted in partnership with Australian Government agencies. The trial is a “framed field experiment” in which subjects drawn from the ranks of senior and executive-level officers in Australian Government agencies are asked to review job applications for an executive-level position, a familiar task, but understand that this exercise is part of an experiment and their behaviour is being studied. Subjects either review applications in the usual way based upon curriculum vitae in which the gender of applicants is identifiable or they review the same applications in de-identified (gender blind) form.

Experimental Design Details

The trial is an individually randomized controlled trial, conducted in partnership with around Australian Public Service (APS) agencies. The trial is be a “framed field experiment” in which subjects drawn from the ranks of senior and executive-level officers in these agencies are asked to review job applications for an executive-level position, a familiar task, but understand that this exercise is part of an experiment and their behaviour is being studied. Subjects either review applications in the usual way based upon curriculum vitae in which the gender of applicants is identifiable or they review the same applications in de-identified (gender blind) form. A total sample of approximately 4,000 subjects will provide statistical power to detect substantial (≥5%) impacts on primary outcomes due to gender-blinding of applications.

Randomization Method

Randomization done by a computer algorithm

Randomization Unit

Individual

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

N/A

Sample size: planned number of observations

1,600 individuals (government officers)

Sample size (or number of clusters) by treatment arms

1,600 individuals (government officers)

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

A sample of 1,600 provides 99% power for a two-sided test (with 2.5% alpha level) to detect a 5% change the percentage of women rated as ‘potentially suitable’ for the job or shortlisted (from 43% to 45.2%). It also provides 99% power to detect a 5% change in the average evaluation scores for female candidates (from 7 to 7.35 out of 10), and 83% power to detect a 5% change in the average shortlist rankings of women (on a 1-5 ordinal scale). Notice that the alpha level for these tests for effects for each outcome is set to 2.5% so that the overall Type One error rate for testing for statistical significance of effects is kept at the standard 5% level.

Supporting Documents and Materials

IRB

INSTITUTIONAL REVIEW BOARDS (IRBs)

IRB Name

BETA Ethics Committee

IRB Approval Date

2016-10-21

IRB Approval Number

ETH 2016-001

Analysis Plan

Analysis Plan Documents

Unconscious Bias Tables of Results from Analysis of Primary Outcomes