Unconscious Bias in Hiring in the Australian Public Service: Evidence from a Framed Field Experiment in Shortlisting of Job Candidates

Last registered on December 20, 2017

Pre-Trial

Trial Information

General Information

Title
Unconscious Bias in Hiring in the Australian Public Service: Evidence from a Framed Field Experiment in Shortlisting of Job Candidates
RCT ID
AEARCTR-0001783
Initial registration date
November 22, 2016

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
November 22, 2016, 9:03 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
December 20, 2017, 10:48 AM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
Harvard University

Other Primary Investigator(s)

Additional Trial Information

Status
Completed
Start date
2016-10-24
End date
2016-12-09
Secondary IDs
Abstract
Women are under-represented in management and executive-level positions in the private sector and in many areas in the Australian government. This may reflect gender discrimination in hiring and promotion decisions. Such discrimination is generally difficult to overcome because cognitive biases can be so internalized that individuals are unaware that their decision making processes are affected. Creating gender-blind processes for reviewing job applications may provide an effective way to mitigate unconscious bias. The proposed study aims to assess the magnitude of gender bias in standard Australian government processes for selecting job candidates and whether the introduction of a gender-blind approach to reviewing job applications can help eliminate bias. The primary outcomes we will measure are reviewers’ initial assessments of whether applicants are ‘potentially suitable’ to the job or not, the rating scores (0-10 cardinal scale) reviewers give to applicants they initially select as ‘potentially suitable’ for the job, and whether any of the ‘potentially suitable’ applicants are recommended among the top 5 applicants to be shortlisted for further consideration for appointment to a senior position; in each case we will compare average outcomes when the applicants are identified in review materials as female or male, and when they are not identified by gender.
External Link(s)

Registration Citation

Citation
Hiscox, Michael. 2017. "Unconscious Bias in Hiring in the Australian Public Service: Evidence from a Framed Field Experiment in Shortlisting of Job Candidates." AEA RCT Registry. December 20. https://doi.org/10.1257/rct.1783-3.0
Former Citation
Hiscox, Michael. 2017. "Unconscious Bias in Hiring in the Australian Public Service: Evidence from a Framed Field Experiment in Shortlisting of Job Candidates." AEA RCT Registry. December 20. https://www.socialscienceregistry.org/trials/1783/history/24277
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
The trial is an individually randomized controlled trial. Specifically it is a “framed field experiment” in which subjects are drawn from the population of interest and complete a familiar task, something they would normally perform in a naturally-occurring context, but understand that in this instance they are completing the task as part of an experiment and their behaviour is being studied.

Individuals who agree to participate in the study and register online will be randomly assigned to review materials in either the de-identified (treatment) or usual (control) condition, with pre-randomization stratification based upon agency, age, gender, and job level. If our sample of participants is large enough, we will have two control groups and one treatment group such that the only difference between the two control groups is that each of the applicant CVs will have the first name swapped to a similar first name of the opposite gender (e.g. Jane Smith in control group 1 would be John Smith in control group 2). Having these two control groups would allow us to fully control for all other characteristics of the CVs while experimentally altering only gender (i.e. the first name in the CVs).
Intervention Start Date
2016-11-14
Intervention End Date
2016-11-22

Primary Outcomes

Primary Outcomes (end points)
The key outcome variables are the percentage of women initially recommended as ‘potentially suitable’ for the job, the percentage of women shortlisted for the job, and the average suitability score (on a 0-10 scale) of women among the ‘potentially suitable’ applicants.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The trial is an individually randomized controlled trial, conducted in partnership with Australian Government agencies. The trial is a “framed field experiment” in which subjects drawn from the ranks of senior and executive-level officers in Australian Government agencies are asked to review job applications for an executive-level position, a familiar task, but understand that this exercise is part of an experiment and their behaviour is being studied. Subjects either review applications in the usual way based upon curriculum vitae in which the gender of applicants is identifiable or they review the same applications in de-identified (gender blind) form.
Experimental Design Details
The trial is an individually randomized controlled trial, conducted in partnership with around Australian Public Service (APS) agencies. The trial is be a “framed field experiment” in which subjects drawn from the ranks of senior and executive-level officers in these agencies are asked to review job applications for an executive-level position, a familiar task, but understand that this exercise is part of an experiment and their behaviour is being studied. Subjects either review applications in the usual way based upon curriculum vitae in which the gender of applicants is identifiable or they review the same applications in de-identified (gender blind) form. A total sample of approximately 4,000 subjects will provide statistical power to detect substantial (≥5%) impacts on primary outcomes due to gender-blinding of applications.
Randomization Method
Randomization done by a computer algorithm
Randomization Unit
Individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
N/A
Sample size: planned number of observations
1,600 individuals (government officers)
Sample size (or number of clusters) by treatment arms
1,600 individuals (government officers)
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
A sample of 1,600 provides 99% power for a two-sided test (with 2.5% alpha level) to detect a 5% change the percentage of women rated as ‘potentially suitable’ for the job or shortlisted (from 43% to 45.2%). It also provides 99% power to detect a 5% change in the average evaluation scores for female candidates (from 7 to 7.35 out of 10), and 83% power to detect a 5% change in the average shortlist rankings of women (on a 1-5 ordinal scale). Notice that the alpha level for these tests for effects for each outcome is set to 2.5% so that the overall Type One error rate for testing for statistical significance of effects is kept at the standard 5% level.
IRB

Institutional Review Boards (IRBs)

IRB Name
BETA Ethics Committee
IRB Approval Date
2016-10-21
IRB Approval Number
ETH 2016-001
Analysis Plan

Analysis Plan Documents

Unconscious Bias Tables of Results from Analysis of Primary Outcomes

MD5: af4bf71bf6e8b8a1105cac479ec60070

SHA1: a455bbd299fef75ab1d6bc8400985857f366485e

Uploaded At: November 22, 2016

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
November 22, 2016, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
November 25, 2016, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
n/a
Was attrition correlated with treatment status?
No
Final Sample Size: Total Number of Observations
2,108
Final Sample Size (or Number of Clusters) by Treatment Arms
709 treatment (deidentification), 705 (control group 1), 604 (control group 2)
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
No
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials