Nonbinary hiring discrimination and the politicization of pronouns: A resume audit study

Last registered on March 04, 2024

Pre-Trial

Trial Information

General Information

Title
Nonbinary hiring discrimination and the politicization of pronouns: A resume audit study
RCT ID
AEARCTR-0011183
Initial registration date
April 28, 2023

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
May 03, 2023, 4:25 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
March 04, 2024, 10:47 PM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Primary Investigator

Affiliation
University of Toronto

Other Primary Investigator(s)

Additional Trial Information

Status
Completed
Start date
2023-05-18
End date
2024-03-04
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Thousands of randomly generated fictitious resumes will be sent in response to online job postings in a selection of US cities. A matched pair of resumes will be sent to each job posting, where resumes are similar except for a characteristic of interest: pronoun disclosure. To investigate hiring discrimination against nonbinary applicants, employer response rates will be compared for resumes which disclose "they/them" pronouns (signals the applicant is nonbinary), disclose binary pronouns congruent with sex implied by name (contains political and other signals associated with pronoun disclosure in general), and do not disclose pronouns (control group). Multiple hypotheses will be explored, including whether discrimination varies by geographic politics; sex implied by name; the extent to which the occupation is customer-facing; female-dominated, male-dominated, and non-dominated occupations; and years of relevant work experience.

Registration Citation

Citation
Eames, Taryn. 2024. "Nonbinary hiring discrimination and the politicization of pronouns: A resume audit study." AEA RCT Registry. March 04. https://doi.org/10.1257/rct.11183-3.1
Experimental Details

Interventions

Intervention(s)
A resume audit study design will be leveraged to estimate hiring discrimination against nonbinary applicants (signaled on resumes via “they/them” pronouns listed below the name) and cisgender applicants who disclose pronouns (signaled on resumes via binary pronouns congruent with name-implied sex—for example, “she/her” for Madeline and “he/him” for William). Outcomes among these two treatment groups will be compared to a control applicant who does not disclose pronouns on their resume.
Intervention (Hidden)
Intervention Start Date
2023-05-18
Intervention End Date
2023-10-20

Primary Outcomes

Primary Outcomes (end points)
Positive employer response to job applications is the outcome of interest. Emails and phone lines will be set up and carefully tracked to identify applicants for whom employers respond.
Primary Outcomes (explanation)
Employer response will be viewed as "positive" if they contact the applicant and either offer an interview or request the applicant contact them.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
A team of research assistants will search major job posting websites (Indeed, Monster, Craigslist, etc.) for occupation vacancies in a selection of United States CBSAs. When an appropriate job posting is found, a pair of fictitious, randomized, matched, formatted resumes will be generated and sent in in response. To reduce cost, job postings will only be applied to if the application process involves uploading a resume PDF and answering simple, standardized questions that can be easily determined from the randomized resume (e.g., how many years of relevant experience do you have?) or that can have a standard general response (e.g., can you reliably commute to work at this location?—yes). When applying, information on job posting, employer, and resume characteristics will be recorded in an encrypted database.

Resume characteristics (name, pronouns, objective, work experience, education, skills, certifications) are randomly generated using a program from Lahey and Beasley (2009). During randomization, some characteristics will be matched same (i.e., if the first resume is randomly assigned characteristic A, then the matched pair will also be given characteristic A) and other characteristics matched different (i.e., if the first resume is randomly assigned characteristic A, then the matched pair will be randomly assigned a characteristic aside from A). The characteristic of interest is pronouns, where the research objective is to determine whether similar applicants with different pronoun disclosures receive different positive employer response rates. Once resume content is generated for the matched pair, each will be randomly given one of two possible resume formats.

Employer response will be carefully tracked via phone and email. For each CBSA, two phone lines will be set up using an area code local to the area. To limit fraud detection by email providers and job boards, there are in total two female names and two male names used in each state (e.g., all matched resume pairs in Colorado where the name-implied sex is female will use the same two names). Emails are specific to names, and each name will always use the same phone number when applying in a given city. As a result, with 12 names, 12 emails will be set up in total. Phone voicemails and emails will be monitored on an ongoing basis to identify applications which receive a positive employer response.
Experimental Design Details
Randomization Method
Randomization is done in office by a computer. In particular, the computer program by Lahey and Beasley (2009) is used to construct resume randomization logic. A detailed breakdown is included in the pre-analysis plan.
Randomization Unit
Randomization occurs at the firm (job posting) level.

Matched pairs of resumes are generated simultaneously, where they are matched same in some characteristics (e.g., years of relevant work experience) and matched different in other characteristics (e.g., name). The set of inputs over which randomization occurs is occupation-specific for some characteristics (e.g., work experience) and independent of occupation for other characteristics (e.g., name). These matched pairs are both sent in response to one job posting by a particular firm (location and employer) for a particular occupation.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
3,240 pairs of matched resumes sent to the same number of job postings
Sample size: planned number of observations
6,480 total resumes
Sample size (or number of clusters) by treatment arms
Approximately 2160 resumes with "they/them" pronouns, 1080 resumes with binary pronouns congruent with sex implied by name, 3,240 control resumes.

Of these, approximately half will have male-sounding names and half female-sounding names. Approximately half will be in Republican geographies and half in Democratic geographies. Approximately one third will be in female-dominated, male-dominated, and non-dominated occupations. Approximately one third will be in high, medium, and low customer interaction occupations.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
For simplicity, I denote applicants who send resumes with nonbinary pronouns “T1” matched to control “C1,” and resumes with binary pronouns congruent with name-implied sex “T2” matched to control resumes with no pronouns “C2.” All of the following is for 80% power, 5% statistical significance. Minimum Detectable Effect (MDE) for all of the following is in terms of percentage point differences between treatment and control positive employer response. All MDE reported are associated with a one-sided proportion test. When comparing T1 to C1 or T2 to C2 and assuming the control group has a 10% callback rate: Full sample (2,160 pairs): -2.20% Half sample (1,080 pairs): -3.05% Third sample (720 pairs): -3.70% Assuming the control group has a 15% callback rate: Full sample (2,160 pairs): -3.08% Half sample (1,080 pairs): -3.67% Third sample (720 pairs): -4.57% When comparing T1 to T2 and assuming the T2 has a 10% callback rate: Full sample (3,240 total): -2.65% Half sample (1,620 total): -3.71% Third sample (1,090 total): -4.47% Assuming the T2 has a 15% callback rate: Full sample (3,240 total): -3.21% Half sample (1,620 total): -4.47% Third sample (1,080 total): -5.38% When comparing T1 in group A to T1 in group B and assuming the T1(A) has a 10% callback rate: Half sample (1,620 total): -3.05% Third sample (1,090 total): -3.70% Assuming the T1(A) has a 15% callback rate: Half sample (1,620 total): -3.67% Third sample (1,080 total): -4.57%
Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Toronto Social Sciences, Humanities & Education Review Ethics Board
IRB Approval Date
2023-03-27
IRB Approval Number
44259
Analysis Plan

Analysis Plan Documents

Pre-Analysis Plan

MD5: 398449a32291703e1d5f46cad5d8b8e9

SHA1: 1bdd1ae8ea518f820569a3d76699528df3040b2a

Uploaded At: May 17, 2023

Populated Pre-Analysis Plan

MD5: 53049dd8a60ed8f728f01aa317057f05

SHA1: 0eab316927f3c087a76d9b74cfc925167bb044d2

Uploaded At: March 04, 2024

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials