Job Applications and LLMs

Last registered on April 16, 2024

Pre-Trial

Trial Information

General Information

Title
Job Applications and LLMs
RCT ID
AEARCTR-0013355
Initial registration date
April 13, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
April 16, 2024, 3:24 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Affiliation
Tilburg University

Other Primary Investigator(s)

PI Affiliation
Tilburg University
PI Affiliation
Tilburg University
PI Affiliation
Tilburg University

Additional Trial Information

Status
In development
Start date
2024-04-15
End date
2024-05-31
Secondary IDs
J08, J64, C90
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
The use of Large Language Models (LLM) could have a profound impact on the labor market. There is already extensive research on the impact of AI automation on existing jobs, but little is known about the use of LLMs and AI tools on the selection process leading to new hires. For entry level positions, applicants are usually required to submit a CV and a cover letter in order to send a signal to the new employer about their skills. Writing a good cover letter that impresses an employer requires time and effort, and was traditionally interpreted as an effective way to send a signal about applicants’ relevant skills and motivation for the job. However, with the emergence of LLMs, writing a good quality cover letter has become much easier, and hence cover letters are a less reliable signal of an applicant’s quality. This is further confounded by the insights from the academic literature on the productivity effects of LLMs, which find that LLMs help everyone, however help the low-performers disproportionately more (Noy and Zhang, 2023; Dell’Acqua et al., 2023). Therefore, the difference in quality of cover letters between good and less good applicants is likely to decrease as a result of the use of LLMs – making cover letters a less reliable signal of the applicant’s quality.
External Link(s)

Registration Citation

Citation
Abbas Nejad, Kian et al. 2024. "Job Applications and LLMs." AEA RCT Registry. April 16. https://doi.org/10.1257/rct.13355-1.0
Experimental Details

Interventions

Intervention(s)
Students are randomly assigned to control group (C), treatment 1 (T1), or treatment 2 (T2). All groups undergo a training session, after that they will write a cover letter, for a hypothetical job description of one of six participating firms, based on their preference. The C group receives a generic, unrelated training and is blocked from accessing LLMs when they write a cover letter. T1 receives the same training as C but they are allowed to use LLMs to write their cover letter. T2 receives a training on how to write effective LLM prompts, before being able to use LLMs to write their cover letter.
Intervention Start Date
2024-04-15
Intervention End Date
2024-05-31

Primary Outcomes

Primary Outcomes (end points)
Evaluation of quality of cover letter and CV, based on a pre-specified and standardized grading criteria.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Time us, browser history, LLM prompts.
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Participants will be randomized across three treatments, see above. Participants will first receive a short training, before writing a cover letter to a hypothetical position at a firm of their choosing (among the six partner firms in this study). Cover letters and CVs will be evaluated by recruiters from the six firms who are blind to the interventions and experiment.
Experimental Design Details
Not available
Randomization Method
Randomization done by a computer – stratified based on: gender, GPA, BSc vs. MSc student.
Randomization Unit
Individual-level
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
600
Sample size: planned number of observations
600 individuals
Sample size (or number of clusters) by treatment arms
Divided equally across treatment arms.

If we have a sample of less than 400 participants, we will drop T1 in order to maximize power.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
0.281 standard deviations, for an 80% power at the 5% level. Assumptions: explanatory power of stratified variables of outcome variable: 0%; attrition rate: 0%, two-sided t-test. This is a very conservative estimate as we will stratify on many variables that have been documented in the academic literature to influence employment prospects; and because the test is two-sided.
IRB

Institutional Review Boards (IRBs)

IRB Name
TiSEM Institutional Review Board
IRB Approval Date
2024-03-18
IRB Approval Number
IRB FUL 2024-002