Cover Letter Evaluations and LLMs

Last registered on November 15, 2024

View Trial History

Pre-Trial

Trial Information

General Information

Title

Cover Letter Evaluations and LLMs

RCT ID

AEARCTR-0014784

Initial registration date

November 07, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

November 15, 2024, 1:40 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

Netherlands

Region

Prolific

Primary Investigator

Name

Till Wicker

Affiliation

Tilburg University

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Niccolo Zaccaria

PI Affiliation

Tilburg University

Contact Investigator

PI Name

Giuseppe Musillo

PI Affiliation

Tilburg University

Contact Investigator

PI Name

Kian Abbas Nejad

PI Affiliation

Tilburg University

Contact Investigator

Additional Trial Information

Status

In development

Start date

2024-11-11

End date

2024-11-29

Keywords

Labor

Additional Keywords

Job Applications, Cover Letter, LLMs

JEL code(s)

J08, J64, C90

Secondary IDs

Prior work

This trial is based on or builds upon one or more prior RCTs.

Abstract

The use of Large Language Models (LLM) could have a profound impact on the labor market. The first part of this research project (AEARCTR-0013355) illustrates that the quality of cover letters improves as job seekers can use LLMs (such as ChatGPT) when writing their cover letter – as evaluated by recruiters who were blind to the treatment arms. However, the second part of the research project focuses on recruiters, to understand their perceptions of the use of LLMs in cover letters and job applications more generally. This is done through an online lab experiment, with recruiters from Prolific.

External Link(s)

Registration Citation

Citation

Abbas Nejad, Kian et al. 2024. "Cover Letter Evaluations and LLMs." AEA RCT Registry. November 15. https://doi.org/10.1257/rct.14784-1.0

Sponsors & Partners

Experimental Details

Interventions

Intervention(s)

Recruiters are randomly assigned to control group (C), partial info (T1), or full info (T2). All recruiters will be shown five cover letters that they have to evaluate based on a pre-defined evaluation criteria. These five cover letters are random drawn from a sample of 12 cover letters, written by job seekers who participated in AEARCTR-0013355. All cover letters refer to the same job description, which recruiters are shown. Half of the cover letters were written with the assistance of ChatGPT, and half were not. Recruiters in the C group are not informed about this possibility, while recruiters in T1 are informed that cover letters may be written with the use of ChatGPT. Recruiters in T2 get told exactly which cover letters are written with the use of ChatGPT.

Intervention (Hidden)

Intervention Start Date

2024-11-11

Intervention End Date

2024-11-29

Primary Outcomes

Primary Outcomes (end points)

Evaluation of quality of cover letter and CV, based on a pre-specified and standardized grading criteria.

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Time spent evaluating the cover letter. Responses to post-experimental questionnaire.

Secondary Outcomes (explanation)

Experimental Design

Recruiters will be randomized across three treatments, see above. Recruiters will first see the job description, as well as evaluation criteria to be used, before being shown a cover letter. They are shown one cover letter at a time, and asked to evaluate it based on the pre-specified rubric. After evaluating five cover letters, recruiters will be asked general questions about cover letters, recruitment, and the use of LLMs.

Experimental Design Details

Randomization Method

Randomization done by Qualtrics, without stratification.

Randomization Unit

Individual

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

400 individuals.

Sample size: planned number of observations

400 individuals.

Sample size (or number of clusters) by treatment arms

400 divided across 3 treatments, equaling 133. However, every participant will evaluate multiple cover letters.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Each recruiter will evaluate 5 cover letters, hence we will have 2000 observations. We will cluster standard errors at the recruiter level, and include recruiter and applicant fixed effects. Due to the large number of observations, we are well-powered. With 400 observations, we would have 80% power to detect a two-sided effect size of 0.34 s.d., while with 2000 observations, we could detect an effect size of 0.15 s.d. (assuming independent observations). As we cluster standard errors, our power will be in between the two bounds.

Supporting Documents and Materials

IRB