Experimental Design Details
To address our research question, we will conduct a field experiment in three steps.
Step 1: Job Posting and Applicant Assessment
We will create a research assistant (RAs) position and post the job advertisement campus-wide at the University of Michigan. The RA role focuses on replication analysis, including verifying computational reproducibility, identifying coding errors, and creating structured replication packages. The position requires technical ability in statistical analysis and code review, as well as a high level of effort demonstrated through attention to detail, persistence and so on. Applicants will submit their resumes and cover letters, which will serve as the basis for evaluation.
To assess the impact of actual AI use, each applicant has two cover letters: one written independently without AI (under our supervision), and one we generate using AI with standardized prompts based on their resume.
Specifically, to make sure that applicants write the human-generated cover letter without any AI assistance, they will be invited to a lab session where they will independently draft and submit their cover letter within 45 minutes.
To provide a benchmark to incentivize recruiters' evaluations in the future, we will include a series of assessment tasks which measures applicants' true technical ability and willingness to exert effort. The first task consists of statistical problems designed to evaluate technical skills. The second task measures effort, requiring applicants to extract information from the title pages of academic papers. The third task is a more comprehensive mini replication task, which requires both technical ability and effort to complete. All tasks will be time-limited.
Step 2: Recruiter Selection
We will recruit two groups of recruiters to evaluate the applicants and make hiring decisions. Our study involves two types of evaluators: PhD students with direct experience hiring RAs for similar roles, and HR professionals from Prolific. This allows us to compare decisions from domain experts (PhD students) against those from professional recruiters with broader hiring experience. We will approach the PhD students through the target email service provided by the university. The HR professionals will be recruited directly from Prolific, an online experiment platform.
Step 3: Recruiter Evaluations
Each recruiter will review a series of applicant profiles, each including a resume and a cover letter. Here we employ a 2x2 experimental design: To manipulate the actual use of AI, either the human- or AI-generated cover letter will be randomly assigned to each profile. To manipulate recruiters’ perceptions, we will provide a noisy signal indicating whether AI assistance is likely to be used in the cover letter.
Recruiters will evaluate each applicant profile on (1) technical ability; (2) willingness to exert effort; and (3) an overall evaluation as a final hiring recommendation.
The procedures to review each profile are as follows. First, recruiters see the resume and cover letter and report their prior belief about the likelihood that the cover letter is AI-generated. Next, they receive the signal with an overall accuracy of 75%, indicating whether the cover letter is likely to be human- or AI-generated. We then elicit their posterior belief about the likelihood to investigate how much they update. Lastly, their evaluations of technical ability and willingness to exert effort are elicited as beliefs about the applicant’s performance in the assessment tasks and are incentivized. Their overall evaluations are provided by reporting a Willingness-to-Hire (WTH) score on a scale of 1 to 10. The ranking based on the WTH scores will be incentivized according to applicants’ rankings in the mini replication task completed during the intern period.
For calibration purposes, after completing their initial evaluations, recruiters will have an opportunity during the final review stage of the study to revisit and potentially revise their assessments of each applicant's technical ability, willingness to exert effort, and overall evaluation. During this review stage, recruiters will again have access to each applicant's resume and cover letter to inform their final assessments. They will be told that our final hiring decisions will be made based on all recruiters’ aggregate overall evaluations.