Worth the Effort: Measuring and Predicting Investment in a Costly Application Process

"Applicant" subjects will face random variation in the marginal cost of applying for a particular job. They may face no marginal cost, an effort cost, a monetary cost, or both an effort and a monetary cost.

"Hiring recruiter" subjects will be asked to hire a fixed proportion of applicants from one of the four treatment arms.

2023-04-10

2023-07-01

1) Whether or not a subject applies for each of the job opportunities they are shown.
2) Which resumes a recruiter selects.
3) Whether or not hired applicants actually follow through and attempt the task they are hired for.
4) Whether or not hired applicants are able to successfully complete the task they are hired for.

The tasks for which subjects will apply are modified GRE exam questions in either English or math. The math task will consist of two multiple choice tests, and success will be defined as answering over 50% of questions correctly on both tests. For the English task, there is a multiple choice test followed by an Analytical Writing prompt. Success on the English task will involve answering over 50% of questions correctly on the multiple choice test and receiving a score of at least 4/6 on the Analytical Writing prompt. The prompt responses will be reviewed by external evaluators recruited by the PI.

Note: Original pre-registration had "success" measured as scoring exactly 50% or greater on any particular task. AFTER hiring decisions have been made, but BEFORE hired subjects are invited to actually complete the assigned task(s), the indicator of success has been amended as scoring ABOVE 50% on any particular task (at least 51% on any given multiple choice test and at least 4/6 on the English essay task).

We will randomly vary the cost of applying for each of two job opportunities shown to survey respondents. We will observe their resume and demographic information as well as their application decisions. We will then conduct a hiring experiment wherein we will randomly vary the treatment-type of resumes shown to "recruiter" subjects and then observe the recruiters' hiring decisions. Finally, we will observe hired applicants' performance on the assigned tasks.

The target population is current full-time U.S.-based undergraduate students aged 18-25 AND individuals aged 18-25 who already have undergraduate degrees but have not engaged in any graduate study. There are a total of just under 3,000 such subjects on the platform. The limitations of the prescreening options offered on the subject recruitment platform being used for this experiment preclude recruiting from two disjoint populations for the same study at the same time. Thus, we will recruit exclusively from the population current undergraduate students for the first week of the experiment; after one week, we will begin recruiting from a similar sample of recent college graduates, with no graduate education, aged 25 and under, then we will return to recruiting from undergraduate students for the following week, and so on until 800 applicant subjects have been recruited.

For the application stage, the experiment will be run in consecutive "waves" where the total number of subjects to be recruited will alternate between 50 and 100 subjects. The PI will flip a coin, and if it lands on heads, the first wave will have 100 subjects in it; if it lands on tails, the first wave will have 50 subjects in it. The wave size will alternate after that point until subject total is at maximum. This is done to test whether the size of the visible recruitment pool affects a subject's willingness to apply.

Hiring recruiter subjects will be randomly assigned to view a random subset of resumes from one treatment arm (control, money, effort, money + effort). The chances of being assigned to each of these treatment arms will be scaled based on how many applications we receive in each treatment arm (e.g. if 15% of applications come from the money + effort treatment arm, the chance of being assigned to evaluate resumes from that arm will be 15%).

In addition to "hiring recruiters", there will also be "evaluating recruiters" whose only task will be to review resumes and describe how qualified they feel they are. Since evaluating recruiters will only see resumes regardless of the resume-holder's treatment status, the resumes they see will be randomly selected from the full subject pool.

There will be two "attention checks" included in the first portion of the applicant survey, before applicants see which treatment arm they have been sorted into. If a participant incorrectly answers both attention check questions, they will be immediately excluded from the survey.

There will also be four comprehension questions included in the applicant survey after treatment is assigned. Any respondent who responds to at least two of these four comprehension questions incorrectly will still be allowed to complete the entire survey, but will be classified as a "confused" subject for the purposes of analysis. Analyses will be presented separately for the full respondent pool and for the non-"confused" subject pool.

Randomization performed on a computer and via coin flip.

Randomization into treatment will be done at the individual level.

No

800 "applicant" subjects
31* "hiring recruiter" subjects
27 "evaluating recruiter" subjects

*This quantity is an estimate based on estimated application rates gleaned from a pilot study.

Each applicant subject will view two job postings, so there will be two primary outcome (apply or don't apply) observations per subject, for a total of 1600. Each "hiring" recruiter will view 30 resumes, so there will be ~28x30 = 840 primary outcome (hire or no hire) observations. Each "evaluating" recruiter will view ~30 resumes, such that there will be one evaluation of every subject resume, for a total of 800 observed resume evaluations.

Applicant subjects:
200 control
200 effort cost
200 monetary cost
200 monetary AND effort cost

Hiring recruiter subjects (estimates based on pilot):
11 for control group
6 for monetary cost treatment
8 for effort cost treatment
6 for monetary AND effort cost treatment