Endogenous Information Acquisition in Candidate Evaluation

2023-03-09

2023-03-20

We will have three primary outcome measures:
1. The final classification a decision-maker provides for a candidate: this is the incentivized classification that the decision-maker provides for each candidate that they are assigned. The decision-maker can classify each candidate as either “above-the-bar” or “below-the-bar.” For decision-makers in the exogenous treatment, the final classification occurs after receiving 5 quality signals for a candidate. For decision-makers in the endogenous treatment, the final classification is made when the decision-maker chooses to stop acquiring additional signals and provide a classification (so after anywhere between 0 – 5 signals are observed). We will compare these final classifications across treatment and candidate gender and assess accuracy rates.
2. The final evaluation a decision-maker provides for a candidate: the incentivized likelihood of being “above-the-bar” that the decision-maker provides for each candidate that they are assigned. After each quality signal that the decision-maker views, the decision-maker is asked to provide an incentivized estimate of the likelihood that the candidate they are assessing is “above-the-bar.” For decision-makers in the exogenous treatment, the final evaluation occurs after receiving 5 quality signals for a candidate. For decision-makers in the endogenous treatment, the final evaluation is made when the decision-maker chooses to stop acquiring additional signals and provide a classification (so after anywhere between 0 – 5 signals are observed). We will compare these final evaluations across treatment and candidate gender and assess accuracy rates.
3. Number of signals acquired: the number of costly signals purchased by a decision-maker for a given candidate (in the endogenous information treatment). We will compare the number of signals acquired by gender and prior beliefs.

In addition to these primary outcome measures, we are also interested in the following secondary outcome measures:
• The initial evaluation a decision-maker provides for a candidate: the incentivized likelihood of being “above-the-bar” that the decision-maker provides for each candidate that they are assigned, prior to viewing any signals.
• The interim evaluations a decision-maker provides for a candidate: the incentivized likelihood of being “above-the-bar” that the decision-maker provides for each candidate that they are assigned after any particular signal.
• Belief revisions: the change in a decision-maker’s estimated likelihood that a candidate is above-the-bar after seeing a positive (or negative) signal of quality.

This is an online study conducted on Prolific. The main task of a participant is to classify candidates as above or below-the-bar.

First, we introduce participants to the task of evaluation of candidates. These candidates are past study participants who completed a math and science test and were classified as above or below-the-bar based upon having a test score above a set threshold. We use these past study participants to construct “pools” of candidates for our study participants to assess.

Before assessing candidates, each evaluator (main study participant) is provided with information about the candidate pool from which their candidates are chosen. In particular, they are told the share of male candidates who were above-the-bar (likelihood of a given male candidate being above-the-bar) and the share of female candidates who were above-the-bar.

Evaluators are randomly-assigned to one of four different candidate pools. There are 50 total candidates in each pool. The pools vary in the likelihood of male and female candidates being above-the-bar:

(1) 40-40 pool: 40% of female candidates and 40% of male candidates were above-the-bar
(2) 60-60 pool: 60% of female candidates and 60% of male candidates were above-the-bar
(3) 40-60 pool: 40% of male candidates and 60% of female candidates were above-the-bar
(4) 60-40 pool: 60% of male candidates and 40% of female candidates were above-the-bar

After describing the selected pool to the evaluator, we randomly-draw a candidate from the described pool for the evaluator to assess. Participants see some basic information about this candidate, including their gender and their answers to some (arguably irrelevant) questions (i.e., do you prefer mountain vacations or beach vacations?). Participants provide an incentivized estimated likelihood of that candidate being above-the-bar.

Participants can pay to acquire additional information about the candidate. We refer to these as “signals” – a signal can be either good or bad. The probability with which a good signal is drawn depends upon the true performance of the candidate:
• A candidate who is above-the-bar produces a good signal with probability of 75% and a bad signal with probability of 25%
• A candidate who is below-the-bar produces a good signal with probability of 25% and a bad signal with probability of 75%.

Our primary treatment randomization is whether evaluators can choose whether or not to acquire signals about the candidate. We call this endogenous or exogenous information acquisition:
• Exogenous information acquisition: the evaluator must acquire 5 signals about the candidate before making a classification. After each signal, they provide an updated incentivized likelihood of the candidate being above-the-bar. After acquiring all 5 signals, they make a final binary classification of the candidate, classifying the candidate as either “above-the-bar” or “below-the-bar.”
• Endogenous information acquisition: the evaluator can choose to acquire up to 5 signals about the candidate before making a classification. After each signal, they provide an updated incentivized likelihood of the candidate being above-the-bar. At any point, the evaluator can choose to stop acquiring signals and make a final classification, classifying the candidate as either “above-the-bar” or “below-the-bar.”

Both the random assignment to candidate pools and the random assignment to information treatment is across-participant. All evaluators evaluate 5 candidates from their selected pool, chosen randomly with replacement. They receive no feedback or information on their decision-making until the end of the study.

We will pay 1 out of every 10 participants (randomly-selected) additional incentive pay. For those participants:

At the conclusion of the study, we randomly select one candidate that participant evaluated. For that randomly selected candidate, we pay a bonus of $7 if their final classification of the chosen candidate is correct, and a bonus of $0.50 if they incorrectly classified the randomly-chosen candidate. In addition, we deduct $0.05 from the bonus payment for every signal they acquired for the randomly-chosen candidate in the endogenous treatment. We also randomly select one likelihood slider and pay them an extra $1 if within 10pp of the true likelihood.

Computer

We randomize treatment assignment at the individual level. Randomization by the computer.

No

3,200 participants

3,200 participants

400 participants in each of the 4 pools across 2 treatment conditions (exogenous and endogenous information acquisition), for a total of 3,200 participants.