|
Field
Experimental Design (Public)
|
Before
This is an online study conducted on Prolific. The main task of a participant is to classify candidates as above or below-the-bar.
First, we introduce participants to the task of evaluation of candidates. These candidates are past study participants who completed a math and science test and were classified as above or below-the-bar based upon having a test score above a set threshold. We use these past study participants to construct “pools” of candidates for our study participants to assess.
Before assessing candidates, each evaluator (main study participant) is provided with information about the candidate pool from which their candidates are chosen. In particular, they are told the share of male candidates who were above-the-bar (likelihood of a given male candidate being above-the-bar) and the share of female candidates who were above-the-bar.
Evaluators are randomly-assigned to one of four different candidate pools. There are 50 total candidates in each pool. The pools vary in the likelihood of male and female candidates being above-the-bar:
(1) 40-40 pool: 40% of female candidates and 40% of male candidates were above-the-bar
(2) 60-60 pool: 60% of female candidates and 60% of male candidates were above-the-bar
(3) 40-60 pool: 40% of male candidates and 60% of female candidates were above-the-bar
(4) 60-40 pool: 60% of male candidates and 40% of female candidates were above-the-bar
After describing the selected pool to the evaluator, we randomly-draw a candidate from the described pool for the evaluator to assess. Participants see some basic information about this candidate, including their gender and their answers to some (arguably irrelevant) questions (i.e., do you prefer mountain vacations or beach vacations?). Participants provide an incentivized estimated likelihood of that candidate being above-the-bar.
Participants can pay to acquire additional information about the candidate. We refer to these as “signals” – a signal can be either good or bad. The probability with which a good signal is drawn depends upon the true performance of the candidate:
• A candidate who is above-the-bar produces a good signal with probability of 75% and a bad signal with probability of 25%
• A candidate who is below-the-bar produces a good signal with probability of 25% and a bad signal with probability of 75%.
Our primary treatment randomization is whether evaluators can choose whether or not to acquire signals about the candidate. We call this endogenous or exogenous information acquisition:
• Exogenous information acquisition: the evaluator must acquire 5 signals about the candidate before making a classification. After each signal, they provide an updated incentivized likelihood of the candidate being above-the-bar. After acquiring all 5 signals, they make a final binary classification of the candidate, classifying the candidate as either “above-the-bar” or “below-the-bar.”
• Endogenous information acquisition: the evaluator can choose to acquire up to 5 signals about the candidate before making a classification. After each signal, they provide an updated incentivized likelihood of the candidate being above-the-bar. At any point, the evaluator can choose to stop acquiring signals and make a final classification, classifying the candidate as either “above-the-bar” or “below-the-bar.”
Both the random assignment to candidate pools and the random assignment to information treatment is across-participant. All evaluators evaluate 10 candidates from their selected pool, chosen randomly with replacement. They receive no feedback or information on their decision-making until the end of the study.
At the conclusion of the study, we randomly select one candidate that participant evaluated. For that randomly selected candidate, we pay a bonus of $7 if their final classification of the chosen candidate is correct, and a bonus of $0.50 if they incorrectly classified the randomly-chosen candidate. In addition, we deduct $0.05 from the bonus payment for every signal they acquired for the randomly-chosen candidate.
|
After
This is an online study conducted on Prolific. The main task of a participant is to classify candidates as above or below-the-bar.
First, we introduce participants to the task of evaluation of candidates. These candidates are past study participants who completed a math and science test and were classified as above or below-the-bar based upon having a test score above a set threshold. We use these past study participants to construct “pools” of candidates for our study participants to assess.
Before assessing candidates, each evaluator (main study participant) is provided with information about the candidate pool from which their candidates are chosen. In particular, they are told the share of male candidates who were above-the-bar (likelihood of a given male candidate being above-the-bar) and the share of female candidates who were above-the-bar.
Evaluators are randomly-assigned to one of four different candidate pools. There are 50 total candidates in each pool. The pools vary in the likelihood of male and female candidates being above-the-bar:
(1) 40-40 pool: 40% of female candidates and 40% of male candidates were above-the-bar
(2) 60-60 pool: 60% of female candidates and 60% of male candidates were above-the-bar
(3) 40-60 pool: 40% of male candidates and 60% of female candidates were above-the-bar
(4) 60-40 pool: 60% of male candidates and 40% of female candidates were above-the-bar
After describing the selected pool to the evaluator, we randomly-draw a candidate from the described pool for the evaluator to assess. Participants see some basic information about this candidate, including their gender and their answers to some (arguably irrelevant) questions (i.e., do you prefer mountain vacations or beach vacations?). Participants provide an incentivized estimated likelihood of that candidate being above-the-bar.
Participants can pay to acquire additional information about the candidate. We refer to these as “signals” – a signal can be either good or bad. The probability with which a good signal is drawn depends upon the true performance of the candidate:
• A candidate who is above-the-bar produces a good signal with probability of 75% and a bad signal with probability of 25%
• A candidate who is below-the-bar produces a good signal with probability of 25% and a bad signal with probability of 75%.
Our primary treatment randomization is whether evaluators can choose whether or not to acquire signals about the candidate. We call this endogenous or exogenous information acquisition:
• Exogenous information acquisition: the evaluator must acquire 5 signals about the candidate before making a classification. After each signal, they provide an updated incentivized likelihood of the candidate being above-the-bar. After acquiring all 5 signals, they make a final binary classification of the candidate, classifying the candidate as either “above-the-bar” or “below-the-bar.”
• Endogenous information acquisition: the evaluator can choose to acquire up to 5 signals about the candidate before making a classification. After each signal, they provide an updated incentivized likelihood of the candidate being above-the-bar. At any point, the evaluator can choose to stop acquiring signals and make a final classification, classifying the candidate as either “above-the-bar” or “below-the-bar.”
Both the random assignment to candidate pools and the random assignment to information treatment is across-participant. All evaluators evaluate 5 candidates from their selected pool, chosen randomly with replacement. They receive no feedback or information on their decision-making until the end of the study.
We will pay 1 out of every 10 participants (randomly-selected) additional incentive pay. For those participants:
At the conclusion of the study, we randomly select one candidate that participant evaluated. For that randomly selected candidate, we pay a bonus of $7 if their final classification of the chosen candidate is correct, and a bonus of $0.50 if they incorrectly classified the randomly-chosen candidate. In addition, we deduct $0.05 from the bonus payment for every signal they acquired for the randomly-chosen candidate in the endogenous treatment. We also randomly select one likelihood slider and pay them an extra $1 if within 10pp of the true likelihood.
|