Experimental Design
At the start of the experiment, participants are informed that 400 U.S. adults, representative of the U.S. population, took part in an online 20-questions Math and Science Quiz and briefed on the topics covered in the quiz. The experiment is divided into two parts. In each part they can earn an additional $5. At the end of the study, one of these parts will be randomly selected for bonus payment.
In Part 1, participants review short CVs of eight pre-selected workers and estimate for each worker the likelihood that their performance is in the top 50% relative to the performance of the other 400 workers ("top performer"). The order in which the workers are presented is randomized. The CVs include information on gender (female/male), level of education (no bachelor's degree / bachelor's degree or higher), and month of birth (even/odd). Participants then receive performance predictions for each worker from a machine learning algorithm and have the opportunity to revise their initial estimates. They are informed that the algorithm is trained on the remaining workers from the full 400-worker sample and uses all the CV variables in the baseline condition (excluding gender in the first treatment condition and exluding month of birth - even vs. odd numbered - in the second treatment), along with the workers' performance on a prior math and science quiz to make the performance predictions. Performance on the other quiz is strongly correlated with Math and Science Quiz performance, making the predictions informative. Prediction results remain constant as gender (month of birth) is neither a significant predictor nor correlated with significant predictors. Participants are shown the eight workers in the same order as before and must submit their final estimates, which will be used to determine their $5 bonus payment.
Part 2 follows immediately, in which participants complete a simple logic task after completing the hiring task. In the hiring task, participants are again presented with the eight workers and are asked to decide whether to hire them, making a yes/no decision for each individual. Participants are told that they will receive $2.50 if they solve the logic task correctly. For the bonus payment, one of the hiring decisions is randomly selected. If the selected decision involves hiring a top performer, the participant earns $5. If the selected decision is to hire a worker who is not a top performer, the participant earns $0. If the selected decision is not to hire the worker, the participant keeps their $2.50. The hired worker in the randomly selected decision receives $2.50 regardless of performance.
Participants are then asked to estimate the accuracy of the algorithm (incentivized). The experiment concludes with a brief survey on attitudes toward algorithms and technology, knowledge about algorithms, perceptions of gender discrimination in the U.S., and a demographic questionnaire.
Comprehension questions are presented throughout the experiment, which participants must answer to proceed. All beliefs are elicited using the stochastic Becker-DeGroot-Marschak method.
The quiz was previously conducted as part of a separate online Math and Science Quiz involving 400 U.S. adults, representative of the U.S. adult population, and consisted exclusively of ASVAB questions. These adults completed two similar 20-question quizzes consecutively. The variables presented in the CVs and used for the algorithm’s predictions were selected based on results from a separate survey of 300 U.S. adults (representative sample) that measured beliefs about differences in Math and Science Quiz performance by gender, education level, and month of birth. The algorithm is trained on the sample of 400 workers, excluding the eight selected workers. The algorithm is based on a logistic regression model.