Sample size (or number of clusters) by treatment arms
We plan to collect data from 2,500 Prolific participants who act as recruiters. We base our planned sample size on power calculations using a pilot. We will require that all participants who act as recruiters have completed at least 100 past studies with an approval rate of 95% or above. We randomly assign 50% of recruiters to evaluate blind resumes, and the other 50% to evaluate non-blind resumes.
We also plan to collect data from 4,000 Prolific participants who act as job candidates. We will require that all candidates have completed at least 100 past studies with an approval rate of 95% or above. We will use Prolific's filters to aim for balance on our key variables. In particular, we will impose that Prolific field candidate data as follows:
- 1,000 men, age 45+
- 1,000 men, age under 45
- 1,000 women, age 45+
- 1,000 women, age under 45
Given the available participant pool on Prolific, we anticipate being able to fulfill our full data collection while maintaining these restrictions. However, after 2 weeks of data collection for the candidate survey, if we have not hit 4,000 candidate participants, then we will drop these demographic restrictions and recruit the remainder of the sample. We anticipate that collecting 4,000 candidate responses will yield enough high quality data (as identified by our exclusion restrictions described below) to remain well-powered for our main hypotheses.
Importantly, these sample sizes correspond to the unrestricted sample, i.e. the responses we collect before eliminating participants who do not pass our attention and understanding checks.
We include attention questions in both recruiter and candidate surveys, as suggested by Haaland et al. (2023). In both surveys, we ask: ``The next question is about the following problem. In questionnaires like ours, sometimes there are participants who do not carefully read the questions and just quickly click through the survey. This means that there are a lot of random answers which compromise the results of research studies." To recruiters, we then ask ``To show that you read our questions carefully, please enter twenty as your answer to the following question. How many resumes did you just evaluate?". To candidates, we ask ``To show that you read our questions carefully, please enter twenty as your answer to the following question. How many different resumes did we show you?". We plan to exclude from our final sample any respondent who does not answer this question correctly (we accept typos). We ask this question after the price lists in both surveys.
We also include questions to measure whether respondents understand basic instructions.
For recruiters, we ask after presenting the list instructions: ``If you are selected for extra payment and you chose to hire the candidate from the computer's chosen row, what bonus payment will you receive?
- $0
- 50 cents ($0.50) per question answered correctly by the candidate I matched with
- 100 cents ($1) per question answered correctly by the candidate I matched with"
For candidates, we ask after presenting the first resume: ``What information was on your resume for this opportunity?
- Your favorite subject, Your educational attainment, Sample performance on the technical test
- Your favorite subject, Your age, Your educational attainment, Your sex, Sample performance on the technical test"
Respondents have the opportunity to modify their answer if they fail to provide a correct answer. However, we plan to exclude from our main analysis any respondent who did not answer this question correctly the first time.
Finally, in the lists in both surveys, we added strictly dominated choices to check whether respondents provided rational responses. We plan to restrict our main analysis to participants who submitted turning points that are not strictly dominated. For recruiters, the maximum payment they can receive when hiring the candidate is 500 cents (in case candidates answered all ten test questions correctly). A rational recruiter who understood the list instructions correctly should not provide a turning point above that threshold. We plan to exclude recruiters who submit at least two answers above 500 cents. For candidates, the rational threshold is 100 cents. We plan to exclude any candidate whose turning point is above that threshold. An irrational turning point on any list will lead to the exclusion of all answers from the participant in our main analysis. For recruiters, we allow one mistake (that is, an answer above 500), but we plan to exclude the list where that mistake is.
Finally, we added a timer to measure how much time participants spend on each instruction page (respondents do not know about the timer). We plan to exclude from the main analysis any respondent who does not spend sufficient time reading the main instructions.
To determine a reasonable threshold, we use findings from research in reading and cognitive psychology that highlights the trade-off between speed and accuracy in reading. This research estimates that the average silent reading speed for English readers of non-fiction is around 250 words per minute, and that thorough comprehension drops past two or three times that reading speed (Brysbaert, 2019; Rayner et al., 2016). Qualtrics states the average human reading speed is 300 words per minute and uses this speed to estimate the survey duration (see https://www.qualtrics.com/support/survey-platform/survey-module/survey-checker/survey-methodology-compliance-best-practices/). We plan to restrict our final sample to a reading speed of approximately 400 words per minute on the main instructions. We use Qualtrics timer embedded within our survey pages in order to measure how long participants spend on each page of the survey. Participants who do not spend the required time on the price list instructions page are excluded from the final sample:
- For recruiters: the price list instructions page includes 266 words (excluding the list example), which corresponds to a minimum reading speed of 40 seconds.
- For candidates, the price list instructions page includes 242 words (excluding the list example), which corresponds to a minimum reading speed of 36 seconds.