Experimental Design Details
We will recruit participants on mTurk for our study, which consists of two stages, with the second stage occurring at a specified time and date one week after the first stage (known to them upfront). Participants will be encouraged to take stage 1 only if they can return for stage 2, and we will pay participants for each stage separately. We will strongly incentivize participants to complete both stages by providing a minimum payment of $6.50 for stage 2, while offering only $1.50 for completing stage 1 (each stage takes the same amount of time, roughly 15-20 minutes).
Participants who agree to take part in stage 1 will then be asked to answer 60 multiple-choice trivia questions. Each trivia question will be accompanied with a question asking the participant to report their confidence in the answer they chose on a six-point scale from “No confidence at all” to “Certain.” Each pair of questions (one trivia question and one confidence reporting question) will be presented on its own screen, and participants will not be able to go back and change their answers. The order in which the trivia questions appear will be randomized across participants. We will also use five randomly-placed attention check questions to make sure that participants were paying attention. After participants complete the survey, they will be asked to confirm that they intend to participate in stage 2 at the specified time and date
The second stage of the study will be designed to explore participants’ likelihood of engaging in herding behavior, and how that propensity varies under different financial incentive levels. Returning participants will see the same 60 trivia questions they saw in stage 1, and will be again asked to answer them. However, there will be two key differences in the process. First, when participants are presented with a trivia question in stage 2, they will also be reminded of their answer to the question in stage 1 and the most popular answer to the question from stage 1 across all participants. Second, in stage 2 participants will be financially incentivized to answer each question correctly, with questions assigned to one of four different levels of financial incentive: $0, $1, $2, or $3. Participants will be told at the beginning of the second stage that correct answers to the 60 questions will be associated with financial incentives of $0, $1, $2, or $3, and that one question from each of the four possible financial incentive levels will be randomly selected to be used for payment. We will randomly vary the order that questions appeared and the financial incentives associated with each question, across participants, while keeping the difficulty level of questions similar across incentive levels.
Incentives and ordering will be randomized as follows: participants will be randomly assigned to one of four possible “versions” of the stage 2 survey, designed to allow us to evenly vary the size of the financial incentive associated with any given question. Before stage 2 is implemented, the 60 questions will be classified into one of fifteen difficulty levels (four questions per difficulty level) based on the percent of participants who answered the question correctly in stage 1. Financial incentives will then randomly be assigned to questions in the four versions such that each question will appear with a different incentive in each of the four versions, and each incentive will only appear once per difficulty level in each of the four versions.
To prevent participants from looking up the answers when money is on the line, we will require participants to explicitly agree to not look up the answers prior to answering the questions with financial incentives. Additionally, participants will be told that if there are any signs of cheating (higher number of clicks or longer answer times relative to stage 1 for that participants), they will not receive any payment. This was an effective deterrent in our first iteration of this study.
After participants complete the trivia question section of the survey, they will be asked to answer a handful of demographic questions, at which point they self-report their age, gender, ethnicity, education, and geographic region of residence.