Experimental Design Details
In the ego treatment, subjects first complete logic puzzles presented as parts of an IQ test. For the logic puzzles, we use Raven's Progressive Matrices, a widely administered test which was originally developed as a method to gauge general intelligence in a non-verbal setting.
Subjects' performance on these puzzles are evaluated as a mixture of speed and accuracy and they are incentivized to perform as well as they can.
For the non-ego treatment, subjects do not complete IQ puzzles. Instead, they report their beliefs (described below) about a random integer drawn uniformly from 1 to 100.
During the course of the experiment, participants report their belief about either performing above a given rank in a pool of other subjects at IQ puzzles (ego treatment) or about whether the random number is above a given value (non-ego treatment). In both cases, the elicited belief regards a binary event and is reported as a probability. It is incentivized using the lottery method.
They report these beliefs multiple times before and after receiving feedback, the structure of which is detailed below.
For all treatments, the participants first receive noisy feedback regarding the binary outcome (whether they outrank a given threshold in the ego treatment and whether the random number is above a given threshold in the non-ego treatment). For clearer comprehension, the participants are told that there are 6 truth-telling gremlins and 2 lying gremlins. They are shown a visualization of drawing the gremlins with replacement, and receive feedback from a gremlin of unknown type. Participants receive three signals from this set of gremlins, reporting their beliefs after each signal. Although not described to participants as such, this unbiased noisy feedback round can be considered a "practice round" -- it was designed with the intent to slowly introduce participants to various components of the experiment before introducing them to the biases.
The participants are then introduced to potential biases. They are told that in addition to the truth-telling and lying gremlins, there are two additional types of gremlins. First, the Yay Sayer gremlin delivers positive news regardless of the state of the world. In contrast, the Nay Sayer gremlin always delivers negative news.
Along with the 6 truth-tellers and 2 liars, there will be either 2 Yay Sayers or 2 Nay Sayers in the urn to be drawn with replacement. As described above, the exact procedure in which these biased gremlins will be added will depend on the treatment arm.
To ensure that the participants understand the nature of the biases, we add a comprehension check: we ensure that they understand that adding a positive bias will increase the likelihood of the positive signal, yet reduce the meaningfulness of the signal, and similarly for a negative bias.
In the chosen-bias treatment arm, the participants are given the choice between Yay Sayer and Nay Sayer biases for each round. In contrast, in the forced-choice forced bias treatment arm, the bias is randomly selected at the beginning of each round and disclosed to the participant.
Elicitations about Signal Attribution
In all treatment arms, if an agent receives a signal that is concurrent with the bias (i.e., a ``yes" signal under Yay Sayers or a ``no" signal under Nay Sayers), they are asked what they think is the probability the signal came from the uninformative biased gremlin. This elicitation will take place after agents report their new belief about their rank.
Metacognition (About Self and Others)
After participants finish the belief elicitation portion of the study described above, subjects are asked in an incentive-compatible manner what they believe the effect of each bias was on the final beliefs that they just reported and the final beliefs of other participants who had previously taken the study. In particular, they are asked how the belief biases (relative to a Bayesian benchmark) depend on the bias selection. For individuals in the chosen-bias treatment arm, they are also incentivized to report their beliefs about bias selection behavior. These questions are asked separately about other MTurkers and about the given participant.
We conclude the experiment with standard demographic questions to ensure that our samples are balanced on major demographics. We query for gender, age, education, income, occupation type, as well as media usage.