The experiment is conducted on mturk. See the supplementary doc for the MTurk instructions, which explain the design.
We present subjects with information on the composition of a particular box. The box is equally likely to have 9 blue balls and 1 yellow ball, and 1 blue ball and 9 yellow balls. We then present subjects with information about the composition. This information is generated in two ways, adjusted so that they yield comparable belief updates:
(1) Showing the subjects either balls which are from an alternative "noise box" containing 5 blue balls and 5 yellow balls, or
(2) Drawing a ball from the box and switching the observation's color. (We dub this "swapping")
We then compare beliefs when subjects are provided information about their past draws. Specifically, for all subjects and in all rounds, we present information with two draws from the box, as described above. We subsequently will provide another signal with probability .5, and a "fact-check" with probability .5.
In case (1), a fact-check informs the subject of whether or not a (randomly chosen) past draw was from a noise box. There are four kinds of fact checks which we vary across subjects; they differ on whether (a) all balls can be fact checked, or only draws from the noise box, and (b) all colors can be fact checked, or only particular colors.
In case (2), a fact check informs the subject of whether or not the previous ball was swapped. Here, there are two kinds of fact checks which we vary across subjects, which differ depending on whether all colors can be fact checked, or only particular colors.
The first 32 rounds involve subjects reporting beliefs on the composition of the box. The next 12 rounds involve subjects reporting beliefs about whether a fixed previous draw was accurate. For these rounds, we default to asking subjects about the first observation they see; if, however, a fact-check provides information on this draw, we instead ask about the second observation.
Of note, relative to our previous design, this experiment introduces (2), and additionally allows us to study how beliefs update if information in fact suggested past observations were in fact more accurate than initially suggested, only less.