Experimental Design Details
Respondents are shown 40 images. Each image is either a word or a number, and appears in either blue or orange font. Our primary outcome of interest is respondents answer to the following question: "Suppose the computer randomly chose an image from the images you just saw. It is orange. What is the percent chance that it is a word?" We have eight total treatments, which differ only in the distribution of words, numbers, and colors. We will certainly run the first six treatments, but we may decide against running the final two if maximum possible sample sizes at the lab are smaller than we hope.
Treatment 1: 10 orange words, 10 orange numbers, 0 blue words, 20 blue numbers.
Treatment 2: 10 orange words, 10 orange numbers, 10 blue words, 10 blue numbers.
Treatment 3: 10 orange words, 10 orange numbers, 20 blue words, 0 blue numbers.
Treatment 4: 14 orange words, 6 orange numbers, 0 blue words, 20 blue numbers.
Treatment 5: 14 orange words, 6 orange numbers, 6 blue words, 14 blue numbers.
Treatment 6: 14 orange words, 6 orange numbers, 20 blue words, 0 blue numbers.
Treatment 7: 11 orange words, 9 orange numbers, 20 blue words, 0 blue numbers.
Treatment 8: 9 orange words, 11 orange numbers, 0 blue words, 20 blue numbers.
Our theory predicts both a bias towards 50-50 and that, when there are more blue words, subjects will believe that a randomly chosen orange image is less likely to be a word, and vice versa. We first ask the probability of being a world conditional on the randomly chosen image being orange, then conditional on it being blue, then supposing they do not know it's color. Our primary outcome is the first question, and the other two are secondary. After each of these questions, respondents are asked how certain they are of their answers.
After this, respondents answer the memory questions described above.