Elicitation of beliefs in light of decision environment changes: A laboratory experiment

Probability distribution participants sample from might change.

2023-11-01

2024-01-15

(i) The subjective probability of an observed outcome.
(ii) The subjective probability of an unobserved outcome (any other possible outcome).
(iii) The ratio between subjective probabilities of two outcomes.

(i) This will be elicited multiple times. There will be one such entry for each already observed outcome after each of the 40 draws for each of the 3 urns.
(ii) Participants will be asked to state a subjective probability for any other outcome not yet observed.
(iii) This ratio will be calculated for all the pairs of observed outcomes in each draw stage where this is possible (i.e. after observing two or more outcomes).

(i) Demographics (age, gender, field of study, risk attitude), number of correctly solved Raven matrices (out of 12), number of outcomes participants thought might be in the urn

(i) Answers will be used as control variables.

The experiment consists of three main parts. In each part, participants take 40 samples out of a virtual urn containing different colored marbles. Participants do not know the underlying distribution of the urn. After each sample draw, they are asked to state how many marbles of i) every color sampled so far and ii) any other possible color, not yet sampled, they think the urn contains. These beliefs are incentivized using the Karni method, a variant of the Becker-DeGroot-Marschak mechanism for probabilities. During the sample process, the underlying probability distribution might change. The main parts of the study are followed by an incentivized short version of Raven’s progressive matrices and a brief demographic questionnaire.

The experiment consists of three main parts. In each part, participants take 40 samples out of a virtual urn containing different colored marbles. Participants do not know the underlying distribution of the urn. After each sample draw, they are asked to state how many marbles of i) every color sampled so far and ii) any other possible color, not yet sampled, they think the urn contains. These beliefs are incentivized using the Karni method, a variant of the Becker-DeGroot-Marschak mechanism for probabilities. During the sample process, the underlying probability distribution might change. For the first part, this is possible after the 30th draw, for the second part after the 20th draw, and for the third part after the 10th draw. This is communicated to the participants. We use two treatment conditions: In the Announcement treatment, participants only learn that the distribution might change. In the Information treatment, they additionally learn that the new distribution will always have 5 equiprobable outcomes.

The main parts of the study are followed by an incentivized short version of Raven’s progressive matrices and a brief demographic
questionnaire.

Our specific hypotheses to be tested are included in the uploaded document.

Randomization of sample draws and colors per urn done by the experimental software, treatments are assigned per session.

Samples and colors are randomized on the individual level, treatment is the same for all participants of a session.

No

240 laboratory participants

240 laboratory participants

120 participants per treatment: 120 Announcement, 120 Information

Test if the subjective probability of the residual is different between the treatments (Information vs. Announcement treatment): As participants sample for some time, it is not entirely clear which standard deviation to use. In our last experiment, participants sampled for 30 rounds inf for different tasks. After half of the rounds (when participants have already some feeling for the probabilities), the standard deviations range from 17 to 21. Assume the worst case of a standard deviation of 21. After the change in environment becomes possible and participants suspect that this happened, participants in the Information treatment should converge to a residual probability of 0 (as they know the true probabilities are 0.2 for every outcome). At 120 participants we could still detect differences up to 8 probability points at alpha=0.05 and beta=0.8. Testing if the ratio after observing a yet unobserved event increases, that is participants update to put more weight on the subjectively more likely event: According to a power analysis, we can positively identify a ratio of 55 to 45 participants increasing their ratio at an alpha-level of below 0.05 at 240 participants (sign test). Testing in the Information treatment if the subjective probabilities deviate from the known value of 20, we can still identify effects up to around 5.5 percentage points difference, assuming a standard deviation of 20, alpha=0.05 and beta=0.8. Note that participants sample over multiple rounds, hence we cannot entirely pinpoint when participants might (i) suspect the environment changed and (ii) update their ratios. This will be taken into account in the analysis by looking at how beliefs developed and using multiple procedures to corroborate the results.