Predicting experimental results: The role of incentives, anchoring, and experience

We let subjects predict the outcome of an online survey experiment and investigate the causal effects of (i) providing monetary incentives for correct predictions, (ii) providing an anchor of control-group behavior in the experiment, and (iii) participating in the experiment on the accuracy of subjects’ predictions. Therefore, we implement an online-survey experiment among a representative sample of adults aged 18 to 69 years in Germany, and additionally survey a sample of experts (economics professors) from a regularly conducted German expert sample.
Respondents in the representative sample are randomized into six different experimental groups. The first three groups participate in an “information experiment” on preferences for increased school spending. Group 1 receives no information when stating spending preferences; group 2 is informed about average current school spending per student, and respondents in group 3 have the option to acquire the spending-information provided to group 2. Groups 4, 5, and 6 participate in and “prediction experiment” and predict the results of the “information experiment”. Group 4 receives no anchor and no incentive when stating the prediction. Group 5 is offered a monetary incentive for a correct prediction, and group 6 is provided with an anchor. In addition, we study the causal effect of experience on prediction accuracy by letting groups 1, 2, and 3 predict the results in the “information experiment” after participating in that experiment. The experts complete the same prediction task as group 4. Comparing prediction accuracy between experts and non-experts, we investigate whether providing incentives, anchors, and experience to non-experts improves prediction accuracy of the general population towards the levels of experts.

2020-06-03

2020-06-17

Our primary outcomes of interest are respondents’ predictions of average answers in the three groups of the “information experiment”.

The predictions of respondents in TREATMENT 4 (elicited in stage 2) will serve as the benchmark. Comparing these predictions to the predictions in TREATMENT 5 (stage 2), TREATMENT 6 (stage 2), and TREATMENTS 1 to 3 (stage 3) will allow us to assess the causal effects of providing monetary incentives, benchmarks, and experimental experience, respectively, on prediction accuracy. In addition, we will let experts (economics professors) predict experimental results using the questions of TREATMENT 4 (stage 2) as an additional benchmark.

Treatment-effect heterogeneities in the “prediction experiment” by prior beliefs (elicited in stage 1), and respondents’ highest educational degree.

We will investigate treatment-effect heterogeneities in the “prediction experiment” by prior beliefs elicited in stage 1, which will allow us to assess whether respondents’ prediction quality is affected by own knowledge of education spending, i.e. the topic covered in “information experiment”. In addition, we will investigate heterogeneous treatment effects by respondents’ highest educational degree.

We conduct the experiment in a sample of 6,000 adults aged between 18 and 69 years. The survey is conducted in cooperation with respondi. The recruitment and polling is managed by respondi, who collect the data via an online platform. That is, our participants answer the survey questions autonomously on their own digital devices. Randomization is carried out by respondi at the individual level, using a computer. In addition, the expert sample from the “ifo Ökonomenpanel” will cover around 200 economics professors in Germany.

Our experiment in the representative sample is structured as follows:
Treatment 1:
Stage 1: prior beliefs on average school spending
Stage 2: preferences for school-spending increases (no information)
Stage 3: predicting experimental results
Stage 4: open-ended question on prediction formation
Stage 5: closed-ended question on prediction formation

Treatment 2:
Stage 1: prior beliefs on average school spending
Stage 2: preferences for school-spending increases (information provision)
Stage 3: predicting experimental results
Stage 4: open-ended question on prediction formation
Stage 5: closed-ended question on prediction formation

Treatment 3:
Stage 1: prior beliefs on average school spending
Stage 2: preferences for school-spending increases (voluntary information acquisition)
Stage 3: predicting experimental results
Stage 4: open-ended question on prediction formation
Stage 5: closed-ended question on prediction formation

Treatment 4:
Stage 1: prior beliefs on average school spending
Stage 2: predicting experimental results
Stage 3: open-ended question on prediction formation
Stage 4: closed-ended question on prediction formation

Treatment 5:
Stage 1: prior beliefs on average school spending
Stage 2: predicting experimental results (with incentives)
Stage 3: open-ended question on prediction formation
Stage 4: closed-ended question on prediction formation

Treatment 6:
Stage 1: prior beliefs on average school spending
Stage 2: predicting experimental results (with anchor)
Stage 3: open-ended question on prediction formation
Stage 4: closed-ended question on prediction formation

Our design in the expert sample is structured as follows:
Stage 1: prior beliefs on average school spending
Stage 2: predicting experimental results (from the representative sample)
Stage 3: open-ended question on prediction formation
Stage 4: closed-ended question on prediction formation

Randomization is carried out by the survey company respondi, using a computer.

at the individual level

No

6,000 adults aged 18 – 69 years + 200 experts (economics professors)

6,000 adults aged 18 – 69 years + 200 experts (economics professors)

6,000 adults aged 18 – 69 years, approx. 1,000 will be assigned to each of the treatment groups. 200 experts (economics professors)