Predicting experimental results: The role of incentives, anchoring, and experience

We let subjects predict the outcome of an online survey experiment and investigate the causal effects of (i) providing monetary incentives for correct predictions, (ii) providing an anchor of control-group behavior in the experiment, and (iii) participating in the experiment on the accuracy of subjects’ predictions. Therefore, we implement an online-survey experiment among a representative sample of adults aged 18 to 69 years in Germany, and additionally survey a sample of experts (economics professors) from a regularly conducted German expert sample.
Respondents in the representative sample are randomized into six different experimental groups. The first three groups participate in an “information experiment” on preferences for increased school spending. Group 1 receives no information when stating spending preferences; group 2 is informed about average current school spending per student, and respondents in group 3 have the option to acquire the spending-information provided to group 2. Groups 4, 5, and 6 participate in and “prediction experiment” and predict the results of the “information experiment”. Group 4 receives no anchor and no incentive when stating the prediction. Group 5 is offered a monetary incentive for a correct prediction, and group 6 is provided with an anchor. In addition, we study the causal effect of experience on prediction accuracy by letting groups 1, 2, and 3 predict the results in the “information experiment” after participating in that experiment. The experts complete the same prediction task as group 4. Comparing prediction accuracy between experts and non-experts, we investigate whether providing incentives, anchors, and experience to non-experts improves prediction accuracy of the general population towards the levels of experts.

Subjects in the representative sample will conduct the experiment via an online platform. All wordings presented below are translations of the original German question wording. Subjects will be randomly assigned to one of six experimental groups, and each subject takes decisions in several consecutive stages.

TREATMENT 1:

Stage 1: prior beliefs on average school spending

[Question wording:]
“What do you guess, how much is spent on average each year per student on public general schools in Germany?”

[Answer categories:]
Open field + “Euro”

Stage 2: preferences for school-spending increases

[Question wording:]
“In your opinion, should public spending for schools in Germany increase, decrease, or stay the same?”

[Answer categories:]
“strongly increase/increase/decrease/strongly decrease/stay about the same”

Stage 3: predicting experimental results

[Question wording:]
“Now you should predict the answers of other respondents in this survey as accurately as possible.

In the previous question we randomly divided participants into three groups and asked them about government spending on schools.

Group 1 answered the following question:
“In your opinion, should public spending for schools in Germany increase, decrease, or stay the same?”

Group 2 answered the following question:
“Public education spending in Germany amounts on average to 7,300 Euro per student annually.
In your opinion, should public spending for schools in Germany increase, decrease, or stay the same?”

Group 3 answered the following question:
“In your opinion, should public spending for schools in Germany increase, decrease, or stay the same?”
Before answering the question, group 3 also had the opportunity to find out about current public education spending per student through a click. The click reveals the following information: “Public education spending in Germany amounts on average to 7,300 Euro per student annually.”

What percentage of respondents do you think answers "strongly increase" or "increase”?
Remember that this is a survey of people between the ages of 18 and 69.
[…]
And what percentage of respondents in group 3 do you estimate clicks to access the information?”

[Answer categories:]
Open field + “percentage in group 1 answer "strongly increase" or "increase””
Open field + “percentage in group 2 answer "strongly increase" or "increase””
Open field + “percentage in group 3 answer "strongly increase" or "increase””
Open field + “percent retrieve the information”

Stage 4: open question on prediction formation

[Question wording:]
“Next, we would like to know how you came up with your predictions in the previous question.
Please briefly describe as accurate as possible how you arrived at your answer.”

[Answer category:]
Open field

Stage 5: closed-ended question on prediction formation

[Question wording:]
“Now we would again like to know how you came up with your predictions in the previous question.
To what extent do you agree with the following statements?”

[Items:]
“I have tried hard to predict as accurate as possible.”
“I have searched the Internet for information in order to predict as accurate as possible.”
“I know of scientific studies on support for education spending and have based my answers on them.”

[Answer categories:]
“completely agree/rather agree/rather disagree/completely disagree/neither agree nor disagree”


TREATMENT 2:

Stage 1: see TREATMENT 1

Stage 2: preferences for school spending increases

[Question wording:]
“Public education spending in Germany amounts on average to 7,300 Euro per student annually.
In your opinion, should public spending for schools in Germany increase, decrease, or stay the same?”

[Answer categories:]
“strongly increase/increase/decrease/strongly decrease/stay about the same”

Stage 3, 4, and 5: see TREATMENT 1

TREATMENT 3

Stage 1: see TREATMENT 1

Stage 2: preferences for school spending increases

[Question wording:]
“In your opinion, should public spending for schools in Germany increase, decrease, or stay the same?
Klick here to find out how much the government currently spends per student.” [The click reveals the following information: “Public education spending in Germany amounts on average to 7,300 Euro per student annually.”]

Stage 3, 4, and 5: see TREATMENT 1

TREATMENT 4:

Stage 1: see TREATMENT 1

Stage 2: predicting experimental results

[Question wording:]
“Now you should predict the answers of other respondents in this survey as accurately as possible.

Some other respondents are randomly divided into three groups and asked about government spending on schools.

Group 1 answered the following question:
“In your opinion, should public spending for schools in Germany increase, decrease, or stay the same?”

Group 2 answered the following question:
“Public education spending in Germany amounts on average to 7,300 Euro per student annually.
In your opinion, should public spending for schools in Germany increase, decrease, or stay the same?”

Group 3 answered the following question:
“In your opinion, should public spending for schools in Germany increase, decrease, or stay the same?”
Before answering the question, group 3 also had the opportunity to find out about current public education spending per student through a click. The click reveals the following information: “Public education spending in Germany amounts on average to 7,300 Euro per student annually.”

What percentage of respondents do you think answers "strongly increase" or "increase”?
Remember that this is a survey of people between the ages of 18 and 69.
[…]
And what percentage of respondents in group 3 do you estimate clicks to access the information?”

[Answer categories:]
Open field + “percentage in group 1 answer "strongly increase" or "increase””
Open field + “percentage in group 2 answer "strongly increase" or "increase””
Open field + “percentage in group 3 answer "strongly increase" or "increase””
Open field + “percent retrieve the information”

Stage 3 and 4: See stage 4 and 5 of TREATMENT 1

TREATMENT 5:

Stage 1: see TREATMENT 1

Stage 2: predicting experimental results with incentives

[Question wording:]
Identical to stage 2 of TREATMENT 4, except for the following note:
“If your predictions are correct, you will be paid an additional reward of 1 Euro. For further information on the payment and the definition of correct predictions please click here.” [The click reveals the following information: “After you have submitted your predictions, one prediction will be randomly selected. If your prediction for the selected question is approximately correct, you will receive the additional reward.”]

Stage 3 and 4: See stage 4 and 5 of TREATMENT 1
[Question wording:]
Identical to stage 4 and 5 of TREATMENT 1, except for the following note:
“Note: Your answers to this question do not affect whether you will receive the additional reward for your predictions in the previous question.”


TREATMENT 6:

Stage 1: see TREATMENT 1

Stage 2: predicting experimental results with anchor

[Question wording:]
Identical to stage 2 of TREATMENT 4, except for the following note:
“Information: In a similar survey in 2019, 78 percent of 18 to 69 year-olds answered the question for group 1 with “strongly increase” or “increase”.”

Stage 3 and 4: See stage 4 and 5 of TREATMENT 1

2020-06-03

2020-06-17

Our primary outcomes of interest are respondents’ predictions of average answers in the three groups of the “information experiment”.

The predictions of respondents in TREATMENT 4 (elicited in stage 2) will serve as the benchmark. Comparing these predictions to the predictions in TREATMENT 5 (stage 2), TREATMENT 6 (stage 2), and TREATMENTS 1 to 3 (stage 3) will allow us to assess the causal effects of providing monetary incentives, benchmarks, and experimental experience, respectively, on prediction accuracy. In addition, we will let experts (economics professors) predict experimental results using the questions of TREATMENT 4 (stage 2) as an additional benchmark.

Treatment-effect heterogeneities in the “prediction experiment” by prior beliefs (elicited in stage 1), and respondents’ highest educational degree.

We will investigate treatment-effect heterogeneities in the “prediction experiment” by prior beliefs elicited in stage 1, which will allow us to assess whether respondents’ prediction quality is affected by own knowledge of education spending, i.e. the topic covered in “information experiment”. In addition, we will investigate heterogeneous treatment effects by respondents’ highest educational degree.

We conduct the experiment in a sample of 6,000 adults aged between 18 and 69 years. The survey is conducted in cooperation with respondi. The recruitment and polling is managed by respondi, who collect the data via an online platform. That is, our participants answer the survey questions autonomously on their own digital devices. Randomization is carried out by respondi at the individual level, using a computer. In addition, the expert sample from the “ifo Ökonomenpanel” will cover around 200 economics professors in Germany.

Our experiment in the representative sample is structured as follows:
Treatment 1:
Stage 1: prior beliefs on average school spending
Stage 2: preferences for school-spending increases (no information)
Stage 3: predicting experimental results
Stage 4: open-ended question on prediction formation
Stage 5: closed-ended question on prediction formation

Treatment 2:
Stage 1: prior beliefs on average school spending
Stage 2: preferences for school-spending increases (information provision)
Stage 3: predicting experimental results
Stage 4: open-ended question on prediction formation
Stage 5: closed-ended question on prediction formation

Treatment 3:
Stage 1: prior beliefs on average school spending
Stage 2: preferences for school-spending increases (voluntary information acquisition)
Stage 3: predicting experimental results
Stage 4: open-ended question on prediction formation
Stage 5: closed-ended question on prediction formation

Treatment 4:
Stage 1: prior beliefs on average school spending
Stage 2: predicting experimental results
Stage 3: open-ended question on prediction formation
Stage 4: closed-ended question on prediction formation

Treatment 5:
Stage 1: prior beliefs on average school spending
Stage 2: predicting experimental results (with incentives)
Stage 3: open-ended question on prediction formation
Stage 4: closed-ended question on prediction formation

Treatment 6:
Stage 1: prior beliefs on average school spending
Stage 2: predicting experimental results (with anchor)
Stage 3: open-ended question on prediction formation
Stage 4: closed-ended question on prediction formation

Our design in the expert sample is structured as follows:
Stage 1: prior beliefs on average school spending
Stage 2: predicting experimental results (from the representative sample)
Stage 3: open-ended question on prediction formation
Stage 4: closed-ended question on prediction formation

Randomization is carried out by the survey company respondi, using a computer.

at the individual level

No

6,000 adults aged 18 – 69 years + 200 experts (economics professors)

6,000 adults aged 18 – 69 years + 200 experts (economics professors)

6,000 adults aged 18 – 69 years, approx. 1,000 will be assigned to each of the treatment groups. 200 experts (economics professors)