Impediments to yardstick competition in federal systems: Experimental survey evidence from German citizens

We investigate whether providing citizens with information about their state’s relative educational performance affects satisfaction with education policy, and policy support for increased comparative educational testing. Survey respondents are randomized into five experimental groups, and complete five consecutive stages. In stage 1, they state the importance of cross-state comparability of educational performance. In stage 2, they estimate the rank of their state in the latest comparative math-achievement test. In stage 3 and 4, respondents state their satisfaction with their state’s education policy, and support for the policy proposal to enhance comparative educational testing. In group 1, respondents answer these two questions without any further information. In groups 2 and 3, they receive information about their state’s relative educational performance. In groups 4 and 5, respondents have the option to retrieve this educational-performance information through clicking on a link. In stage 5, all respondents estimate the share of state parliamentarians who support the policy proposal to enhance comparative educational testing.
We implemented a second, independently randomized, experiment when eliciting policy support in stage 4. Before answering this question, half of respondents are informed that the average answer will be forwarded to their state politicians. This allows us to assess whether highlighting the consequentiality of the survey answers affects answering behaviour.

Subjects will conduct the experiment via an online platform.
Subjects will be randomly assigned to one of five experimental groups, and each subject takes decisions in five consecutive stages.

TREATMENT 1:

Stage 1: Perceived importance of cross-state comparability.

[Question wording:]
“How important do you think it is that student performance is comparable between the federal states?”

[Answer categories:]
5-point scale: 1=”very important”, 2=” rather important”, 3=”neither important nor unimportant”, 4=”rather unimportant”, 5=”very unimportant”


Stage 2: Beliefs about state performance.

[Question wording:]
“A recent educational study has compared the mathematics performance of 9th grade students in the 16 German states. What do you estimate, which place did the pupils in your federal state occupy? (1 is the best place, 16 the worst place)”

[Answer categories:]
Open field + “place”


Stage 3: Satisfaction with state’s education policy.

[Question wording:]
“How satisfied are you with education policy in your federal state?”

[Answer categories:]
5-point scale: 1=”very satisfied”, 2=” rather satisfied”, 3=”neither satisfied nor unsatisfied”, 4=”rather unsatisfied”, 5=”very unsatisfied”


Stage 4: Policy support for increased comparative educational testing.

[Question wording:]
“Do you favor or oppose the following reform proposal: Nationwide comparative tests of mathematics achievement will be introduced in all school types. The tests will take place regularly every two years starting in the 5th grade. The average results by federal state will be published to compare student performance across federal states.”

[Answer categories:]
5-point scale: 1=”strongly favor”, 2=”somewhat favor”, 3=”neither favor nor oppose”, 4=”somewhat oppose”, 5=”strongly oppose”


Stage 5: Beliefs about parliamentarian support for enhanced-comparability policy.

[Question wording:]
“What do you estimate, what percentage of the politicians in the state parliament of your federal state is "strongly" or "somewhat" in favor of the reform proposal mentioned in the previous question to introduce nationwide standardized comparative tests for pupils?”

[Answer categories:]
Open field + “percent”


TREATMENT 2:

Stage 1: See TREATMENT 1

Stage 2: See TREATMENT 1.

Stage 3: Satisfaction with state’s education policy + information about own performance.

[Question wording:]
“The students in your federal state ranked [show rank] out of 16 in the educational study mentioned in the previous question.
How satisfied are you with education policy in your federal state?”

[Answer categories:]
5-point scale: 1=”very satisfied”, 2=” rather satisfied”, 3=”neither satisfied nor unsatisfied”, 4=”rather unsatisfied”, 5=”very unsatisfied”

Stage 4: See TREATMENT 1.

Stage 5: See TREATMENT 1.


TREATMENT 3:

Stage 1: See TREATMENT 1

Stage 2: See TREATMENT 1.

Stage 3: Satisfaction with state’s education policy + information about performance rank.

[Question wording:]
“In the educational study mentioned in the previous question, students in the different federal states took the following places: [Show ranking]
How satisfied are you with education policy in your federal state?”

[Answer categories:]
5-point scale: 1=”very satisfied”, 2=” rather satisfied”, 3=”neither satisfied nor unsatisfied”, 4=”rather unsatisfied”, 5=”very unsatisfied”

Stage 4: See TREATMENT 1.

Stage 5: See TREATMENT 1.

TREATMENT 4:

Stage 1: See TREATMENT 1

Stage 2: See TREATMENT 1.

Stage 3: Satisfaction with state’s education policy + information acquisition about own performance.

[Question wording:]
“Click here if you want to find out what place the students in your federal state occupied in the educational study mentioned in the previous question.
[Click: show information from TREATMENT 2]
How satisfied are you with education policy in your federal state?”

[Answer categories:]
5-point scale: 1=”very satisfied”, 2=” rather satisfied”, 3=”neither satisfied nor unsatisfied”, 4=”rather unsatisfied”, 5=”very unsatisfied”

Stage 4: See TREATMENT 1.

Stage 5: See TREATMENT 1.

TREATMENT 5:

Stage 1: See TREATMENT 1

Stage 2: See TREATMENT 1.

Stage 3: Satisfaction with state’s education policy + information acquisition about performance rank.

[Question wording:]
“Click here if you want to find out which places students in the different federal states occupied in the educational study mentioned in the previous question.
[Click: show information from TREATMENT 3]
How satisfied are you with the education policy in your federal state?”

[Answer categories:]
5-point scale: 1=”very satisfied”, 2=” rather satisfied”, 3=”neither satisfied nor unsatisfied”, 4=”rather unsatisfied”, 5=”very unsatisfied”

Stage 4: See TREATMENT 1.

Stage 5: See TREATMENT 1.

2020-06-03

2020-06-17

Citizens’ satisfaction with their state’s education policy, and support for the policy proposal to enhance comparative educational testing.

Satisfaction and policy support in TREATMENT 1 (elicited in stage 3 and 4) will serve as a benchmark. Comparing these answers to TREATMENT 2 and 3 will allow us to assess the causal effect of providing relative educational-performance information. Comparing answers to TREATMENT 4 and 5 reveals the causal effect of offering the option to acquire this information.

Heterogeneity by baseline importance of comparability, prior beliefs about the state’s performance, actual relative performance, and political ideology.

We conduct the experiment in a sample of 10,000 adults aged between 18 and 69 years. The survey is conducted in cooperation with the survey institute responi. The recruitment and polling is managed by respondi, who collect the data via an online platform. That is, our participants answer the survey questions autonomously on their own digital devices. Randomization is carried out by respondi at the individual level, using a computer.

Our experiment is structured as follows:
Respondents will be randomly assigned (between subject) to treatment 1 (p=1/3), treatment 2 (p=1/6), treatment 3 (p=1/6), treatment 4 (p=1/6), or treatment 5 (p=1/6). In our main analysis, we plan to pool treatments 2 and 3, and treatments 4 and 5, respectively.


Treatment 1:
Stage 1: Perceived importance of cross-state comparability.
Stage 2: Beliefs about state performance.
Stage 3: Satisfaction with state’s education policy.
Stage 4: Policy support for increased comparative educational testing.
Stage 5: Beliefs about parliamentarian support for enhanced-comparability policy.

Treatment 2:
Stage 1: Perceived importance of cross-state comparability.
Stage 2: Beliefs about state performance.
Stage 3: Satisfaction with state’s education policy + information about own performance.
Stage 4: Policy support for increased comparative educational testing.
Stage 5: Beliefs about parliamentarian support for enhanced-comparability policy.

Treatment 3:
Stage 1: Perceived importance of cross-state comparability.
Stage 2: Beliefs about state performance.
Stage 3: Satisfaction with state’s education policy + information about performance rank.
Stage 4: Policy support for increased comparative educational testing.
Stage 5: Beliefs about parliamentarian support for enhanced-comparability policy.

Treatment 4:
Stage 1: Perceived importance of cross-state comparability.
Stage 2: Beliefs about state performance.
Stage 3: Satisfaction with state’s education policy + information acquisition about own performance.
Stage 4: Policy support for increased comparative educational testing.
Stage 5: Beliefs about parliamentarian support for enhanced-comparability policy.

Treatment 5:
Stage 1: Perceived importance of cross-state comparability.
Stage 2: Beliefs about state performance.
Stage 3: Satisfaction with state’s education policy + information acquisition about performance rank.
Stage 4: Policy support for increased comparative educational testing.
Stage 5: Beliefs about parliamentarian support for enhanced-comparability policy.

Randomization is carried out by the survey company respondi, using a computer.

at the individual level

No

10,000 individuals (adults aged between 18 and 69 years)

10,000 individuals (adults aged between 18 and 69 years)

Treatment 1: 3,333; treatments 2 to 5: 1,666 each.