Persistent Stories or Persistent Numbers: Effects of Narrative versus Statistical Evidence on Policy Preferences

Last registered on April 01, 2022

Pre-Trial

Trial Information

General Information

Title
Persistent Stories or Persistent Numbers: Effects of Narrative versus Statistical Evidence on Policy Preferences
RCT ID
AEARCTR-0008821
Initial registration date
February 08, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
February 10, 2022, 7:37 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
April 01, 2022, 4:43 AM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
The University of Tokyo

Other Primary Investigator(s)

PI Affiliation
The University of Tokyo
PI Affiliation
The University of Tokyo

Additional Trial Information

Status
In development
Start date
2022-03-07
End date
2023-03-22
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
1. Motivation
Many research studies have shown that intervention effects are short-lived. However, we consider that the short life of a treatment effect originates to a substantial degree a uni-dimensional experimental setting, such as requiring greater efforts to save energy or towels, power, or water. In fact, many essential issues of our lives do not involve uni-dimensional decision-making. When a legislature approves a budget for the coming year, it does not necessarily increase or decrease the taxation rate. Instead, it discusses a possible budget vector while viewing the scholar of the vector, or the revenue, as given. Improvement in public policy does not necessarily mean a larger size of the government but instead could mean wiser spending given a constant revenue. Traditional uni-dimensional design implicitly assumes additional effort or tax payments and might discourage long-term improvement in behavior or spending.

To address this issue, we conceive a multidimensional public policy space in which hypothetical policy packages are randomly generated by a fully randomized conjoint design. The policy space contains a possible shrinking of government size as well as spending for public policies. If a respondent wants to keep the size of government constant or even shrink it, she or he still might want to relieve poverty by reallocating spending in other directions. By allowing for such choices, we can accurately identify treatment effects on policy preferences.

Additionally, possibly different impacts of different modes of information have been discussed. We use both statistical evidence and narratives to exchange findings and ideas on a daily decision-making basis. While we implicitly believe that statistical or narrative evidence might be more effective under specific circumstances, empirical results remain mixed. We estimate the possible difference in choices within our hypothetical public policy space.

2. Objective
We investigate (1) whether information provision affects public policy preferences in the long term by a three-wave panel conjoint experiment performed from March 2022 to September 2022 and to February 2023. If we find that information provision does affect policy preferences, we investigate (2) whether narrative or statistical evidence affects policy preferences stronger/longer. Rom among several critical issues, we adopt poverty relief as a focus and study the relative preference for poverty relief compared with other dimensions of public policy; public works, education, public health care, the public pension fund, and fiscal retrenchment.

3. Setting and design
We conduct a panel internet survey by recruiting a nonprobability sample of 15,000 Japanese adults on the first wave in March 2022 and send consecutive surveys to the same respondents in September 2022 and February 2023. In each wave, we implement a fully randomized conjoint experiment that shows 2 hypothetical allocations (indicated by the percentage of increased consumption tax) to 5 dimensions of public policy: 1) public works, 2) education, 3) public health care, 4) the public pension fund, and 5) poverty relief. Any residual is assumed to be spent for government debt redemption. Respondents choose between the 2 hypothetical policy packages in a task. We assign 5 tasks to each respondent at each wave.

4. Treatment
In the first wave of March 2022, we provide the randomly selected respondents with statistical information on the relative poverty rate in Japan or a narrative on the relative poverty in Japan. Statistical evidence is cited from the National Livelihood Survey 2019 conducted by the Ministry of Health, Labour and Welfare, the government of Japan (https://www.mhlw.go.jp/toukei/saikin/hw/k-tyosa/k-tyosa19/index.html).

5. Results
5.1 Persistency of intervention effects
We investigate how long the effects of information treatment on preferences for public policies persist.

5.2 Levels and lengths of different modes of information
We investigate whether either narrative evidence or statistical evidence for relative poverty in Japan affects policy preferences more strongly than the other and whether either of the effects persists longer than the other.
External Link(s)

Registration Citation

Citation
Kawata, Keisuke, Kenneth McElwain and Masaki Nakabayashi. 2022. "Persistent Stories or Persistent Numbers: Effects of Narrative versus Statistical Evidence on Policy Preferences." AEA RCT Registry. April 01. https://doi.org/10.1257/rct.8821
Experimental Details

Interventions

Intervention(s)
1. Information treatment
In the first wave planned in March 2022, we provide the randomly selected respondents with statistical information about the relative poverty rate in Japan or a narrative on the relative poverty in Japan. Statistical evidence is cited from the National Livelihood Survey 2019 conducted by the Ministry of Health, Labour and Welfare, the government of Japan (https://www.mhlw.go.jp/toukei/saikin/hw/k-tyosa/k-tyosa19/index.html).

2. Choice of hypothetical policy packages
We adopt a fully randomized conjoint design to generate 2 5-dimensional public policy package options. Since each level (i.e., budget allocation percentage) of an attribute (i.e., a policy) is selected in forming a policy package, we can identify a change in preference accompanying a change in attribute levels as a causal effect from the latter to the former.
Intervention Start Date
2022-03-07
Intervention End Date
2022-03-22

Primary Outcomes

Primary Outcomes (end points)
1. We investigate whether the provision of information regarding poverty affects public policy preference.
2. If we find an effect of information provision mode, we measure how long the effect persists.
3. If we find an effect of information provision, we also investigate whether narrative or statistical evidence has a greater or more persistent impact on policy preference than the other.
4. If either narrative or statistical evidence has a greater or more persistent impact on policy preference than the other, we investigate whether the difference is associated with respondents' choice of the mode of information (i.e., of narrative or statistical evidence).
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
1. Survey design
1. Panel experiment of information treatment effects
We conduct a panel Internet survey by recruiting a nonprobability sample of 15,000 Japanese adults on the first wave in March 2022 and send the consecutive surveys to the same respondent in September 2022 and February 2023. On each wave, we implement a fully randomized conjoint experiment that shows 2 hypothetical allocations indicated by the percentage of raised consumption tax to 5 dimensions of public policies; 1) public works; 2) education; 3) public healthcare; 4) public pension; 5) poverty relief. Any residual is assumed to be spent for government debt redemption. Respondents choose between the 2 hypothetical policy packages at a task. We assign 5 tasks to each respondent on each wave.

2. Information treatment
In the first wave planned in March 2022, we provide the randomly selected respondents with statistical information about the relative poverty rate in Japan or a narrative on the relative poverty in Japan. Statistical evidence is cited from the National Livelihood Survey 2019 conducted by the Ministry of Health, Labour and Welfare, the government of Japan (https://www.mhlw.go.jp/toukei/saikin/hw/k-tyosa/k-tyosa19/index.html).

3. Background characteristics
We survey demographic characteristics, educational backgrounds, income, household income, partisanship, self-conceived right-leaning, preference for the size of government, and self-conceived social status.
Experimental Design Details
Not available
Randomization Method
Randomization will be done in a survey company by a computer.
Randomization Unit
15,000 respondents.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
15,000 individuals.
Sample size: planned number of observations
15,000 individuals.
Sample size (or number of clusters) by treatment arms
5,000 individuals are the control arm and 10,000 individuals are the treatment arm.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
Ethical Review Board, Institute of Social Science, The University of Tokyo
IRB Approval Date
2022-02-07
IRB Approval Number
87