|
Field
Abstract
|
Before
How do people update their beliefs on contentious political issues when they receive new information? Do they do so in a rational manner or do they refuse to update their beliefs? To investigate this, we perform a survey experiment on Amazon Mechanical Turk. We study the extent to which people claim their beliefs on a highly salient political issue would change depending on pertinent empirical facts (provided as hypotheticals). We then compare these responses to how beliefs actually change when people learn new empirical facts, which yields insight about the prevalence of motivated reasoning. We apply this approach to a range of issues -- police shootings of minorities, climate change, affirmative action, income taxation of the top 1%, economic mobility, zoning laws, and the Olympics -- which vary in their political polarization. Within the same framework, we explore whether it is possible to de-bias individuals from engaging in motivated reasoning by asking (and reminding) individuals about how their beliefs would change as a function of varying hypotheticals. In an alternative experiment, we provide individuals with informative but noisy signals, which accordingly may either conform with or contradict their ideological preconceptions.
|
After
How do people update their beliefs on contentious political issues when they receive new information? Do they do so in a rational manner or do they refuse to update their beliefs? To investigate this, we perform a survey experiment through survey panel company Prolific. We study the extent to which people claim their beliefs on highly salient political issues would change depending on pertinent empirical information (provided as hypotheticals). We then compare these responses to how beliefs actually change when people are given empirical information, which yields insight about the prevalence of motivated reasoning. We apply this approach to a range of issues -- police shootings of minorities, climate change, affirmative action, income taxation of the top 1%, economic mobility, transgender participation in sports, crime in Republican- and Democratic-run cities, gun control, and the Olympics. Within the same framework, we explore whether it is possible to de-bias individuals from engaging in motivated reasoning by asking individuals about how their beliefs would change as a function of varying hypotheticals.
|
|
Field
Trial Start Date
|
Before
August 05, 2021
|
After
November 09, 2023
|
|
Field
Trial End Date
|
Before
August 25, 2021
|
After
November 30, 2023
|
|
Field
Last Published
|
Before
August 05, 2021 03:47 PM
|
After
November 09, 2023 12:59 AM
|
|
Field
Intervention Start Date
|
Before
August 05, 2021
|
After
November 09, 2023
|
|
Field
Intervention End Date
|
Before
August 25, 2021
|
After
November 30, 2023
|
|
Field
Primary Outcomes (End Points)
|
Before
We are interested in the extent to which individuals' reported normative beliefs change in response to new information. In particular, we will compare:
(a) People's ex ante beliefs about political issues (using answers of people in the control and hypothetical groups)
(b) People's claimed responsiveness to information (using answers of people in the control and hypothetical groups)
(c) People's ex post beliefs about political issues (using answers of people in all three groups -- in particular, the wedge between the treatment and hypothetical groups)
Furthermore, we are interested in the gap between the extent to which individuals claim, hypothetically, their beliefs would change if a given piece of information was true versus the extent to which they actually change when individuals are told the information is, in fact, true.
And we are interested in the extent to which our approach of eliciting responses to hypotheticals before providing the true information can be used as a tool for de-biasing.
|
After
We study the extent to which individuals' reported normative beliefs change in response to new information. In particular, we will examine:
(a) People's ex ante beliefs about political issues (using answers of people in the control group)
(b) People's claimed responsiveness to information (using answers of people in the hypothetical group)
(c) People's ex post beliefs about political issues (using answers of people in the treatment group)
(d) People’s ex post beliefs about political issues when potentially constrained by previously-stated hypothetical beliefs (using answers of people in the hypothetical group when they are subsequently given the treatment information)
With these components, we are interested in the gap between the extent to which individuals claim, hypothetically, their beliefs would change if a given piece of information was true versus the extent to which they actually change when individuals are told the information is, in fact, true. This is evidence of motivated reasoning. Numerically, this is given by the wedge between (c) and (b), relative to the wedge between (c) and (a). (This can be measured for any given demographic or pre-treatment subgroup.)
Second, we are interested in the extent to which our approach of eliciting responses to hypotheticals before providing the true information can be used as a tool for de-biasing. Numerically, this is given by the wedge between (d) and (c) in aggregate and at the individual level.
Third, we are interested in the magnitude of the responsiveness of hypothetical beliefs to hypothetical information, relative to the cross-sectional relationship between normative beliefs and empirical beliefs in the control group. To make this more concrete, how much of the partisan difference in normative beliefs is claimed as being due to differences in information.
|
|
Field
Primary Outcomes (Explanation)
|
Before
Depending on the exact normative question, we expect either a negative or positive relationship between beliefs and the hypothetical information. To enable standardization across categories (such as when pooling the data), we will rotate the responses accordingly. In particular, the values for the answers to the taxes, mobility, and zoning questions will be reversed.
|
After
|
|
Field
Experimental Design (Public)
|
Before
Our survey experiment will be posted on Amazon Mechanical Turk, advertised with a base compensation of $2.50 and an incentive bonus of up to $1 for participants. Individuals who opt to take our survey will answer a series of questions about their beliefs relating to several different political issues. The issues will include police shootings of minorities, climate change, affirmative action, income taxation of the top 1%, economic mobility, zoning laws, and the Olympics.
Once an individual opts to take the survey, they will begin by answering a series of demographic questions. After this, for each of five of the above seven issue areas (randomly selected), the respondent will be randomly sorted into either (i) the control group, (ii) the information treatment group, or (iii) the hypothetical treatment group. Independently, they will be sorted into an "A" or "B" group, which receive slightly different orders of normative beliefs questions. In other words, an individual may be sorted into the control group ver. A for the climate change issue, the hypothetical group ver. B for the affirmative action issue, and the treatment group ver. A for the income taxation issue.
Regardless of which group the individual is sorted into, they will be asked for their belief on an empirical fact relating to a political issue. After this,
In the control group, respondents will be asked for their normative beliefs on the issue (both the A and B questions).
In the information treatment group, respondents will be presented with the true answer to the above empirical question corresponding to the issue. Then, they will be asked for their normative beliefs on the issue (both the A and B questions).
In the hypothetical group, respondents will be asked their normative beliefs on the issue (both the A and B questions). They will then be presented, randomly, with three hypotheticals asking what their normative beliefs would be IF the true answer to the empirical question was X, for three randomly-selected values of X (one of which is the true value). They will be asked the A version of the normative beliefs question if they were sorted into group A and the B version if they were sorted into group B.
This process is then repeated four more times for four more issues.
Comparing the extent to which individuals claim they will hypothetically update their beliefs (for the true value of X) to the extent to which they actually do update their beliefs when presented with the true information will yield information on the extent to which motivated reasoning is occurring. In addition, we will test how much people claim they will respond to new information.
In particular, we will compare:
(a) People's ex ante beliefs about political issues (using answers of people in the control and hypothetical groups)
(b) People's claimed responsiveness to information (using answers of people in the control and hypothetical groups)
(c) People's ex post beliefs about political issues (using answers of people in all three groups -- in particular, the wedge between the treatment and hypothetical groups)
At the end of the survey, for each issue on which individuals were sorted into a hypothetical group, they will receive a follow-up. They will be given the information treatment and then asked their normative beliefs on the issue (both A and B). Comparing the extent to which they then update their beliefs on A and on B will yield information on the efficacy of using the hypotheticals approach as a tool for de-biasing.
In particular, we will test:
(a) Whether the apparent degree of motivated reasoning is reduced by this process of first posing hypothetical questions
(b) Whether the apparent degree of motivated reasoning differs between the specific normative question on which the respondent previously answered the hypothetical (e.g., A) versus the normative question which they were not posed hypotheticals about (e.g., B). To the extent that de-biasing is more successful for A vis-a-vis B (in this example) this suggests a demand effect rather than a true de-biasing
In a follow-up survey one week later, we will again ask people these normative beliefs questions (both the A and B versions) in order to examine the extent to which these effects persist. The mechanical demand effects whereby people in the hypothetical group feel constrained to be consistent with their hypothetical answers (e.g., for A but not constrained for B) should be substantially alleviated one week later, and this will allow us to determine whether any de-biasing is, in fact, real.
|
After
Our survey experiment will be posted on Prolific and distributed to their nationally-representative panel of respondents. Individuals who opt to take our survey will answer a series of questions about their beliefs relating to several different political issues. The issues will include police shootings of minorities, climate change, affirmative action, income taxation of the top 1%, economic mobility, transgender participation in sports, crime in Republican- and Democratic-run cities, gun control, and the Olympics.
Once an individual opts to take the survey, they will begin by answering a series of demographic questions. After this, for each of five of the above nine issue areas (randomly selected), the respondent will be randomly sorted into either (i) the control group, (ii) the information treatment group, or (iii) the hypothetical treatment group. Independently, they will be sorted into an "A" or "B" group, which receive slightly different orders of normative beliefs questions. In other words, an individual may be sorted into the control group ver. A for the climate change issue, the hypothetical group ver. B for the affirmative action issue, and the treatment group ver. A for the income taxation issue.
Regardless of which group the individual is sorted into, they will be asked for their belief on an empirical fact relating to a political issue. After this,
In the control group, respondents will be asked for their normative beliefs on the issue (both the A and B questions).
In the information treatment group, respondents will be presented with the true answer to the above empirical question corresponding to the issue. Then, they will be asked for their normative beliefs on the issue (both the A and B questions).
In the hypothetical group, respondents will be presented, randomly, with three hypotheticals asking what their normative beliefs would be IF the true answer to the empirical question was X, for three randomly-selected values of X (one of which is the true value). They will be asked the A version of the normative beliefs question if they were sorted into group A and the B version if they were sorted into group B.
This process is then repeated four more times for four more issues.
Comparing the extent to which individuals claim they will hypothetically update their beliefs (for the true value of X) to the extent to which they actually do update their beliefs when presented with the true information will yield information on the extent to which motivated reasoning is occurring. In addition, we will test how much people claim they will respond to new information.
Concretely, we will examine
(a) People's ex ante beliefs about political issues (using answers of people in the control group)
(b) People's claimed responsiveness to information (using answers of people in the hypothetical group)
(c) People's ex post beliefs about political issues (using answers of people in the treatment group)
(d) People’s ex post beliefs about political issues when potentially constrained by previously-stated hypothetical beliefs (using answers of people in the hypothetical group when they are subsequently given the treatment information)
At the end of the survey, for each issue on which individuals were sorted into a hypothetical group, they will receive a follow-up. They will be given the information treatment and then asked their normative beliefs on the issue (both A and B). Comparing the extent to which they then update their beliefs on A and on B will yield information on the efficacy of using the hypotheticals approach as a tool for de-biasing.
In particular, we will test:
(a) Whether the apparent degree of motivated reasoning is reduced by this process of first posing hypothetical questions
(b) Whether the apparent degree of motivated reasoning differs between the specific normative question on which the respondent previously answered the hypothetical (e.g., A) versus the normative question which they were not posed hypotheticals about (e.g., B). To the extent that de-biasing is more successful for A vis-a-vis B (in this example) this suggests a demand effect rather than a true de-biasing. To the extent that this attempted de-biasing is at least partially unsuccessful, this is indicative of the strength of motivated reasoning on political views; not only are people able to discount unfavorable information, they would have done so even in the face of information they had recently claimed would change their mind.
In a follow-up survey one week later, we will again ask people these normative beliefs questions (both the A and B versions) in order to examine the extent to which these effects persist. The mechanical demand effects whereby people in the hypothetical group feel constrained to be consistent with their hypothetical answers (e.g., for A but not constrained for B) should be substantially alleviated one week later, and this will allow us to determine whether the information treatment is persistent and any de-biasing is, in fact, real.
|
|
Field
Randomization Method
|
Before
Randomization performed by the survey software (Qualtrics).
|
After
Randomization performed by the survey software (Qualtrics).
For each issue, the arm (control, treatment, hypothetical) that a given individual is sorted into is randomized independently.
For each issue, the order in which Question A and Question B are asked is randomized independently.
For each issue where the individual was in the hypothetical arm, the order in which the follow-up information treatment questions are asked is randomized independently.
|
|
Field
Planned Number of Clusters
|
Before
700 individuals
|
After
2000 individuals
|
|
Field
Planned Number of Observations
|
Before
700 individuals
|
After
2000 individuals
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
700 individuals each answer questions on 5 of 7 issues (randomly-selected). Consequently, each issue will be presented to approximately 500 individuals, of which 1/3 will be sorted into the control group, 1/3 will be sorted into the information treatment group, and 1/3 will be sorted into the hypothetical group.
|
After
2000 individuals each answer questions on 5 of 9 issues (randomly-selected). Consequently, each issue will be presented to approximately 1100 individuals, of which 1/4 will be sorted into the control group, 1/4 will be sorted into the information treatment group, and 1/2 will be sorted into the hypothetical group.
|
|
Field
Building on Existing Work
|
Before
|
After
No
|