Are Preconceptions Postconceptions? Evidence on Motivated Political Reasoning

Last registered on August 05, 2021

Pre-Trial

Trial Information

General Information

Title
Are Preconceptions Postconceptions? Evidence on Motivated Political Reasoning
RCT ID
AEARCTR-0007985
Initial registration date
August 03, 2021

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
August 05, 2021, 5:21 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
August 05, 2021, 3:47 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
Harvard University

Other Primary Investigator(s)

PI Affiliation
Harvard University

Additional Trial Information

Status
In development
Start date
2021-08-05
End date
2021-08-25
Secondary IDs
Abstract
How do people update their beliefs on contentious political issues when they receive new information? Do they do so in a rational manner or do they refuse to update their beliefs? To investigate this, we perform a survey experiment on Amazon Mechanical Turk. We study the extent to which people claim their beliefs on a highly salient political issue would change depending on pertinent empirical facts (provided as hypotheticals). We then compare these responses to how beliefs actually change when people learn new empirical facts, which yields insight about the prevalence of motivated reasoning. We apply this approach to a range of issues -- police shootings of minorities, climate change, affirmative action, income taxation of the top 1%, economic mobility, zoning laws, and the Olympics -- which vary in their political polarization. Within the same framework, we explore whether it is possible to de-bias individuals from engaging in motivated reasoning by asking (and reminding) individuals about how their beliefs would change as a function of varying hypotheticals. In an alternative experiment, we provide individuals with informative but noisy signals, which accordingly may either conform with or contradict their ideological preconceptions.
External Link(s)

Registration Citation

Citation
Lilley, Matthew and Brian Wheaton. 2021. "Are Preconceptions Postconceptions? Evidence on Motivated Political Reasoning." AEA RCT Registry. August 05. https://doi.org/10.1257/rct.7985-1.1
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Intervention Start Date
2021-08-05
Intervention End Date
2021-08-25

Primary Outcomes

Primary Outcomes (end points)
We are interested in the extent to which individuals' reported normative beliefs change in response to new information. In particular, we will compare:
(a) People's ex ante beliefs about political issues (using answers of people in the control and hypothetical groups)
(b) People's claimed responsiveness to information (using answers of people in the control and hypothetical groups)
(c) People's ex post beliefs about political issues (using answers of people in all three groups -- in particular, the wedge between the treatment and hypothetical groups)

Furthermore, we are interested in the gap between the extent to which individuals claim, hypothetically, their beliefs would change if a given piece of information was true versus the extent to which they actually change when individuals are told the information is, in fact, true.

And we are interested in the extent to which our approach of eliciting responses to hypotheticals before providing the true information can be used as a tool for de-biasing.
Primary Outcomes (explanation)
Depending on the exact normative question, we expect either a negative or positive relationship between beliefs and the hypothetical information. To enable standardization across categories (such as when pooling the data), we will rotate the responses accordingly. In particular, the values for the answers to the taxes, mobility, and zoning questions will be reversed.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Our survey experiment will be posted on Amazon Mechanical Turk, advertised with a base compensation of $2.50 and an incentive bonus of up to $1 for participants. Individuals who opt to take our survey will answer a series of questions about their beliefs relating to several different political issues. The issues will include police shootings of minorities, climate change, affirmative action, income taxation of the top 1%, economic mobility, zoning laws, and the Olympics.

Once an individual opts to take the survey, they will begin by answering a series of demographic questions. After this, for each of five of the above seven issue areas (randomly selected), the respondent will be randomly sorted into either (i) the control group, (ii) the information treatment group, or (iii) the hypothetical treatment group. Independently, they will be sorted into an "A" or "B" group, which receive slightly different orders of normative beliefs questions. In other words, an individual may be sorted into the control group ver. A for the climate change issue, the hypothetical group ver. B for the affirmative action issue, and the treatment group ver. A for the income taxation issue.

Regardless of which group the individual is sorted into, they will be asked for their belief on an empirical fact relating to a political issue. After this,

In the control group, respondents will be asked for their normative beliefs on the issue (both the A and B questions).

In the information treatment group, respondents will be presented with the true answer to the above empirical question corresponding to the issue. Then, they will be asked for their normative beliefs on the issue (both the A and B questions).

In the hypothetical group, respondents will be asked their normative beliefs on the issue (both the A and B questions). They will then be presented, randomly, with three hypotheticals asking what their normative beliefs would be IF the true answer to the empirical question was X, for three randomly-selected values of X (one of which is the true value). They will be asked the A version of the normative beliefs question if they were sorted into group A and the B version if they were sorted into group B.

This process is then repeated four more times for four more issues.

Comparing the extent to which individuals claim they will hypothetically update their beliefs (for the true value of X) to the extent to which they actually do update their beliefs when presented with the true information will yield information on the extent to which motivated reasoning is occurring. In addition, we will test how much people claim they will respond to new information.

In particular, we will compare:
(a) People's ex ante beliefs about political issues (using answers of people in the control and hypothetical groups)
(b) People's claimed responsiveness to information (using answers of people in the control and hypothetical groups)
(c) People's ex post beliefs about political issues (using answers of people in all three groups -- in particular, the wedge between the treatment and hypothetical groups)

At the end of the survey, for each issue on which individuals were sorted into a hypothetical group, they will receive a follow-up. They will be given the information treatment and then asked their normative beliefs on the issue (both A and B). Comparing the extent to which they then update their beliefs on A and on B will yield information on the efficacy of using the hypotheticals approach as a tool for de-biasing.

In particular, we will test:
(a) Whether the apparent degree of motivated reasoning is reduced by this process of first posing hypothetical questions
(b) Whether the apparent degree of motivated reasoning differs between the specific normative question on which the respondent previously answered the hypothetical (e.g., A) versus the normative question which they were not posed hypotheticals about (e.g., B). To the extent that de-biasing is more successful for A vis-a-vis B (in this example) this suggests a demand effect rather than a true de-biasing

In a follow-up survey one week later, we will again ask people these normative beliefs questions (both the A and B versions) in order to examine the extent to which these effects persist. The mechanical demand effects whereby people in the hypothetical group feel constrained to be consistent with their hypothetical answers (e.g., for A but not constrained for B) should be substantially alleviated one week later, and this will allow us to determine whether any de-biasing is, in fact, real.
Experimental Design Details
EMPIRICAL FACT QUESTIONS

- Police shootings: In 2015, the Washington Post began to log every fatal shooting by an on-duty police officer in the United States.

For context:
As of 2019, the US population was approximately 12.8% Black.
In 2019, according to the FBI, among people arrested for violent crimes, approximately 36.4% were Black.

What is your best estimate of the percentage of people fatally shot by police (from 2015 through 2020) who were Black (amongst those whose race was known)?


- Climate change: Climate scientists at NASA's Goddard Institute for Space Studies maintain records on global average temperatures since 1880.

How many of the 20 hottest years on record would you estimate have occurred since the year 2000, according to the NASA data?


- Affirmative action: Elite colleges and universities such as Harvard have a very restricted number of slots to which they admit applicants for undergraduate study each year.

Harvard states that it "does not discriminate against applicants from any group in its admissions processes."

According to data released by Harvard University, between 1995 and 2013, the average SAT score for admitted White students was 2234 out of 2400.

What is your best estimate of the average SAT score (out of 2400) for admitted African-American students over the same time period, according to Harvard's data?


- Income taxation: Every year, the Internal Revenue Service (IRS) releases data on income and taxation in the United States. From this data, the share of federal income tax paid by the top 1% of income-earners can be computed.

For context:
According to the IRS, 21% of all income was earned by the top 1% of income-earners.

What is your best estimate of the share of federal income tax that was paid by the top 1% of income-earners in 2019, according to the IRS data?


- Economic Mobility: The Internal Revenue Service (IRS) maintains an archive of historical tax return data from individual taxpayers. This data can be linked across generations and used to compute estimates of social mobility -- such as the fraction of children born to families in the bottom 20 percentiles of income who themselves made it into the top 20 percentiles of income at the same point later in their life.

Consider the poorest 20 percentiles of children, as measured by their parents' income in early adulthood. Then:
If there was no relationship between parent and child income, 20% of these children would themselves make it into the top 20 percentiles of income at the same age later in life.
0% corresponds to none of these children making it into the top 20 percentiles of income at the same age later in life.
100% corresponds to all of these children making it into the top 20 percentiles of income at the same age later in life.

Consider the poorest 20 percentiles of children, as measured by their parents' income in early adulthood.

What is your best estimate of the share of these children who themselves make it into the top 20 percentiles of income at the same age later in life, according to the IRS data?


- Zoning: Zoning laws regulate land-use. They restrict the types of buildings and/or activities that are permitted in certain areas within communities. Common motives for zoning laws include environmental concerns, neighborhood beautification, or preferences of the existing population for maintaining the community's size and character.

Research has suggested, however, that by constraining the supply of housing, restrictive zoning laws can increase rents and home prices.

For context:
According to an index compiled by the Cato Institute which measures land-use regulation, California has the 4th-most restrictive land-use regulations in the United States while Nevada has the 26th-most restrictive land-use regulations.
According to the US Census Bureau, as of 2019, the Las Vegas metropolitan area has 0.7% of total US population while the Los Angeles metropolitan area has 5.7% of total US population.

The Census Bureau compiles data on the number of building permits issued for the construction of new single-family homes in the United States. Between 2015 and 2019, 1.2% of those permits were issued in Las Vegas.

What percentage of permits do you think were issued in Los Angeles?


- Olympics: The modern Olympics games have occurred since 1896. According to the International Olympic Committee (IOC), counting both summer and winter Olympics (through 2018), the ten athletes who have won the most gold medals have won a total of 99 gold medals between them.

Of course, some sports provide more opportunities for an individual athlete to win many medals.

What is your best estimate of the number of gold medals won by American swimmer Michael Phelps?


NORMATIVE BELIEFS QUESTIONS

Police Shootings:
Normative Belief Question A: Do you believe that police in the US are systemically racist against Black people?
Normative Belief Question B: Do you support “defunding the police,” which entails reducing funding for police departments and reallocating that funding toward social services such as counseling, healthcare, and public housing?

Climate Change:
Normative Belief Question A: Do you support stronger government action to combat climate change, even if it requires higher taxes?
Normative Belief Question B: Do you believe that the global average temperature is warming due primarily to human activity?

Affirmative Action:
Normative Belief Question A: Do you think the Supreme Court should ban universities from using race as a factor in admission decisions? (Universities argue that affirmative action is key to improving representation of minorities.)
Normative Belief Question B: Do you think that affirmative action causes lower performing and less deserving students from underrepresented minority groups to take places that otherwise would be earned by more deserving students of other races?

Income Taxation:
Normative Belief Question A: Do you support higher taxes on the top 1% of income-earners?
Normative Belief Question B: Do you think the top 1% of income-earners excessively use loopholes to avoid paying their fair share of taxes?

Economic Mobility:
Normative Belief Question A: Do you believe that hard work doesn't generally bring success; luck and connections are more important?
Normative Belief Question B: Do you think that most wealthy people are only wealthy because of inheritance and/or family privilege?

Zoning:
Normative Belief Question A: Do you think zoning laws (often promoted for their environmental, aesthetic, and other benefits) are too restrictive and should be relaxed?
Normative Belief Question B: Do you think that differences in zoning laws substantially explain why some parts of the US (e.g. San Francisco, New York City) have extremely high housing prices relative to other areas (e.g. Houston, Orlando)?

Olympics:
Normative Belief Question A: Do you believe Michael Phelps is the greatest Olympic athlete of all time?
Normative Belief Question B: Do you believe Michael Phelps is the greatest swimmer of all time?
Randomization Method
Randomization performed by the survey software (Qualtrics).
Randomization Unit
Randomization will occur at the individual level.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
700 individuals
Sample size: planned number of observations
700 individuals
Sample size (or number of clusters) by treatment arms
700 individuals each answer questions on 5 of 7 issues (randomly-selected). Consequently, each issue will be presented to approximately 500 individuals, of which 1/3 will be sorted into the control group, 1/3 will be sorted into the information treatment group, and 1/3 will be sorted into the hypothetical group.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials

Documents

Document Name
Survey Flow and Survey Questionnaire
Document Type
survey_instrument
Document Description
Contains survey flow and questionnaire as output by Qualtrics.
File
Survey Flow and Survey Questionnaire

MD5: b085c403bab0c0b6918a562ac52469e1

SHA1: 29af933da38676e7da2c2847b20e2c12ebb479be

Uploaded At: August 05, 2021

IRB

Institutional Review Boards (IRBs)

IRB Name
Harvard University Committee on the Use of Human Subjects (CUHS)
IRB Approval Date
2021-04-08
IRB Approval Number
IRB20-1201

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials