Debiasing Motivated Reasoning Through Learning: Evidence from an Online Experiment

Last registered on July 08, 2019

View Trial History

Pre-Trial

Trial Information

General Information

Title

Debiasing Motivated Reasoning Through Learning: Evidence from an Online Experiment

RCT ID

AEARCTR-0004401

Initial registration date

July 01, 2019

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

July 08, 2019, 11:33 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

United States of America

Region

Primary Investigator

Name

Michael Thaler

Affiliation

UCL

Contact Primary Investigator

Other Primary Investigator(s)

Additional Trial Information

Status

In development

Start date

2019-07-07

End date

2019-12-28

Keywords

Other

Additional Keywords

Motivated reasoning, debiasing, biased beliefs, polarization, overconfidence, fake news

JEL code(s)

C91, D83, D84, D91, L82

Secondary IDs

Abstract

When people receive information about controversial issues such as immigration policies, upward mobility, and racial discrimination, the information often evokes both what they currently believe and what they are motivated to believe. Motivated reasoning posits that people misupdate from information by treating these motivated beliefs as an extra signal. The main objective of this experiment is to test whether a "debiasing treatment" can attenuate motivated reasoning. This treatment tells people when they have erred and asks them their opinion on whether motivated reasoning played a role. Relating motivated reasoning to other biases and phenomenons, this experiment tests whether the debiasing treatment reduces polarization, makes people less overprecise, and has them form more accurate assessments of the veracity of news sources.

External Link(s)

Registration Citation

Citation

Thaler, Michael. 2019. "Debiasing Motivated Reasoning Through Learning: Evidence from an Online Experiment." AEA RCT Registry. July 08. https://doi.org/10.1257/rct.4401-1.0

Former Citation

Thaler, Michael. 2019. "Debiasing Motivated Reasoning Through Learning: Evidence from an Online Experiment." AEA RCT Registry. July 08. https://www.socialscienceregistry.org/trials/4401/history/49469

Sponsors & Partners

Experimental Details

Interventions

Intervention(s)

When people receive information about controversial issues, the information often evokes both what they currently believe and what they are motivated to believe. The theory of motivated reasoning posits that people misupdate from information by treating these motivated beliefs as an extra signal. This intervention aims to attenuate the bias of motivated reasoning through a treatment in which people learn about past errors.

Intervention Start Date

2019-07-08

Intervention End Date

2019-07-09

Primary Outcomes

Primary Outcomes (end points)

Motivated reasoning

Primary Outcomes (explanation)

Motivated reasoning is measured as a directional deviation from Bayes' rule. Subjects are asked to assess the probability that a message comes from a news sources that tells the truth (or lies). This design is constructed such that Bayesians will not update about the veracity about a news source regardless of the message. However, motivated reasoners will give higher veracity assessments to sources that tell them when it tells them something they are more motivated to believe. Motivated reasoning is measured by regressing veracity assessment on message type (pro-motivated belief / anti-motivated belief).

Secondary Outcomes

Secondary Outcomes (end points)

1. Dummy for changing beliefs

2. Confidence: Prediction of performance relative to 100 others

3. Performance: Subjects' performance on news assessments

4. Overprecision: 1/2 minus the probability that 50% confidence interval contains the answer

5. Polarization: Whether subjects' beliefs move away from or towards the mean belief

6. Trust in news: Overall assessments

Secondary Outcomes (explanation)

1. Whether subjects change their beliefs in the direction of the message statement; e.g. if the message says "The answer is greater than 57." then this takes value 1 if and only if the subject's second guess is greater than 57.

2. Subjects' prediction of where they ranked in performance on the study is elicited towards the end of the experiment.

3. On every news assessment, subjects earn a score which is proportional to the probability they win a bonus prize. Performance on a given question is equal to this score.

4. Subjects' median belief, 25th percentiles, and 75th percentiles are elicited on each question. Overprecision is a dummy that takes 0.5 if the correct answer is not within the CI and -0.5 if the correct answer is within the CI. That is, overprecision is equal to 0.5 - P(answer within 50\% CI). Overprecision is positive (negative) when the CI contains the true answer less (greater) than 50\% of the time.

5. Whether subjects are more likely to change their beliefs in the direction of the message statement when the message tells them to move away from the mean population belief.

6. Assessment of the veracity of news sources.

Experimental Design

There are two main parts to this design: Identifying Motivated Reasoning and Debiasing Treatment.

Identifying Motivated Reasoning:

The main test of this in the experiment involves three steps. See the analysis plan for more details.

1. Beliefs: Subjects are asked to guess the answers to questions like the refugee one above. Importantly, they are asked and incentivized to guess their median belief (i.e. such that find it equally likely for the answer to be above or below their guess). They are also asked and incentivized for their interquartile range.

2. News: Subjects receive a binary message from one of two news sources: True News and Fake News. The message from True News is always correct, and the message from Fake News is always incorrect. The probability of either source is 1/2 and iid across questions. This is the main (within-subject) treatment variation.

The message says either "The answer is greater than your previous guess of [previous guess]." or "The answer is less than your previous guess of [previous guess]." Note that the message space is different for each subject since subjects have different priors. These customized messages are designed so that they have the same subjective likelihood of occurring.

3. Assessment: After receiving the message, subjects assess the probability that the source was True News on a scale from 0/10 to 10/10 and are incentivized to state their true belief. This is the main outcome measure. The page is identical to the beliefs page but the guess boxes are replaced with assessment choices. The effect of variation in news on veracity assessments is the primary outcome variable for identifying motivated reasoning.

The general point of this setup is that subjects receive messages that compare the answer to their median, so they should not rationally update their assessment based on the message. Directionally different assessments are difficult to reconcile with Bayesian updating; they are also difficult to reconcile with general misweighting of priors (since the prior of source is fixed at 1/2) or likelihoods (each message is equally likely, so the message is uninformative about source veracity). However, these deviations can be explained by motivated reasoning.

The most direct test is to hypothesize what people are motivated to believe, and compare their assessments on "Pro-Motive" news and "Anti-Motive" news. If Pro-Motive news is trusted more than Anti-Motive news, this indicates that motivated reasoning is likely with these hypothesized motives is at play.

Interacting this assessment gap with a treatment provides an estimate for the effect of the treatment on the degree of the motivated reasoning bias. If this gap is positive, and the interaction of news type (Pro/Anti-Motive) and treatment is negative, then the treatment likely is effective in debiasing subjects.

Subjects see 14 rounds of questions; on politicized and performance-related topics, the random binary messages are coded as Pro-Motive or Anti-Motive using a hypothesized table of motives.

On several topics, messages are also coded as Optimistic or Pessimistic using a hypothesized table of motives.

Debiasing Treatment:

Starting after round 3, subjects in this treatment receive an additional page if their previous news veracity assessment scored fewer points than if they had answered "5/10 chance it's True News; 5/10 chance it's Fake News".

This page informs subjects about their performance on that question and the correct news source. It also gives subjects two survey questions about whether they think their erroneous assessment was due to motivated reasoning.

Outcomes mainly look at the intent-to-treat effect. The control group and the treatment group (up until round 3) cannot see this page, while the treatment group (starting in round 4) can; these are the groups for comparison in analysis.

Experimental Design Details

Randomization Method

Randomization for all treatments done by computer

Randomization Unit

Pro-Motive / Anti-Motive news randomized at question level.

Questions randomized at individual level.

Control / debiasing treatment randomized at individual level.

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

1050 individuals (~90% of whom are in main analysis)

Sample size: planned number of observations

Approximately 13500 news assessments (~90% of which are in main analysis)

Sample size (or number of clusters) by treatment arms

350 individuals control, 700 individuals debiasing treatment

Approximately 6100 Pro-Motive news, 6100 Anti-Motive news

Approximately:
3200 Pro-Motive news in debiasing traetment
2900 Pro-Motive news in control
3200 Anti-Motive news in debiasing traetment
2900 Anti-Motive news in control

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Supporting Documents and Materials

IRB