Exploring the external validity of debunking “fake news” on social media via fact checking and media literacy interventions

Last registered on July 08, 2024

Pre-Trial

General Information

Title

RCT ID

AEARCTR-0013629

Initial registration date

July 06, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

July 08, 2024, 1:55 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

Germany

Region

Primary Investigator

Name

Lara Marie Berger

Affiliation

University of Cologne

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Anna Kerkhof

PI Affiliation

ifo Institute for Economic Research

Contact Investigator

PI Name

Johannes Münster

PI Affiliation

University of Cologne

Contact Investigator

PI Name

Felix Mindl

PI Affiliation

University of Cologne

Contact Investigator

Additional Trial Information

Status

In development

Start date

2024-07-08

End date

2024-08-19

Keywords

Behavior, Other

Additional Keywords

fake news, fact check, media literacy, social media, misinformation

JEL code(s)

Secondary IDs

Prior work

This trial is based on or builds upon one or more prior RCTs.

Abstract

This study builds on our results from the project "Debunking “fake news” on social media: short-term, long-term, and spillover effects of fact check and media literacy interventions", which was pre-registered under the RCT ID AEARCTR-0008199. In our original study we analyzed the effectiveness of fact-checking and media literacy interventions in the context of Covid-19- and other health-related posts on social media. With this follow-up study we aim to better understand the external validity of our findings by taking the interventions to a new context (replacing the fake news and facts about Covid-19 with fake news and facts about environmental topics). Additionally, we want to understand if and how the degree of the credibility of a certain fake is related to the effectiveness of an intervention.

External Link(s)

Registration Citation

Citation

Berger, Lara Marie et al. 2024. "Exploring the external validity of debunking “fake news” on social media via fact checking and media literacy interventions." AEA RCT Registry. July 08. https://doi.org/10.1257/rct.13629-1.0

Sponsors & Partners

Experimental Details

Interventions

Intervention(s)

1) Fact check intervention
2) Media literacy intervention

Each intervention is tested for high-credibility fakes and low-credibility fakes (2x3 design).

See Experimental Design for details.

Intervention (Hidden)

Intervention Start Date

2024-07-08

Intervention End Date

2024-08-19

Primary Outcomes

Primary Outcomes (end points)

Perceived credibility of the “fake news”

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Perceived credibility of facts, Time spent with the “Tips how to spot fake news”, Time spent with the fact checks, Time spent with the Facebook postings (fakes and facts), Time spent with the credibility questions

Secondary Outcomes (explanation)

Experimental Design

We will expose the participants to fakes and facts on environmental and health-related topics that we retrieve from Facebook.

The participants will be randomly divided into six groups:
- Participants who are exposed to high-credibility fakes and facts --> Control group high
- Participants who are exposed to low-credibility fakes and facts --> Control group low
- Participants who are exposed to high-credibility fakes and facts and fact checks that debunk the fake news --> Fact check intervention high
- Participants who are exposed to low-credibility fakes and facts and fact checks that debunk the fake news --> Fact check intervention low
- Participants who are shown Facebook’s “Tips how to spot false news” and are afterwards exposed to high-credibility fakes and facts --> Media literacy intervention high
- Participants who are shown Facebook’s “Tips how to spot false news” and are afterwards exposed to low-credibility fakes and facts --> Media literacy intervention low

The environment-related fakes participants are exposed to have been rated as more-credible or less-credible in a pre-study. Participants in the "high" groups are exposed to two fakes which have been rated as more-credible, while participants in the "low" groups see two fakes which have been rated as less-credible. The facts and health-related fakes and facts are identical in all groups. The order in which the fakes and facts are displayed will be randomized. We also randomize whether participants in the fact check group see the fact check before or after the corresponding fake.

To study the effects of the fact check and the media literacy intervention, we will compare participants’ perceived credibility of the “fake news” to participants who saw the fakes and facts, but did not receive an intervention (always comparing the high groups to one another and the low groups to the other low groups). To study the effect of the degree of credibility we will compare the coefficients for the treatment effects in the high groups to the coefficients for the treatment effects in the low groups.

Estimation procedure:
Our main regressions estimate the effect of the treatment group dummies on the perceived credibility of “fake news”. Control variables include age, gender, family status, income, education, “big 5”, political preferences, and prior knowledge on current events, health, and nutrition.

We hypothesize the following effects:

1) The fact checking intervention reduces the credibility of the „fake news” as compared to participants without intervention.
2) The media literacy intervention reduces the credibility of the „fake news” as compared to participants without intervention.
3) The effects of both interventions are higher for fakes which are ex-ante rated as less credible.

Experimental Design Details

Randomization Method

computer

Randomization Unit

individual; within-individual for the analysis of the degree of credibility

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

Sample size: planned number of observations

1800 individuals

Sample size (or number of clusters) by treatment arms

600 high control group, 600 low control group, 600 high fact-check group, 600 low fact-check group, 600 high media literacy group, 600 low media literacy group (3600 total)

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Supporting Documents and Materials

IRB