Debunking “fake news” on social media: short-term, long-term, and spillover effects of fact check and media literacy interventions

Last registered on September 08, 2021

Pre-Trial

Trial Information

General Information

Title
Debunking “fake news” on social media: short-term, long-term, and spillover effects of fact check and media literacy interventions
RCT ID
AEARCTR-0008199
Initial registration date
September 07, 2021
Last updated
September 08, 2021, 5:18 PM EDT

Locations

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information

Primary Investigator

Affiliation
ifo Institute for Economic Research

Other Primary Investigator(s)

PI Affiliation
University of Cologne
PI Affiliation
University of Cologne
PI Affiliation
University of Cologne

Additional Trial Information

Status
In development
Start date
2021-09-10
End date
2021-11-30
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
We conduct a large-scale online survey experiment to study which remedies facilitate the distinction between false and correct information on social media. In particular, we examine the short-term, long-term, and spillover effects of fact check vs. media literacy interventions on participants’ factual knowledge, behavioral intentions, and the perceived credibility of information retrieved from social media. To this end, we expose participants to false and correct statements that we retrieve from Facebook. Then, we compare participants who obtain a brief media literacy training in terms of Facebook’s “Tips how to spot false news" before being exposed to the Facebook posts with participants who receive fact checks that debunk the false statements. As far as we know, we are the first who pursue a clean and sophisticated comparison of fact check and media literacy interventions as a means to debunk “fake news.”
External Link(s)

Registration Citation

Citation
Kerkhof, Anna et al. 2021. "Debunking “fake news” on social media: short-term, long-term, and spillover effects of fact check and media literacy interventions." AEA RCT Registry. September 08. https://doi.org/10.1257/rct.8199-1.0
Sponsors & Partners

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information
Experimental Details

Interventions

Intervention(s)
1) Fact check intervention
2) Media literacy intervention

See Experimental Design for details
Intervention Start Date
2021-09-10
Intervention End Date
2021-10-15

Primary Outcomes

Primary Outcomes (end points)
1) Participants’ factual knowledge on topics covered by the “fake news”
2) Perceived credibility of the “fake news”
3) Behavioral intentions:
a) Probability to get Covid-19-vaccine. Note: participants who are not vaccinated yet will be asked about their probability to get vaccinated as soon as they re-ceive an offer. Participants who are already vaccinated will be asked about the probability to get a booster in case the gov-ernment recommended this.
b) Probability to consume dietary supplements
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
1) Participants’ factual knowledge on topics covered by facts
2) Perceived credibility of facts
3) Time spent with the “Tips how to spot fake news”
4) Time spent with the fact checks
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We will expose the participants to fakes and facts on Corona vaccines and on nutrition that we retrieve from Facebook.

The participants will be randomly divided into five groups:
- Participants who are not exposed to any fakes or facts at all
- Participants who are exposed to fakes and facts
- Participants who are exposed to fakes and facts and fact checks that debunk the fake news on vaccines (not the fakes on nutrition) --> Fact check intervention
- Participants who are shown Facebook’s “Tips how to spot false news” and are afterwards exposed to fakes and facts --> Media literacy intervention
- Participants who are exposed to facts and the fact checks on vaccines

To study the short-term effects of the fact check and the media literacy intervention, we will compare participants’ factual knowledge, perceived credibility of the “fake news”, and their behavioral intentions w.r.t. Corona vaccines to participants who saw the fakes and facts, but did not receive an intervention.

To examine potential spillover effects of the fact check and media literacy interven-tion, we show the participants fakes and facts on an unrelated topic (dietary sup-plements) without further interventions. Then, we will again compare factual knowledge, perceived credibility of the “fake news”, and behavioral intentions of participants in the fact check and in the media literacy group to participants who saw the fakes and facts, but did not receive an intervention.

Finally, to study the long-term effects of the fact check and media literacy interven-tion, we re-invite our participants about one week later and repeat our analyses.

Estimation procedure:
Our baseline regressions will be OLS estimations with treatment group dummies as main explanatory variables. The dependent variables are perceived credibility of the “fake news”, factual knowledge on Corona vaccines and dietary supplements, and self-reported probabilities to get vaccinated and to consume dietary supplements. Control variables include age, gender, family status, income, education, “big 5”, po-litical preferences, and prior knowledge on current events, health, and nutrition. As participants may skip over the fact checking and media literacy intervention, our baseline analysis is equivalent to an ITT analysis. To obtain estimates for the LATE of our interventions, we will also conduct an IV estimation, where we use the time spent with the fact checks or the “Tips how to spot false news” as endogenous explanatory variable and assignment to treatment as instrument.

We hypothesize the following effects:

1a) In the short-run, the fact checking intervention reduces the credibility of and increases factual knowledge about the corrected „fake news” as compared to participants without intervention.

1b) In the short-run, participants who received the fact checking intervention are more likely to state that they are willing to get vaccinated against Covid-19 than participants without intervention.

2a) In the short- and in the long-run, the media literacy intervention reduces the credibility of and increases factual knowledge about all „fake news” as compared to participants without intervention.

2b) In the short- and in the long-run, participants who received the media literacy intervention are more likely to state that they are willing to get vaccinated against Covid-19 and abstain from dietary supplements than participants without intervention.

2c) The long-run effects of the media literacy intervention are smaller than its short-run effects.

The idea is as follows:
While fact checking is likely to be an effective tool to debunk the corrected messages themselves, its effect may be limited to these messages. Thus, the intervention re-duces the credibility of the “fake news” on Corona vaccines that we display in Ses-sion 1 as compared to participants without intervention. As a result, factual knowledge on Corona vaccines is larger than in the group without intervention and participants are more likely to get vaccinated. In contrast to that, we do not expect to see spillover effects on “fake news” about dietary supplements or on “fake news” on Corona vaccines in Session 2.
The media literacy intervention conveys general skills that can be broadly applied. Hence, we expect that the intervention reduces the credibility of “fake news” on Co-rona vaccines and dietary supplements that we display in Session 1 and 2 as com-pared to participants without intervention. As a result, factual knowledge on Corona vaccines and dietary supplements is larger than in the group without intervention and participants are more likely to get vaccinated and less likely to consume sup-plements. As some participants may have forgotten the “Tips how to spot false news”, we expect the effects to be smaller in Session 2 than in Session 1.

Subgroup analyses:
To date, about half of the German population is already fully vaccinated against Covid-19 and we expect a similar fraction of our participants to be vaccinated, too. As vulnerability to “fake news” may depend on prior knowledge and experiences, we will split the sample into vaccinated and non-vaccinated participants and explore heterogeneous effects. Similarly, we will partition our sample into participants who agree with the official Corona remedies vs. those who don’t to see if and how prior beliefs and attitudes moderate the effects of our interventions. Finally, an interesting subgroup analysis will be to partition participants into those who spend much and those who spend few time on social media to see if experience and prior exposure to “fake news” from social media plays a role.
Experimental Design Details
Not available
Randomization Method
The participants will be randomly allocated into the groups by the market research company that we are going to cooperate with

Randomization Unit
Randomization unit: Individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
NA
Sample size: planned number of observations
3000 participants (individuals) divided into five groups for the first wave, Planning with 20% attrition after 1 week: about 2400 participants (individuals) divided into five groups for the second wave
Sample size (or number of clusters) by treatment arms
600 participants (individuals) per group for the first wave
Planning with 20% attrition after 1 week: about 480 participants (individuals) per group for the second wave
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
Research Ethics Review; Faculty of Management, Economics, and Social Sciences; University of Cologne
IRB Approval Date
2021-05-04
IRB Approval Number
210014JM