The spread of (mis)information: A social media experiment in Pakistan

Last registered on September 27, 2022

Pre-Trial

Trial Information

General Information

Title
The spread of (mis)information: A social media experiment in Pakistan
RCT ID
AEARCTR-0009954
Initial registration date
September 25, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
September 27, 2022, 11:58 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
UC Riverside

Other Primary Investigator(s)

PI Affiliation
UC Davis
PI Affiliation
Lahore University of Management Science
PI Affiliation
University of Michigan

Additional Trial Information

Status
On going
Start date
2020-03-20
End date
2022-10-01
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
This randomized experiment takes place on a social media platform in Pakistan called Baang. It measures the impact of two treatments that fully control access to misinformation relative to a more standard ex-post approach to controlling misinformation. In the treatment versions of the platform, moderation is ex-ante: all user-generated posts are reviewed by a moderator before being posted on the platform. In the remove treatment, user content that includes COVID-19 misinformation is simply never posted to the platform, while in the sunshine treatment, user content that contains COVID-19 misinformation is posted along with a rebuttal that debunks the post. The experiment focuses on misinformation regarding COVID-19 and is combined with an intervention to disseminate official information about the pandemic on the platform.
External Link(s)

Registration Citation

Citation
Hirshleifer, Sarojini et al. 2022. "The spread of (mis)information: A social media experiment in Pakistan." AEA RCT Registry. September 27. https://doi.org/10.1257/rct.9954-1.0
Experimental Details

Interventions

Intervention(s)
Intervention Start Date
2020-06-27
Intervention End Date
2020-08-27

Primary Outcomes

Primary Outcomes (end points)
The main outcomes in the experiment examine exposure and engagement at the user-level for three sources of information. We particularly focus on the impact of the treatments on the official information posts, since these posts were designed to provide high-quality information about COVID-19. We also examine two sources of user-generated information: useful posts that contain valuable information about COVID-19 and misinformation posts that contained false information about COVID-19.
Primary Outcomes (explanation)
For each of the three types of information, we conduct our analysis for three exposure and engagement measures. The focus of our analysis is on exposure to information, which is measured through minutes spent listening to a given type of post. In addition, we consider two measures of engagement. We examine number of shares separately, since it is the primary outcome of interest for researchers focusing on the determinants of the spread of misinformation. We also examine an standardized index of the other measures of engagement: comments, likes, and dislikes.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
This experiment tested three approaches to addressing misinformation on the platform using two treatments and a control. The two treatments rely on ex-ante moderation, which means that all user-generated content on the platform was reviewed by a moderator before being made publicly available. In the removal treatment, we never posted the content identified as misinformation related to COVID-19. In the sunshine treatment, we posted all user-generated misinformation content, but we included a specific rebuttal with each piece of content that conveyed information from trusted sources such as the WHO. These rebuttals played automatically immediately after the misinformation content, and were identified as official responses from the platform.
Experimental Design Details
These two treatments are compared against a control condition that relies on ex-post community-based moderation. This approach to moderation is similar to that of many social media platforms, and it was the standard on Baang before the study began. In the control, all user-created content is available immediately as it was posted, but users can tag messages as potential COVID-19 misinformation. These tagged posts are then sent to moderators who remove them from the platform if they are found to be misinformation.
In it important to note that users in all three conditions were exposed to the same content in general. The only two exceptions were that users in the removal treatment were not exposed to COVID-19 misinformation posts, and users in the sunshine treatment were exposed to the official rebuttals. Otherwise, if a user in the removal or sunshine treatment posts to platform, users in the control condition will see that post immediately. Meanwhile, if a user in the control condition posts to the platform, those posts were available to them immediately, and would become available to users in the two treatment conditions once they were moderated.
Randomization Method
The randomization took place automatically as people called into the platform. The randomization design accounted for networks of users and minimized potential spillovers. Specifically, treatment assignment depended on how a user reached the platform for the first time after the randomization began. Original users, who called in directly, were randomly assigned to one of the three conditions when they called into the platform. Referral users, who called in because they were forwarded content from the platform by another user, were assigned to the same condition as the user who forwarded them content. Regardless of how a user came to the platform initially, once a user was assigned to a condition, they remained in that condition every time they called into the platform thereafter.
Randomization Unit
Since the condition assignment of an original user determines the assignment of their referral users, this study relies on cluster-level random assignment. Each of the clusters in the study includes a single original user and their referral users if any.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
1408
Sample size: planned number of observations
3698
Sample size (or number of clusters) by treatment arms
Of our 3698 users, 1153 were in the remove treatment arm, 1258 were in the sunshine treatment arm, and 1287 were in the control arm.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
Lahore University of Management Sciences
IRB Approval Date
2020-03-03
IRB Approval Number
03032021AAA-FWA-00019408
IRB Name
UC Davis
IRB Approval Date
2021-02-18
IRB Approval Number
1717776-1

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials