The Impact of Comments on Social Media Campaigns

Last registered on July 29, 2024

View Trial History

Pre-Trial

Trial Information

General Information

Title

The Impact of Comments on Social Media Campaigns

RCT ID

AEARCTR-0013812

Initial registration date

July 22, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

July 29, 2024, 4:35 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

United States of America

Region

Primary Investigator

Name

Lena Song

Affiliation

UIUC

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Dante Donati

PI Affiliation

Columbia University

Contact Investigator

Additional Trial Information

Status

In development

Start date

2024-07-01

End date

2026-07-01

Keywords

Behavior

Additional Keywords

JEL code(s)

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

We measure the causal impact of comment sections on social media posts promoting racial justice. Conducted in two phases, it involves an online field experiment with over millions of Facebook users. We first collect and analyze comments to understand how political ideology influences engagement with posts. We then evaluates how exposure to different comment narratives affects subsequent engagement (comments, views), intentions (clicks), and off-platform behaviors (donations, petitions).

External Link(s)

Registration Citation

Citation

Donati, Dante and Lena Song. 2024. "The Impact of Comments on Social Media Campaigns." AEA RCT Registry. July 29. https://doi.org/10.1257/rct.13812-1.0

Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Experimental Details

Interventions

Intervention(s)

Our main intervention is to manipulate the comment section on Facebook ads.

Intervention (Hidden)

We will use a function on Facebook that allows us to hide comments in the comment section of our promoted posts. Users on Facebook who are exposed to our post will see ads with exogenously varying comment sections.

Intervention Start Date

2024-07-01

Intervention End Date

2026-07-01

Primary Outcomes

Primary Outcomes (end points)

engagement (comments, views), intentions (clicks), and off-platform behaviors (donations, petitions)

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

We will use tools on Facebook to vary what users see in the comment section of our posts.

Experimental Design Details

We first run to generate comments, targeting different zip codes by ideology (proxied by 2020 election Republican vote share)

Using these ads with comments, we then use a Facebook ad feature called re-marketing with social proof, to re-market them to 500 new zip codes.

We Randomization at the zip code level with four arms:
- All Comments: Ads with original comment section
- No Comments: Ads with new comments or all comments hidden
- Progressive Comments: Ads with all progressive comments hidden
- Conservative Comments: Ads with all conservative comments hidden

Randomization Method

randomization done by a computer (by Meta AB testing technology)

Randomization Unit

at the individual level

Was the treatment clustered?

Yes

Experiment Characteristics

Sample size: planned number of clusters

200 zip codes sets X 4 conditions per zip code = 800 clusters

Sample size: planned number of observations

3 million facebook users

Sample size (or number of clusters) by treatment arms

750,000 control (no comments), 750,000 treatment 1 (all comments), 750,000 treatment 2 (progressive comments), 750,000 treatment 3 (conservative comments)

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Supporting Documents and Materials

IRB