The Impact of Comments on Social Media Campaigns

Last registered on May 14, 2025

View Trial History

Pre-Trial

Trial Information

General Information

Title

The Impact of Comments on Social Media Campaigns

RCT ID

AEARCTR-0013812

Initial registration date

July 22, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

July 29, 2024, 4:35 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated

May 14, 2025, 8:12 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country

United States of America

Region

Primary Investigator

Name

Lena Song

Affiliation

UIUC

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Dante Donati

PI Affiliation

Columbia University

Contact Investigator

Additional Trial Information

Status

In development

Start date

2025-01-15

End date

2026-07-01

Keywords

Behavior

Additional Keywords

JEL code(s)

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

We measure the causal impact of comment sections on social media posts promoting racial justice. Conducted in two phases, it involves an online field experiment with over millions of Facebook users. We first collect and analyze comments to understand how political ideology influences engagement with posts. We then evaluates how exposure to different comment narratives affects subsequent engagement (comments, views), intentions (clicks), and off-platform behaviors (mailing list sign-up, donations).

External Link(s)

Registration Citation

Citation

Donati, Dante and Lena Song. 2025. "The Impact of Comments on Social Media Campaigns." AEA RCT Registry. May 14. https://doi.org/10.1257/rct.13812-2.1

Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Experimental Details

Interventions

Intervention(s)

Our main intervention is to manipulate the comment section on Facebook ads.

Intervention (Hidden)

We will use a function on Facebook that allows us to hide comments in the comment section of our promoted posts. Users on Facebook who are exposed to our post will see ads with exogenously varying comment sections.

Intervention Start Date

2025-03-25

Intervention End Date

2026-07-01

Primary Outcomes

Primary Outcomes (end points)

engagement (comments, views), intentions (clicks), and off-platform behaviors (mailing list sign-up, donations)

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

We will use tools on Facebook to vary what users see in the comment section of our posts.

Experimental Design Details

We first run to generate comments, targeting different zip codes by ideology (proxied by 2020 election Republican vote share)

Using these ads with comments, we then use a Facebook ad feature called re-marketing with social proof, to re-market them to new zip codes.

There are four arms:
- No Comments: Ads with all comments hidden
- Mixed Comments: Ads with progressive and conservative comments shown
- Progressive Comments: Ads with only progressive comments shown
- Conservative Comments: Ads with only conservative comments shown

Randomization Method

randomization done by a computer (by Meta AB testing technology)

Randomization Unit

at the individual level

Was the treatment clustered?

Yes

Experiment Characteristics

Sample size: planned number of clusters

Around 150 clusters

Sample size: planned number of observations

1.5-2 million Facebook users

Sample size (or number of clusters) by treatment arms

Around 450,000 control (no comments), 450,000 treatment 1 (mixed comments), 450,000 treatment 2 (progressive comments), 450,000 treatment 3 (conservative comments)

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Supporting Documents and Materials

IRB