Improving User Experience on Social Media. A Field Experiment (Part II, Survey)

Last registered on October 03, 2022

Pre-Trial

Trial Information

General Information

Title
Improving User Experience on Social Media. A Field Experiment (Part II, Survey)
RCT ID
AEARCTR-0010138
Initial registration date
September 27, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
October 03, 2022, 5:12 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
The University of Chicago

Other Primary Investigator(s)

PI Affiliation
The University of Chicago
PI Affiliation
Columbia University
PI Affiliation
The Mozilla Foundation

Additional Trial Information

Status
On going
Start date
2022-07-06
End date
2023-02-01
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
This is a pre-registration for the endline survey of the following study: Improving User Experience on Social Media. A Field Experiment, registered AEARCTR-0009628.

External Link(s)

Registration Citation

Citation
Beknazar-Yuzbashev, George et al. 2022. "Improving User Experience on Social Media. A Field Experiment (Part II, Survey)." AEA RCT Registry. October 03. https://doi.org/10.1257/rct.10138-1.0
Experimental Details

Interventions

Intervention(s)
This is a pre-registration for the endline survey of the following study: Improving User Experience on Social Media. A Field Experiment, registered AEARCTR-0009628.

Here, we pre-register outcomes that we elicit during the endline survey. The pre-registration preceeds the beginning of data collection for the survey, scheduled to start on 9/28/2022.
Intervention Start Date
2022-07-20
Intervention End Date
2022-09-30

Primary Outcomes

Primary Outcomes (end points)
Outcomes elicited during the endline survey for the study:
1. Willingness to pay (or accept) for keeping the browser extension installed for another month.
2. Index of self-reported well-being.
3. Toxicity ratings of online comments.
Primary Outcomes (explanation)
Ad 1. We relied on a dynamic MPL approach, with a range of values for WTP from -$6 to $6. We randomly selected 10 participants for whom one of the MPL choices is implemented.

Ad 2. We asked the following questions:

(1) Click the point on the scale below that you feel is most appropriate in describing you. Over the past 6 weeks, I thought I was:
1 (not a very happy person), 2, 3, 4, 5, 6, 7 (a very happy person).

(2) Indicate your agreement with the statement below. During the past 6 weeks, I was satisfied with my life.
Possible answers: Strongly disagree, Disagree, Slightly disagree, Neither agree nor disagree, Slightly agree, Agree, Strongly agree.

(3) Below are some ways you might have felt or behaved in the past 6 weeks. Please tell us how much of the time during the past 6 weeks:
... you felt depressed.
... you felt anxious.
... you were absorbed in doing something worthwile.
... you felt bored.
Possible answers for each statement: 1. None or almost none of the time, 2., 3., 4. All or almost all of the time.

Based on the six responses, we will craete one index of well-being by averaging the standardized values – for each response we will substract the mean and divide by the standard deviation.

Ad 3. We provide the participants with the following instructions:

Lastly, we would like to ask you to rate toxicity of seven real comments posted online. Before you provide the ratings, please read the definitions of toxicity below.
Very Toxic: a very hateful, aggressive, or disrespectful comment that is very likely to make you leave a discussion or give up on sharing your perspective.
Toxic: a rude, disrespectful, or unreasonable comment that is somewhat likely to make you leave a discussion or give up on sharing your perspective.

Please rate the toxicity of the following comments. We kept the original spelling, including any mistakes.

The statements are as follows (displayed in random order):
(a) So tired of all these Portlanders moving to California and ruining the culture. When will it stop?!?
(b) The sky is falling! The sky is falling! This country has elected a black man twice! And now a woman! Please God! Make America Great Again! We're going down in flames!!!
(c) Only an individual living in the bubble of Ottawa and "working" as a professor can come up with this nonsense.....wow!
(d) can we just exterminate this scumbag already? every miserable breath he takes is a slap in the face to people who understand the privilege of parenthood.
(e) If you want more EmX transit route, more bike lanes, and less traffic lanes, vote for either Joshua Skov or Chris Wig. If not Goggle the others and make sure they are not part of the arrogant progressive movement, which is ruining Eugene.
(f) FYI: "LOL" does not turn a pack of malicious lies into amusingly subversive wisdom.
(g) Sadly, you are probably right. Things in Africa have gone from bad to worse for gay people, thanks mostly to American evangelical homophobes like Scott Lively exporting their hate to foreign shores.

To aggregate the resposnes, we first compute the proportion of people who reported each statement to be ‘Toxic’ or ‘Very Toxic’. Then, we average the proportions across the seven statements to report the final outcome.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
This is a pre-registration for the endline survey of the following study: Improving User Experience on Social Media. A Field Experiment, registered AEARCTR-0009628.

Here, we pre-register outcomes that we elicit during the endline survey. The pre-registration preceeds the beginning of data collection for the survey, scheduled to start on 9/28/2022.
Experimental Design Details
We recruit participants to install a browser extension called "Social Media Research", which has the ability to hide toxic content (posts, comments, replies) on Twitter, Facebook, and YouTube.

Users are randomized into one of two groups. In the treatment group, during the intervention period all content exceeding the threshold level of toxicity is hidden (please see Intervention section for a description on how we assign the toxicity scores). In the control group, no content is hidden. In both groups, the extension loads replies which Twitter places under the “Show more replies” at the bottom of the comment sections (where more toxic content is placed by Twitter). This increases the variation in exposure to toxic content between the groups.

After installation, the user enters an observation period of 2 weeks (measured individually from the day they first open one of the three platforms in their browser with the extension installed) where the extension collects data on their activity but there is no hiding intervention in both groups. Subsequently, the intervention begins for the treatment group.

The endline survey was administered after the end of the six-week intervention period for all participants. To evaluate the effect of the intervention on the survey outcomes (described in this pre-registration), we will use two-sided t-tests for differences in means between the treatment groups.

Based on comparing Twitter handles that we collect in the endline survey and those which are captured by the extension, we will drop all participants whom we know were not active (as detected by the extension) on day 46 of the study or later.
Randomization Method
After the user installs the browser extension (and agrees to the data collection), the extension (using JavaScript) generates a random number between 0 and 1. If the number exceeds 0.5, the user is assigned one of the treatments. If it is below 0.5, the user is assigned the other.
Randomization Unit
Individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
N/A
Sample size: planned number of observations
We consider the endline survey for the study registered in AEARCTR-0009628. Given the survival rates in the study, we expect fewer than 700 responses.
Sample size (or number of clusters) by treatment arms
We consider the endline survey for the study registered in AEARCTR-0009628. Given the survival rates in the study, we expect fewer than 350 responses per treatment group.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
Social and Behavioral Sciences Institutional Review Board The University of Chicago
IRB Approval Date
2022-02-28
IRB Approval Number
IRB22-0073
IRB Name
Morningside IRB, Columbia University
IRB Approval Date
2022-03-10
IRB Approval Number
AAAT9887

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials