Toxicity and User Engagement: Online Study

Information on the intervention is hidden until the end of the trial.

2024-09-16

2025-03-31

1. User engagement

We measure whether the participant clicks “View 3 comments” at the bottom of the treated post to uncover the comment section. The link is formatted and situated within the post in a way that closely resembles how such links appear on Facebook. On click, an image with comments (following Facebook formatting) is displayed.

2. Willingness to accept for transcribing 100 social media posts

Participants use a slider from $2 to $30 to choose WTA with precision of $0.1. The person with the lowest compensation will be invited to transcribe 100 social media posts and paid the second smallest compensation. If there are multiple people with the smallest compensation, one of them will be chosen at random. The lower bound is selected to reflect that the minimum compensation per hour is $8 and that the task will take at least 15 minutes.

Heterogeneity:

First, we will look at heterogeneity with respect to: gender and minority status.

We classify participants as members of a minority group if they (1) selected an answer other than “white” in the ethnicity question, (2) identify as a Hispanic, Latino, or a person of Spanish origin, or (3) selected a different sexual orientation than heterosexual.

Moreover, as religiosity shapes norms of behavior, we will look at heterogeneity with respect to being religious (i.e., choosing a different answer than “no religion” when asked about religion or belief).

1.Recall of toxicity

A randomly selected subsample of participants (1/3rd) will be asked whether or not any of the posts that they were asked to transcribe were toxic.

A standard definition of toxicity will be provided: “A statement is toxic if it is a rude, disrespectful, or unreasonable comment that is somewhat likely to make you leave a discussion or give up on sharing your perspective.”

2. Recall of hate speech

A randomly selected subsample of participants (1/3rd) will be asked whether or not any of the posts that they were asked to transcribe contain hate speech.

Meta’s definition of hate speech will be provided: “Hate speech is a direct attack against people on the basis of protected characteristics: race, ethnicity, national origin, disability, religious affiliation, caste, sexual orientation, sex, gender identity, and serious disease.”

3. Recall of treatment posts

A randomly selected subsample of participants (1/3rd) will be asked which of the four options provided was one of the posts they were asked to transcribe.

The options will be similar to each other. For example, for Treatment 1 and Control 1, we will use:
- What is so difficult to understand? I hate Mondays.
- What did you just say? You hate Mondays?
- What is so difficult to understand? I hate Blacks.
- What did you just say? You hate Blacks?

4. Entertainment rating of posts

We will ask participants to rate on a scale from 0 to 100 (using a slider) how entertaining the training transcription posts were.

We will look at the same angles of heterogeneity as for the primary outcomes.

Information on the experimental design is hidden until the end of the trial.

Not available

Qualtrics randomization

Individual

No

N/A

We plan to recruit approximately N=1,000 participants per group (survey completions), which gives a total of about 4,000 people for the whole study. To increase power, we will also include pilot observations (approximately 100 observations per group).

We will randomly assign individuals to four experimental groups (Treatment 1, Control 1, Treatment 2, Control 2) with equal probabilities.

Our sample size gives us enough power to detect effect sizes close to 0.12 s.d for the primary outcomes (comparing each treatment with its associated control group), which is sufficient to detect main effects found in the pilot study. Additionally, for user engagement (primary outcome 1), we consider a pooled comparison (Treatment 1 + Treatment 2 vs. Control 1 + Control 2). For this test, we will have ex-ante power to detect the effect size of 0.09 s.d. Our MDEs reflect the subtlety of the intervention - we change little about the structure of the treatment posts. Furthermore, they are lower than the minimum MDE recommended for information provision experiments (0.15 s.d.) by Haaland et al. (2023). At the same time, we take into account that certain types of outcomes, including willingness to accept for tasks/work, are more inelastic to information interventions than other outcomes typical for this type of experiment.