Social Media Toxicity and Political Attitudes

Information on the intervention is hidden until the end of the trial.

2024-07-10

2025-03-31

1. Dehumanization

2. Support for partisan violence

3. Affective polarization

4. Political knowledge

Outcomes 1-4 will be measured in all surveys (baseline, midline, endline).

Heterogeneity:
We will look at heterogeneity with respect to: political affiliation and minority Details are provided below.

Re 1: Dehumanization will be measured using four survey questions:

- How evolved do [outparty members] seem? (0-100 Slider, Extremely unevolved, Extremely evolved, pictures of humanoid figures at different levels of evolution are included)
- To what extent do you think [outparty members] are capable of experiencing sensations such as hunger, fear, pain, and pleasure? (101 pt slider, Not at all, A great deal)
- To what extent do you think [outparty members] are capable of engaging in thought processes such as planning, reasoning, and remembering? (101 pt slider, Not at all, A great deal)
- How demonic or saintly do [outparty members] seem? (0-100 Slider, Extremely demonic, Extremely saintly, pictures of angelic/human/demonic figures included)

We will standardize scores for each statement and add them up to create a single index measure.

Re 2: Support for partisan violence will be measured using the following survey questions:

- When, if ever, is it OK for [inparty] to send threatening and intimidating messages to [outparty] leaders? (101 pt scale, Never, Always)
- When, if ever, is it OK for an ordinary [inparty member] in the public to harass an ordinary [outparty member] on the Internet, in a way that makes the target feel frightened? (101 pt scale, Never, Always)
- How much do you feel it is justified for [inparty members] to use violence in advancing their political goals these days? (101 pt scale, Not at all, Extremely)
- How much do you feel it is justified for [inparty members] to use violence if the [outparty] wins more races in the next election? (101 pt scale, Not at all, Extremely)

We will standardize scores for each statement and add them up to create a single index measure.

Re 3: Affective polarization will be measured using the following survey questions:

- [APOUT1] Please indicate how you feel toward [outparty members] using the scale below. 100 means that you feel very favorably or warm toward them, 0 that you feel very unfavorable or cold, and 50 are neutral.
- [APIN1] Please indicate how you feel toward [inparty members] using the scale below. 100 means that you feel very favorably or warm toward them, 0 that you feel very unfavorable or cold, and 50 are neutral.
- [APOUT2] How comfortable are you having friends who are [outgroup members]? (101 pt scale Not at all to Completely)
- [APIN2] How comfortable are you having friends who are [ingroup members]? (101 pt scale Not at all to Completely)

We will compute an index of affective polarization using the following formula [(APIN1-APOUT1)+(APIN2-APOUT2)]/2.

Re 4: Participants will be asked questions about their political knowledge in each of the surveys.

Of the following news events, which ones do you think are true events that occurred in the last month, and which ones do you think are false and did not occur? (True, False, Unsure)

There will be five statements in each survey, taken from headlines 2 weeks before the survey, with 2-3 modified to be false.

1. Spiral of silence

2. Moral disengagement

3. Intergroup empathy

4. Meta-perceptions

5. Committing partisan violence

6. Behavioral outcomes measured by the extension

Outcomes 1-4 will be measured in all surveys (baseline, midline, endline).

Heterogeneity:
We will look at the same angles of heterogeneity as for the primary outcomes.

Re 1: The spiral of silence is based on the following questions:

1) How often do you talk about politics? (“Never or almost never,” “Rarely,” “Sometimes,” “Often,” or “Very often”)
2) How comfortable or uncomfortable would you feel giving your honest opinions in a social media discussion on the issues below? (“Very comfortable,” “Fairly comfortable,” “Neither comfortable nor uncomfortable,” “Fairly uncomfortable,” or “Very uncomfortable”)
- Race relations in the United States
- Abortion
- Immigration policy
- Foreign aid
- Climate Change

We will encode choices with numerical scores (1-4 or 1-5), standardize scores for each of the six statements above, and add them to create a single spiral of silence index.

Re 2: How much do you agree or disagree with the following statements, on a scale from 1=Strongly disagree to 7=Strongly agree?

1) [outparty members] are a serious threat to the United States and its people.
2) [outparty members] are not just worse for politics - they are downright evil.
3) Many [outparty members] lack the traits to be considered fully human - they behave like animals.

We will standardize scores for each statement and add them up to create a single index measure.

Re 3: How much do you agree or disagree with the following statements?
1) I find it difficult to see things from [outgroup members] point of view.
2) It is important to understand [outgroup members] by imagining how things look from their perspective.

Re 4: Generally speaking, would you say that most people can be trusted, or that you can't be too careful in dealing with people? (101 pt scale, Can't be too careful… most people can be trusted)

Re 5: Imagine that the human figure you see below represents a [counter partisan] voter. Now, you can release any feelings you experience towards [counter partisans] by inserting from 0 pins to a maximum of 10 pins into the figure. Each click on the body represents a pin you insert. Feel free to insert the number of pins that you want. When you feel you are done click next.

Re 6: We will measure user engagement, including the number of posts and comments displayed to the user, the number of ad impressions, time spent on the treated platforms, the number of likes/reactions. Additionally, we will measure content production, including the quantity of posts and comments made by the user. We will standardize values for each outcome and add them up to create a single engagement index measure.

Our focus in measuring behavioral outcomes is comparing the Perspective treatment with Alienation+Perspective treatment, to illuminate how the additional reliance on Alienation API affects user engagement and content production. We will test the hypothesis using between subjects variation (comparing the two groups).

Information on the experimental design is hidden until the end of the trial.

Not available

After the user installs the browser extension (and agrees to the data collection), the extension assigns the user one of the experimental groups. The following example visualizes our approach well. The extension generates a random number between 0 and 1. If the number exceeds 0.4, the user is assigned one of the treatments: if it is between 0.4 and 0.7 to the treatment that uses Perspective API as classifier, if it is above 0.7 to the treatment that additionally uses Alienation as classifier. If it is below 0.4, the user is assigned to the control group: if it is between 0 and 0.2 it is assigned to the control group with no hiding, while if it is between 0.2 and 0.4 then it is assigned to the control group with random hiding.

Between the midline and the endline, we are planning to cross-randomize the respondents in both control groups into two groups that will either experience an engagement intervention or not. Randomization will be performed by the extension.

Individual

No

N/A

The number of observations depends on how successful we will be in promoting the browser extension (it will be promoted through advertisement on social media). We estimate to be able to recruit approximately 2,500 users.

We will randomly assign individuals to treatment groups with the following probabilities: 40% to the control group (equal split between pure control with no hiding and control with random hiding), 30% to a treatment group that uses Perspective API classifier, and 30% to a treatment group that uses Alienation classifier.

We use the software GPower to derive the optimal sample size for the experiment. The family test is a one-sided t test and the statistical test is a linear bivariate regression with two groups and differences between slopes (diff-in-diff). We assume that parameters alpha and 1-beta are equal to 0.05 and 0.8, we fix that the allocation ratio of the respondents of the treatment group relative to the control group is 1.5, we assume a standard deviation for the outcomes being the same across groups and equal to 1, and we target a minimum detectable effect of 0.1SD. This implies a total sample of 2,578 respondents (divided in 1,547 in the treatment group and 1,031 in the control group).