Perception of browser extension

Information on the intervention is hidden until the end of the trial.

The intervention is introduced via a browser extension that participants install for the purpose of the study. The extension can modify content that the participants see on Facebook.

The purpose of this intervention is to test whether different forms of delay affect user experience. Based on pilot evidence, we expect that delay does not affect experience; however, if we find that it does after scaling, we will use these estimates to adjust our findings in Beknazar et al., “Toxic content and user engagement on social media: Evidence from a field experiment.”

There are three treatment groups: CONTROL, HIDING NORMAL, HIDING LONG.

In the CONTROL group, the extension does not take any action.

In the HIDING NORMAL group, the extension randomly hides posts and comments on Facebook. Ads are never hidden. The hiding probability is calibrated such that the extension should be hiding, on average, 5% of total content. The extension introduces delay in hiding of the elements. In all cases, the delay is equal to 400 milliseconds (the time between an element is identified and the time it is is hidden).

In the HIDING LONG group, the extension randomly hides content with the same probability as in the HIDING NORMAL group. However, the delay is 1300 milliseconds.

2026-01-12

2026-01-31

The primary outcome is an index of user experience that is based on three measures: (1) rating of user experience, (2) willingness to accept for testing the extension again, (3) reporting experiencing delays on Facebook.

The index is constructed as the equally weighted average of the standardized three outcomes above. Each outcome is standardized using the control-group mean and standard deviation, and outcomes (2) and (3) enter with a negative sign so that higher values of the index consistently indicate better user experience.

We use the following question to elicit the rating of user experience.

How would you rate your overall experience navigating Facebook while using the extension?
- Very negative
- Somewhat negative
- Neutral
- Somewhat positive
- Very positive

We encode each of the answers using a scale from 1 to 5, with 1 being “very negative” and 5 being “very positive”.

We use the following question to elicit the willingness to accept.

Imagine we asked you to repeat this testing task in the future. What is the minimum amount of money in US dollars you would require to do it again?

Note: please be truthful. The computer will select a random amount and you will be invited to participate again and offered this amount if your required compensation is equal to or below the computer's offer.

Participants can use a slider from $0 to $10 to answer the question. We also note that this outcome was not incentivized in the pilot data.

Lastly, we use the following question to elicit whether a participant noticed any delays.

Did you notice any delay or change in the quality of your Facebook experience while the extension was active?
- Yes
- No

Information on the experimental design is hidden until the end of the trial.

We recruit participants on Prolific. We restrict the pool of eligible participants to individuals (1) residing in the US, (2) on desktop devices, (3) using Google Chrome browser, (4) using Facebook regularly (at least once in the previous month).

Participants are instructed to install a browser extension which can modify the content they see on Facebook. They are not provided with any specific information on the extension functionality. After the installation, they are redirected to Facebook and asked to spend 5 minutes using the platform. We ask participants to use Facebook as they normally would.

The extension randomizes participants into one of three treatment groups: CONTROL, HIDING NORMAL, HIDING LONG. See INTERVENTION section for details.

This experiment follows up on the field experiment in the paper “Toxic Content and User Engagement on Social Media: Evidence from a Field Experiment”. In the paper, we provide causal experimental evidence that exposure to toxicity increases user engagement. The experiment was conducted with a browser extension that hides toxic content in the treatment group and does not change user experience in the control group.

In this trial, we aim to understand whether some of the engagement effect in the field experiment is due to delays introduced by the hiding process or the hiding process itself. And if so, the data that we collect would allow us to appropriately bound the effects that we find in the field experiment.

We will conduct two main hypothesis tests.

We will compare HIDING NORMAL to CONTROL to test whether hiding elements on Facebook affects user experience.

We will compare HIDING LONG to HIDING NORMAL to test whether, conditional on hiding elements, a long delay affects user experience relative to a short delay.

Both the browser extension and the treatment calibration were designed to match the previous field experiment. In the field experiment, about 5% of content was hidden in the treatment group on Facebook. We calibrated the random hiding probability in the browser extension to match that hiding probability. The median hiding delay in the field experiment was about 400 milliseconds. Hence, in HIDING NORMAL, we set the delay to be 400 milliseconds in all cases. Lastly, to understand the impact of long delays, we set the delay to be 1300 milliseconds in HIDING LONG. This is close to the 95th percentile of hiding delay in the field experiment.

The browser extension randomizes people into the treatment groups using JavaScript.

Individual

No

N/A

We plan to recruit 1,500 individuals. Since only regular Facebook users and survey takers on Google Chrome desktop browser are eligible, we might not be able to recruit as many on Prolific. In such a case, we will attempt recruitment via other platforms. To help with the issue of scarcity of participants, we will pool our pilot sample (239 individuals) in the final sample.

500 individuals per treatment arm

Power calculations reveal that this sample size is enough for an MDE of 0.1774 SD.