Social media advertising loads as prices

See pre-analysis plan

2024-08-22

2024-12-31

Time. The main outcome of interest is the amount of active time spent on Facebook and on two different types of content displayed on Facebook: ads and organic posts. Active time spent is calculated as described in Beknazar, Jiménez-Durán, McCrosky, and Stalinski (2022).
Another main outcome of interest is the time substitution to Twitter and Reddit, where we can also measure active time, as well as YouTube, where we only measure the approximate time the browser window was open.

Engagement with ads. Another main outcome of interest is the level of consumer engagement and activity of consumers with the ad content. Our main measure is an index (Kling, Liebman, and Katz, 2007) constructed with the number of impressions (views) and clicks (although we will also report the effect on each of these outcomes separately).

Valuation. Another main outcome of interest is the post-intervention vs. pre-intervention change in the (incentivized) Willingness-to-Accept (WTA) to deactivate social media for four weeks. We will elicit the WTA in the baseline survey and in the endline survey.

Besides these websites, a secondary outcome of interest is the time spent on other social media platforms as in Beknazar, Jiménez-Durán, McCrosky, and Stalinski (2022). Moreover, we consider the substitution in terms of time spent on the top-10 websites visited by each participant during baseline.

We will also report as a secondary outcome of interest the time spent on the advertiser’s website (capped at 5 minutes for each visit), and “bounce backs” (returns in 30 seconds or less) to Facebook.
Besides this, another secondary outcome of interest is user reactions to ads (i.e., likes/shares), in the treatment arms where we can collect these.


As a secondary outcome of interest, we will elicit the WTP to keep the browser extension installed for four more weeks. We will elicit it using a take-it-or-leave-it approach, randomly varying the amount that we offer to participants.

Other secondary outcomes of interest are the amount of content consumed and produced, and the toxicity of content produced. Besides these outcomes, we will report auxiliary outcomes to understand Facebook’s response to the intervention and to rule out potential confounders. For example, we will compute the ad load “offered” (before hiding) to participants and a measure of how targeted the ads offered are. We also have access to participants’ Twitter handles (collected using the browser extension) and will compute, using the API, the number of posts they produce and people they follow, to measure substitution patterns more precisely. Along these lines, we will measure the change in the percent of social media use on browser vs mobile, which we will elicit in the baseline and endline surveys. We will measure time spent on mobile using screenshots from users’ phones. We will also ask for screenshots of targeting data on Facebook.

PE Outcomes. We will also have a set of secondary outcomes related to political economics. First, we will elicit the propensity to share fake news during the intake and endline surveys. We will also extract urls to websites shared and observed by the participants to examine their interaction with non-fake-news websites (we will use the list of news websites from Gregory Martin, Andrey Simonov, and Shoshana Vasserman. “Beyond the Paywall: Measuring Supply and Demand for Online News in a Rapidly Changing News Environment”, work in progress) as well as fake-news websites (we will use the list from Melnikov, 2021 and Lasser and Rupp, 2022) and the propensity to share political information. Lastly, using the available information about our participants we will match them to voter records to obtain a measure of turnout.

See PAP for more details

Not available

Computer randomization
We are stratifying/blocking randomization by day of recruitment

Individual level

No

We anticipate recruiting approximately 1,500 participants using Facebook ads. The ad campaigns are selected to optimize conversions, which we measure as participants finishing the intake survey that opens after installing our browser extension. We are targeting desktop ads to US-based adults with UK or US English language who use Chrome to access Facebook and exclude those who use Firefox, Safari, or Edge to access Facebook. At the start of recruitment, the extension does not work for Edge, but we are planning to extend support to that browser, at which point we will remove Edge from the exclusion criteria.


We exclude from the sample those participants who:
Do not have English set as their interface language on Facebook;
Are not using Chrome or Edge (the latter – if we manage to support it);
Appear to be using bots or create multiple accounts in order to game the system and get more rewards than one per individual (for example, during piloting, we noticed a single individual with browser language set to “zh-CN” who has managed multiple consecutive installations and survey completions from different IP addresses).
Uninstall the extension upon installing it, before the treatment period begins.
Do not use their Facebook account for at least 1 minute per week during the baseline period.

At the time of pre-registration we are planning to apply for different grants requesting additional funding. If the grants get approved, we will expand our sample size by approximately 600 more participants (this number depends on the price of recruitment ads on Facebook, which are hard to predict in an election cycle).

Same as clusters.

Observations equally split across all four treatment arms

We conducted a power analysis using pilot data (from 31 users in the quantity reduction condition, 28 users in the replacement control condition, 31 in the targeting reduction condition, and 120 in the pure control condition). This intervention lasted for 13 days and had a baseline of 14 days. We conducted 100 simulations at different sample sizes of our main empirical specification (diff-in-diff) and a more conservative specification that relies only on difference in means (cross-sectional specification). We did this for both the quantity reduction and the targeting reduction interventions. Based on these simulations, we require a sample of at least 1,200 participants to have 80% power with conventional size. For this reason, we pre-register a sample of around 1,500 participants. Note also that we expect the effect sizes to be larger during the main intervention, which will last longer than the pilot intervention. However, the smallest effect size of the cross-sectional difference in means is 0.11 SD, which would require a sample size of 2,500 in a design with two treatment arms. Even if the difference in means is not our main specification, we will attempt to expand our number of observations, which will be contingent on having a grant approved. See PAP for more details

Ad Loads - PAP.pdf