AEA RC T Registr y

The American Economic Association's registry for randomized controlled trials

Facebook users will be exposed to an ad campaign featuring a set of videos about Thanksgiving travel. 6 doctors from MGH, Harvard and Lynn Community health center have each recorded one short message (15 seconds) using an identical script: “This Thanksgiving, the best way to show your love is to stay home. If you do visit, wear a mask at all times. I'm Dr. XX from XX, and I'm urging you: don't risk spreading COVID. Stay safe, stay home.”

The video messages will be posted to a project Facebook page titled "The Doctors for Coronavirus Prevention Project". Users will see the ads and the name of the Facebook group associated with the ads on their feeds. Watching the videos in the ad posts is completely optional for Facebook users.

Within treated geographies, Facebook's algorithm will be used to place the ads. We expect each treated individual to see the ads approximately 3 times over the course of the ad campaign.

2020-11-14

2020-11-29

Our most important outcome is mobility. We are also very interested in whether our campaign affected COVID-19 measures. However, we acknowledge that our study is much better powered to detect impacts on mobility.

Mobility and Travel: different outcomes are observable from Facebook at the county and zip code level due to data privacy concerns
- Zip code-level:
- Fraction of individuals staying in the same geographical location (1mi x 1mi area) (measured daily).
- Fraction of individuals leaving the geographical location (1mi x 1mi area) each day (measured daily)
- County-level:
- “Stay put” metric. SOURCE: https://data.humdata.org/dataset/movement-range-maps
- “Change in movement” metric: SOURCE: https://data.humdata.org/dataset/movement-range-maps
- Colocation data: Data marks counts of people from county pairs (A,B) who are observed 5 minutes in the same place. (Weekly)

Covid 19:
- Case rates/hospitalization/deaths:
- Zip: available from local public health departments for the 13 states in the Phase I experiment.
- County: available from local public health departments, nation-wide. Available for download from the New York Times.

Symptoms:
- Facebook COVID-19 symptoms survey. https://dataforgood.fb.com/docs/covid-19-symptom-survey-request-for-data-access/

Intended behaviors and beliefs: These outcomes all come from Facebook’s “Brand Lift” tool. In each zip code of the Phase 1 study, a hold-out sample is randomly selected in target zips to not receive the ad. We can compare treated and control responses within zip.
- Survey Questions:
- Ad recall
- Mask intentions
- Thanksgiving travel intentions
- How much should people try to stay home?

Additional outcomes, data pending:
- Mobility data from Safegraph (zip code level).
- MGH COVID-19 symptom tracking app data.
- Stay put, Change in movement, co-location at the zip code level.
- Dynata survey data on plans to travel for Thanksgiving, sampled from Nov. 13-24

Individuals in treatment geographies selected by Facebook's algorithms to be exposed to the ad campaign view videos on their Facebook feeds approximately 3 times during the duration of the campaign. Individuals in control geographies receive no treatment.

We rolled out the campaign in 2 phases:

Phase 1:
13 states were chosen based on the availability of zip-code level COVID-19 data. We used a 2-stage randomization with both county and zip-code level variation:
- 50% of counties in the 13 states were assigned to Low Intensity treatment
- 25% of zip codes in the low intensity counties were assigned to treatment
- 50% of counties in the 13 states were assigned to High Intensity treatment
- 75% of zip codes in the high intensity counties were assigned to treatment
The campaign began on November 14 and will run through November 29.

Phase 2:
Facebook was interested in scaling up the campaign to 15 additional states with high COVID-19 case rates and relatively low levels of compliance with preventative behaviors. Given Facebook's budget constraint, we randomized 80% of counties in these 15 states to receive the treatment. The campaign began on November 23 and will run through November 29.

Not available

Randomization was done in Stata for both phases.

In Phase 1, we use a 2-stage randomization at the county and zip code levels.
In Phase 2, randomization is at the county level. Ad targeting is based at the zip code level. Zip codes are assigned to the county with which it shares the largest area.

Individual Facebook users are targeted by Facebook's algorithms.

Yes

Phase I: 820 counties, 6998 zip codes in 13 states.
Phase 2: 1063 counties in 15 additional states.

We approximate that 40,000,000 Facebook users will be exposed to the ads.

Phase I: 50% of the 820 counties are treated. 50% of the 6998 zip codes are treated.
Phase 2: 80% of the 1063 counties are treated.

Thanksgiving COVID Messaging Analysis Plan