COVID-19 Thanksgiving Messaging at Scale

Last registered on June 29, 2021

Pre-Trial

Trial Information

General Information

Title
COVID-19 Thanksgiving Messaging at Scale
RCT ID
AEARCTR-0006821
Initial registration date
November 25, 2020
Last updated
June 29, 2021, 11:14 AM EDT

Locations

Region

Primary Investigator

Affiliation
MIT

Other Primary Investigator(s)

Additional Trial Information

Status
Completed
Start date
2020-11-14
End date
2021-06-01
Secondary IDs
Abstract
In prior work, we have found that video messages delivered by doctors increase knowledge of COVID-19, and in some cases, lead to viewers taking actions consistent with the messaging. In a scale-up phase, we plan to use Facebook ads to show a 15 second video clip recorded by 6 MGH, Harvard and Lynn Community health center doctors to approximately 40,000,000 Facebook users. The ads will be shown between November 14-29 and focus on staying safe – limiting travel, social distancing and mask-wearing. We are randomizing exposure to the ad campaign at the ZIP code and county level to study whether the videos change mobility and Thanksgiving holiday travel and whether there are any detectable impacts on the spread of COVID-19. Using information about Facebook social network connections and mobility patterns between counties, we also plan to measure whether exposure to the ad campaign spilled over along the network.

Following the completion of the Thanksgiving intervention, we implemented a follow-on Christmas intervention with the same two-stage research design, re-randomized in the same 13 states. Videos recorded by nurses and physicians encouraging people to stay home for Christmas were again pushed as ads to Facebook users.

External Link(s)

Registration Citation

Citation
Duflo, Esther . 2021. "COVID-19 Thanksgiving Messaging at Scale." AEA RCT Registry. June 29. https://doi.org/10.1257/rct.6821-2.0
Sponsors & Partners

Partner

Type
private_company
Experimental Details

Interventions

Intervention(s)
Facebook users will be exposed to an ad campaign featuring a set of videos about Thanksgiving travel. 6 doctors from MGH, Harvard and Lynn Community health center have each recorded one short message (15 seconds) using an identical script: “This Thanksgiving, the best way to show your love is to stay home. If you do visit, wear a mask at all times. I'm Dr. XX from XX, and I'm urging you: don't risk spreading COVID. Stay safe, stay home.”

The video messages will be posted to a project Facebook page titled "The Doctors for Coronavirus Prevention Project". Users will see the ads and the name of the Facebook group associated with the ads on their feeds. Watching the videos in the ad posts is completely optional for Facebook users.

Within treated geographies, Facebook's algorithm will be used to place the ads. We expect each treated individual to see the ads approximately 3 times over the course of the ad campaign.

The same intervention protocols were followed again prior to the Christmas holiday with a new set of messages recorded by doctors and nurses.
Intervention Start Date
2020-11-14
Intervention End Date
2020-11-29

Primary Outcomes

Primary Outcomes (end points)
Our most important outcome is mobility. We are also very interested in whether our campaign affected COVID-19 measures. However, we acknowledge that our study is much better powered to detect impacts on mobility.

Mobility and Travel: different outcomes are observable from Facebook at the county and zip code level due to data privacy concerns
- Zip code-level:
- Fraction of individuals staying in the same geographical location (1mi x 1mi area) (measured daily).
- Fraction of individuals leaving the geographical location (1mi x 1mi area) each day (measured daily)
- County-level:
- “Stay put” metric. SOURCE: https://data.humdata.org/dataset/movement-range-maps
- “Change in movement” metric: SOURCE: https://data.humdata.org/dataset/movement-range-maps
- Colocation data: Data marks counts of people from county pairs (A,B) who are observed 5 minutes in the same place. (Weekly)

Covid 19:
- Case rates/hospitalization/deaths:
- Zip: available from local public health departments for the 13 states in the Phase I experiment.
- County: available from local public health departments, nation-wide. Available for download from the New York Times.

Symptoms:
- Facebook COVID-19 symptoms survey. https://dataforgood.fb.com/docs/covid-19-symptom-survey-request-for-data-access/


Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Intended behaviors and beliefs: These outcomes all come from Facebook’s “Brand Lift” tool. In each zip code of the Phase 1 study, a hold-out sample is randomly selected in target zips to not receive the ad. We can compare treated and control responses within zip.
- Survey Questions:
- Ad recall
- Mask intentions
- Thanksgiving travel intentions
- How much should people try to stay home?

Additional outcomes, data pending:
- Mobility data from Safegraph (zip code level).
- MGH COVID-19 symptom tracking app data.
- Stay put, Change in movement, co-location at the zip code level.
- Dynata survey data on plans to travel for Thanksgiving, sampled from Nov. 13-24

Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Individuals in treatment geographies selected by Facebook's algorithms to be exposed to the ad campaign view videos on their Facebook feeds approximately 3 times during the duration of the campaign. Individuals in control geographies receive no treatment.

We rolled out the campaign in 2 phases:

Phase 1:
13 states were chosen based on the availability of zip-code level COVID-19 data. We used a 2-stage randomization with both county and zip-code level variation:
- 50% of counties in the 13 states were assigned to Low Intensity treatment
- 25% of zip codes in the low intensity counties were assigned to treatment
- 50% of counties in the 13 states were assigned to High Intensity treatment
- 75% of zip codes in the high intensity counties were assigned to treatment
The campaign began on November 14 and will run through November 29.

Phase 2:
Facebook was interested in scaling up the campaign to 15 additional states with high COVID-19 case rates and relatively low levels of compliance with preventative behaviors. Given Facebook's budget constraint, we randomized 80% of counties in these 15 states to receive the treatment. The campaign began on November 23 and will run through November 29.
Experimental Design Details
Randomization Method
Randomization was done in Stata for both phases.
Randomization Unit
In Phase 1, we use a 2-stage randomization at the county and zip code levels.
In Phase 2, randomization is at the county level. Ad targeting is based at the zip code level. Zip codes are assigned to the county with which it shares the largest area.
Christmas: same as Phase I

Individual Facebook users are targeted by Facebook's algorithms.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
Phase I: 820 counties, 6998 zip codes in 13 states.
Phase 2: 1063 counties in 15 additional states.
Christmas: 767 counties, 6716 zip codes (For this wave, we excluded rural counties with high Trump vote shares in the 2020 election out of caution due to the polarization about COVID, as we didn't want to create backlash. We also included zip codes even if they did not report COVID-19 cases.)
Sample size: planned number of observations
We approximate that 40,000,000 Facebook users will be exposed to the ads.
Sample size (or number of clusters) by treatment arms
Phase I: 50% of the 820 counties are treated. 50% of the 6998 zip codes are treated.
Phase 2: 80% of the 1063 counties are treated.
Christmas: same as Phase I (50% of counties and zips in the sample frame)
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We determined that a sample of 820 counties would provide 80% power to detect effect sizes of 0.2 standard deviations for county-level outcomes, comparing intervention (H) vs. control (L), using only average county-level data. For outcomes with zip code level data, using intra-class correlations of 0.2 (0.475) assuming clusters of equal size, a sample of 6,998 zip codes would provide 80% power to detect effect sizes of 0.057 (0.072) standard deviations (using zip code level data).
IRB

Institutional Review Boards (IRBs)

IRB Name
Massachusetts Institute of Technology
IRB Approval Date
2020-11-11
IRB Approval Number
2003000118
Analysis Plan

Analysis Plan Documents

Thanksgiving COVID Messaging Analysis Plan

MD5: 918f0510ed15e3e7822a6ad471ea4984

SHA1: 496fea1cfdd7a4b8be58a00530b5d37b1e80c70f

Uploaded At: November 25, 2020

Post-Trial

Post Trial Information

Study Withdrawal

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
December 31, 2020, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
February 11, 2021, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
We conducted the Thanksgiving intervention as anticipated. Our final sample included 6998 zip codes in 820 counties. We restricted the sample to only the zip codes that reported COVID-19 infections at baseline. We also ran an almost-identical experiment for the Christmas holiday, using physician- and nurse-recorded messages that pertained to Christmas travel. We re-randomized using the same sample. However, prior to the Christmas campaign, 60 fully rural counties in the top tercile of votes for Donald Trump in the 2020 election were removed from the study. This was done out of caution and to avoid adverse effects. The research team was concerned that the messaging campaign might have adverse unintended effects in very rural, heavily Republican-leaning counties given the growing polarization in December. The remaining sample had 767 counties. We included in the campaign all zip codes in the intervention in the selected counties (even if they could not be matched to COVID-19 infection data). For the COVID-19 outcomes, we have a final sample of 6716 zip codes.
Was attrition correlated with treatment status?
No
Final Sample Size: Total Number of Observations
Thanksgiving: 6998 zip codes, 820 counties.
Christmas: 6716 zip codes, 767 counties.
Recall, Facebook Movement Range mobility data is available at the county level, while COVID-19 data is at the zip code level.
Final Sample Size (or Number of Clusters) by Treatment Arms
See the Consort diagram (Figure 1) in the Supplemental Materials for a detailed break-down of sample sizes by treatment and campaign.
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Abstract
During the COVID-19 epidemic, many health professionals started using mass communication on social media to relay critical information and persuade individuals to adopt preventative health behaviors. Our group of clinicians and nurses developed and recorded short video messages to encourage viewers to stay home for the Thanksgiving and Christmas Holidays. We then conducted a two-stage clustered randomized controlled trial in 820 counties (covering 13 States) in the United States of a large-scale Facebook ad campaign disseminating these messages. In the first level of randomization, we randomly divided the counties into two groups: high intensity and low intensity. In the second level, we randomly assigned zip codes to either treatment or control such that 75% of zip codes in high intensity counties received the treatment, while 25% of zip codes in low intensity counties received the treatment. In each treated zip code, we sent the ad to as many Facebook subscribers as possible (11,954,109 users received at least one ad at Thanksgiving and 23,302,290 users received at least one ad at Christmas). The first primary outcome was aggregate holiday travel, measured using mobile phone location data, available at the county level: we find that average distance travelled in high-intensity counties decreased by 0.993 percentage points (95% CI -1.616, -0.371, p-value 0.002) the three days before each holiday. The second primary outcome was COVID-19 infection at the zip-code level: COVID-19 infections recorded in the two-week period starting five days post-holiday declined by 3.5 percent (adjusted 95% CI [-6.2 percent, -0.7 percent], p-value 0.013) in intervention zip codes compared to control zip codes.
Citation
arXiv:2106.11012

Reports & Other Materials