Improving survey completions using additional reminder emails and by providing additional information on survey content and impact

Last registered on January 19, 2022

Pre-Trial

Trial Information

General Information

Title
Improving survey completions using additional reminder emails and by providing additional information on survey content and impact
RCT ID
AEARCTR-0008792
Initial registration date
January 11, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
January 19, 2022, 12:29 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Primary Investigator

Affiliation
Scripps Research Translational Institute

Other Primary Investigator(s)

Additional Trial Information

Status
In development
Start date
2022-01-12
End date
2022-03-15
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
This study looks at:
* Determining whether a ‘pure’ reminder is more effective at increasing survey completion in a health research program than a reminder that also communicates the content of the survey and its impact.
* Determining if there are any spillover effects of being in a given reminder group (business-as-usual 2 reminders; larger set of pure reminders; larger set of reminders including information on survey impact and content) on subsequent participation.

The All of Us Research Program (AoURP) is an ambitious effort to gather health data from one million or more people living in the United States to accelerate research that may improve health. The aim is to help build one of the most diverse health databases in history. Researchers will use the data to learn how our biology, lifestyle, and environment affect health. This may one day help them find ways to treat and prevent disease. Understanding the health conditions, lifestyle and environmental conditions of participants are key pieces of information for researchers.

PPI 1-3 are important surveys in which AoURP collects data about the demographic background, health history and lifestyle information about our participants. The Basics survey (PPI 1) informs us of key information about the participant, including their age, race, gender, and education level. The Overall Health (PPI 2) survey asks participants about their physical and mental health backgrounds, which enables researchers to understand the health conditions of the AoURP cohort. The Lifestyle (PPI 3) survey asks about the habits of the participant, including smoking, alcohol, and recreational drug use habits. After primary consent completion, roughly 85% of participants complete the Basics (PPI 1) survey (calculated over December 2019 to November 2021). This number drops to 50-51% for PPI 2 and 3, resulting in an overall PPI 1-3 completion of 50%. This experiment aims to determine if additional reminders and variations in the reminder content can encourage more participants to complete the PPI 1-3 surveys.
External Link(s)

Registration Citation

Citation
Ter Meer, Janna. 2022. "Improving survey completions using additional reminder emails and by providing additional information on survey content and impact." AEA RCT Registry. January 19. https://doi.org/10.1257/rct.8792
Experimental Details

Interventions

Intervention(s)
This study will comprise of three arms:
• The control arm which includes two business-as-usual reminder emails.
• The pure reminder arm includes up to 6 reminders and informs the participant that they need to complete a survey. These emails feature a CTA, but don't go into detail about the survey content or its impact.
• The reminder + content + impact arm includes up to 6 reminders that include a CTA at the top as well as information about the survey content and its impact for AoURP researchers.
Intervention Start Date
2022-01-12
Intervention End Date
2022-03-15

Primary Outcomes

Primary Outcomes (end points)
The primary outcome variable is the proportion of participants that complete the respective surveys (PPI 1, 2 and 3) relative to those that complete primary consent after receiving one of three reminder email interventions.
Primary Outcomes (explanation)
After creating an account, users go through the primary consent module. Once the user has given consent to join the program, they become a program participant. The first task available to participants is Basics (PPI 1). This survey captures key information about the participant, including their age, race, gender and education level. This is the single gate through which all other tasks become available to the participant. Important for this experiment, after PPI 1 is completed, both the Overall Health (PPI 2) survey and the Lifestyle (PPI 3) survey become available and can be completed in any order. Participants that do not complete all of the surveys are served the reminder emails. The success of the interventions will be measured by the proportion of participants that complete the respective surveys (PPI 1, 2 and 3) relative to those that complete primary consent in each group.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The study will take place in the All of Us Research Program (AoURP). This is a longitudinal health research program looking at the impact of genetics and environmental factors on health outcomes, with a particular focus on individuals that are underrepresented in biomedical research. Once the user consents to become a participant, they can share different types of health data with the program, including through surveys, donating a biosample and by linking their Electronic Health Records. Our focus is on participants at the beginning of the journey and their likelihood to complete three important surveys for the program.

AoURP participants are recruited from the general public through advertisements on websites such as webmd.com and nextdoor.com. To date, AoURP has enrolled more than 400,000 participants. We will aim to recruit 21,700 participants. This corresponds to 4,340 participants in the control condition and 8,680 for each of the treatment conditions (17,360 participants for the pooled comparison).

After creating an account on the AoURP platform, users go through the primary consent module. Once the user has given consent to join the program, they become a program participant. Upon completing the primary consent, the participant becomes eligible to complete the Basics survey (PPI 1). This survey informs us of key information about the participant, including their age, race, gender and education level. Once completed, the participant becomes eligible to complete the Overall Health and the Lifestyle surveys. The Overall Health (PPI 2) survey asks participants about their physical and mental health backgrounds, which enables researchers to understand the health conditions of the AoURP cohort. The Lifestyle (PPI 3) survey asks about the habits of the participant, including smoking, alcohol, and recreational drug use habits. Participants are randomized to one of three trial arms after completing primary consent and receive specific reminder messages corresponding to their group assignment if they meet the reminder requirements.

The control includes two business-as-usual reminder emails. The first email is sent 24 hours after completing primary consent AND PPI 1 AND PPI 2 AND PPI 3 are incomplete. The second email is sent 5 days after completing primary consent AND PPI 1 OR PPI 2 OR PPI 3 are incomplete.

The pure reminder arm includes up to 6 pure reminders. The pure reminder emails inform the participant that they need to complete a survey. These emails include a CTA, but don't give detail about the survey content or its impact. In this arm, each PPI survey has two email alternatives for a total of 6 reminder emails. The first PPI 1 email is sent 24 hours after completing primary consent AND PPI 1 is incomplete. The first PPI 2 email is sent 24 hours after completing PPI 1 AND PPI 2 is incomplete. The first PPI 3 email is sent 24 hours after completing PPI 2 AND PPI 3 is incomplete OR 12 days after completing consent AND PPI 1 is complete AND PPI 3 is incomplete. The second PPI 1 email is sent 5 days after completing primary consent AND PPI 1 is incomplete. The second PPI 2 email is sent 5 days after completing PPI 1 AND PPI 2 is incomplete. The second PPI 3 email is sent 5 days after completing PPI 2 AND PPI 3 is incomplete OR after 17 days of completing consent AND PPI 1 is complete AND PPI 3 is incomplete.

The reminder + content + impact arm includes up to 6 reminders that include a CTA at the top as well as information about the survey content and its impact for AoURP researchers. In this arm, each PPI survey has two email alternatives for a total of 6 reminder emails. The first PPI 1 email is sent 24 hours after completing primary consent AND PPI 1 is incomplete. The first PPI 2 email is sent 24 hours after completing PPI 1 AND PPI 2 is incomplete. The first PPI 3 email is sent 24 hours after completing PPI 2 AND PPI 3 is incomplete OR 12 days after completing consent AND PPI 1 is complete AND PPI 3 is incomplete. The second PPI 1 email is sent 5 days after completing primary consent AND PPI 1 is incomplete. The second PPI 2 email is sent 5 days after completing PPI 1 AND PPI 2 is incomplete. The second PPI 3 email is sent 5 days after completing PPI 2 AND PPI 3 is incomplete OR after 17 days of completing consent AND PPI 1 is complete AND PPI 3 is incomplete.

The proportion of participants that complete the respective surveys (PPI 1, 2 and 3) relative to those that complete primary consent after receiving one of three reminder email interventions will be calculated. This measure will assess which intervention was the most successful at improving the conversion rate for the PPI 1-3 surveys.

The study will be active in the AoURP dashboard for an estimated nine weeks allowing for individuals to participate in the study.
Experimental Design Details
Randomization Method
Once participants become eligible for the study, they are randomly assigned a number between 1 and 3, which corresponds to one of the three trial arms. We are overweighting assignment into the treatment. Specifically, for each participant, we generate a random integer ranging from 0 to 4. Individuals being assigned 0 are in the control condition, those assigned 1 or 2 go into the pure reminder condition and those assigned 3 or 4 go into the reminder+ condition.
Randomization Unit
The unit of randomization is at the individual participant level. This happens when participants complete primary consent.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
N/A
Sample size: planned number of observations
Approximately 21,700 persons
Sample size (or number of clusters) by treatment arms
Approximately 4,340 participants in the control condition and 8,680 for each of the treatment conditions (17,360 participants for the pooled comparison).
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We make the following assumptions for our calculations: - We use standard parameters for the alpha and beta thresholds (0.05 and 0.20, respectively). This beta threshold translates to statistical power of 80%. - The baseline conversion rates and estimated weekly traffic are based on platform data analyzed between December 2019 and November 2021 (see section ‘Baseline data’ above). We run calculations for a baseline conversion rate of 50%, which reflects the overall PPI 1-3 conversion rate under the current set of reminders. - We assume three experimental conditions. - In our comparison of the current set of reminders and new reminders, the two treatment conditions will be pooled. Note that we are weighting treatment assignment, with 40% going into each of the treatments and 20% going into the control. We do this to power the comparison between treatments to a smaller effect size. We expect the effect size to be larger between the control and the pooled treatment conditions. This is why we are comfortable allocating more participants to the treatments. In addition, while our confidence intervals for the estimate of the control condition will be larger, the ones for the pooled treatment condition should be more precise as a result of these conditions having more observations. A sample size of 21,700 persons we selected an MDES of ~2.1 pp as a conservative estimate. We believe that this effect size is reasonable for the comparison of the current set of reminders against the expanded set (for which observations from treatments 1 and 2 would be pooled). Whether we will be able to clear the MDES for the comparison between treatment 1 and 2 is less certain.
IRB

Institutional Review Boards (IRBs)

IRB Name
All of Us Institutional Review Board
IRB Approval Date
2021-11-10
IRB Approval Number
2016-05-Supplement-01D

Post-Trial

Post Trial Information

Study Withdrawal

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials