Improving survey completions using additional reminder emails and by providing additional information on survey content and impact

This study will comprise of three arms:
• The control arm which includes two business-as-usual reminder emails.
• The pure reminder arm includes up to 6 reminders and informs the participant that they need to complete a survey. These emails feature a CTA, but don't go into detail about the survey content or its impact.
• The reminder + content + impact arm includes up to 6 reminders that include a CTA at the top as well as information about the survey content and its impact for AoURP researchers.

Update made to registration on 09/22/22:
We are extending this evaluation to two reminders for two additional surveys in the AoU journey called PPI 5 (Health Care Access and Utilization) and PPI 6 (Personal & Family Health History).For each survey, two reminders are sent spaced 5 days apart. The PPI 5 and 6 surveys become available to participants that have completed PPI1 (Basics) and 90 days have passed since program enrollment, which is also when the first reminder for PPI 5 goes out. We will have two conditions: Pure Reminder and Reminder + that follow the same structure as for the PPI 1-3 reminders described above. In both groups, the CTA is in the first paragraph of the email and the email subject line and preview text are the same across groups. In other words, the only difference between the conditions is that the Reminder+ condition gives more detail about the survey content and why the answers are important for health research.

Note that we are choosing not to test the control messages for PPI 5-6 so that we can better power the other two conditions.

2022-01-12

2022-12-30

The primary outcome variable is the proportion of participants that complete the respective surveys (PPI 1, 2 and 3) relative to those that complete primary consent after receiving one of three reminder email interventions.

Update made to registration on 09/22/22:
For the additional evaluation of the PPI 5-6 reminders, we will look at the proportion of participants that complete PPI 5 and 6 relative to those that complete Primary Consent and PPI 1 (Basics) and receive at least one of the reminder messages. We will analyze results for the pooled surveys (Pure Reminder for PPI 1-3 and PPI 5-6 against Reminder+ for PPI 1-3 and PPI 5-6) and separately by PPI 1-3 and PPI 5-6.

After creating an account, users go through the primary consent module. Once the user has given consent to join the program, they become a program participant. The first task available to participants is Basics (PPI 1). This survey captures key information about the participant, including their age, race, gender and education level. This is the single gate through which all other tasks become available to the participant. Important for this experiment, after PPI 1 is completed, both the Overall Health (PPI 2) survey and the Lifestyle (PPI 3) survey become available and can be completed in any order. Participants that do not complete all of the surveys are served the reminder emails. The success of the interventions will be measured by the proportion of participants that complete the respective surveys (PPI 1, 2 and 3) relative to those that complete primary consent in each group.

The study will take place in the All of Us Research Program (AoURP). This is a longitudinal health research program looking at the impact of genetics and environmental factors on health outcomes, with a particular focus on individuals that are underrepresented in biomedical research. Once the user consents to become a participant, they can share different types of health data with the program, including through surveys, donating a biosample and by linking their Electronic Health Records. Our focus is on participants at the beginning of the journey and their likelihood to complete three important surveys for the program.

AoURP participants are recruited from the general public through advertisements on websites such as webmd.com and nextdoor.com. To date, AoURP has enrolled more than 400,000 participants. We will aim to recruit 21,700 participants. This corresponds to 4,340 participants in the control condition and 8,680 for each of the treatment conditions (17,360 participants for the pooled comparison).

After creating an account on the AoURP platform, users go through the primary consent module. Once the user has given consent to join the program, they become a program participant. Upon completing the primary consent, the participant becomes eligible to complete the Basics survey (PPI 1). This survey informs us of key information about the participant, including their age, race, gender and education level. Once completed, the participant becomes eligible to complete the Overall Health and the Lifestyle surveys. The Overall Health (PPI 2) survey asks participants about their physical and mental health backgrounds, which enables researchers to understand the health conditions of the AoURP cohort. The Lifestyle (PPI 3) survey asks about the habits of the participant, including smoking, alcohol, and recreational drug use habits. Participants are randomized to one of three trial arms after completing primary consent and receive specific reminder messages corresponding to their group assignment if they meet the reminder requirements.

The control includes two business-as-usual reminder emails. The first email is sent 24 hours after completing primary consent AND PPI 1 AND PPI 2 AND PPI 3 are incomplete. The second email is sent 5 days after completing primary consent AND PPI 1 OR PPI 2 OR PPI 3 are incomplete.

The pure reminder arm includes up to 6 pure reminders. The pure reminder emails inform the participant that they need to complete a survey. These emails include a CTA, but don't give detail about the survey content or its impact. In this arm, each PPI survey has two email alternatives for a total of 6 reminder emails. The first PPI 1 email is sent 24 hours after completing primary consent AND PPI 1 is incomplete. The first PPI 2 email is sent 24 hours after completing PPI 1 AND PPI 2 is incomplete. The first PPI 3 email is sent 24 hours after completing PPI 2 AND PPI 3 is incomplete OR 12 days after completing consent AND PPI 1 is complete AND PPI 3 is incomplete. The second PPI 1 email is sent 5 days after completing primary consent AND PPI 1 is incomplete. The second PPI 2 email is sent 5 days after completing PPI 1 AND PPI 2 is incomplete. The second PPI 3 email is sent 5 days after completing PPI 2 AND PPI 3 is incomplete OR after 17 days of completing consent AND PPI 1 is complete AND PPI 3 is incomplete.

The reminder + content + impact arm includes up to 6 reminders that include a CTA at the top as well as information about the survey content and its impact for AoURP researchers. In this arm, each PPI survey has two email alternatives for a total of 6 reminder emails. The first PPI 1 email is sent 24 hours after completing primary consent AND PPI 1 is incomplete. The first PPI 2 email is sent 24 hours after completing PPI 1 AND PPI 2 is incomplete. The first PPI 3 email is sent 24 hours after completing PPI 2 AND PPI 3 is incomplete OR 12 days after completing consent AND PPI 1 is complete AND PPI 3 is incomplete. The second PPI 1 email is sent 5 days after completing primary consent AND PPI 1 is incomplete. The second PPI 2 email is sent 5 days after completing PPI 1 AND PPI 2 is incomplete. The second PPI 3 email is sent 5 days after completing PPI 2 AND PPI 3 is incomplete OR after 17 days of completing consent AND PPI 1 is complete AND PPI 3 is incomplete.

The proportion of participants that complete the respective surveys (PPI 1, 2 and 3) relative to those that complete primary consent after receiving one of three reminder email interventions will be calculated. This measure will assess which intervention was the most successful at improving the conversion rate for the PPI 1-3 surveys.

The study will be active in the AoURP dashboard for an estimated nine weeks allowing for individuals to participate in the study.

Update made to registration on 09/22/22:
See additional documentation for message content and trigger conditions of the PPI 5-6 messages.

Once participants become eligible for the study, they are randomly assigned a number between 1 and 3, which corresponds to one of the three trial arms. We are overweighting assignment into the treatment. Specifically, for each participant, we generate a random integer ranging from 0 to 4. Individuals being assigned 0 are in the control condition, those assigned 1 or 2 go into the pure reminder condition and those assigned 3 or 4 go into the reminder+ condition.

Update made to registration on 09/22/22:
Participants are re-randomized into a Pure Reminder or Reminder+ group for the PPI 5-6 reminder series. They will be assigned to either group with equal probability.

The unit of randomization is at the individual participant level. This happens when participants complete primary consent.

No

N/A

Approximately 21,700 persons Update made to registration on 09/22/22: Traffic to the platform has slowed since launching the study. We are therefore revising our target sample size to 15,750 persons.

Approximately 4,340 participants in the control condition and 8,680 for each of the treatment conditions (17,360 participants for the pooled comparison).

Update made to registration on 09/22/22:
We are seeking to obtain approximately 6,300 in each of the two treatment arms.

We make the following assumptions for our calculations: - We use standard parameters for the alpha and beta thresholds (0.05 and 0.20, respectively). This beta threshold translates to statistical power of 80%. - The baseline conversion rates and estimated weekly traffic are based on platform data analyzed between December 2019 and November 2021 (see section ‘Baseline data’ above). We run calculations for a baseline conversion rate of 50%, which reflects the overall PPI 1-3 conversion rate under the current set of reminders. - We assume three experimental conditions. - In our comparison of the current set of reminders and new reminders, the two treatment conditions will be pooled. Note that we are weighting treatment assignment, with 40% going into each of the treatments and 20% going into the control. We do this to power the comparison between treatments to a smaller effect size. We expect the effect size to be larger between the control and the pooled treatment conditions. This is why we are comfortable allocating more participants to the treatments. In addition, while our confidence intervals for the estimate of the control condition will be larger, the ones for the pooled treatment condition should be more precise as a result of these conditions having more observations. A sample size of 21,700 persons we selected an MDES of ~2.1 pp as a conservative estimate. We believe that this effect size is reasonable for the comparison of the current set of reminders against the expanded set (for which observations from treatments 1 and 2 would be pooled). Whether we will be able to clear the MDES for the comparison between treatment 1 and 2 is less certain. Update made to registration on 09/22/22: With 6,300 persons in each of the treatment arms, we are powered to detect a difference of around 2.5 percentage points over a baseline completion rate of 50%.