Preferences, beliefs, and demand for vaccines

Details of the interventions are kept in the Hidden Section while the study is in the field.

Abstract: Seasonal diseases such as influenza, COVID-19, and Respiratory Syncytial Virus infect hundreds of thousands of Americans annually, and kill tens of thousands. Safe vaccines exist to reduce the incidence of these diseases, yet demand for these vaccines is low. There is reason to think that vaccine take-up is influenced by perceptions of effectiveness, side effects, aversion to needles, and convenience factors. However there is little evidence on people’s beliefs about these factors, and no research has quantified the contribution of these factors to vaccine take-up. This study will consist of two survey waves, with the surveys administered over the internet. In the first wave, we will use hypothetical choices in an internet survey to assess the roles of effectiveness, side effects, needles, newness, and price in vaccine take-up. We will also ask people what they believe about the effectiveness and side effects of available vaccines. To validate these hypothetical choices, we will also offer subjects incentives to get vaccinated against the flu, at their usual health care provider or pharmacy. We will not administer these vaccines, or otherwise intervene with subjects beyond administering the survey. In a follow up survey, we will ask subjects to upload anonymized proof of their vaccination, reward them if they have gotten vaccinated, and ask some questions about their experience with the vaccination.


A large part of the study involves measurement of preferences, which is described below in the experimental design. There will, though, be three randomized portions of the study that would be considered interventions.

1. We will randomize incentives for getting vaccinated for the seasonal flu among participants who have not already been vaccinated by the time of our first survey. To do this, at the end of the survey participants will make 5 choices between bonus-payments for the follow up survey. In each choice one of the options is a $1 bonus that does not depend on vaccination and the other option is a higher bonus amount that will only be paid if the subject uploads verification they were vaccinated at the second survey. The vaccination-contingent bonus amounts are {$1.50, $3, $10, $20, and $50}. We will randomly select one of these choices to be implemented, with the randomization probability split 49% on $1.50, 1% on $3; 25% on $10, 24% on $20, and 1% on $50. This will create random variation in the incentive amount conditional on choice patterns.

2. The second intervention occurs within the first survey. All subjects will be asked to estimate some statistics about flu prevalence and vaccine effectiveness for the general U.S. population from the prior year. Subjects will be randomized with 50% chance to see estimates of the true rates of these statistics prior to answering questions about their beliefs about flu incidence, vaccine effectiveness, and vaccine side-effect rates for themselves.

3. At the end of the first survey we will randomize individuals who were offered the incentive bonus questions and who indicated a WTA < $50 to a "nudge" treatment with probability 50% that encourages them to make a plan for their vaccination if they want to get vaccinated and provides links to locations where they can schedule vaccines.

2024-11-04

2024-12-16

1. Conjoint-model estimated WTA for vaccination at both subjective and objective rates for vaccine characteristics.
2. Reported belief in percentage reduction in chance of getting the flu with vaccine (perceived effectiveness)
3. Reported belief in difference in likelihood of experiencing illness consistent with severe side effects if vaccinating relative to if not (perceived severe side-effect rate).
4. WTA switching point for vaccination in incentivized bonus choices.
5. Vaccination status at end-line survey
6. Reported experience of illness consistent with vaccine side effects at endline survey.

The WTA measure for vaccination will come from estimating a discrete choice, mixed logit model on the hypothetical vaccine choice data. This estimation will include the vaccine characteristics varied in the conjoint as characteristics, including an inside option indicator.. The estimated parameters of the mean and variance of the weights on these features will be based on demographic characteristics of age (binary split at age 50), education (binary split at college+), and a total score on a two-question vaccine hesitancy scale. Our Bayesian estimation procedure results in an individual specific distribution of preferences. We also elicit individual-specific beliefs about prevailing vaccine characteristics. Given a draw of preferences and an individuals’ beliefs, we calculate the expected utility from each option in each choice (taking expectation before logit error is known). Integrating over individual preference distributions, we obtain choice probabilities for each group.

Vaccination status at end-line survey will be measured with two approaches that we consider a bounding:
1. Simply use the self-reported vaccination status in the end-line survey for all individuals.
2. Use the self-reported vaccination status in the end-line survey for those not randomized to incentives. Those randomized to incentives are marked as vaccinated only if their vaccination status is verified in the end-line.

Stated beliefs about own likelihood in absence of vaccination of contracting the flu, having a severe case of the flu, being hospitalized for the flu, and dying from the flu.

Details of the experimental design are kept in the Hidden Portion while the study is in the field.

We will recruit U.S. adult subjects from the online study platform Prolific. The target sample will be residents of states with online systems for verifying vaccination. We will use quota sample to obtain 10% of respondents age 65 or older. See additional experimental design details for exclusion criteria.

In the first survey, participants will answer questions about their current status and intentions about getting the seasonal flu vaccines. The primary measurement collection will use a conjoint analysis to measure preferences for flu vaccination by asking participants to make hypothetical choices between flu vaccines that differ on important features such as vaccine efficacy, severe side-effect rates, delivery mode, convenience, and additional financial incentives for vaccination. We will use the answers to these questions to estimate a discrete choice model for vaccination that allows us to a) analyze the distribution of payments that people would require in order to get vaccinated and b) to conduct counterfactual exercise of vaccination rates under different beliefs about vaccine features, such as efficacy and side-effects rates. We will also elicit beliefs about vaccine efficacy and severe side-effect rates from participants. By comparing average beliefs to clinical-trial data using the conjoint analysis, we can examine the extent to which improved information about vaccines might affect vaccination rates. The elicitation of beliefs will involve a randomized information treatment arm that informs individuals about the true baseline rates of efficacy and side effects in the general population before eliciting the subjects' beliefs for themselves. Finally, in the first survey we will elicit an incentivized measure of WTA for flu vaccination by asking participants who were not already vaccinated at the time of this survey to make choices between bonus payments for a follow up survey that are or are not contingent on the subject showing proof of vaccination in the follow-up survey. The answers to these questions provide a method for validating the predictions from the conjoint analysis. We will randomly select from bonus options and use this randomization to generate experimental variation in incentives for vaccination conditional on choices over incentives. Among those who were presented with the choices about bonus incentives and selected one of the incentives for of less than $50, we will randomize at a rate of 50% to a nudge treatment that encourages making a plan for vaccination and gives links to resources for scheduling a vaccine. In the follow up survey participants will be asked about whether they were vaccinated during the time between the first and second survey and about whether they experienced illness that would be consistent with severe side effects from vaccination.

Planned exclusion criteria and variable selection include:

1. We will drop Prolific ids with multiple survey attempts in Survey Round 1.
2. We will exclude from our primary analysis any subjects who report having a contraindication for flu vaccination.
2. Our primary analysis will exclude subjects who fail either of two attention checks and subjects who have any disagreements between their stated preference for option A vs. B in the paired choice in the conjoint with their choice of vaccine when comparing those same options to an outside no vaccine option.
3. We will drop respondents with missing values for key demographics that will be used in the conjoint estimation: age, education, and two-question vaccine hesitancy scale.
4. The model specification for the conjoint follows Moshary et al. (2023 AERI). Utility for individual i from each choice option will be a function of a constant (=1 for all vaccines), vaccine effectiveness, side effect rate, needle indicator, convenient indicator, and dollar reward. We assume that preference parameters are normally distribution conditional on demographics. Conditional on preferences, the error term will be assumed to be type-1 extreme value. We assume the mean and variance of preference parameters depend on age (above/below 50) , education (less than 4 years college, or not), and vaccine hesitancy (above/below median on total score to our two vaccine hesitancy questions). Individual level parameters will be estimated using the Metropolis-in-Gibbs sampler as described in Rossi, Allenby, and McCulloch (2005). We constrain the coefficient on the reward and vaccine effectiveness to be non-negative and the coefficient on the side effect to be non-positive.
5. We are planning the following tests of the validity of the conjoint model predictions: a) We will simulate the likelihood of selecting a vaccine with parameters set to match objective estimates of flu vaccine characteristics from the conjoint model and compare the average simulated take-up rate to the average stated take-up rate to the matching question in our survey (note this question is not used to estimate the conjoint), b) We will simulate the likelihood of getting vaccinated from the conjoint model and compare the average predicted flu vaccine take-up rate for our sample to the average flu vaccine take-up reported in the BRFSS re-weighted to match the age distribution in our sample. For the simulation using the conjoint estimates as inputs for our primary validation test we will use subjects' subjective beliefs about vaccine effectiveness and severe side effect rates. For the needle indicator we will assume needle unless the subject states in the survey that when thinking about side effects and vaccine effectiveness they were anticipating a nasal spray. For the convenience indicator we will set the indicator to match usual care if the individual states that getting vaccinated would be a minor inconvenience or worse, c) We will use the same parameters as in b to simulate the likelihood of choosing each of the incentivized bonus options at the end of the survey using the estimated error term in the model to account for uncertainty about vaccination when evaluating the bonus options. We will compare the average predicted choice fractions from the conjoint model to the realized take-up rates.
6.To estimate the treatment effect of incentives on vaccination we will focus attention on "complier groups" based on answers to the bonus incentive questions. For these groups we will report the estimated vaccination differential under randomly assigned $10 and $20 incentives relative to randomly assigned non-incentive status. There are multiple possible complier groups based on answers to the choice of bonus options (those who always choose incentive; always choose sure payment; and those who switch multiple times will all be excluded from this part of treatment analysis). Group 1 are those who choose the non-contingent option at incentives $1.50, but choose the incentive at $10 and higher (note: assignment to this group does not depend on answer to $3 incentive). Group 2 are those who choose the non-contingent bonus at $1.50, $3, and $10, but choose the incentive at $20 and higher. For Group 1 can be randomized into control (if the $1.50 incentive is chosen randomly for them), $10 incentive (if the $10 is chosen randomly) or $20 incentive (if the $20 is chosen randomly). Group 2 can be randomized into control (if $1.50 or $10 is randomly selected for them) or $20 incentive (if the $20 is chosen randomly for them). Those receiving random draws of $3 or $50 incentive option will be excluded from the main estimates since these incentive levels are targeted with low probability. We will report the estimated increase in vaccination from each incentive level by estimating the treatment effect for the $10 and $20 incentives for Group 1 and the $20 incentive for Group 2. We will present these both separately and a pooled effect for the $20 incentive pooling Group 1 and Group 2.
7. We will use the randomization to incentives for complier groups (noted above) to estimate ITT effects of incentive offer on reported rates of illness consistent with side effects.
8. We will also report an estimated treatment effect of vaccination on reported rates of illness consistent with side effects for the complier groups. To do this we will use standard instrumental variables techniques for TOT estimation by using randomly assigned incentive levels for compliers as an instrument for vaccination.
9. We will compare that average estimated treatment effect of vaccination on illness consistent with side effects to the average subjective beliefs about vaccine side effect rates among the population of compliers. We will use a 1-sided t-test to test whether the average subjective side effect rate is higher than the estimated true side effect rate.


Notes:
1. Estimate a TE for $20 switch and $10 switch separately and then average together.
2. Median split on the vacc hesitancy -- so 3 binary variables governing variation in the betas and variance.

Computer randomization via random number generator in Qualtrics. Randomization to the information vs. no-information conditions about population-level flu incidence and vaccine effectiveness will be 50% to each arm. Randomization to the bonus incentive questions will be 49% to $1.50 incentive; 1% on $3; 25% to $10 incentive; 24% to $20 incentive and 1% on $50 incentives.

Individual level.

No

Our sample size is implicitly determined by our budget of $36,000. We do not know our cost per subject ex-ante because it depends in part on subject choices. To maximize enrollment within our budget we will recruit in waves that follow each other quickly in time. After each wave we will calculate "maximum cost", which assumes that everyone responds to the wave 2 survey and earns the maximum possible bonus they are eligible (determined by combination of randomization and subject choices). We will continue adding waves until we exhaust the budget. No analysis will be done on the survey 1 data until all waves have been completed. We are anticipating total enrollment of approximately 2,800 subjects.

See note for planned clusters. Will be the same since unit of analysis is individual.

1. The information treatment arms will each target 50% of the total enrolled sample

2. The bonus incentives for vaccination will be randomized among those who state they have not been vaccinated prior to the study and those who are not contraindicated for the vaccine. The share of the overall enrolled sample eligible for randomization is not known for sure, but based on pilot surveys we estimate that it will be 75% (~2,100). From that sample, individuals will be randomized to incentive bonus eligibility at rates of 49% for $1.50; 1% for $3; 25% for $10; 24% for $20; and 1% for $50.