Intervention (Hidden)
The survey respondents were recruited from a subset of Alignable <www.alignable.com> users who opted to participate in a HBS research survey. Among the users who opted in, about 940 responded to a first survey, where their email addresses were stored for contact in future waves. A different group of users who did not participate in the first wave survey are also considered, but with a smaller set of treatments.
We have piloted baseline response rates by sending 30 test messages with no incentives to the group who participated in the first survey.
The remaining ~910 users will be allocated into 4 groups. Group 1 is the control, with no incentive to take the survey. They will receive the email asking that they take the 10 minute survey that they originally agreed to complete as part of the research. Group 2 is the “gift exchange” group, that will receive the link to a $25 gift card as our thanks for later taking the survey. Group 3 is the “incentive group”, where the email will indicate they will receive a link to a $25 gift card after completing the full survey. Group 4 is the “large incentive group”, where the email will indicate they will receive a link to a $50 gift card after completing the full survey.
We have a budget of about $30,000 for incentives. Alignable has agreed to send emails to new users as well, but the proportion receiving contingent $25 incentives will be determined based on the remaining budget after observing responses (and expenses) in the initial wave.
This section is updated to the protocol came about on September 30, 2020 because we have budget leftover to offer incentives in a second round. This second round builds in lottery incentives versus baseline incentives. We will have 4 groups: a group with no incentives, a group with $25 conditional on completing the survey, a group with a $25 lottery, and a group with a $50 lottery. This second wave has about 21,091 total participants, but about 9,000 of them are core participants with a response to a prior survey where we wish to track follow up outcomes.
For the second wave, we have a bit over 21,091 subjects who are in the pool of eligible emails. 3,893 of these users participated in the prior survey wave. (An additional 5,000 users are of special interest for a different project because they participated in a different core survey where we intend to link responses over time.) We have 525 $25 gift cards available to allocate to different treatment arms, but logistics to target the special interest users versus the other users will be difficult. Instead, we will randomize across arms, but the randomization will have different allocations because the lottery is less expensive than a conditional $25 incentive for everyone.
The treatment arms will have the following numbers of participants: 525 will be in the arm where users receive a $25 gift card upon completing the survey. 1400 will be in the arm with a $25 lottery conditional upon completion. 1400 be in the arm with a $50 lottery conditional upon completion. The remainder will be in the control group that gets a thank you message.
On April 2, 2022 an additional survey will go live to a sample of about 100,000 Alignable businesses chosen at random from the Alignable database. All users will get a solicitation email asking them to take the survey. A small number of users will be chosen at random and their message will indicate they will be eligible for a $25 gift card upon survey completion. A different randomly chosen group will be informed they will be entered into a drawing for a $25 gift card.
This survey is designed to test whether the incentives attract users with different opportunity costs, as measured by hours per week spent working or on household tasks like childcare. We will also test motivation by looking at free text responses and how these differ between groups. One free text response is about government policy toward business owners, which will allow us to measure respondents’ desire to express their viewpoint. We would expect incentivized respondents will supply lower effort (measured by response length and time spent) on this question. A second free text question is about vaccine policy, a polarizing question, which will allow us to test whether non-incentivized respondents have more extreme positions. We will operationalize this using sentiment analysis.