Survey Incentives

Last registered on April 01, 2022

Pre-Trial

Trial Information

General Information

Title
Survey Incentives
RCT ID
AEARCTR-0006138
Initial registration date
July 17, 2020

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
July 20, 2020, 11:38 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
April 01, 2022, 11:03 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
Harvard Business School

Other Primary Investigator(s)

PI Affiliation
Harvard
PI Affiliation
Harvard

Additional Trial Information

Status
In development
Start date
2020-07-14
End date
2022-04-16
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Survey response rates are often low. To encourage survey completion, participants are sometimes offered financial incentives. In a survey of small business owners, this project explores the impact of incentives on survey responses. Business owners will be offered one of four treatments: (1) in the control group, business owners will receive a message encouraging them to complete the survey; (2) in the gift treatment, business owners will receive a similar message and will also be offered a gift card prior to starting the survey with no conditions attached; (3) in the incentives treatment ($25), business owners will be offered a similar message and also a gift card of the same dollar amount, but to be paid after survey completion (and conditional on completing the full survey); (4) the large incentives treatment ($50) is the same as (3) but with $50 instead of $25. We will analyze the survey response rates and survey completion rates under each condition, as well as the characteristics of businesses responding under each condition. We will also investigate messaging effects for repeat users in a panel survey and new users who are being recruited to join a panel. The experiment uses a partially adaptive design.
External Link(s)

Registration Citation

Citation
Cullen, Zoe , Michael Luca and Christopher Stanton. 2022. "Survey Incentives." AEA RCT Registry. April 01. https://doi.org/10.1257/rct.6138-3.0
Experimental Details

Interventions

Intervention(s)
We are conducting a multi-wave panel survey on small business owners’ perceptions and actions in the face of COVID-19. This research evaluates different inducements for survey participation.
Intervention (Hidden)
The survey respondents were recruited from a subset of Alignable <www.alignable.com> users who opted to participate in a HBS research survey. Among the users who opted in, about 940 responded to a first survey, where their email addresses were stored for contact in future waves. A different group of users who did not participate in the first wave survey are also considered, but with a smaller set of treatments.

We have piloted baseline response rates by sending 30 test messages with no incentives to the group who participated in the first survey.

The remaining ~910 users will be allocated into 4 groups. Group 1 is the control, with no incentive to take the survey. They will receive the email asking that they take the 10 minute survey that they originally agreed to complete as part of the research. Group 2 is the “gift exchange” group, that will receive the link to a $25 gift card as our thanks for later taking the survey. Group 3 is the “incentive group”, where the email will indicate they will receive a link to a $25 gift card after completing the full survey. Group 4 is the “large incentive group”, where the email will indicate they will receive a link to a $50 gift card after completing the full survey.

We have a budget of about $30,000 for incentives. Alignable has agreed to send emails to new users as well, but the proportion receiving contingent $25 incentives will be determined based on the remaining budget after observing responses (and expenses) in the initial wave.


This section is updated to the protocol came about on September 30, 2020 because we have budget leftover to offer incentives in a second round. This second round builds in lottery incentives versus baseline incentives. We will have 4 groups: a group with no incentives, a group with $25 conditional on completing the survey, a group with a $25 lottery, and a group with a $50 lottery. This second wave has about 21,091 total participants, but about 9,000 of them are core participants with a response to a prior survey where we wish to track follow up outcomes.

For the second wave, we have a bit over 21,091 subjects who are in the pool of eligible emails. 3,893 of these users participated in the prior survey wave. (An additional 5,000 users are of special interest for a different project because they participated in a different core survey where we intend to link responses over time.) We have 525 $25 gift cards available to allocate to different treatment arms, but logistics to target the special interest users versus the other users will be difficult. Instead, we will randomize across arms, but the randomization will have different allocations because the lottery is less expensive than a conditional $25 incentive for everyone.

The treatment arms will have the following numbers of participants: 525 will be in the arm where users receive a $25 gift card upon completing the survey. 1400 will be in the arm with a $25 lottery conditional upon completion. 1400 be in the arm with a $50 lottery conditional upon completion. The remainder will be in the control group that gets a thank you message.

On April 2, 2022 an additional survey will go live to a sample of about 100,000 Alignable businesses chosen at random from the Alignable database. All users will get a solicitation email asking them to take the survey. A small number of users will be chosen at random and their message will indicate they will be eligible for a $25 gift card upon survey completion. A different randomly chosen group will be informed they will be entered into a drawing for a $25 gift card.
This survey is designed to test whether the incentives attract users with different opportunity costs, as measured by hours per week spent working or on household tasks like childcare. We will also test motivation by looking at free text responses and how these differ between groups. One free text response is about government policy toward business owners, which will allow us to measure respondents’ desire to express their viewpoint. We would expect incentivized respondents will supply lower effort (measured by response length and time spent) on this question. A second free text question is about vaccine policy, a polarizing question, which will allow us to test whether non-incentivized respondents have more extreme positions. We will operationalize this using sentiment analysis.
Intervention Start Date
2020-07-20
Intervention End Date
2022-04-14

Primary Outcomes

Primary Outcomes (end points)
Our primary outcomes of interest are the response to the email by clicking through to the survey, survey completion in this wave, and differences in the distribution of user characteristics to complete the survey.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Time spent in the survey. Survey response and completion in future waves of the multi-wave survey. In the final survey in the Alignable pool, we will also look at measures of expressiveness and sentiment.
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Please see details under intervention.
Experimental Design Details
Randomization Method
In office on computer
Randomization Unit
Individuals
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
910
Sample size: planned number of observations
910
Sample size (or number of clusters) by treatment arms
227
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
Harvard University
IRB Approval Date
2020-06-23
IRB Approval Number
IRB20-0465
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials