x

We are happy to announce that all trial registrations will now be issued DOIs (digital object identifiers). For more information, see here.
A randomized experiment to test the effect of truth-telling incentivisation methods on survey responses
Last registered on October 18, 2019

Pre-Trial

Trial Information
General Information
Title
A randomized experiment to test the effect of truth-telling incentivisation methods on survey responses
RCT ID
AEARCTR-0004860
Initial registration date
October 18, 2019
Last updated
October 18, 2019 10:41 AM EDT
Location(s)

This section is unavailable to the public. Use the button below to request access to this information.

Request Information
Primary Investigator
Affiliation
Erasmus University Rotterdam
Other Primary Investigator(s)
PI Affiliation
Erasmus University Rotterdam
PI Affiliation
Erasmus University Rotterdam
Additional Trial Information
Status
In development
Start date
2019-10-21
End date
2020-04-30
Secondary IDs
Abstract
Choice-matching is a recently developed method for eliciting honest responses to multiple choice questions in surveys (Cvitanić et al., 2019). Toussaert (2018) has also recently developed a method for incentivising predictions of one's own future behavior which avoids creating incentives to alter actual behavior. We test both of these methods in a general population survey on physical activity preferences and behavior by randomly assigning respondents to one of three versions of the same survey: one which uses choice-matching and the Toussaert method, another which uses choice-matching only, and a third which uses neither method. We test the effect of these methods on survey responses.
External Link(s)
Registration Citation
Citation
O Ceallaigh, Diarmaid, Kirsten I.M. Rohde and Hans van Kippersluis. 2019. "A randomized experiment to test the effect of truth-telling incentivisation methods on survey responses." AEA RCT Registry. October 18. https://doi.org/10.1257/rct.4860-1.0.
Sponsors & Partners

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information
Experimental Details
Interventions
Intervention(s)
Intervention Start Date
2019-10-21
Intervention End Date
2020-04-30
Primary Outcomes
Primary Outcomes (end points)
1. Test of Choice-matching: Responses to a range of questions on physical activity preferences and behavior, as well as questions on monetary preferences, in waves 1-3 of the survey (the survey is longitudinal and has three waves, each approximately 2 weeks apart).

2. Test of Toussaert method: Response to question eliciting predictions of respondents on their physical activity (in hours) over the coming two weeks made in waves 1 and 2 of the survey (predictions of physical activity are not asked in wave 3). Henceforth we call this question the "PA prediction question".

3. Test of attrition: Respondent attrition rates between surveys.

4. Test of distortion of physical activity behavior: Responses to question eliciting self-reported actual number of hours of physical activity over the previous two weeks (henceforth called the "PA actual question"), accuracy of prediction made in the "PA prediction question".
Primary Outcomes (explanation)
1. Questions relevant for choice-matching include: monetary and physical activity time preferences elicited using choice lists, monetary risk preferences elicited using a lottery task, self-reported actual number of hours of physical activity over the previous two weeks, stated ideal hours of physical activity over the next two weeks, the PA prediction question, stated demand for physical activity commitment devices.
2. Toussaert method: The only question relevant for the Toussaert method is the PA prediction question. For respondents assigned to the Toussaert method group, the PA prediction question will be incentivised using the Toussaert method. The PA prediction question in this case involves asking respondents to predict the physical activity hours of another respondent who is similar to themselves (similar in terms of their responses to other physical activity preferences and behavior questions). These respondents will also receive a secondary qualitative question asking them to state to what extent they predict that their own physical activity over the next two weeks will differ from their stated ideal physical activity for those two weeks ("will be much lower", "will be a little lower"...etc.). Respondents assigned to the other two groups (who don't receive the Toussaert method) will get a PA prediction question asking them to simply predict their own number of physical activity hours. They will not receive the secondary qualitative question.
3. Attrition is the % of respondents who complete a wave of the survey but don't complete the next wave.
4. The accuracy of responses to the "PA prediction question" is calculated by comparing these predictions in wave 1(2) to the responses given in the "PA actual question" in wave 2(3).
Secondary Outcomes
Secondary Outcomes (end points)
Secondary Outcomes (explanation)
Experimental Design
Experimental Design
We randomly assign respondents to one of three versions of the same survey: one which uses choice-matching and the Toussaert method, another which uses choice-matching only, and a third which uses neither method. Further details are provided in the Hidden Experimental Design which will be visible after the completion of the study.
Experimental Design Details
Not available
Randomization Method
Randomization carried out automatically in Qualtrics online survey software.
Randomization Unit
Individual
Was the treatment clustered?
No
Experiment Characteristics
Sample size: planned number of clusters
N/A
Sample size: planned number of observations
Invitations to take part in this survey will be sent out to approximately 65,000 members of the Lifelines general population cohort study in the Netherlands. We estimate that approximately 15,000 respondents will complete all three waves of the survey. We estimate that approximately 20,000 will complete wave 1, 16,000 will complete wave 2 and 15,000 will complete wave 3. 1. Number of observations for tests of choice-matching: Estimated 51,000 completed survey responses (20,000 in wave 1, 16,000 in wave 2, 15,000 in wave 3). 2. Number of observations for tests of the Toussaert Method: Estimated 36,000 completed survey responses (20,000 in wave 1, 16,000 in wave 2). 3. Number of observations for test of attrition: Attrition between wave 1 and 2 - Estimated 20,000 respondents. Attrition between wave 2 and 3 - Estimated 16,000 respondents. 4. Number of observations for test of distortion of behavior: Estimated 31,000 completed survey responses (16,000 in wave 2, 15,000 in wave 3)
Sample size (or number of clusters) by treatment arms
1. Tests of choice-matching: Estimated 17,000 survey responses in each of the three arms (treatment A, B and control) (6,666 per arm in wave 1, 5,333 per arm in wave 2, 5,000 per arm in wave 3).
2. Tests of the Toussaert Method: Estimated 12,000 survey responses in each of the three arms (treatment A, B and control) (6,666 per arm in wave 1, 5,333 per arm in wave 2).
3. Test of attrition: Attrition between wave 1 and 2 - Estimated 6,666 respondents per arm. Attrition between wave 2 and 3 - Estimated 5,333 respondents per arm.
4. Test of distortion of behavior: Estimated 10,333 survey responses in each of the three arms (treatment A, B and control) (5,333 per arm in wave 2, 5,000 per arm in wave 3).
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB
INSTITUTIONAL REVIEW BOARDS (IRBs)
IRB Name
Medisch Ethische Toetsingscommissie METC UMC Groningen
IRB Approval Date
2019-09-03
IRB Approval Number
METc 2019/464