A randomized experiment to test the effect of truth-telling incentivisation methods on survey responses

2019-10-21

2021-07-31

1. Test of Choice-matching: Responses to a range of questions on physical activity preferences and behavior, as well as questions on monetary preferences, in waves 1-3 of the survey (the survey is longitudinal and has three waves, each approximately 2 weeks apart).

2. Test of Toussaert method: Response to question eliciting predictions of respondents on their physical activity (in hours) over the coming two weeks made in waves 1 and 2 of the survey (predictions of physical activity are not asked in wave 3). Henceforth we call this question the "PA prediction question".

3. Test of attrition: Respondent attrition rates between surveys.

4. Test of distortion of physical activity behavior: Responses to question eliciting self-reported actual number of hours of physical activity over the previous two weeks (henceforth called the "PA actual question"), accuracy of prediction made in the "PA prediction question".

1. Questions relevant for choice-matching include: monetary and physical activity time preferences elicited using choice lists, monetary risk preferences elicited using a lottery task, self-reported actual number of hours of physical activity over the previous two weeks, stated ideal hours of physical activity over the next two weeks, the PA prediction question, stated demand for physical activity commitment devices.
2. Toussaert method: The only question relevant for the Toussaert method is the PA prediction question. For respondents assigned to the Toussaert method group, the PA prediction question will be incentivised using the Toussaert method. The PA prediction question in this case involves asking respondents to predict the physical activity hours of another respondent who is similar to themselves (similar in terms of their responses to other physical activity preferences and behavior questions). These respondents will also receive a secondary qualitative question asking them to state to what extent they predict that their own physical activity over the next two weeks will differ from their stated ideal physical activity for those two weeks ("will be much lower", "will be a little lower"...etc.). Respondents assigned to the other two groups (who don't receive the Toussaert method) will get a PA prediction question asking them to simply predict their own number of physical activity hours. They will not receive the secondary qualitative question.
3. Attrition is the % of respondents who complete a wave of the survey but don't complete the next wave.
4. The accuracy of responses to the "PA prediction question" is calculated by comparing these predictions in wave 1(2) to the responses given in the "PA actual question" in wave 2(3).

We randomly assign respondents to one of three versions of the same survey: one which uses choice-matching and the Toussaert method, another which uses choice-matching only, and a third which uses neither method. Further details are provided in the Hidden Experimental Design which will be visible after the completion of the study.

We randomly assign respondents to one of three versions of the same survey: one which uses choice-matching and the Toussaert method (treatment A), another which uses choice-matching only (treatment B), and a third which uses neither method (no-incentives).
The following questions are incentivised using choice-matching in treatment A and B, and unincentivised in the no-incentives group: monetary and physical activity time preferences elicited using choice lists, monetary risk preferences elicited using a lottery task, self-reported actual number of hours of physical activity over the previous two weeks, stated ideal hours of physical activity over the next two weeks, stated demand for physical activity commitment devices.
The PA prediction question is incentivised using the Toussaert method in treatment A, incentivised using Choice-matching in treatment B, and unincentivised in the no-incentives group.
Choice-matching is implemented as per the canonical version described in Cvitanić et al. (2019) (Cvitanić, J., Prelec, D., Riley, B., and Tereick, B. 2019. "Honesty via Choice-Matching." American Economic Review: Insights, 1 (2): 179-92.).
The Toussaert method we use is based on the method used by Toussaert (2018) (Toussaert, S. 2018. "Eliciting Temptation and Self‐Control Through Menu Choices: A Lab Experiment". Econometrica, 86(3), 859-889.). We elicit a respondent's prediction of the number of hours of physical activity of a similar other over the next two weeks (as described in the Primary Outcomes section above). The accuracy of this prediction is incentivised using monetary rewards. We use this prediction as a proxy for their prediction of their own physical activity. We use the secondary qualitative prediction of physical activity (as described in the Primary Outcomes section above) to validate this proxy by analysing the correlation between the proxy prediction of a respondent and their predictions as per this secondary qualitative question.
1. To test Choice-matching, we compare Choice-matching incentivised survey responses in treatment A and B to the corresponding unincentivised responses in the no-incentives group.
2. To test the Toussaert method, we compare responses to the PA prediction question in treatment A to the responses to the PA prediction question in treatment B and the no-incentives group. As outlined above, for treatment A respondents we also analyse the correlation between their PA prediction question response and their response to the secondary qualitative question.
3. Test of attrition: We test if the respondent attrition rate in this three-wave survey differs between groups. For example, attrition may be lower in the treatment groups compared to the no-incentives group if the incentives provide extra motivation for respondents to complete each wave. Alternatively, incentives may crowd out intrinsic motivation, creating higher attrition rates in the treatment groups.
4. Test of distortion of behavior: We test for differences between groups in their responses to the PA actual question and also in the accuracy of their responses to the PA prediction question. For example, directly asking respondents for a quantitative prediction of their own physical activity in the PA prediction question (as is done in treatment B and the no-incentives group) may cause respondents to change their physical activity behavior in the subsequent two weeks to match their prediction because of a desire to be consistent. This motivation to change behavior may not be so strong for treatment A respondents, who only make predictions of their own behavior in qualitative terms (secondary qualitative question). If this were the case, we may see a difference between treatment A and the other two groups in responses to the PA actual question in waves 2 and 3 (treatment B and no-incentives may have larger responses if they were motivated to do more physical activity to match their prediction). We may also see a difference in the accuracy of predictions made in the PA prediction question in waves 1 and 2 (treatment B and no-incentives may be more accurate if they changed their behavior to match their prediction).
Note: The tests of choice-matching will be carried out using survey responses from all three waves of the survey. The tests of the Toussaert method will only be carried out using wave 1 and wave 2 survey responses, as the Toussaert method is not used in wave 3, as outlined previously. The test of distortion of behavior will use data from all three waves, but will only use respondents who get as far as completing at least wave 2 of the survey. This is because the relevant outcome variables are responses to the PA actual question in waves 2 and 3, and accuracy of the PA prediction question (PA actual question response in wave 2(3) minus PA prediction question wave 1(2)).

Randomization carried out automatically in Qualtrics online survey software.

Individual

No

N/A

Invitations to take part in this survey will be sent out to approximately 65,000 members of the Lifelines general population cohort study in the Netherlands. We estimate that approximately 15,000 respondents will complete all three waves of the survey. We estimate that approximately 20,000 will complete wave 1, 16,000 will complete wave 2 and 15,000 will complete wave 3. 1. Number of observations for tests of choice-matching: Estimated 51,000 completed survey responses (20,000 in wave 1, 16,000 in wave 2, 15,000 in wave 3). 2. Number of observations for tests of the Toussaert Method: Estimated 36,000 completed survey responses (20,000 in wave 1, 16,000 in wave 2). 3. Number of observations for test of attrition: Attrition between wave 1 and 2 - Estimated 20,000 respondents. Attrition between wave 2 and 3 - Estimated 16,000 respondents. 4. Number of observations for test of distortion of behavior: Estimated 31,000 completed survey responses (16,000 in wave 2, 15,000 in wave 3)

1. Tests of choice-matching: Estimated 17,000 survey responses in each of the three arms (treatment A, B and control) (6,666 per arm in wave 1, 5,333 per arm in wave 2, 5,000 per arm in wave 3).
2. Tests of the Toussaert Method: Estimated 12,000 survey responses in each of the three arms (treatment A, B and control) (6,666 per arm in wave 1, 5,333 per arm in wave 2).
3. Test of attrition: Attrition between wave 1 and 2 - Estimated 6,666 respondents per arm. Attrition between wave 2 and 3 - Estimated 5,333 respondents per arm.
4. Test of distortion of behavior: Estimated 10,333 survey responses in each of the three arms (treatment A, B and control) (5,333 per arm in wave 2, 5,000 per arm in wave 3).