Honesty Interventions in Online Surveys

The goal of this study is to gain a better understanding of the extent to which (and possibly the conditions under which) low-cost honesty interventions can be used to address dishonesty and shirking among survey participants in online surveys.

2023-03-17

2023-03-23

Responses/Behavior in Game Designed to Elicit Individual-Level Measure of (Dis)Honesty: To measure individual-level dishonesty directly, the survey includes a game with a material incentive to behave dishonestly. Every respondent is provided with a screen introducing the opportunity to earn a bonus. At the top of the screen, the respondent is told: "You have now answered approximately half of the questions in this survey. You now have the opportunity to earn a bonus [reward]" (emphasis in the original). Below this statement, participants are shown an image of two tumbling dice, underneath which they are told: "After you click 'Next', you will be shown an image of six dice. If two of the dice show the same number of eyes, you will receive a bonus as follows: If 1 is the highest number that appears twice, you will get the basic Bonus of € 0.50; If 2 is the highest number that shows twice, you get double the basic bonus: € 1.00; if 3 is the highest number that shows twice, you will get triple the basic bonus: €1.50; etc."
On the next page, respondents are shown an image consisting of the surfaces of six dice. 95% of respondents are shown an image containing two dice showing the number 1 (no other number twice; see Analysis Plan for details). Underneath the image, all participants are asked: "Each of the six dice shows a number of eyes. You get a bonus if a number appears exactly twice. Does a number appear twice on your screen?" [In German: "Jeder der sechs Würfel zeigt eine Zahl von Augen an. Sie erhalten einen Bonus, wenn eine Zahl genau zweimal vorkommt. Kommt bei Ihnen eine Zahl zweimal vor?"] This statement and question is supposed to heighten the ambiguity about whether we have any knowledge of the number shown twice, allowing for the possibility that respondents can behave dishonestly (see below) without detection. The question is followed by a simple Yes – No choice.
If the participant clicks "Yes", s/he gets the immediate (on-page) feedback: "Congratulations! Please remember the number that appears twice." If the respondent selects No, the statement "What bad luck! We are sorry that you did not roll a bonus." appears. Participants who click "No" skip the next page.
On the next page, participants are asked: "Which number appeared twice on the dice? To claim your bonus, select that number from the drop-down menu:" The dropdown menu shown for this question contains the options 1 through 6. As participants select a number from the menu, a statement appears underneath: "The bonus to which you are entitled is € 0.50. This bonus will be credited to you by Mingle/Meinungsplatz after completion of the project (in approx. 10-14 days)" if they select "1"; "The bonus to which you are entitled is €1.00 if they select "2"; etc., such that the specific amount filled in, depending on the number selected, so as to remind participants of the monetary reward available to them for being dishonest. While €3 might be only a modest material incentive, it is twice the "incentive pay" awarded to respondents by the survey company for participating in the survey, suggesting that participants who might deviate from the social norm of honest behavior in response to material rewards should thus be incentivized to behave dishonestly.

For all respondents who were shown the number "1" twice, any deviance from "1" constitutes a direct, individually attributable measure of dishonesty with a range from 1 (low) to 5 (high/maximum level of dishonesty), which can be compared across individuals. We expect respondents who receive the honesty treatment (groups T1 or T3) to exhibit a lower likelihood of behaving dishonestly (reporting that they are entitled to a higher bonus than suggested by the image of the six dice shown to them) and/or lower level of dishonesty (C > T3 ≈ T1). See analysis plan for details.

Outcomes (possible effects of the experimental treatments) will also be measured in six other ways (see Analysis Plan for details):
(1) Time spent on the survey as a whole:
(2) Time spent on each page;
(3) Responses to attention check questions;
(4) Responses to factual attention questions;
(5) Responses to honesty check questions;
(6) Participants' level of (dis)agreement with the items of the Social Desirability–Gamma Short Scale.

See analysis plan for detail on each of the outcome variables. For example, for the first measure, we expect respondents who, in the beginning of the survey, received the attentiveness treatment (T2) to spend more time on average to complete the survey than respondents in the control group (C). As honesty implies thoughtful considerations of the questions and response options, we also expect respondents who have received the honesty treatment (T1) to spend more time on taking the survey than respondents in the control group, though this effect might be less strong than the effect of specifically committing to attentiveness. Theory and prior work does not allow us to develop a clear expectation as to whether, in the combined treatment (T3) the effect of the honesty treatment should be a reinforcing complement or a substitute for the attentiveness treatment, such that we would expect the pattern of time spent to be: T3 ≥ T2 > T1 > C.

For the initial ("pilot") study, we split participants into three treatment groups and one control group of about 250 participants, each. The treatment groups received (1) an honesty treatment, (2) an attentiveness treatment or (3) an honestyXattentiveness treatment.

The experimental design is described in some detail under "Intervention" and "Outcomes" above and in more detail in the Analysis Plan. Additional self-reported outcome measures or conditioning variables are discussed in the Analysis Plan.

Selection into one of the three treatment groups or the control group will be done via a "randomizer" in the survey software Qualtrics (with equal distribution across the four groups). Within each treatment group and the control group, participants are further assigned to subgroups for substantively unrelated questions via nested randomizers.

individual

No

1 country

For initial pilot study: 1000 (+ up to 10%) inhabitants of Germany (Respondi/Bilendi online sample)

ca. 250

to be identified via pilot study