Lying about Cheating

The interventions are described in the hidden part of the registry.

2015-08-14

2015-08-28

Fraction of subjects who truthfully report having cheated is the main outcome variable.

Secondary outcome variables include responses to the set of questions included at the end of the survey about corruption, bribery, and expectations.

The average treatment effect of interest is the difference between the aggregate response bias in two experimental conditions.

The fraction of subjects who actually cheat in an experimental condition is the difference between the theoretical distribution of the outcome of the draw, and the self-reported distribution in that experimental condition. (Randomization implies that this should be roughly equal across experimental conditions).

The aggregate response bias in an experimental condition is the difference between the fraction who admit to having cheated, and the fraction of subjects who actually cheated.

These variables are described more precisely in the attached PDF document.

The experimental design is described in the hidden part of the registration for this study.

In this study I propose an experimental design to measure the causal effect of different survey question techniques on reporting bias for a sensitive question concerning cheating. This research design makes it possible to estimate the true aggregate level of cheating – the key quantity that has remained elusive in most previous studies. In the core part of the study, subjects are privately exposed to a draw from a known distribution and they are remunerated in proportion to the value of the draw that they report having obtained (thereby providing an incentive to exaggerate). They are then asked whether they reported their draw accurately or exaggerated. In each experimental condition, subjects are asked this question using a different survey technique: direct questioning, indirect questioning, RRT, and ICT. This makes it possible to estimate the average causal effect of each survey technique on response bias by comparing self-reported cheating with directly-estimated cheating, for each of the experimental conditions.

Laboratory experiments are often criticized on the grounds that experimenter demand effects (EDE) could distort subject behavior. Even when EDE distorts behavior, this provides cause for concern only when the distortion could in principle spuriously produce the key results -- distortions orthogonal to the behavior of interest could be safely ignored (Zizzo 2010). There is good reason to expect that EDE are not a cause of concern in the present study. First, it is possible that EDE could influence incentives to cheat (i.e., to misreport one's draw in order to receive a higher payment). However, the rate of cheating is not the estimand of interest. Moreover, any such distortion would be identical in all the experimental conditions. Second, it is not easy to see how EDE might distort the estimand of interest -- i.e., the difference in the average proportion who report having cheated and the estimated rate of cheating, in each experimental condition. One would have to argue, among other things, that different modes of asking (each associated with an experimental condition) are subject to different degrees of EDE. While I see no evident manner in which this might be the case, I nevertheless take several measures to mitigate potential EDE.

First, non-deceptive obfuscation is implemented by adding a filler section. The filler section makes it very difficult for the subject to figure out exactly what the researcher is after, since both the filler and core sections are plausibly of interest in the context of the study. Second, non-deceptive obfuscation is utilized within the core section of the study, in order to further decrease the possibility that subjects could figure out what the experimenter is after (or even what the central topic of the study is). Third, incentivized questions are utilized in both the core and the filler sections of the study, so that differences in incentives are not a clear indicator of researcher interest. Finally, to avoid EDE stemming from the physical presence of a researcher in the room (e.g., via eye contact, or tone of voice), the study is conducted entirely online.

More concretely, the first section of the study will focus on the use of small change -- a topic of salient historical and contemporary relevance due a traditional scarcity of small-denomination currency in Mexico. This section includes an incentivized question with the structure of a “beauty contest” (asking subjects which coin denomination they believe is most frequent among participants in the study). The second section will ask the subject to toss a coin several times and report the number of heads, providing incentives for subjects to exaggerate. The subject will then be asked whether she misreported. The manner of asking is the treatment variable, and it will vary across the experimental conditions. The four conditions are: direct questioning, indirect questioning, ICT, and RRT. This is the core section. The filler section provides a plausible alternative reason for asking subjects to have a coin at hand. The core section will additionally contain filler material emphasizing the topic of informal decision-making devices (such as the coin-toss or rock-paper-scissors), to provide a plausible alternative reason for asking subjects to toss the coin.

Further details are provided in the attached document and links.

Randomization by computer.

Individuals.

No

I cannot know what the response rate will be. I expect to have 200-300 subjects total.

Design is not clustered; see above.

Treatment conditions 1, 2 and 4 each have a 24% probability of arising; treatment condition 3 has a 28% probability.

Since this pre-registration text precedes the running of the pilot and of the full study, I rely on the literature to obtain a rough sense of the adequate sample size. I focus mainly on treatment effects taking direct questioning (condition 1) as the control condition, and each of the other three conditions as separate treatments. I focus on two studies where the sensitive item concerns some form of cheating, and on a general meta-analysis: • Ocantos et al (2012) study vote buying using the ICT. They find that only 2.4% (s.e.=0.6%) of subjects reported receiving an individual gift in echange for their vote during a 2008 election in Nicaragua when asked directly, but 24% (s.e.=5.5%) reported having received such a gift when asked through the ICT. Assuming equal variances for the treatment and control groups, and supposing the variance is numerically equal to that estimated for the treatment group (a conservative choice in this case), power is close to 1.0 even for a sample size as small as 30 (since the effect size is so large). (Note: the formula I'm using, in Latex code, is: \beta=\Phi(\left[|\mu_{t}-\mu_{c}|\sqrt{N}\right]/2\sigma-\Phi^{-1}(1-\alpha/2)), where \beta denotes power, \Phi is the cumulative Normal distribution, \Phi^{-1} is the inverse of such distribution, \sigma is the standard deviation of the outcome for both the treatment and the control groups, N is the sample size, and \alpha is the level of statistical significance (Gerber and Green 2012)). • Van der Heijden et al (2000) compare RRT with direct questioning in both a face-to-face survey and a computer-assisted survey. The proportion of subjects known to have engaged in income fraud who admitted to it was 43% (s.e.=6.8%), 25%(s.e.=4.4%), and 19%(5.8%) respectively for RRT, face-to-face direct questioning, and computer direct questioning. The effect size here is also so large that even 30 subjects suffices to attain power close to 1.0 (assuming, for example, that the standard error of the outcomes is 6.8%, and choosing 43%-25%=18% as the magnitude of the treatment effect). • In a meta-analysis of RRT, Lensvelt-Mulders (2005) find that the mean percent underestimation of a sensitive item using RRT is 38% (s.e.=.099), while it is 42% (s.e.=.099) in face-to-face interviews, 46% (s.e.=.138) in phone interviews, 47% (s.e.=.14) in self-administered questionnaires, and 62% (s.e.=.191) in computer-assisted surveys. Comparing the rate of reporting of the sensitive behavior under RRT with that under self-administered questionnaires, and taking the standard error to be 0.14, a sample size of 75 is necessary to attain power of 0.8. In sum, while I face considerable uncertainty about effect sizes and variances before running the study, an N of 50 to 70 per treatment condition seems reasonable. It is not clear whether a small pilot study (with 15-30 respondents) would suffice to reduce this uncertainty in a meaningful way. To further improve power, to improve covariate balance, and to reduce the variability, I will estimate treatment effects adding pre-treatment covariates as control variables, and (alternatively) I will implement sequential blocking on pre-treatment covariates (after having collected the data, but without utilizing outcome data for blocking; see Moore and Moore 2011).