Does monitoring adversely affect worker performance? Evidence from a natural field experiment

A control device is implemented in a natural work environment. We will investigate how workers react to such an implementation of control: Is there a negative behavioural reaction, i.e. do workers lower their effort once control is implemented?

Two groups:
1. Control group - incomplete contract (no control device)
2. Treatment group - contract with a minimum performance requirement (a weak control device)

2018-12-10

2019-01-11

Primary outcome is how workers effort is affected by the random assignment to the control and treatment group. Hence, key outcome variable is workers effort. Effort respectively shirking is measured by the number of clicks on the opt-out button, measuring effort with regard to the incentivised effort dimension, i.e. supply.

Primary outcome variables are POST_OO and delta_OO:
1) POST_OO = number of clicks on the opt-out option after treatment induction (task 2), being a proxy of effort respectively shirking for each individual i, directly observed
2) PRE_OO = number of clicks on the opt-out option in the pre-treatment stage (task 1), a proxy of effort respectively shirking for each individual i, directly observed
3)
delta_OO = POST_OO - PRE_OO representing the difference in shirking frequency between post-treatment and pre-treatment stage, for each individual i, constructed

For a discussion concerning the exogenous variables/regressors, please refer to the attached pre-analysis plan.

Also, we will employ alternative dependent variables (non-incentivised effort dimensions) which are valid proxies for effort respectively shirking, too: First, the number of errors (representing quality) and second, the time used to complete the task.
We also construct further outcome variables - please refer to the attached pre-analysis plan for a detailed discussion.

Please refer to the pre-analysis plan.

The (public) description of the experimental design is kept to its minimum to avoid experimenter demand effects. The (hidden) description will become public once the study is completed and will help you to understand the design in further detail.

In short, in a real labor market, we create two groups: One group receives an incomplete contract and is not subject no any controlling or monitoring device. The other group receives a more complete contract with a minimum performance requirement as a control device. First, all workers are assigned to an incomplete contract (HIT1). In a second stage, workers are assigned to one of the two mentioned groups (HIT2) and perform again a real-effort task. Some days or weeks after the real-effort task took place, workers demographics and social preferences are elicited by employing the Global Preference Survey (Falk et al., 2018).

This study is divided into three separate tasks classified into two parts: Part A includes the field experiment with two separate real-effort tasks to elicit workers effort, while the subsequent Part B elicits workers social preferences.

Part A: Field experiment
The experiment is conducted in an online labor market intermediary: we recruit workers
through Amazon Mechanical Turk ("AMT"). We play the principal or employer ("requester")
and offer a one-time employment contract with a fixed reward in case of a so-called Human
Intelligence Task ("HIT") completion. Agents ("workers") are not aware that they participate
in an experiment and engage in a naturalistic real-effort task commonly posted on AMT:
extracting information out of a picture in order to categorize these. Concretely, we present
workers with pictures from game-play situations of a lacrosse game. We ask workers to extract
the following information out of that picture: the jersey number of the player in the foreground,
the color of its jersey, the total count of light and dark colored jerseys, and the total count of
referees. Pictures vary in the degree of difficulty, requiring a different degree of effort to solve. First, a pre-treatment stage (HIT1) is conducted where all workers are subject to a nocontrol
environment. This stage has a two-fold purpose: first, HIT1 serves a lock-in task with the goal to reduce attrition once treatment is induced. Second, we are able to collect
pre-treatment individual performance characteristics.
Workers are presented 20 pictures. For each picture, workers need to decide whether they
can solve that picture. This is the case if all requested information is visible ("Clear image, all
info visible"-button). Workers can also decide to opt-out. The opt-out button ("Unclear image,
not all info visible"-button) is the truthful response if workers cannot solve a picture; e.g. if
the picture is blurry or the requested information is not identifiable. Such an opt-out option
is very commonly used on AMT and hence natural to workers. The opt-out option allows for
cheap shirking since in HIT1 all workers are automatically paid regardless of their output -
no worker is subject to a control mechanism. Thanks to such a button, we are able to induce
variation in workers’ effort, measured through the number of pictures solved. Once workers have
completed all 20 images, they are paid USD 1 and are granted a qualification on AMT. With
this qualification, they have the opportunity to do a different set of 20 pictures in another HIT.
This is the treatment stage (HIT2) where the contract of workers is varied. The control group
receives the same contract as in HIT1 and is again not subject to any control mechanism. For
the treatment group, a control mechanism in the form of a minimum performance requirement
x is implemented.

 NC - no MPR (control group)
Same incomplete contract as in pre-treatment stage HIT1: no minimum performance
requirement implemented (x=0).
 WC - low MPR
We implement a weak, inefficient control device by setting a low minimum performance
requirement allowing workers to click on the opt-out option relatively often, that is 8
times out of 20 (x=12).
At the end of HIT2, we elicit (i) individual fairness perceptions with regard to the reward
(ii) intrinsic motivation to fulfill the task by asking workers if they play or regularly watch
lacrosse and (iii) an additional variable controlling for the device workers are using.

Part B: Preference elicitation
Some weeks after the field experiment, we will invite all workers who completed the treatment
stage to participate in an academic study (HIT3). In this stage, we will (i) collect demographic
data and (ii) employ the streamlined method of the Global Preference Survey ("GPS"). This
stage is identical for both groups.

Randomization done by a computer software (Otree).

Individual

No

no clustered treatment.

We will recruit 506 workers by setting the number of individual assignments on AMT for HIT1 to 506, anticipating a final sample of 248 subjects (due to attrition).

The resulting sample size yields 124 subjects per group who have completed all three tasks, that is HIT1, HIT2 and HIT3, so in total 248 subjects.

A one-sided two-sample t test power calculation is computed based on the data presented in the attached pre-analysis plan. Power is set to 0.9, while alpha is set to 0.05. Our hypothesis is directional, and that is why we use a one-sided power calculation. The pilot data reveals that subjects in treatment NC click on average 3.147 times (out of 20) the opt-out button, while subjects in WC click it 3.75 times. Hence, the effect size results to 0.603. Standard deviation for NC yields 1.4798, while for WC 1.7413. The resulting sample size (one-sided) yields 124 subjects per group.

Pre-analysis Plan 8 March 2020 - REPLICATION

Pre-analysis Plan 22 Oct 2018

Memo concerning pre-analysis plan 22 Oct 2018 (dated 10 Dec 18)