The effects of incentives in non-routine analytical team tasks

We implement three across-subject treatments: a control condition, an incentive condition in which a bonus incentive is framed as a gain, and an incentive condition in which a bonus incentive is framed as a loss.

Additionally, we perform three within-subject treatments (described in the experimental details).

2021-08-16

2021-10-31

Our primary outcome variables concern the different effort dimensions that may be affected by incentives:

Skill-to-task matching,
Information sharing,
Communication, and
(their interaction with) overall performance.

Skill-to-task matching - primary outcomes:

i) the likelihood of selecting the person with the highest Raven test score (extensive margin) in part III and

ii) the difference between the test score of the selected team member and the highest test score in the team (intensive margin). Using statistical tests and regression analysis, we will study whether these outcomes differ significantly between control and incentive conditions. Our design further allows us to investigate the causal effect of incentives on the finishing time in the individual logical sub-task in part III. Finally, to illustrate the relationship between cognitive skills and performance in part III, we plan to test for the correlation between measures of logical
skills and performance in the respective sub-task.


Information sharing - primary outcomes:

i) the number of team members sharing information (which is received at the beginning of the experiment) in part II.2 (the "counting eagles" sub-task). We will compare this number across treatments using statistical tests and regression analyses.


Communication - primary outcomes:

i) Within-team-difference in times needed to solve subpart 1.2 and 2.1
Here, we explicitly test whether the effect of incentives works (causally) through communication. To do so, we exploit the within-team variation of the possibility to communicate in the two similar sub-tasks (part I.2 and part II.1). We compare within-team differences in finishing times for the two sub-tasks (Difference_i= time in "with communication"-version - time in "no communication"- version) across treatment conditions, using statistical tests and regressions.


Incentives and overall performance - primary outcomes:

Concerning the robustness of our results from the field experiments, our pre-registered primary outcomes are

i) the time needed to complete the whole team task,
ii) the fraction of teams receiving the bonus across treatment conditions, and
iii) the fraction of teams completing the task in 60 minutes, which we compare using statistical tests and regression analyses.

Skill-to-task matching,
Information sharing,
Communication,
(their interaction with) overall performance, and
Geolocation of team members during the task.

Skill-to-task matching - secondary outcomes:

These secondary measures stem from (post-experimental) survey questions on why a person was selected in part III and transcribed audio recordings. This allows us to judge the effects of incentives on perceived reasons for the selection of team members and the content of communication preceding the selection decision from the beginning of Part III onward (using text analysis).
Further, we plan to study the time teams devoted to choosing a member for the individual task across treatment conditions.


Information sharing - secondary outcomes:

To analyze the causal effect of incentives on performance through information sharing, we will also compare:

i) the time needed to complete part II.2 with and without incentives.

Further, to illustrate the relationship between helpful information sharing and performance in part II.2, we will test for

ii) Spearman's correlation between the number of team members sharing information and performance (time spent) in part II.2.


Communication - secondary outcomes:

i) the spoken time related to the team task divided by the overall spoken time,
ii) the overall spoken time divided by the total time spent in the team task, and
iii) the spoken time related to the team task divided by the total time spent in the team task.

We plan to compare these measures across treatments using statistical tests and regression analyses which allows us to study how incentives affect the extent of relevant communication and the total extent of communication within the team task.

Incentives and overall performance - secondary outcomes
Additionally, we plan to test whether teams differ in their completion times at each consecutive sub-task in parts I and II depending on whether or not they face incentives.

Geo-location of team members during the task
We also track where team members are standing during the task using a tracking system (recorded for each team member every second) and plan to analyze in exploratory analyses whether incentives affect efforts in movement or closeness of team members.

Our experiment is based on a board game version of a real-life escape game. The setting resembles the same important features as our field study while it is also implementable in the laboratory. We alter the board game to explicitly study the causal effects of team incentives on the three effort dimensions respondents of a survey conducted earlier rated most important:

First, we test if incentives causally affect whether teams assign the most skilled team member to a specific sub-task (skill-to-task matching) and analyze how this subsequently maps into performance.

Second, we investigate the causal effect of incentives on the likelihood of team members sharing relevant information (information sharing) to facilitate task completion. Additionally, we show how more or less information sharing subsequently translates into completion times.

Third, we study the causal effect of incentives on communication and provide additional causal evidence on how communication subsequently maps into performance.

Following our previously conducted natural field experiment, we also implement three treatments across subjects. In BGControl, teams are not provided with any monetary incentives and have 60 minutes to complete the task for a flat payment of 15euro (irrespective of success). In BGGain45, subjects can receive an additional team bonus of 30 euro if they succeed in completing the task in less than 45 minutes, and treatment BGLoss45 implements the same treatment but with a loss frame. Teams always consist of three participants.

The experiment consists of three parts (part I, part II, and part III), and each part contains several sub-parts (e.g. part I.2 denotes subpart 2 of part I). Before the start of the experiment, participants undergo Raven's progressive matrices test. Upon completion of the test, participants receive feedback only about their own test score. Afterward, they receive several materials, and each team member additionally receives a text that contains (different) information which can be shared with other team members in part II.2.

The information to share is part of a longer text with otherwise entertaining yet not helpful information and participants will not be able to deduct that some information may be valuable until they reach part II.2.

Subjects then commence with part I.1. The exact design and tasks of all parts are described in the additional (hidden) section Experimental Design. After completion of this part, subjects advance to part I.2 and subsequently to part II.1, which we use to study the effects of incentives through verbal communication. We deliberately design parts I.2 and II.1 to be similar, yet challenging to subjects, and prohibit subjects to use verbal communication in one of the two parts. This is announced only at the beginning of the respective part and enforced by the threat of exclusion through the experimenter. We then compare performance in part I.2 relative to performance in part II.1 between our incentive treatments and control. The (exogenously created) absence of verbal communication thus allows us to determine the causal effect of communication on performance, and whether this effect is dependent on the provision of monetary incentives in a difference in differences analysis. To avoid order effects, we randomly vary whether verbal communication is prohibited in part I.2 or in part II.1, and additionally flip the two parts for half of all sessions.

In part II.2, subjects can share the information distributed before the start of the experiment. Importantly, the information provided is sufficient, but not necessary to arrive at the correct solution. Alternatively, subjects can decide to not share their information and use the materials provided to work on the part's solution. By comparing the differences in how much information is shared across treatments with and without incentives, we can determine the causal effect of incentives on information sharing. Additionally, we will be able to see to what extent information sharing correlates with eventual performance in task II.2.

In part III, subjects are asked to select one team member for a task that requires logical reasoning. Participants may disclose their score from the Raven test taken prior to the experiment to aid the team's choice. They are, however, not explicitly reminded of the test and do not receive suggestions that this test may have predictive power for performance in the task to follow. While the chosen team member works on the logical reasoning task, secluded from the rest of her group, the other team members work on other tasks needed to finish the experiment. By comparing the differences in cognitive test scores of the team member chosen and the team member with the highest score between our incentive conditions, we can estimate the causal effect of incentives on skills-to-task matching. Additionally, we can correlate the aforementioned test differences with subsequent performance in the logical reasoning task. After the conclusion of part III, subjects fill in some questionnaires and are paid and subsequently dismissed from the laboratory.

To avoid time trends in the data affecting our results, we plan to run three sessions concurrently, such that each treatment will be conducted at the same time and day.

We plan to recruit 360 participants to form 120 teams consisting of three members each and run the experiment in Munich (Germany), Tilburg (The Netherlands), or elsewhere (depending on where Covid-19 restrictions allow us to recruit participants and run the experiment). Akin to the earlier framed field experiment, we assign two-thirds of teams to the incentive treatment (40 to BGGain45, 40 to BGLoss45), and one-third to BGControl. With a statistical power of 80 percent, this enables us to detect relevant effect sizes for the most important treatment comparisons: the effects of incentives (pooling BGGain45 and BGLoss45) on communication, information sharing, and skill-to-task matching. For instance, assuming similar performance distributions as in our framed field experiment, we are able to detect reductions in completion times of 3 minutes or larger.

The non-routine team task is framed as a secret mission, in which participants need to gain access to the palace of the leader of a fictitious country (part I), find some secret information in the palace (part II), and escape (part III), all within 60 minutes.

As participants arrive in the laboratory, they are randomized into teams of three and then assigned to different rooms (with treatment having been randomized across rooms as well).

After receiving instructions from the experimenter, participants undergo a cognitive skill test at separated workstations (we will use Raven's (1998) progressive matrices). Subjects receive their own test scores as private information and are not informed about their team members' performance in the test.

Then, participants start working jointly at a table with a tablet computer placed in the middle. The tablet computer serves to transmit task solutions by participants but also displays hints for progressing in the task at pre-specified times. Therefore, all teams receive the exact same hints at the same time, a feature adapted from the underlying board game.

Subjects receive pen and paper, a decoding sheet, and each team member receives a text containing information about the layout of the leader's office in the palace. This text mostly contains useless but entertaining information, but also, and different for each team member, some information that helps to find the solution to part II.2. Participants are not told about the value of combining their personal information.

After subjects indicate to be ready to commence the experiment, a 60-minute clock is started on the tablet computer and they receive an envelope containing the materials for part I.1. These are a name tag with an empty field at the bottom titled `personal code', an invitation letter to the palace opening containing the information to 'bring your personal code', another decoding sheet displaying a matrix of numbers, several keys, and a white paper strip with small dots and stripes on both sides. The tablet computer asks subjects to enter their personal code, which can be found by combining the dots and stripes shown on both sides of the paper strip. The resulting pattern can then be decoded (using the decoding sheet distributed initially) to obtain the personal code.

After subjects have entered their personal code, they receive an envelope with the materials for part I.2. The materials consist of 5 different flags, an invitation card reminding subjects not to speak (if communication is prohibited in part I.2), a text of the country's national anthem, and a note from the country's leader, saying that the combination of the country's flag and the personal code will yield the solution to part I.2. To arrive at the solution, participants must study the anthem's text to identify the correct flag. They can then use the decoding sheet from part I.1 to identify the correct four-digit number needed to solve the quests in part I.2. Using the keys handed out in part I.1 (which bear single-digit numbers), subjects need to select the four keys (in the right order) on the tablet computer to end part I.

After they managed to do so, the experimenter distributes materials for part II.1. In part II.1, participants receive information cards for five different fictitious enemy countries (with a map of each country and some basic info such as GDP, the strength of armed forces, and other information), a solution sheet containing a matrix that will yield two of the four correct keys to terminate part II, and a speech by the leader describing the country he considers to be the greatest enemy (containing a reminder not to speak should verbal communication be prohibited in part II.1). Selecting the greatest enemy country can be achieved by combining clues from the speech with the information on the country information cards and then making use of the matrix on the solution sheet.

Verbal communication is randomly prohibited in either part I.2 or part II.1.
This ban is implemented by the experimenter under the threat of exclusion and the experimenter also announces as soon as communication possibilities are restored. In half of all sessions, the contents of part II.1 and part I.2 are exchanged to avoid order effects.

After part II.1 is completed, participants receive materials for part II.2.
These are a picture of the leader's office, as well as instructions to `count the golden eagles' displayed there, as well as a sheet translating Roman into Arabic numerals. Participants can simply search for all golden eagles in the picture, but they can also arrive at the solution by sharing the information they received prior to the experiment. Two of the three participants received information about the number of golden eagles in certain parts of the room at the beginning of the experiment, which combined yields the total. This number, translated into Roman numerals yields the last two keys, as all keys (in addition to single-digit Arabic numbers) also each bear a Roman numeral. Entering all four keys on the tablet computer ends part II.

For part III, subjects are asked to select a team member for an individual task requiring logical reasoning. They are not reminded of the Raven test results obtained before the experiment and are not made aware of a possible correlation between the ability to perform in the individual task of part III and this test. They may, however, themselves take the initiative and discuss the results if they so wish. After the team decided on a member, this member is moved to a secluded desk, where he/she receives materials and instructions. The individual task requires sorting 8 picture cards (with pictures on both sides) to be sorted into a 2 times 4 matrix based on a number of logical statements accompanying the instructions (e.g. `the green flower pot can never be next to the green portrait'). By combining all statements, only one possible solution for arranging the picture cards remains.

Meanwhile, the remaining two group members work on a variety of diverse tasks. They need to detect a pattern in a sequence of numbers and continue the sequence, find an object hidden in a stereoscopic image, arrange keys in a specific fashion so they form the shape of a number, and use a key to follow a drawn path on a paper slip to unveil some letters. The solutions to these four tasks yield the four keys to ending part III and thus the game, while the solution to the individual task done by the third team member yields the order in which the keys have to be entered.

After participants have entered the correct four keys (or if the 60 minutes have expired, whichever occurs first), the task ends.

Participants are then asked to fill in some questionnaires. The questionnaires include questions on why a specific person was chosen for the individual task in part III, the questions on leadership used in the framed field experiment, a rating of the statements regarding all ten dimensions used in a survey we previously conducted, as well as general demographics such as age and gender and experience with escape room (board) games. If participants were assigned to a bonus condition and managed to (did not manage to) complete the task within 45 minutes, they then receive the bonus in BGGain45 (hand the bonus back in BGLoss45). All participants also receive the participation fee and are subsequently dismissed from the laboratory.

We will run all three planned treatments (BGControl, BGGain45, BGLoss45) in parallel (if recruitment allows for doing so) in three different rooms. This randomization is done at the individual level (random assignment of each participant to a three-person team and thereby to one of three rooms, participants draw a card when arriving at the lab). Further, we randomly assign different orders of parts 1.2 and 2.1 as well as whether communication is not allowed in parts 1.2 or 2.1 to different rooms. This randomization is done in the office by a computer.

Each team of three is randomly assigned to either BGControl, BGGain45, or BGLoss45.

Each room is randomly assigned different orders of part 1.2 and 2.1 as well whether communication not allowed in part 1.2 or 2.1 to different rooms.

No

120 teams (360 individuals)

120 groups (360 individuals)

40 teams (120 individuals) in BGControl, 40 teams (120 individuals) in BGGain45, and 40 teams (120 individuals) BGLoss45

Our power calculations are based on earlier findings of a framed field experiment and on assumptions about the data generating process and performances in the respective sub-tasks of the current study: Using a sample of 120 groups allows us to identify pooled incentive effect sizes of about 0.547 standard deviations in two-sample t-tests with a statistical power of 80 percent at the five-percent significance level. That is, if we observe similar finishing times and variances as in the framed field experiments, we can identify effects of incentives (pooled) on the remaining time that are larger than 3 minutes and 13 seconds. Power is expected to be lower for binary outcomes such as finishing within 60 or 45 minutes. Using a Chi-squared-test, we can identify effect sizes larger than 17 to 27 percentage points, depending on the fraction of subjects finishing the task in BGControl within 45 or 60 minutes. In order to calculate power for the effects of incentives on the three different effort dimensions, we have to make additional assumptions on finishing times in the respective sub-tasks, as well as on the fractions of teams in BGControl who choose the best-skilled person for the individual logical task (in Part III), and the fraction of teams that decide to share the relevant information in part II.2 (the "counting eagles" sub-task). For example, concerning information sharing, we can identify increases in the share of teams, in which all team members share relevant information, of at least 22 percentage points when assuming that 10 percent (or 65 percent) of teams do so in BGControl and similarly for skill-to-task matching when focusing on teams choosing the best-skilled team member. Measuring the effect of incentives on skill-to-task matching, we will proxy cognitive ability by either performance in Raven's progressive matrices (Raven, 1998). Using a short version of Raven's progressive matrices (12 out of the original 48 matrices) and assuming an identical mean and standard deviation in Raven scores (9.36, 2.41) as observed in Kocher, Trautmann, and Xu (2019) (who also used a student sample) we are able to identify effects larger than 1.32 solved matrices. Concerning the effects of incentives on communication, assuming control teams spend about 60 percent talking about task-related issues (with a standard deviation of 25 percent), we can identify differences in means larger than 14 percentage points. Finally, concerning the effects of incentives through communication, we calculate power based on simulations (1000 repetitions) for a Wilcoxon rank-sum test, two-sample t-test, and a regression that takes task fixed effects into account. We assume that teams in BGControl need on average 6 minutes to complete the no communication sub-task (with a standard deviation of 1.5 minutes). With communication, we assume that teams need 40 seconds less, i.e. they need 320 seconds (with a standard deviation of 60 seconds). In addition, our simulation includes potential task fixed effects (flag vs. countries task), which we assume to be normally distributed around zero with a standard deviation of 30 seconds. Finally, our simulations take into account that groups performing fast on the first task (no communication) are likely performing also fast on the second task. Assuming that performance with communication will be determined by 70 percent through performance in the no-communication task and by 30 percent through the assumptions on the communication task, our approach allows us to identify reductions larger than 30 seconds in the communication sub-task, conditional on incentives reducing the time in the no-communication task only mildly (at most by 5 seconds). Given our assumptions on standard deviations in finishing times for the no communication sub-task itself, we can identify pooled incentive effects larger than 27.5 seconds in the latter using a two-sample t-test.