The effect of incentives in a puzzle-solving game

Last registered on January 16, 2023

Pre-Trial

Trial Information

General Information

Title
The effect of incentives in a puzzle-solving game
RCT ID
AEARCTR-0009301
Initial registration date
April 25, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
April 28, 2022, 6:07 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
January 16, 2023, 12:19 PM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation

Other Primary Investigator(s)

PI Affiliation
U.S. Air Force Academy
PI Affiliation
U.S. Air Force Academy
PI Affiliation
U.S. Air Force Academy

Additional Trial Information

Status
Completed
Start date
2022-08-08
End date
2022-12-16
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Competition tied in with self-evaluation can trigger dishonesty in even the most honest person. In many professional settings, including in some parts of the United States military, subordinates are required to provide significant input about their work performance to their superiors as a part of the performance appraisal process.  In some cases, subordinates are expected to draft their own performance appraisals.  These appraisals may then be used to rank-order subordinates to determine who receives a limited number of rewards, such as a “Definitely Promote” (DP) rating of an officer in the United States military. The objective of this study is to examine the effects of two alternative incentive systems designed to simulate alternative performance reward systems on stated performance in a matrix-solving game.
External Link(s)

Registration Citation

Citation
Silz-Carson, Katherine et al. 2023. "The effect of incentives in a puzzle-solving game." AEA RCT Registry. January 16. https://doi.org/10.1257/rct.9301-3.0
Experimental Details

Interventions

Intervention(s)
The objective of this study is to examine the effects of two alternative incentive systems on stated performance in a matrix-solving game . In this game, participants are given a set of 20 matrices containing 12 numbers each. For each matrix, participants are asked to select the two numbers that sum to 10. After completing all 20 matrices, participants will report on a separate page the total number of matrices that they believe that they solved correctly. Participants will be compensated based on their stated number of matrices solved using one of two possible incentive systems. Participants will be randomly assigned to one of two incentive systems to determine their experimental compensation.
Intervention Start Date
2022-08-15
Intervention End Date
2022-09-30

Primary Outcomes

Primary Outcomes (end points)
Outcome 1: Stated performance in Incentive System #1
Outcome 2: Stated performance in Incentive System #2
Primary Outcomes (explanation)
Overstatement is the difference between the number of matrices that a participant claims to solve and the number of matrices that they actually solve (observable based on their responses to the 20 matrix problems). Overstatement = Stated number solved - actual number solved

Secondary Outcomes

Secondary Outcomes (end points)
None
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The objective of this study is to examine the effects of two alternative incentive systems on stated performance in a matrix-solving game . In this game, participants are given a set of 20 matrices containing 12 numbers each. For each matrix, participants are asked to select the two numbers that sum to 10. After completing all 20 matrices, participants will report on a separate page the total number of matrices that they believe that they solved correctly. Participants will be compensated based on their stated number of matrices solved using one of two possible incentive systems. Participants will be randomly assigned to one of two incentive systems to determine their experimental compensation.
Experimental Design Details
The objective of this study is to examine the effects of two alternative incentive systems on stated performance in the matrix-solving game used by Rigdon and D’Esterre (2015). In this game, participants are given a set of 20 matrices containing 12 numbers each . For each matrix, participants are asked to select the two numbers that sum to 10. After completing all 20 matrices, participants will report on a separate page the total number of matrices that they believe that they solved correctly. Participants will be compensated based on their stated number of matrices solved using one of two possible incentive systems:

Incentive System #1: Base rate + Bonus if solve at least 15 matrices
Incentive System #2: Base rate + Bonus if solve at least 15 matrices AND are in the top 10% of subjects assigned to this incentive system (including ties)

Participants will be randomly assigned to one of the two incentive systems. Incentive System #1 is designed to simulate most standard employee performance appraisal systems, in which employees become eligible for performance awards (e.g. time off awards, cash bonuses, step increases in pay) only if they exceed some minimum standard of performance (e.g. In the Federal employee system, employees must rate at least a 3 on a 5-point scale to be eligible for these awards). Incentive System #2 also requires participants to meet this minimum standard, but adds competition in the form of a tournament. It is designed to simulate the competition that occurs in performance evaluation systems that only reward the top n% of employees, such as the P/DP promotion rating system that exists for members of the military. The null hypothesis to be tested is:

H0: The amount of overstatement of performance in Incentive System #1 and Incentive System #2 are the same.

We expect to be able to reject this hypothesis. Although our prior is that there will be more overstatement in Incentive system #2, some prior research (e.g. Cadsby, Song, and Tapon (2010)) has found more overclaims in target-based schemes than in tournament schemes. Thus, there is a possibility that there will be more overstatement in Incentive System #1 than in Incentive System #2. For this reason, we use a 2-sided alternative hypothesis when specifying the parameters of the power analysis.
Randomization Method
Participants will be randomly assigned to one of two incentive systems by a computer.
Randomization Unit
Random assignment occurs at the individual participant level
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
N/A - observations in this experimental design are not clustered
Sample size: planned number of observations
150 individual participants - 75 per incentive system
Sample size (or number of clusters) by treatment arms
75 individual participants in incentive system #1
75 individual participants in incentive system #2
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
A difference of means test of the null hypothesis that the amount of overstatement is the same in Incentive Systems #1 and #2 results in a minimum detectable effect size of 1-1.5 puzzles. The analysis assumes independent random samples and assumes the following parameters, which were drawn from the literature on prior studies that have used the matrix puzzle test. Average overstatement in Incentive System #1: 1 puzzle Standard deviation of overstatement (both Incentive Systems): 3 puzzles Level of significance: 0.05 Power: 0.8.
Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
U.S. Air Force Academy Institutional Review Board
IRB Approval Date
2022-06-15
IRB Approval Number
FAC20220019E

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
November 01, 2022, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
November 01, 2022, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
150 (75 per treatment)
Was attrition correlated with treatment status?
No
Final Sample Size: Total Number of Observations
150 (75 per treatment)
Final Sample Size (or Number of Clusters) by Treatment Arms
75 incentive system #1; 75 incentive system #2
Data Publication

Data Publication

Is public data available?
No

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Program Files

Program Files
No
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials