Incentive Contracts, Effort Costs, and Productivity in Teams and Individuals: Experimental Evidence from Computer Programmers

We will study our research question on computer science students hired to complete a computer programming task. We randomized the temperature (control: 24 degrees Celsius or warm:29 degrees Celsius) of the room they perform the task in and whether the task is completed individually or in teams.

Our study has 4 treatment groups as summarized below:

1. Individual based production, control room temperature
2. Individual based production, warm room temperature
3. Team based production, control room temperature
4. Team based production, warm room temperature

2022-10-03

2022-11-15

Programmer effort
Programmer performance
Programmer productivity

Programmer effort will be measured as number of key strokes, mouse clicks, and keyboard strikes in total and over time.
Programmer performance will be measured as total number of successfully completed features, and total number of successfully completed features weighted by number of total feature submissions.
Programmer productivity will be measured as the number of features successfully completed divided by the sum of the total number of clicks and characters typed.

Programmer self-reported satisfaction with performance
Programmer measure of task difficulty

To test our research questions, we will hire undergraduate computer science students enrolled in 4-year college programs in Dhaka to complete a computer-programming job that asks programmers to implement five feature additions to an existing Java-based script. Participants will be given four and a half hours to complete the task. All features can be implemented independently of each other and are designed to be of similar difficulty. Layered onto this program is task tracking that allows us to track effort over time by collecting data on key strokes, pixels scrolled, and clicks participants make each minute. The program also has a tool that automatically checks the functionality of features upon submission, and programmers can submit each feature multiple times until they have confirmation that they have completed it correctly. The number of times each feature is submitted to be evaluated by the programmer is also stored by the program.

We chose the population of Dhaka computer science undergraduate students for our study for several reasons. First, Dhaka experiences very high temperatures in summer; the average maximum temperature in September is 32 degrees Celsius and the average minimum is not much lower at 26 degrees Celsius. Second, the IT sector contributes significantly to Bangladesh’s economy; in 2017, it was estimated that the IT sector in Bangladesh was worth $600 million and accounted for about 250,000 jobs . Ensuring IT worker productivity is maintained as temperatures rise is, therefore, likely to be important to the continued economic development of the country.

Our randomized control trials involve a two treatments that will be randomly assigned across session-rooms. To generate temperature variation in our study, we will randomly assign the air conditioner temperature in each room that participants are working in to be 24 degrees or 29 degrees Celsius. We will run two session-rooms at a time with at most 10 participants per room for both the individual sessions and the team sessions. We have selected the temperature values 24 degrees or 29 degrees to generate sufficient temperature variation across participants, and to ensure that we are not subjecting participants to uncomfortably high temperatures.


Lastly, we will randomly vary whether the task is to be completed in pairs of two programmers or individually. Within the team sessions, pairs will be randomly assigned and will be instructed to work on the task according to a pair programming format where members take turns being the driver (typing) and navigator (coming up with solutions). With 10 participants per session room, each session room assigned to the team treatment will have at most 5 teams. This treatment is intended to establish whether the impacts of higher temperature differ when team work is required relative to when it is not. In the team sessions, bonuses will be paid for joint performance such that if a team successfully implements a feature, both team members will receive an equivalent bonus.

Team and individual conditions was randomized across session days using Stata's generate uniform distribution number assignment command (gen uniform())
Room temperature was randomized across rooms within session days using Stata's generate uniform distribution number assignment command (gen uniform())
Participants were randomly assigned to session days using Stata's generate uniform distribution number assignment command (gen uniform())

We have two levels of randomization to first ensure session types are randomly assigned across outside air temperatures and air qualities, and second to ensure participants are not self-selecting into particular day-types or to session days also being attended by someone they know.
Randomization units:
1. Experimental sessions
2. Individual participants

No

N/A

200 work-unit observations (100 individual workers, 100 pairs of workers) or 300 workers in total.

50 individuals in Individual Production, Control temp; 50 individuals in Individual Production, Warm temp; 50 individuals in Team Production, Control temp; 50 individuals in Team Production, Warm temp

Using data from a pilot we ran to ensure the computer program was running effectively, we observed a mean number of total characters typed among programmers working individually of 13999.55 (standard deviation of 8609.67). With 50 individuals each in the hot and cool rooms, to detect a significant effect 80% of the time, we need a mean difference in number of total characters typed of 4,872, or about 35% change in effort relative to the baseline mean. The mean number of features completed in total was 2.21 (sd of 0.713) so to detect a significant effect on programmer performance 80% of the time, our mean detectable effect size for the effect of temperature on individual productivity of 0.404 features.