Team Production in International Labor Markets: Experimental Evidence from the Field

Last registered on July 06, 2016

Pre-Trial

Trial Information

General Information

Title
Team Production in International Labor Markets: Experimental Evidence from the Field
RCT ID
AEARCTR-0001402
Initial registration date
July 01, 2016

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
July 01, 2016, 9:04 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
July 06, 2016, 9:44 AM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region
Region
Region
Region

Primary Investigator

Affiliation
UC San Diego

Other Primary Investigator(s)

Additional Trial Information

Status
Completed
Start date
2013-01-06
End date
2016-06-30
Secondary IDs
University of Toronto Social Sciences and Humanities Research Ethics Board, protocol reference #28221
Abstract
Co-workers are increasingly diverse in their nationality and skill sets. This paper studies the effect of diversity on how workers are organized using data from a field experiment conducted in an environment where diversity is pervasive. Findings show that team organization improves outcomes when workers are from the same country. The opposite is true when workers are nationally diverse. These results are more pronounced for teams of workers with specialized skills. Further investigation of the data suggests that diverse teams have difficulty communicating despite having a shared language. I find no evidence that preferences for or expectations about working with someone from the same country is affecting team performance.
External Link(s)

Registration Citation

Citation
Lyons, Elizabeth. 2016. "Team Production in International Labor Markets: Experimental Evidence from the Field." AEA RCT Registry. July 06. https://doi.org/10.1257/rct.1402-4.0
Former Citation
Lyons, Elizabeth. 2016. "Team Production in International Labor Markets: Experimental Evidence from the Field." AEA RCT Registry. July 06. https://www.socialscienceregistry.org/trials/1402/history/9286
Experimental Details

Interventions

Intervention(s)
I
Intervention Start Date
2013-01-06
Intervention End Date
2013-06-20

Primary Outcomes

Primary Outcomes (end points)
Team performance, both total output and output per hour
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Identifying the causal impact of national diversity on the value of teamwork is problematic because teams are selected and not randomly assigned. This endogenous team formation makes it difficult to disentangle both the value of teamwork as compared to independent work and the differential impact of teamwork for nationally diverse teams. For example, it is plausible that employers and workers choose which jobs to complete as a team and that they choose who should work together in teams according to their performance expectations. To address these concerns, I test the effects of national diversity on work team success by conducting a field experiment on the world's largest online contract labor platform, oDesk.

To perform the study, I set up an employer account on the oDesk website and posted jobs to attract contractor bids. I hired contractors who met the requirements for participation in the experiment and randomly put them into groups of two to complete a programming task. I define teams in this study as pairs of contractors working towards a common goal.

There are several reasons I chose to run this experiment on oDesk. First, oDesk provides a team room application which allows employers to put hired contractors into teams so that they can work together online. In particular, contractors working in the same team room can monitor what the others are doing and chat with each other through instant messaging. Contractors can only be logged into one team room at a time. Employers on oDesk frequently require that multiple hires work in the same team room. Second, the team room facilitates contractor monitoring by employers through frequent screen shots, memos, work diaries, and activity meters. As a result, the majority of employers on oDesk require that their hires complete their tasks while logged into their team room. Another important feature of oDesk for this research is the option to interview. oDesk encourages employers to interview contractors they are interested in hiring before making any offers. These interviews can take place through messages exchanged on the site. Lastly, oDesk is an international labor market which contractor participants have selected into knowing they would almost surely be working with employers and contractors from other countries. Therefore, we might expect to see less distaste for working with a nationally diverse organization than in more local labor markets.

To test my research question, I needed contractors to be assigned a task for which collaboration is not unnatural. At the same time, because I am comparing team work to independent work, contractors working on their own should be capable of completing at least some portion of the task. In addition, because I am interested in testing the coordination and communication costs associated with national diversity in teams, free-riding on the assigned task should be difficult. The task also needed to be consistent with the types of tasks posted on oDesk so that contractors did not become suspicious about the purpose of the job. Moreover, an objective evaluation of contractor performance was necessary. Finally, it must have been possible to complete the task remotely. Given these constraints, I developed a web development task that required both back-end (PHP) and front-end programming (Javascript) to assign contractors hired for the experiment. Hired contractors were asked to add a list of features in these languages to existing code. This is similar to a web development task for which a group of workers are responsible for both the design and functionality of a web page.

To reinforce the collaboration aspect of this job and to reduce free-riding in teams, I hired one contractor in each pairing to complete the Javascript portion of the task and another to complete the PHP portion. The team rooms further reduce the potential for free-riding because contractors are aware that their individual performance can be monitoring by employers through the team room application. I assigned each pair three features to add to the code, one that required only Javascript programming, one that required only PHP programming, and one that required both. Including both individual tasks and a team task accomplishes two objectives. First, it allows for both independent and for team work so that neither of the organizational form treatments lead to contractor frustration. Second, it allows me to disentangle difficulties associated with collaboration from difficulties associated with the job in general. All hired contractors received the same list of features to be added, and I instructed them to work on the features that corresponded to the language for which they were hired. Nothing in the instructions or the task design explicitly prevents contractors from working on all three features of the task, including the parts that are not in the language they were hired for. The reasons a single contractor may not work on both languages are first, that some contractors are only familiar in coding one of the two languages, and second, that they have been asked to focus on one of the two languages. Importantly, the feature that requires both programming languages to be completed requires that the codes from each language work together. A contractor who only knows one of the two languages would not be able to complete this feature on his or her own. Therefore, contractors in the independent work treatment would not be able to complete all three features unless they are familiar with both PHP and Javascript.

In order for the contractors in the teamwork condition to be able to work with each other, I gave them a total of eight hours to complete the task; they had to work these hours within the same period as the other contractors in their groups.

I randomly assigned each hired contractor to one of four types of groups of two, and all groups were made up of one contractor hired to work on the Javascript code and one to work on the PHP code. The experiment had two treatments: pairs were either from different countries or not and were either permitted to work in teams or not. Pairs in the team treatment worked in the same team room as each other, and those in the independent work treatment worked in separate team rooms and were unaware of who their pair mate was so that team work was not an option. The purpose of this experimental design is to compare the benefits of teamwork in homogeneous teams with the benefits of teamwork in nationally diverse teams. This allows me to identify how organizing workers into teams compares with organizing them as independent workers, and how this is affected by national diversity.

To control for country-specific differences in contractor quality, I limited the number of countries from which I hired. Based on information about the set of applicants my job postings attracted during the pilot phase of my study, I included English speaking contractors from India, Pakistan, and Bangladesh in the experiment. Contractors from these countries frequently applied to my job postings and bid an amount that was within the hourly wage criteria. Moreover, workers in these countries have very similar time zones making it easier to allow for remote team work among them.

Using an employer account on oDesk, I attracted contractor bids by posting two job types. One type of job posting asked for bids from contractors able to code in PHP and the other for bids from contractors able to code in Javascript. Contractors therefore selected into the programming language they preferred. To minimize attrition among hired contractors, the job postings described the type of work the task required, the date the work needed to be done on, and the number of hours contractors had to complete the task. The job postings also specified that the maximum bid that would be accepted ($4.00 USD). The job postings did not mention that the work would be completed in teams nor did they mention anything about country-specific requirements.

The first applicant to each job posting who bid at most $4.00 and who was from one of the countries included in my experiment received a list of interview questions. The purpose of the questions was to get an idea of the level of Javascript and PHP knowledge each contractor had, their English language abilities, and the countries they have lived in. I gave each interviewee two hours to reply to the interview questions before I interviewed the next applicant who met the bid and country requirements. I hired the first interviewee to reply with the exception of interviewees who did not provide pertinent answers due to their lack of English language ability.

Once hired, contractors were sent the appropriate instructions and the file with the code to be edited. There were four versions of the job instructions to reflect whether the contractor was hired for the Javascript or the PHP portion of the job and whether pairs were permitted to work as a team or not. Instructions do not vary across the national diversity treatment.

In addition to describing the features to be added to the code, the instructions noted the timeline for the task and the country that contractors' co-worker lived in because it is not possible to restrict access to this information in the team work treatment, so I controlled for this knowledge across treatment groups to ensure that it was not driving my findings.

I asked contractors who had not turned in their work by the deadline to submit any work they had completed. Once contractors had turned in their work or after the deadline had passed, I asked them whether they would be willing to answer a few questions about the job for a $0.50 bonus. The purpose of the survey is to obtain a measure of how well contractors think they did on the job and their perceptions on teamwork.
Experimental Design Details
Randomization Method
A coin flip determined whether or not the second contractor interviewed to be hired for a pair of contractors would be from the same country as the first or not. If the first contractor selected for the interview did not respond to the interview request, a coin flip would again determine whether the second contractor interviewed for the second hire in a pair of contractors would be from the same country as the first or not. Subsequently, a coin flip determined whether or not a pair of contractors would work as a team.
Randomization Unit
The level of randomization is a pair of contractor workers.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
160 pairs of contractors
Sample size: planned number of observations
160 pairs of contractors, 320 contractors total
Sample size (or number of clusters) by treatment arms
40 pairs of contractors nationally homogeneous independent work, 40 pairs of contractors nationally diverse independent work, 40 pairs of contractors nationally homogeneous team work, 40 pairs of contractors nationally diverse team work,
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Based on pilot study: 40 observations per group to obtain statistically signi ficant differences in number of features completed on the task between the groups at the 5% level with a power of 80%.
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
June 30, 2013, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
August 01, 2014, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
162 pairs of contractors
Was attrition correlated with treatment status?
No
Final Sample Size: Total Number of Observations
324 contractors
Final Sample Size (or Number of Clusters) by Treatment Arms
40 pairs nationally homogeneous teamwork, 40 pairs nationally homogeneous independent work, 41 pairs in the nationally diverse teamwork, 41 pairs in the nationally diverse independent work
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Abstract
Co-workers are increasingly diverse in their nationality and skill sets. This paper studies the effect of diversity on how workers are organized using data from a field experiment conducted in an environment where diversity is pervasive. Findings show that team organization improves outcomes when workers are from the same country. The opposite is true when workers are nationally diverse. These results are more pronounced for teams of workers with specialized skills. Further investigation of the data suggests that diverse teams have difficulty communicating. I find no evidence that preferences for or expectations about working with someone from the same country is affecting team performance.
Citation
Lyons, Elizabeth (2016). "Team Production in International Labor Markets: Experimental Evidence from the Field"

Reports & Other Materials