Experimental Design
Identifying the causal impact of national diversity on the value of teamwork is problematic because teams are selected and not randomly assigned. This endogenous team formation makes it difficult to disentangle both the value of teamwork as compared to independent work and the differential impact of teamwork for nationally diverse teams. For example, it is plausible that employers and workers choose which jobs to complete as a team and that they choose who should work together in teams according to their performance expectations. To address these concerns, I test the effects of national diversity on work team success by conducting a field experiment on the world's largest online contract labor platform, oDesk.
To perform the study, I set up an employer account on the oDesk website and posted jobs to attract contractor bids. I hired contractors who met the requirements for participation in the experiment and randomly put them into groups of two to complete a programming task. I define teams in this study as pairs of contractors working towards a common goal.
There are several reasons I chose to run this experiment on oDesk. First, oDesk provides a team room application which allows employers to put hired contractors into teams so that they can work together online. In particular, contractors working in the same team room can monitor what the others are doing and chat with each other through instant messaging. Contractors can only be logged into one team room at a time. Employers on oDesk frequently require that multiple hires work in the same team room. Second, the team room facilitates contractor monitoring by employers through frequent screen shots, memos, work diaries, and activity meters. As a result, the majority of employers on oDesk require that their hires complete their tasks while logged into their team room. Another important feature of oDesk for this research is the option to interview. oDesk encourages employers to interview contractors they are interested in hiring before making any offers. These interviews can take place through messages exchanged on the site. Lastly, oDesk is an international labor market which contractor participants have selected into knowing they would almost surely be working with employers and contractors from other countries. Therefore, we might expect to see less distaste for working with a nationally diverse organization than in more local labor markets.
To test my research question, I needed contractors to be assigned a task for which collaboration is not unnatural. At the same time, because I am comparing team work to independent work, contractors working on their own should be capable of completing at least some portion of the task. In addition, because I am interested in testing the coordination and communication costs associated with national diversity in teams, free-riding on the assigned task should be difficult. The task also needed to be consistent with the types of tasks posted on oDesk so that contractors did not become suspicious about the purpose of the job. Moreover, an objective evaluation of contractor performance was necessary. Finally, it must have been possible to complete the task remotely. Given these constraints, I developed a web development task that required both back-end (PHP) and front-end programming (Javascript) to assign contractors hired for the experiment. Hired contractors were asked to add a list of features in these languages to existing code. This is similar to a web development task for which a group of workers are responsible for both the design and functionality of a web page.
To reinforce the collaboration aspect of this job and to reduce free-riding in teams, I hired one contractor in each pairing to complete the Javascript portion of the task and another to complete the PHP portion. The team rooms further reduce the potential for free-riding because contractors are aware that their individual performance can be monitoring by employers through the team room application. I assigned each pair three features to add to the code, one that required only Javascript programming, one that required only PHP programming, and one that required both. Including both individual tasks and a team task accomplishes two objectives. First, it allows for both independent and for team work so that neither of the organizational form treatments lead to contractor frustration. Second, it allows me to disentangle difficulties associated with collaboration from difficulties associated with the job in general. All hired contractors received the same list of features to be added, and I instructed them to work on the features that corresponded to the language for which they were hired. Nothing in the instructions or the task design explicitly prevents contractors from working on all three features of the task, including the parts that are not in the language they were hired for. The reasons a single contractor may not work on both languages are first, that some contractors are only familiar in coding one of the two languages, and second, that they have been asked to focus on one of the two languages. Importantly, the feature that requires both programming languages to be completed requires that the codes from each language work together. A contractor who only knows one of the two languages would not be able to complete this feature on his or her own. Therefore, contractors in the independent work treatment would not be able to complete all three features unless they are familiar with both PHP and Javascript.
In order for the contractors in the teamwork condition to be able to work with each other, I gave them a total of eight hours to complete the task; they had to work these hours within the same period as the other contractors in their groups.
I randomly assigned each hired contractor to one of four types of groups of two, and all groups were made up of one contractor hired to work on the Javascript code and one to work on the PHP code. The experiment had two treatments: pairs were either from different countries or not and were either permitted to work in teams or not. Pairs in the team treatment worked in the same team room as each other, and those in the independent work treatment worked in separate team rooms and were unaware of who their pair mate was so that team work was not an option. The purpose of this experimental design is to compare the benefits of teamwork in homogeneous teams with the benefits of teamwork in nationally diverse teams. This allows me to identify how organizing workers into teams compares with organizing them as independent workers, and how this is affected by national diversity.
To control for country-specific differences in contractor quality, I limited the number of countries from which I hired. Based on information about the set of applicants my job postings attracted during the pilot phase of my study, I included English speaking contractors from India, Pakistan, and Bangladesh in the experiment. Contractors from these countries frequently applied to my job postings and bid an amount that was within the hourly wage criteria. Moreover, workers in these countries have very similar time zones making it easier to allow for remote team work among them.
Using an employer account on oDesk, I attracted contractor bids by posting two job types. One type of job posting asked for bids from contractors able to code in PHP and the other for bids from contractors able to code in Javascript. Contractors therefore selected into the programming language they preferred. To minimize attrition among hired contractors, the job postings described the type of work the task required, the date the work needed to be done on, and the number of hours contractors had to complete the task. The job postings also specified that the maximum bid that would be accepted ($4.00 USD). The job postings did not mention that the work would be completed in teams nor did they mention anything about country-specific requirements.
The first applicant to each job posting who bid at most $4.00 and who was from one of the countries included in my experiment received a list of interview questions. The purpose of the questions was to get an idea of the level of Javascript and PHP knowledge each contractor had, their English language abilities, and the countries they have lived in. I gave each interviewee two hours to reply to the interview questions before I interviewed the next applicant who met the bid and country requirements. I hired the first interviewee to reply with the exception of interviewees who did not provide pertinent answers due to their lack of English language ability.
Once hired, contractors were sent the appropriate instructions and the file with the code to be edited. There were four versions of the job instructions to reflect whether the contractor was hired for the Javascript or the PHP portion of the job and whether pairs were permitted to work as a team or not. Instructions do not vary across the national diversity treatment.
In addition to describing the features to be added to the code, the instructions noted the timeline for the task and the country that contractors' co-worker lived in because it is not possible to restrict access to this information in the team work treatment, so I controlled for this knowledge across treatment groups to ensure that it was not driving my findings.
I asked contractors who had not turned in their work by the deadline to submit any work they had completed. Once contractors had turned in their work or after the deadline had passed, I asked them whether they would be willing to answer a few questions about the job for a $0.50 bonus. The purpose of the survey is to obtain a measure of how well contractors think they did on the job and their perceptions on teamwork.