Experimental Design
Workers were first randomized into either the control group or one of two treatment groups: the detailed evaluation or the coarse evaluation treatment group. Randomization into any treatment group was stratified on prior oDesk experience, such that workers without oDesk experience had a higher chance of being in any treatment group (32 percent) than experienced workers (15 percent). Conditional on receiving any treatment, all workers had a 50 percent chance of receiving the detailed evaluation treatment. Inexperienced workers constituted approximately three quarters of each treatment group.
The coarse evaluation treatment was designed to be equivalent to being hired (and, thus, evaluated) by a typical employer in the marketplace. The detailed evaluation treatment was identical to the coarse evaluation treatment except that it provided the market with more information about some workers’ job performance. Workers in both treatment groups were hired and given a maximum of ten hours over one week to enter the data. They were told that if, after spending ten hours on the task, they had not completed it, they should send the file back unfinished. The following objective measures of workers’ performance were used: their data entry speed, their error rate, the date they returned the data file, and three measures of whether they had followed the data entry instructions. All hired workers were rated on a one-to-five scale using a weighted average of workers’ scores on these performance measures. The distribution of scores from the job was designed to match the distribution of scores low-wage data entry workers received in the marketplace, adjusted for the fact that a worker in the sample was more likely to be inexperienced than a typical oDesk worker. The scores were calculated in the same way for workers in both treatment groups. Approximately 18 percent of workers did not return the file or log any hours. Under oDesk’s protocol, these workers were not rated. Thus, the treatments should be considered as an intent to hire.
The particular treatment group to which workers were assigned affected only the type of comment workers were eligible to receive. No workers in either treatment group received a comment if they earned a rating below three. The remaining workers in the coarse evaluation treatment received an uninformative comment. The remaining workers in the detailed evaluation treatment received a detailed comment if they scored at least a four and an uninformative comment if they scored between three and four. Workers in the detailed evaluation treatment did not know that they would receive a detailed evaluation until it was posted.
The uninformative comment was chosen to be short and positive, like most of the comments in the marketplace. The detailed comment provided objective information on the worker’s data entry speed and accuracy, whether the worker met the deadline, and whether she followed the job’s instructions. Additionally, it repeated the uninformative comment, so the only difference between the two comment types was the objective information provided in the detailed evaluation.