Experimental Design
Context. Participants (“Spectators”) are informed at the outset that a group of workers were recruited to collect online information about US public companies under a flat payment. Each worker spent 5 minutes on each of two tasks, Task 1 focusing on financial firms and Task 2 on service firms. After the tasks ended, an evaluator assigned each worker two task scores, one for each task, to record the number of firms for which accurate information was collected.
Scoring was carried out over two days, Day 1 and Day 2. Each worker was randomly assigned to be scored on one of the two days. The key difference between the two days lies in the accuracy of Task 2 scores: while all Day 1 Task 2 scores accurately reflect true performance, a random half of Day 2 workers received an inflated Task 2 score—one point higher than their true score. This random grade inflation renders Task 2 scores invalid for, and biased in favor of, Day 2 workers. By contrast, Task 1 scores are accurate for all workers, regardless of scoring day. Comprehension checks are used to ensure participants understand this, and the information is reiterated throughout the survey.
Allocation decisions. Spectators’ main task is to decide which of two workers should receive an additional \$3 bonus, based on information provided in a report card. Each spectator makes four allocation decisions, structured across two experimental parts, with two conditions per part. The two parts differ in the scoring days of the two candidate workers, which determines the accuracy of their Task 2 scores:
• Accurate Info part: Both workers were scored on Day 1, so their Task 2 scores are accurate.
• Biased and Invalid Info part: One worker was scored on Day 1 and the other on Day 2. Because only Day 2 scores are subject to random inflation, Task 2 scores in this part are invalid for and biased in favor of the Day 2 worker.
Within each part, spectators make two allocation decisions under different informational conditions:
• Task 2 Excluded condition: The report card displays each worker’s scoring day and Task 1 score, but omits Task 2 scores.
• Task 2 Included condition: The report card displays the scoring day along with both Task 1 and Task 2 scores.
The two experimental parts as well as the two conditions within each part are presented to spectators in random order.
To make an allocation decision for a part-condition, spectators choose and commit to a contingent plan that specifies who receives the bonus under every possible realization of the report card. The candidate plans available to spectators are listed below. These plans are presented to spectators in random order.
Task 2 Excluded
Accurate Info part:
Plan A: Flip a coin to decide who gets the bonus, regardless of their scores.
Plan B: Give the bonus to the worker with the higher Task 1 score. If their Task 1 scores are the same, flip a coin to decide.
Biased and Invalid Info part:
Plan A: Same as above (flip a coin regardless of scores).
Plan B: Same as above (give bonus to higher Task 1 scorer; flip a coin if tied).
Task 2 Included
Accurate Info part:
Plan B: Same as above (higher Task 1 score; coin flip if tied).
Plan C: Give the bonus to the worker with the higher total score (Task 1 + Task 2). If their total scores are the same, flip a coin to decide.
Biased and Invalid Info part:
Plan B: Same as above (higher Task 1 score; coin flip if tied).
Plan D: Give the bonus to the worker with the higher total score (Task 1 + Task 2). If their total scores are the same, give the bonus to the Day 1 worker.
Plan D comes with an explanation: “Under this rule, the one extra point the Day 2 worker may have got only matters when the two workers have the same true total performance. When their true total performance is the same, the Day 2 worker gets the bonus only if they received the extra point. Otherwise, the Day 1 worker receives the bonus. If one worker has a higher true total performance, that worker always gets the bonus.”
Spectators are asked to provide open-ended explanations for their plan choices under the Task 2 Included condition.
Implementation of allocation plans. Each spectator is informed that one of their four allocation decisions—corresponding to a specific part-condition—may be randomly selected for real implementation. If selected, we first match the spectator with a pair of workers whose scoring days are consistent with the designated part. We then generate a report card for the two workers, containing the information specified by the selected condition. Finally, the spectator’s chosen allocation plan for that part-condition is automatically applied to the report card to determine which worker receives the bonus.
Information preferences. In each part, we elicit spectators’ preferences over the inclusion of Task 2 scores in the report card after they have chosen and committed to an allocation plan under both informational conditions. Specifically, we ask whether they would prefer the report card to include or exclude Task 2 scores, or whether they are indifferent. Spectators who express a preference for inclusion or exclusion are given the opportunity to confirm their choice by typing a sentence. Confirming a preference for inclusion (or exclusion) increases the likelihood that Task 2 scores will actually be included (or excluded) on the report card if that part-condition is selected for implementation. Finally, all spectators are asked to provide an open-ended justification for their information preference.