Human Oversight in AI-Based Performance Evaluation: A Field Experiment

2026-12-01

2027-02-28

Outcome variables are drawn from two sources:
• Behavioral performance data from FocalCompany's internal information system, including: number of stores visited, visit duration, tasks completed, task characteristics, or quality-related indicators. These data are available for the pre-experimental period as well as during the experiment, enabling difference-in-differences analyses.
• Self-reported measures from a three-wave survey (pre, mid, and post intervention), capturing agents' perceived work approach, caution, effort, and perceptions of the evaluation system.

The study uses a two-condition between-participants field experiment conducted in collaboration with FocalCompany, a Taiwanese company that provides field services for consumer goods producers. FocalCompany employs approximately 100 field agents who regularly visit and serve retail locations and document their completed tasks by uploading pictures via the company app. FocalCompany is implementing an AI-based checking system to evaluate these pictures, and our study accompanies this implementation. Field agents are organized in eight existing work teams; to reduce spillovers among employees who occasionally work together, treatment assignment occurs at the team level.
Condition 1 – AI only: Employees are informed that uploaded pictures of completed tasks are evaluated by an AI-based checking system that provides immediate feedback.
Condition 2 – AI plus human oversight: Employees are informed that pictures are initially checked by AI, but may additionally be reviewed by a human evaluator to detect possible inconsistencies.
Participants will be informed about the AI pilot and the accompanying research, but will not be informed that different teams are assigned to different conditions. This incomplete disclosure is necessary to avoid contamination and demand effects. The manipulation does not affect formal compensation or require participants to perform tasks beyond their regular work responsibilities.
In addition to the field experiment, a small number of voluntary interviews will be conducted with managers and field agents before the experiment (to understand organizational context) and after the experiment (to interpret findings and understand potential null results). A three-wave survey is administered before, during, and after the intervention.

No pilot study was conducted. The survey instrument was developed and refined prior to the pre-registration, but no data collection involving the experimental manipulation has taken place.

Not available

FocalCompany's approximately 100 field agents are organized into eight geographically separated work teams across different regions of Taiwan. Within each region, teams are assigned to conditions in equal numbers using a computer-generated random draw, ensuring balanced treatment assignment across regions. Within teams, agents occasionally interact, but cross-team contact is rare, limiting the risk of contamination across conditions.

The randomization unit is the work team. All members of a given team are assigned to the same condition.

Yes

8 work teams, with 4 teams assigned to each condition.

Behavioral data from the company's internal information system will be available for all participating field agents (approximately 100). Survey observations may be lower depending on participation rates; to incentivize completion, participants will receive a financial incentive for completing each survey wave, the details of which are to be determined. We expect up to approximately 100 survey responses per wave.

4 teams per treatment arm. The exact number of individual agents per arm depends on team sizes. We expect approximately 50 agents per condition, though the precise split will be determined by the actual team composition at the time of the intervention.