The Effects of Human-Supervised AI Agents on Customer-Service Productivity and Performance

Last registered on September 25, 2026
View Trial History

Pre-Trial

Trial Information

General Information

Title
The Effects of Human-Supervised AI Agents on Customer-Service Productivity and Performance
RCT ID
AEARCTR-0019572
Initial registration date
September 17, 2026

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
September 25, 2026, 9:48 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name
Chao Gu
Affiliation
Zhejiang university
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
zhe Yuan
PI Affiliation
Zhejiang University
Contact Investigator
PI Name
zhe Yuan
PI Affiliation
Zhejiang University
Contact Investigator

Additional Trial Information

Status
On going
Start date
2026-05-30
End date
2026-11-18
Keywords
Behavior, Firms & Productivity, Governance, Labor, Other
Additional Keywords
Artificial Intelligence
JEL code(s)
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
We will conduct a two-arm, individual-level trial among customer-service representatives who handle asynchronous consumer complaints and dispute-resolution cases at a large global e-commerce company. Eligible representatives will be randomly assigned in equal proportions to a treatment group that receives access to a human-supervised AI agent or to a control group that continues using the existing customer-service workflow. The trial will take place within the company’s ordinary operations. We will use administrative records from the experimental period to compare the two groups’ work efficiency and service performance. The primary analysis will estimate the intention-to-treat effect of being assigned access to the AI agent.
External Link(s)

Registration Citation

Citation
Gu, Chao, zhe Yuan and zhe Yuan. 2026. "The Effects of Human-Supervised AI Agents on Customer-Service Productivity and Performance." AEA RCT Registry. September 25. https://doi.org/10.1257/rct.19572-1.0
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
We will conduct a worker-level randomized controlled trial in the customer-service division of a large global e-commerce company. Eligible customer-service representatives who handle asynchronous consumer complaints and dispute cases will be randomly assigned in equal proportions to a treatment or control group. Treatment representatives will receive access to a large-language-model-based agent embedded in their existing work interface. The agent uses available case information to recommend and, where authorized, carry out resolution steps under the department’s standard operating procedures. Representatives can monitor the agent, review its proposed actions, interrupt the automated process, and take over the case at any time. Control representatives will continue using the existing workflow without access to the agent.
Intervention Start Date
2026-09-01
Intervention End Date
2026-09-18

Primary Outcomes

Primary Outcomes (end points)
The work-efficiency index will combine:
- The number of eligible cases completed per scheduled working hour.
- Average active handling time per completed case, coded so that shorter handling time represents better performance.
- Elapsed time from case assignment to final disposition.
- The share of assigned cases completed within the applicable service standard.
The performance and resolution-quality index will combine:
- The company’s standard quality-assurance or case-audit score.
- Compliance with the prescribed operating procedure.
- The share of cases resolved without reopening, escalation, appeal, or reversal within a prespecified follow-up period.
- Customer satisfaction or the corresponding firm performance measure, where routinely collected for the relevant case type.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We will conduct a two-arm cluster-randomized controlled trial across customer-service sites of a large global e-commerce company. Participating sites will be randomly assigned in approximately equal proportions to either a treatment group or a control group. All eligible customer-service representatives at treatment sites will receive access to a human-supervised AI agent, while representatives at control sites will continue using the existing customer-service workflow. We will use administrative records from the experimental period to compare work efficiency and service performance between treatment and control sites. The primary analysis will estimate the intention-to-treat effect of assigning a customer-service site to receive access to the AI agent.
Experimental Design Details
Not available
Randomization Method
Randomization will be conducted in an office by computer using reproducible statistical code. After the participating-site roster has been finalized, each site will be assigned a unique identifier. The code will randomly permute the sites using a prespecified random seed and assign approximately one-half of the sites to treatment and the remainder to control. All eligible customer-service representatives within a site will receive the treatment status assigned to that site. The research team will archive the randomization code, seed, input roster, and resulting site-level assignment file.
Randomization Unit
Customer-service site (cluster). All eligible customer-service representatives within the same site will receive the site’s assigned treatment status. There will be no individual-level randomization within sites.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
We will conduct a cluster-randomized controlled trial in the customer-service division of a large global e-commerce company. Participating customer-service sites will be randomly assigned in approximately equal proportions to a treatment or control group. All eligible customer-service representatives who handle asynchronous consumer complaints and dispute cases at treatment sites will receive access to a large-language-model-based agent embedded in their existing work interface. The agent uses available case information to recommend and, where authorized, carry out resolution steps under the department’s standard operating procedures. Representatives can monitor the agent, review its proposed actions, interrupt the automated process, and take over the case at any time. Representatives at control sites will continue using the existing workflow without access to the agent.
Sample size: planned number of observations
About 2,000 customer servicers
Sample size (or number of clusters) by treatment arms
50 Customer-service sites
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number
Analysis Plan