Justification for AI Use - A Field Experiment with LLM Agents

Last registered on November 26, 2025
View Trial History

Pre-Trial

Trial Information

General Information

Title
Justification for AI Use - A Field Experiment with LLM Agents
RCT ID
AEARCTR-0017259
Initial registration date
November 25, 2025

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
November 26, 2025, 7:11 AM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country
Germany
Region
NRW

Primary Investigator

Name
Leonhard Grabe
Affiliation
Universität zu Köln
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Simon Lübke
PI Affiliation
University of Cologne
Contact Investigator

Additional Trial Information

Status
In development
Start date
2025-11-23
End date
2025-12-31
Keywords
Behavior, Firms & Productivity
Additional Keywords
AI acceptance, customer service, field experiment
JEL code(s)
M11, M21, O33
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
In a field experiment embedded in regular outbound customer service operations, we examine whether a small framing intervention improves customer engagement during fully automated calls conducted by an AI voice system. In these calls, the AI system handles the complete conversation flow and may transfer customers to human agents only when necessary. We compare call outcomes between a control group that receives the standard neutral introductory sentence and a treatment group that receives an enhanced introduction explaining why the AI system is calling and emphasizing the intended benefit for the customer, namely quicker processing of their request. We evaluate whether this additional explanatory sentence increases the likelihood that customers stay in the call, reduces immediate hang-ups, and affects subsequent transfers to human agents. The experiment relies on call-level random assignment and uses only operational data already collected for quality assurance. The aim is to assess whether transparent, benefit-oriented communication about the AI system’s role can measurably improve customer interaction quality in fully automated service calls.

External Link(s)

Registration Citation

Citation
Grabe, Leonhard and Simon Lübke. 2025. "Justification for AI Use - A Field Experiment with LLM Agents." AEA RCT Registry. November 26. https://doi.org/10.1257/rct.17259-1.0
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
We evaluate the effect of using justifications for AI use on the acceptance of AI chatbots. We modify the introduction of an LLM-based chatbot, whose task it is to arrange appointments on behalf of a large technology company via phone call. In the control condition, the agent introduces itself by name and states the purpose of the call. In the treatment condition, we include an additional sentence to justify the delegation of this task to the AI, such as earlier appointments for the consumer. The key outcome variable is the likelihood of succesfully scheduling an appointment with the chatbot.
Intervention (Hidden)
Intervention Start Date
2025-11-23
Intervention End Date
2025-12-18

Primary Outcomes

Primary Outcomes (end points)
The key outcome is a binary indicator whether the agent was successful in scheduling an appointment with the customer (=1) or not (=0).
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary outcomes include whether the call was transferred to a human agent, the total duration of the call, and the share of calls that are terminated without a successful outcome after less than one minute (if the data is available).
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We evaluate the effect of providing justifications for AI use on the acceptance of AI chatbots. The study is conducted using an LLM-based chatbot, whose task it is to arrange appointments on behalf of a large technology company via phone call. We modify whether or not the bot provides and explanation for why the bot was tasked with scheduling the appointments rather than a human customer service agent. We measure the acceptance of the bot using the share of successfully booked appointments and transferred calls, as well as the average duration and sentiment of the call - if this data is available.
Experimental Design Details
Randomization Method
Customers are randomly allocated to one of two AI agents by the firm using a digital coinflip.
Randomization Unit
Randomization was implemented on the individual level and day-level. That is, all customers that are scheduled to be contacted on a given day are allocated to one group.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
8000 customers (4000 per treatment group)
Sample size: planned number of observations
8000 customers (4000 per treatment group)
Sample size (or number of clusters) by treatment arms
4000 customers
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
German Association for Experimental Economic Research e.V.
IRB Approval Date
2025-11-19
IRB Approval Number
v4RViH15
Analysis Plan

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials