An Investigation of AI Chatbots as a Tool for Learning

Last registered on September 21, 2026
View Trial History

Pre-Trial

Trial Information

General Information

Title
An Investigation of AI Chatbots as a Tool for Learning
RCT ID
AEARCTR-0019238
Initial registration date
September 10, 2026

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
September 21, 2026, 8:20 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name
Jenny Ulla
Affiliation
University of Illinois - Urbana Champaign
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Kimberly Mendoza
PI Affiliation
University of Illinois - Urbana Champaign
Contact Investigator
PI Name
Jessica Thornton
PI Affiliation
University of Illinois - Urbana Champaign
Contact Investigator

Additional Trial Information

Status
In development
Start date
2026-11-01
End date
2028-12-31
Keywords
Behavior
Additional Keywords
generative AI, chatbot, human-AI interaction
JEL code(s)
M42, C91, O33
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Professionals increasingly complete work tasks with help from interactive chat-based resources. While these tools can improve immediate task performance, less is known about how they affect whether people actually learn the underlying task. This study examines how the conditions under which individuals receive and record information during a task affect how much they understand and retain about the task itself. Participants take the role of an audit associate completing an accounts payable procedure, with access to a chat resource for guidance while they work. We measure task performance as well as participants' ability to explain and apply the underlying procedure afterward. Findings inform how organizations might structure the use of interactive assistance so that employees continue to build expertise while using these tools.
External Link(s)

Registration Citation

Citation
Mendoza, Kimberly, Jessica Thornton and Jenny Ulla. 2026. "An Investigation of AI Chatbots as a Tool for Learning." AEA RCT Registry. September 21. https://doi.org/10.1257/rct.19238-1.0
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
All participants complete audit procedure and have access to an interactive chat resource for guidance while they work. The chat resource provides procedural guidance about the task. Participants are randomly assigned to conditions that vary who provides the guidance through the chat and how participants are instructed to record it in their working documentation. All other aspects of the task, materials, and instructions are identical across conditions.
Intervention Start Date
2026-11-01
Intervention End Date
2026-12-31

Primary Outcomes

Primary Outcomes (end points)
Primary measures are task performance (errors correctly identified and correct classification of the effect on the account balance) and knowledge acquisition, assessed afterward through a set of questions. Additional measures include manipulation checks, perceived social presence, reliance on and accessibility of the chat, effort, task difficulty, and participant background; chat transcripts are recorded to evaluate the process of knowledge acquisition in our study.
Primary Outcomes (explanation)
None

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
This is a randomized between-participants laboratory experiment. Participants take the role of an audit associate completing a three-way match task, comparing vendor invoices to a client check register to identify errors and indicate the direction and amount of any resulting misstatement. While working, all participants can consult an interactive chat resource that provides procedural and conceptual guidance but does not supply conclusions about specific items, so participants perform the verification themselves. Participants are randomly assigned to conditions that vary aspects of the chat interaction and how they are instructed to record the guidance they receive. Primary outcomes are task performance and participants' subsequent understanding of the underlying procedure, measured through a set of questions.
Experimental Design Details
Not available
Randomization Method
Sessions are conducted in a computer lab and each session is randomly assigned to the source condition (chatbot or audit senior). Within each session, participants are randomly assigned by the survey software to one of the two instruction conditions. Session identifiers are recorded for all participants.
Randomization Unit
We have two levels of randomization. The source factor (chatbot vs. audit senior) is randomized at the session level: each computer-lab session is randomly assigned to one source condition, so all participants within a session share the same source condition. The instruction factor (summarize vs. copy and paste) is randomized at the individual level: within each session, the survey software randomly assigns each participant to one of the two instruction conditions. Session identifiers are recorded for all participants.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
11 sessions (4 chatbot, 7 audit senior)
Sample size: planned number of observations
168 participants
Sample size (or number of clusters) by treatment arms
42 chatbot-summarize condition, 42 chatbot-copy and paste condition, 42 audit senior-summarize condition, and 42 audit senior-copy and paste condition
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Power was calculated in G*Power for a 2 × 2 between-participants ANOVA (fixed effects; numerator df = 1; four cells). Assuming α = 0.05 and 90% power, detecting an effect of f = 0.30 (equivalent to a medium effect size) requires a total sample of 119 participants. We plan to recruit 42 participants for each condition and will reach our sample size needed based on this power analysis. This figure applies to our primary test, the Source × Instruction interaction, and to the Instruction main effect, which are randomized at the individual level within session and are therefore not materially affected by clustering.
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Illinois Office for Protection of Research Subjects
IRB Approval Date
2024-08-20
IRB Approval Number
IRB24-1079
Analysis Plan