Incentivizing Engagement, Resisting Shortcuts: Experimenting with AI-Assisted Learning in a Higher Education Classroom

Last registered on September 21, 2026
View Trial History

Pre-Trial

Trial Information

General Information

Title
Incentivizing Engagement, Resisting Shortcuts: Experimenting with AI-Assisted Learning in a Higher Education Classroom
RCT ID
AEARCTR-0019650
Initial registration date
September 14, 2026

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
September 21, 2026, 9:19 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name
Rong Rong
Affiliation
University of Massachusetts Amherst
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Christian Rojas
PI Affiliation
UMass Amherst
Contact Investigator
PI Name
Hossein Pishro-Nik
PI Affiliation
UMass Amherst
Contact Investigator
PI Name
Heath Hatch
PI Affiliation
UMass Amherst
Contact Investigator

Additional Trial Information

Status
In development
Start date
2026-09-16
End date
2027-02-08
Keywords
Behavior, Education
Additional Keywords
Artificial Intelligence, Classroom Engagement, Higher Education, Student Performance
JEL code(s)
C93, D91
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
AI tools are increasingly available to support student learning, but they carry a built-in pitfall: students are tempted to use them for quick, direct answers rather than genuine problem-solving. AI tutors can be designed to resist this shortcut-taking behavior, but doing so often comes at the cost of low student engagement when their use is voluntary. This study examines that tradeoff in the context of a custom AI tutor deployed in an introductory college physics course over a full semester. Students are randomly assigned to one of three AI tutors that vary in the way a direct answer can be provided. Five class sections are assigned to one control condition and four incentive conditions. Along one dimension, we compare student engagement and learning outcomes across sections that differ in how AI tutor use is incentivized. Along the second dimension, we examine whether delaying the availability of direct answers affects student performance, engagement with the AI tutor, and the tendency to seek shortcuts versus reason through problems incrementally. Student performance will be measured through proctored course exams without AI assistance, and homework performance will be recorded throughout the semester. Students will complete two detailed surveys after each midterm, reporting their experience with the AI tool. The results of the study aim to improve AI tutor designs and incentive conditions that can limit shortcut-seeking behavior while sustaining the engagement needed for productive learning in higher education.
External Link(s)

Registration Citation

Citation
Hatch, Heath et al. 2026. "Incentivizing Engagement, Resisting Shortcuts: Experimenting with AI-Assisted Learning in a Higher Education Classroom." AEA RCT Registry. September 21. https://doi.org/10.1257/rct.19650-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
The intervention is a custom AI chatbot tutor made available to students in an introductory college physics course for use during in-class problem-solving sessions and for independent study at home throughout the semester. Students interact with one of three versions of the tutor, which differ in how readily they provide a direct answer to a question, ranging from a purely Socratic style that never reveals the solution to a style that offers a direct answer on request. Separately, the course varies whether using the tutor in class and/or at home is voluntary or tied to a small extra-credit incentive across the entire semester. One class section serves as a no-chatbot baseline.
Intervention Start Date
2026-09-16
Intervention End Date
2026-12-22

Primary Outcomes

Primary Outcomes (end points)
(1) student performance on course exams (individual portions of two midterms, completed without AI assistance in-class) and graded homework/in-class problem-set items (completed with AI assistance at home);
(2) chatbot engagement, measured by usage frequency, number of conversation turns, and word counts, both in class and at home;
(3) Text content extracted from chatbot conversation
(4) shortcut-seeking behavior, measured by the frequency and timing with which students opt into or request a direct (Didactic) answer rather than continuing Socratic dialogue (only in DO and DDO treatments);
(5) other type of engagement activity (class attendance, Office Hour attendance through Physics Help Center, number of direct email inquiry with TAs or instructors)
(6) self-reported experience and perceived learning, students' self-reported attitudes toward AI use from the two post-midterm surveys (both Likert scale and open ended text entry); graded reflections submitted for the designated chatbot-use homework problem
(7) demographics, from the two post-midterm surveys
Primary Outcomes (explanation)
Engagement will be constructed from chatbot log data (session counts, message counts, conversation turns, and word counts) aggregated by student and by in-class session table ID.
Shortcut-seeking behavior will be constructed from the proportion of conversations (or conversation rounds) in which a student under the DO or DDO condition chooses the direct-answer option, and how quickly they do so.
Learning outcomes will be constructed from standardized exam and homework scores, and, where relevant, from the graded reflection on the designated chatbot-use problem in each problem set.
Text entries extracted from the chatbot will be used in text analysis.

Secondary Outcomes

Secondary Outcomes (end points)
None
Secondary Outcomes (explanation)
None

Experimental Design

Experimental Design
The study uses a two-dimensional, semester-long design within a single introductory physics course, split into several sections (about 100 each). First, at the start of the semester, each of the 11 seating tables (in a section) used during in-class problem solving is assigned to one of three chatbot conversation styles, held fixed for that table all semester (students in that table remain fixed throughout the semester). Second, independent of table assignment, the incentive attached to chatbot use varies by class section across the semester: one baseline section with no chatbot (students instead use an in-person help center), and four subsequent sections following a 2x2 design crossing voluntary vs. incentivized in-class use with voluntary vs. incentivized at-home use. Learning outcomes are assessed using exams completed without AI assistance. Engagement outcomes are assessed using survey response and objective measure of classroom and office hour participation.
Experimental Design Details
Not available
Randomization Method
Students will be randomly assigned to conversation-style condition via a computer-generated random draw process conducted by the research team before the semester begins. The mapping of incentive conditions to specific class sections is assigned based on the instructor preference and constrained by course logistics rather than fully randomized; Instructor H teaches sections 1-3, and will be assigned the baseline section, voluntary in-class + voluntary at-home for (section 2), and voluntary in-class + incentivized at-home (section 3); Instructor P teaches sections 4 and 5, and will be assigned the incentivized in-class + voluntary at-home (section 4), and incentivized in-class + incentivized at-home (section 5). Sections 1-5 are ordered based on the timing of the section on the day of the class.
Randomization Unit
Two units of assignment: (1) seating table (11 tables per section) for the conversation-style dimension; (2) class section for the incentive dimension, applied to the whole section.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
the treatments are clustered into 55 tables (conversation-style dimension) and 5 class sections (incentive dimension);
Sample size: planned number of observations
Approximately 500 students evenly enrolled in 5 sections (exact count to be confirmed after add/drop deadline)
Sample size (or number of clusters) by treatment arms
Conversation style (approx., assuming ~9 students/table): 3 tables (~27*5 students) Socratic; 4 tables (~36*5 students) Delayed-Didactic-Option; 4 tables (~36*5 students) Didactic-Option.
Incentive condition: ~100 students in “baseline” (Section 1); ~100 students in “voluntary in-class + voluntary at-home” (Section 2), ~100 students in “voluntary in-class + incentivized at-home” (Section 3); ~100 students in “incentivized in-class + voluntary at-home” (Section 4), and ~100 students in “incentivized in-class + incentivized at-home” (Section 5)
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
UMass Amherst Institutional Review Board
IRB Approval Date
2026-09-14
IRB Approval Number
8074
Analysis Plan