Agentic AI and Structured vs. Self-guided learning

Last registered on July 28, 2025

View Trial History

Pre-Trial

Trial Information

General Information

Title

Agentic AI and Structured vs. Self-guided learning

RCT ID

AEARCTR-0016373

Initial registration date

July 23, 2025

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

July 28, 2025, 9:07 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name

Sofia Bapna

Affiliation

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

PI Affiliation

PI Name

PI Affiliation

PI Name

PI Affiliation

PI Name

Vandith Pamuru Subramanya Rama

PI Affiliation

Contact Investigator

Additional Trial Information

Status

In development

Start date

2025-07-24

End date

2026-08-01

Keywords

Education

Additional Keywords

JEL code(s)

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

Agentic-AI tools are beginning to be integrated into classrooms to support student learning and prepare them for a workforce where such tools are increasingly common. An agentic-AI tool is deployed in a core analytics course at a top-30 global MBA program; the tool serves as a virtual data scientist and is used by students to solve case assignments. We test which type of guidance is most effective for students using the tool. Students are randomly assigned to one of two conditions: a Structured Learning condition and a Self-guided Learning condition. In each condition, students receive either exact prompts to use for the case or a description of the tool’s capabilities as they pertain to the case. In both conditions, the guidance is intended only to help students get started on solving the assigned cases; neither provides a complete solution. We examine (1) which type of agentic-AI usage guidance improves learning outcomes (captured by performance on a proctored, in-person, closed-book, pen-and-paper final exam), (2) whether usage differs across students based on the type of guidance, and (3) whether the type of guidance has heterogeneous effects on learning outcomes and usage patterns by gender, competence (e.g., prior academic performance), and work experience (e.g., analytics experience).

External Link(s)

Registration Citation

Citation

Pamuru Subramanya Rama, Vandith et al. 2025. "Agentic AI and Structured vs. Self-guided learning." AEA RCT Registry. July 28. https://doi.org/10.1257/rct.16373-1.0

Sponsors & Partners

Experimental Details

Interventions

Intervention(s)

Students are randomly assigned to one of two conditions: Structured Learning or Self-guided Learning.

There are two individual case assignments in the course, posted one week apart. For each assignment, students receive an email from the instructor corresponding to their assigned experimental condition. Once assigned to a condition, students remain in that condition for both assignments. Emails are sent as soon as the assignment is posted on the LMS. Students have four days to complete each assignment (e.g., if the assignment is posted on Thursday (or Friday), the solution is due on Monday (or Tuesday), respectively).

There are multiple sections of the course; some sections post the assignment on Thursday and others on Friday.

The subject line of the emails is: “Instructions for your assignment due on <Month> <Day>.” The email templates are shown below.

Note: ‘Opening Message’ and ‘Instructor Name’ are placeholders that will be kept constant across conditions for each case. Similarly, <Month> <Day> are placeholders indicating the assignment due date.

*Structured Learning - Case 1*
Dear students,
[Opening Message 1] The “Cancer Detection” case on binary classification is now available on the DAB platform. We encourage you to start chatting with the AI agent to learn about this case. To help you get started, consider using the following prompts:
1. Please describe the case.
2. Please provide a correlation heat map of the input variables.
3. Please build a logistic regression model to detect cancer and show the detailed steps.
4. Please show me the first 10 rows in the test set with actual outcome, predicted probability, and predicted class label. Format this as a table.
5. Please test the kNN model for this prediction task. What would be a good value of k?
Best regards,
[Instructor Name]

*Self-guided Learning Case 1*
Dear students,
[Opening Message 1] The “Cancer Detection” case on binary classification is now available on the DAB platform. We encourage you to start interacting with the AI agent to learn about this case. To help you get started, consider exploring the following capabilities of the AI agent:
1. The AI agent can provide a description of the case.
2. The AI agent can create various visualizations of the data.
3. The AI agent can perform manipulation and processing on the data.
4. The AI agent can build and evaluate numeric prediction models, such as linear regression and kNN, on the specific outcome variable you choose.
5. The AI agent can perform systematic model tuning to find a good performing model.
Best regards,
[Instructor Name]

*Structured Learning - Case 2*
Dear students,
[Opening Message 2] The “Catalog Company” case on numeric prediction is now available on the DAB platform. We encourage you to start chatting with the AI agent to learn about this case. To help you get started, consider using the following prompts:
1. Please describe the case.
2. Can you visualize the distribution of the “Spending” variable in a histogram?
3. Build a linear regression model to predict spending. Show the detailed process and report the RMSE and MAPE metrics.
4. Please redo the model by not using the purchase variable and report the full model as well as RMSE.
5. Please build a KNN model to predict spending. Choose a good value k and report RMSE.
Best regards,
[Instructor Name]

*Self-guided Learning Case 2*
Dear students,
[Opening Message 2] The “Catalog Company” case on numeric prediction is now available on the DAB platform. We encourage you to start interacting with the AI agent to learn about this case. To help you get started, consider exploring the following capabilities of the AI agent:
1. The AI agent can provide a description of the case.
2. The AI agent can create various visualizations of the data.
3. The AI agent can perform manipulation and processing on the data.
4. The AI agent can build and evaluate various numeric prediction models, such as linear regression and kNN.
5. The AI agent can perform systematic model tuning to find a good performing model.
Best regards,
[Instructor Name]

Intervention Start Date

2025-07-24

Intervention End Date

2025-08-22

Primary Outcomes

Primary Outcomes (end points)

Final exam score, which is a proxy for learning.

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Number of prompts

Secondary Outcomes (explanation)

Experimental Design

An agentic-AI tool is deployed in a core analytics course in an MBA program, serving as a virtual data scientist to help students solve case assignments. We test the effect of different types of guidance—Structured Learning or Self-guided Learning—on how students use the tool to identify solutions to the assignments. Students are randomly assigned to one of the two conditions and receive case assignment emails from the instructor aligned with their assigned condition. We measure how this guidance influences both students’ usage of the tool and their learning outcome. Additionally, we will examine heterogeneity by gender, competence (e.g., previous academic performance), work experience (e.g analytics experience).

Experimental Design Details

Not available

Randomization Method

Simple randomization using a script (with a seed, so the randomization can be replicated).

Randomization Unit

Individual (student).
The students are distributed across multiple sections of the course. Randomization to the two conditions will occur within each section.
Across all students (combining students across sections) we will check for balance on: gender, sector of experience, years of experience, and undergrad degree type (assuming we receive this data).

Was the treatment clustered?