The Effects of Generative AI on Learning

Last registered on December 04, 2024
View Trial History

Pre-Trial

Trial Information

General Information

Title
The Effects of Generative AI on Learning
RCT ID
AEARCTR-0014930
Initial registration date
December 02, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
December 03, 2024, 1:40 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
December 04, 2024, 11:16 AM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name
Natalie Irmert
Affiliation
Lund University
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Catalina Franco
PI Affiliation
Center for Applied Research (SNF) at NHH – Norwegian School of Economics
Contact Investigator
PI Name
Siri Isaksson
PI Affiliation
Norwegian School of Economics
Contact Investigator

Additional Trial Information

Status
In development
Start date
2024-12-03
End date
2025-12-31
Keywords
Education, Gender
Additional Keywords
Artificial intelligence, learning
JEL code(s)
I21, I24, J16
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Artificial Intelligence (AI) is becoming an increasingly important skill in the labor market, but will it also affect academic success? Recent research shows that current students --who will be facing this rapidly changing labor market-- are adopting AI tools at differential rates based on both gender and ability. Whether AI will affect adopters’ academic success hinges on whether AI interferes with or enhances learning, which in turn depends on whether AI is being used as a substitute for or complement of own effort. If AI harms learning, students with high adoption rates would be worse prepared for the labor market than those with low adoption rates. If AI enhances learning, students who do not become proficient at AI would be left behind. To assess the impact of AI on learning, we run a controlled lab experiment which allows us to restrict and allow access to AI in different between-subject treatment variations in which students learn about a new topic. We explore whether AI is employed in a way that causally creates a gap in learning and as a consequence payoffs. Our results provide evidence on the important question of whether the documented differential adoption and use of AI by gender and ability is likely to create gender gaps in academic success. In addition, we explore several mechanisms such as over-reliance on generative AI tools, engagement and motivation.
External Link(s)

Registration Citation

Citation
Franco, Catalina, Natalie Irmert and Siri Isaksson. 2024. "The Effects of Generative AI on Learning." AEA RCT Registry. December 04. https://doi.org/10.1257/rct.14930-2.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
We design a laboratory experiment with three between subject treatment variations. In the baseline treatment, students learn without access to AI. In the AI-assisted treatment, students learn with access to AI. In the AI-guided treatment, students learn with access to AI and examples of ways to interact with AI to promote learning. This will allow us to answer the crucial question of whether, and for whom, AI is helpful as a learning tool and whether providing guidance of how to use the tool matters.
Intervention Start Date
2024-12-03
Intervention End Date
2024-12-13

Primary Outcomes

Primary Outcomes (end points)
The main outcome variable will be the score that students get on their final exam in stage 3. We will compare across treatments to see whether access to (guided) AI helps or hurts learning outcomes, as measured by the stage 3 score.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Subgroup analysis by gender, GPA.
Secondary outcomes: being a top/bottom scorer, number of practice questions solved, number of practice questions solved correctly, time spent on exam, time spent on practice questions, usage of AI tool, usage of Google Search and Google Translate, number of prompts in ChatGPT, text analysis of prompts in ChatGPT, self-assessed learning, enjoyment of learning, being comfortable with tools, being confident about using tools, effort put in learning process, engagement with learning process, perceptions on over-reliance on tools.
Secondary Outcomes (explanation)
The text analysis of ChatGPT prompts will be exploratory. We will be looking for whether the students asked directly for answers or whether they wrote prompts that would allow them to enhance their learning process. We will look for commonly used words and the content of the prompts and ChatGPT responses. We will use ChatGPT to help evaluate the quality of prompts.

Experimental Design

Experimental Design
Regardless of treatment, subjects will learn about the same topic. The treatment differences lies in whether or not they have access to ChatGPT/ChatGPT with guidance while learning.
Each session will adhere to the same procedure:
-Subjects learn for 15 minutes using written learning materials provided by the researchers.
-Subjects complete practice questions with access to different learning aids depending on treatment.
-Subjects take a 10 minute test without access to any learning aids.
-Subjects take a post-study survey asking them about demographics as well as attitudes and beliefs about AI.
Experimental Design Details
Not available
Randomization Method
Students draw number from a ballot upon arrival in the lab.
Randomization Unit
individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
no clusters
Sample size: planned number of observations
600 students
Sample size (or number of clusters) by treatment arms
200 students control, 200 students AI assisted, 200 students AI guided
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
Nottingham School of Economics Research Ethics Committee
IRB Approval Date
2024-10-16
IRB Approval Number
N/A
Analysis Plan

Analysis Plan Documents

GenAI_learning_PAP.pdf

MD5:

SHA1:

Uploaded At: December 04, 2024