The Effects of Generative AI on Learning

Last registered on December 04, 2024
View Trial History

Pre-Trial

Trial Information

General Information

Title
The Effects of Generative AI on Learning
RCT ID
AEARCTR-0014930
Initial registration date
December 02, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
December 03, 2024, 1:40 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
December 04, 2024, 11:16 AM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country
United Kingdom of Great Britain and Northern Ireland
Region
England

Primary Investigator

Name
Natalie Irmert
Affiliation
Lund University
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Catalina Franco
PI Affiliation
Center for Applied Research (SNF) at NHH – Norwegian School of Economics
Contact Investigator
PI Name
Siri Isaksson
PI Affiliation
Norwegian School of Economics
Contact Investigator

Additional Trial Information

Status
In development
Start date
2024-12-03
End date
2025-12-31
Keywords
Education, Gender
Additional Keywords
Artificial intelligence, learning
JEL code(s)
I21, I24, J16
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Artificial Intelligence (AI) is becoming an increasingly important skill in the labor market, but will it also affect academic success? Recent research shows that current students --who will be facing this rapidly changing labor market-- are adopting AI tools at differential rates based on both gender and ability. Whether AI will affect adopters’ academic success hinges on whether AI interferes with or enhances learning, which in turn depends on whether AI is being used as a substitute for or complement of own effort. If AI harms learning, students with high adoption rates would be worse prepared for the labor market than those with low adoption rates. If AI enhances learning, students who do not become proficient at AI would be left behind. To assess the impact of AI on learning, we run a controlled lab experiment which allows us to restrict and allow access to AI in different between-subject treatment variations in which students learn about a new topic. We explore whether AI is employed in a way that causally creates a gap in learning and as a consequence payoffs. Our results provide evidence on the important question of whether the documented differential adoption and use of AI by gender and ability is likely to create gender gaps in academic success. In addition, we explore several mechanisms such as over-reliance on generative AI tools, engagement and motivation.
External Link(s)

Registration Citation

Citation
Franco, Catalina, Natalie Irmert and Siri Isaksson. 2024. "The Effects of Generative AI on Learning." AEA RCT Registry. December 04. https://doi.org/10.1257/rct.14930-2.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
We design a laboratory experiment with three between subject treatment variations. In the baseline treatment, students learn without access to AI. In the AI-assisted treatment, students learn with access to AI. In the AI-guided treatment, students learn with access to AI and examples of ways to interact with AI to promote learning. This will allow us to answer the crucial question of whether, and for whom, AI is helpful as a learning tool and whether providing guidance of how to use the tool matters.
Intervention (Hidden)
We will collect data from 600 current students at the University of Nottigham at the CedEX Lab. Treatment will be randomized at the individual level.
The three treatments are:
-T1: No access to ChatGPT (Control)
-T2: Access to ChatGPT (AI-access)
-T3: Guided access to ChatGPT (AI-guided)

T1 was designed to mirror learning before prior to ChatGPT and to work as a baseline against which we can measure the impact of giving access to (guided) AI to see if it helps or hurts learning outcomes. Since students would have access to a web-browser and the learning online before, we will provide access to this during stage 2 in T1. We will however block websites providing access to AI sites in order to block the access to AI.
T2 was designed to mirror learning when ChatGPT is available to students without clear guidelines, structure and training- a situation in which it is up to each student whether and how she or he wants to use AI in the learning process. This mirrors the situation which many universities face as they are grappling with the important task of adjusting their policies to the advent of AI. Subjects will access ChatGPT on their browser with a logged in paid account. The account will be provided by the researchers an thus be cleared of any memory. Prompting history will be archived and saved by researchers to be analyzed.
T3: This treatment is the exact same as T2, except that students now also receive guidance (written by the researchers) on how to use ChatGPT. The idea behind this treatment is to explore the potential of providing not just equal access but also equal training and encouragement in AI use for students.
Intervention Start Date
2024-12-03
Intervention End Date
2024-12-13

Primary Outcomes

Primary Outcomes (end points)
The main outcome variable will be the score that students get on their final exam in stage 3. We will compare across treatments to see whether access to (guided) AI helps or hurts learning outcomes, as measured by the stage 3 score.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Subgroup analysis by gender, GPA.
Secondary outcomes: being a top/bottom scorer, number of practice questions solved, number of practice questions solved correctly, time spent on exam, time spent on practice questions, usage of AI tool, usage of Google Search and Google Translate, number of prompts in ChatGPT, text analysis of prompts in ChatGPT, self-assessed learning, enjoyment of learning, being comfortable with tools, being confident about using tools, effort put in learning process, engagement with learning process, perceptions on over-reliance on tools.
Secondary Outcomes (explanation)
The text analysis of ChatGPT prompts will be exploratory. We will be looking for whether the students asked directly for answers or whether they wrote prompts that would allow them to enhance their learning process. We will look for commonly used words and the content of the prompts and ChatGPT responses. We will use ChatGPT to help evaluate the quality of prompts.

Experimental Design

Experimental Design
Regardless of treatment, subjects will learn about the same topic. The treatment differences lies in whether or not they have access to ChatGPT/ChatGPT with guidance while learning.
Each session will adhere to the same procedure:
-Subjects learn for 15 minutes using written learning materials provided by the researchers.
-Subjects complete practice questions with access to different learning aids depending on treatment.
-Subjects take a 10 minute test without access to any learning aids.
-Subjects take a post-study survey asking them about demographics as well as attitudes and beliefs about AI.
Experimental Design Details
All subjects will learn about the same topic: Esperanto. The topic was selected since it is a language that very few people know or have studied and the point is to see how well students can learn about a new topic (rather than measuring their pre-existing skills.
Randomization Method
Students draw number from a ballot upon arrival in the lab.
Randomization Unit
individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
no clusters
Sample size: planned number of observations
600 students
Sample size (or number of clusters) by treatment arms
200 students control, 200 students AI assisted, 200 students AI guided
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
Nottingham School of Economics Research Ethics Committee
IRB Approval Date
2024-10-16
IRB Approval Number
N/A
Analysis Plan

Analysis Plan Documents

GenAI_learning_PAP.pdf

MD5:

SHA1:

Uploaded At: December 04, 2024

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials