It Takes a Village (Plus AI): A Pilot Study on AI-Driven Math Tutoring to Revolutionize Learning

Last registered on October 07, 2024

Pre-Trial

Trial Information

General Information

Title
It Takes a Village (Plus AI): A Pilot Study on AI-Driven Math Tutoring to Revolutionize Learning
RCT ID
AEARCTR-0014505
Initial registration date
October 01, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
October 07, 2024, 7:11 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Primary Investigator

Affiliation
Carnegie Mellon University

Other Primary Investigator(s)

PI Affiliation
Carnegie Mellon University

Additional Trial Information

Status
In development
Start date
2024-04-01
End date
2025-08-01
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Personalized Learning Squared (PLUS) is an edtech platform that uses a hybrid human-AI tutoring intervention to boost learning gains for middle school students from historically underserved communities. We know that tutoring works to improve learning, especially among students who are furthest behind. However, tutoring is a fairly resource intensive and expensive intervention. PLUS connects remote, human tutors to students working on math software in classrooms, and uses insights from the math software to identify needs in real time and help tutors allocate their attention between students. This optimizes student gains and provides a scalable solution to increase math learning. PLUS has been operating in classrooms since the 2022-2023 school year. They have been expanding into new schools and districts, improving the PLUS platform as well as their approach to onboarding and supporting teachers and tutors, and evaluating the impact of the PLUS platform iteratively. They plan to conduct a pilot study during the 2024-2025 school year that will continue to test the implementation of the PLUS program as well as the impact on student learning outcomes.
External Link(s)

Registration Citation

Citation
Koedinger, Kenneth R. and Danielle Thomas . 2024. "It Takes a Village (Plus AI): A Pilot Study on AI-Driven Math Tutoring to Revolutionize Learning ." AEA RCT Registry. October 07. https://doi.org/10.1257/rct.14505-1.0
Experimental Details

Interventions

Intervention(s)
PLUS Tutoring is a math tutoring program that operates within school hours during regular math classes and is designed to provide personalized tutoring for all students in a school. Using data from schools’ existing math practice software (Mathia, iReady, IXL, etc.), the PLUS Tutoring software helps allocate appropriate levels of tutoring to different students. Students in PLUS will receive one of three forms of PLUS tutoring: AI/ed tech tutor added (no human tutoring), on-demand student-initiated tutoring, or data-driven tutor-initiated tutoring. All students in the study will, one day per week during math class, use a device (typically a laptop) with educational technology systems, and open a video-conferencing platform (such as Zoom or Pencil Spaces). Treatment students will receive data-driven tutor-initiated tutoring, with tutors assigned to 2-6 students based on student needs and tutoring skills. Tutors will lead sessions between 2 and 20 minutes to build relationships, identify and address motivational challenges and help get students back on track to making progress. Tutors use a dashboard (PLUS Toolkit) to track student math achievement, practice effort, and prior progress. Comparison group students will have access to on-demand student-initiatives tutoring, with 1 tutor available to serve 10-15 students. During math classes, students may use video-conferencing platforms to request help from a remote tutor. These remote tutoring sessions are also predicted to last roughly 2-20 minutes.
Intervention Start Date
2024-08-19
Intervention End Date
2025-06-16

Primary Outcomes

Primary Outcomes (end points)
Standardized end of year state math assessments (Pennsylvania System of School Assessment-PSSA and the West Virginia General Summative Assessment-WVGSA)
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
We will also measure program and student progress in math using district-approved diagnostic assessments (e.g., i-Ready Diagnostic, Pennsylvania Classroom Diagnostic Tools (CDT), NWEA MAP, Star Assessment Diagnostic) and attendance.
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We will use a regression discontinuity design (RD) to measure the impact of tutor-initiated versus student-initiated tutoring on math learning outcomes among middle school students. Previous test scores will be used as the forcing variable and a score cutoff will determine allocation into tutor-initiated or student-initiated tutoring. Students whose school records do not include this score will be randomly assigned. Spring 2024 PSSA/WVGSA scores will be used as the forcing variable for the RD. Anyone below the median PSSA score for a particular district, school, and grade, will be assigned to tutor-initiated tutoring, while students who score above the median will be assigned to student-initiated tutoring. Students with missing scores will be randomly assigned to one of the two groups. Additionally, we will analyze differences in outcomes across key subgroups, including historically marginalized students, students from low-income backgrounds, students with disabilities, English Language learners, and students from underserved populations. This will allow us to explore how these targeted groups respond to different tutoring models, shedding light on any variations in impact.
Experimental Design Details
Not available
Randomization Method
We will conduct a student-level randomization for students with missing historical PSSA and WVGSA data (done in office by a computer).
Randomization Unit
student / individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
0- no clusters
Sample size: planned number of observations
We anticipate around 3,000 students will participate in this study during the 2024-2025 school year. We anticipate around ⅔ will have test data from the previous year and will be involved in the RD, and the remaining ⅓ will have missing data and will be randomized.
Sample size (or number of clusters) by treatment arms
N/A
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
It Takes a Village (Plus AI): Doubling Math Learning by Optimizing Tutoring from Training to Practice
IRB Approval Date
2024-09-13
IRB Approval Number
Registration No: IRB00000603