Pre-Analysis Plan for A Large RCT of ALTER-Math: AI-Powered Learning by Teaching to Enhance and Renovate Middle School Math Learning

Middle school math, a key milestone to engaging in advanced mathematical study, significantly influences students' STEM education, career trajectories, and the broader national STEM workforce (Kieran, 2004; Matthews & Farmer, 2008). However, at this moment, middle school mathematics education faces major challenges, including the lack of resources and approaches to engage students both cognitively and affectively (Major et al., 2021; Mitchall & Jaeger, 2018; Williams et al., 2018). Nationally, only about 26% of eighth graders are proficient in math, and in some high-poverty districts, the rates are in the single digits (Learning Engineering Virtual Institute, n.d.)​. Such disparities highlight a critical equity issue: without effective interventions, many historically underserved students fall behind in math, limiting their future educational and economic opportunities. Therefore, improving middle school math outcomes is a priority for policymakers focused on closing achievement gaps and promoting workforce readiness. Findings from a recent study at the National Bureau of Economic Research (Guryan et al., 2021) offer a promising solution, where the research demonstrates that individualized tutoring significantly improved annual math learning gains, improved test scores, and reduced course failure rates for historically underserved students. This rigorous research provides stakeholders with an empirically and longitudinally evidenced strategy to support students, especially those from an economically disadvantaged background, by catering to their personalized needs and individual differences.
Augmenting mathematics tutoring with artificial intelligence (AI) tools has demonstrated potential for success. AI presents scalable solutions for students with diverse learning needs through adaptive learning (e.g., DreamBox Learning Inc.), intelligent tutoring systems (e.g., MATHia, formerly known as Cognitive Tutor, Ritter et al., 2007), and teacher-centered AI (e.g., ASSISTments, Heffernan & Heffernan, 2014). Recently, AI technologies have advanced from structured, expert-modeled, and rule-based systems to more dynamic, intuitive, and context-aware methodologies through generative artificial intelligence such as large language models (LLMs). Despite AI's potential for reimagining mathematics education, current implementations often treat students as passive recipients of AI-driven decisions, where AI provides direct answers, guidance, and feedback to students. Supported by the Learning Engineering Virtual Institute (LEVI), our project, AI-powered Learning by Teaching to Enhance and Renovate Math Learning (ALTER-Math), addresses these gaps by transforming students from passive learners into proactive teachers of AI-powered teachable agents, utilizing LLMs and multimodal AI technologies. While AI is yet to be a superior teacher, it is already an excellent student.
Led by the University of Miami (UM) and the University of Utah (Utah), with a collaborative network of experts and engineers from Stanford, Duke, Vanderbilt, and Math Nation, ALTER-Math aims to enhance math learning for middle school students, especially those from low-income households. In ALTER-Math, students guide, teach, and interact with the AI agent in multiple representations (e.g., graphing, drawing, tables, and texts) to solve math problems embedded into different real-world scenarios (e.g., music, environmental issues, and cultural festivals). Through partnership with and direct integration into Math Nation—an online math curriculum with over one million active students annually—we have developed robust cyberinfrastructure guided by effective learning sciences principles, conducted classroom studies showing promising results (Xing et al., 2025), and gained key insights to refine our AI-driven approach.

2026-10-01

2027-03-31

The Florida Assessment of Student Thinking (FAST) will serve as a distal measure of middle school students' mathematics performance. Administered three times a year and aligned with Florida’s BEST standards, FAST provides a comprehensive, computer-adaptive assessment of student growth, with tests lasting 100–120 minutes and featuring multiple-item formats (Florida Department of Education, 2024). We will use the first wave of FAST (PM1) in Fall 2026 as the baseline and the third wave (PM3) in Spring 2027 as the post-intervention measure to evaluate the effects of ALTER-Math.

To assess students’ interest in mathematics, we will use the Student Interest in Mathematics Scale (Wininger et al., 2014), a validated 17-item Likert-scale instrument measuring emotion, value, knowledge, and engagement (Cronbach's α = 0.76–0.87). Lastly, students’ interactive log data will be used as a proxy for engagement, capturing time-on-task, retention, activity frequency, and interactions with AI agents (Kim et al., 2020). Engagement trends will be analyzed weekly across the RCT period to assess growth

To rigorously evaluate the efficacy and scalability of ALTER-Math, we propose a three-level cluster randomized trial (CRT) involving approximately 7,500 students (~50% low-income) and 150 teachers from 53 schools in Florida. Our study will be guided by the overarching research question of How effectively does the AI-powered teachable agent engage students and enhance students’ mathematical achievements? We will also conduct sub-group analysis to answer research questions, including (1) To what extent does the use of ALTER-Math within and outside Math Nation impact students’ mathematical engagement and achievements? (2) How do usage patterns differ between historically marginalized middle school students regarding race, socio-economic status, and English learner status, and their more advantaged peers, when using the AI teachable agent? and (3) To what extent does the AI teachable agent influence math learning outcomes for historically marginalized middle school students compared to their more advantaged peers?

Study Design Overview: The 28-week study (October 2026–April 2027) will be a three-level cluster randomized trial with students nested within teachers nested within schools. Random assignment to treatment and control conditions will occur at the school level. We propose to include 50 middle schools (25 per condition), with an average of 3 teachers per school. Each teacher is expected to teach two classrooms of approximately 25 students, resulting in an estimated total sample size of 7,500 students. To strengthen contrasts, we will include students from Duval County, who do not have access to Math Nation. This allows us to examine the effects of ALTER-Math in a distinct context where Math Nation is not available. For schools with access to Math Nation, we will leverage the platform to randomize access to ALTER-Math. This approach will streamline teacher recruitment and simplify the management of treatment and control groups. For Duval County, we will recruit teachers during in-person professional development (PD) sessions, document their intent to participate, and randomly assign schools to intervention or control conditions. By randomizing at the school level, we minimize potential spillover effects that could occur with student-level randomization and ensure compliance with What Works Clearinghouse (WWC) Standards 5.0 without reservations (U.S. Department of Education et al., 2022). While baseline equivalence on pre-test measures is generally expected in large randomized samples, differences greater than 0.05 but less than 0.25 standard deviations will be addressed by including pre-test measures as covariates in our analytic models.

Not available

Randomization will occur at the school level after teacher recruitment and baseline rosters are finalized. Schools will be assigned in a 1:1 ratio to treatment or business-as-usual control using a reproducible randomization script, with assignment conducted by the research team before implementation begins. To support balance across study conditions while keeping the design transparent, randomization will be stratified by district and Math Nation access context. Within each stratum, schools will be randomly assigned to treatment or control to the extent possible, recognizing that the number of participating schools may vary across districts and strata. After randomization, the research team will examine baseline equivalence across conditions using available student-level characteristics, including gender, FRPM eligibility, and other relevant demographic indicators. If any pre-specified baseline covariate differs across arms by more than 0.25 standard deviations in the provisional draw, we will use restricted randomization or rerandomization before assignments are released. Students and teachers inherit the assignment of their school.

School level randomization

Yes

50 schools, with each school on average having 3 teachers

7,500 students, 150 teachers, and 50 schools

25 schools control, and 25 schools treatment

Using the PowerUp! software (Dong & Maynard, 2013), we estimated the minimum detectable effect size (MDES) achievable with 44 schools, each with an average of 3 teachers (50 students for each teacher). Variance proportions between schools (ICC = 0.38) and teachers (ICC = 0.20) were specified based on prior research (Leite et al., 2022). We also assumed substantial variance explained by the pre-test (R² ranging from 0.37 to 0.87 across levels). The calculated MDES at a power of 0.8 and α = 0.05 is 0.25; the effect size is aligned with our previous large-scale quasi-experiment and pilot RCT (see Viability and Value for details). A 90% retention rate is assumed, leading to an estimated 50 schools needed to maintain sufficient statistical power for the study. The proposed study follows a three-level cluster randomized trial: students (Level 1) are nested within teachers (Level 2), and teachers are nested within schools (Level 3). Randomization will occur at the school level. The following sample size calculation process can be replicated in the PowerUp! software (Dong & Maynard, 2013) based on information provided in the Power Analysis subsection in the Technical Design section. The number of schools estimated is 44. With a 90% retention rate, 50 schools will be recruited.