Experimental Design
RE-ALTER will be conducted to collect evidence of the effect of student use of ALTER-Math on their math learning and engagement. Leveraging Math Nation, we can selectively deploy updates and randomized experiments to ALTER-Math. The pilot study is a two-level cluster randomized trial (CRT) with students clustered within teachers from Florida middle schools. The random assignment will occur at the teacher level with a total of 32 teachers (16 in each condition), and with an average of 2 classrooms per teacher and 25 students per classroom. We propose to randomize at this level to avoid spillover effects that are likely to occur with random assignment at the student level. The control group is a business-as-usual condition, using Math Nation in the same frequency as the experimental group, but without ALTER-Math support. This design meets the standards of the What Works Clearinghouse (WWC) version 5.0 without reservations (U.S. Department of Education et al., 2022). We will test baseline equivalence using balance tests but do not expect imbalances due to the large, random sample.
Methods: We will start recruiting teachers in early Fall through partnered school districts that are committed to using Math Nation (see Appendix A). In late Fall 2024, all participating teachers will complete a synchronous online PD on ALTER-Math ($250 incentive for each teacher participant). Teachers will complete a knowledge measure after the training and have to answer 80% of the items correctly to proceed in the study. Randomized assignment of the control and experiment groups will begin in early Spring 2025, following the PD session, to officially launch the study. If assigned to the experiment group, participating teachers will commit to using ALTER-Math with their students at least one hour per week in early Spring 2025. Regardless of intervention assignment, each teacher will receive $1,000 as compensations. The fidelity of implementation will be checked through system logs. We will communicate regularly with treatment teachers to ensure a high level of fidelity of implementation. The intervention will last three months, which is approximately 12 weeks. Previous studies of AI-based interventions in middle school have shown that regular weekly use during the Spring semester leads to improved student achievement on a high-stakes test administered at the end of the semester (Leite et al., 2022). All participating teachers will have full access to ALTER-Math for the duration of the LEVI project after the experimental study.
Inclusion of underserved students: In order to guarantee the participation of students from economically disadvantaged and racially minority backgrounds, our focus is on partnering with districts that (1) have a substantial student body to maximize our influence and (2) serve a large proportion of financially challenged (using free lunch or reduced-price meal status as a proxy) and minority (available in Math Nation database) students. To this end, we have identified an initial list of K-12 school districts that have committed to a four-year use of Math Nation. We have access to all the data generated by the users from these school districts in Math Nation, including demographics such as race, gender, and schools, behavioral logs, and in-platform assessments.
Data Analysis Plan: We will assess the effect of ALTER-Math on student interest, proximal and distal achievement scores using a two-level ANCOVA model:
Y_jkt=γ_0+γ_1 Z_k+γ_2 (X_jkt-M_k )+γ_3 (M_k-M)+u_k+ε_ijk
where Yjkt is the standardized posttest score on an outcome measure for student j in teacher k in treatment t. Zk is an effect-coded variable indicating the condition to which teacher k was assigned, Xjkt is the pretest score, Mk is a teacher-level mean pretest score, M is the grand mean pretest score, and uk and εjkt are teacher and student-level residuals, respectively. For the FAST, the post-test score is the third assessment given at the end of the school year (April to May, 2025) and the pre-test score is the second assessment administered in mid-year (January, 2025). We will explore the inclusion of fixed district effects during data analysis. Between-teacher and within-teacher covariates by treatment interactions will also be investigated. Standardized mean difference effect sizes will be reported for each analysis.
To assess the effect of ALTER-Math on weekly student engagement across 12 weeks of the Spring semester, we will use the following growth curve model:
ϴ_ijk=δ_0+δ_1 T_ijk+δ_2 Z_k+r_0jk+r_1jk T_ijk+u_k+ε_ijk
where ϴijk is the engagement estimate at week i from 1 to 12 to for student j in teacher k in treatment Z. 𝛿₀ is the average intercept and r0jk is a random intercept of students, which quantifies the student initial engagement status. 𝛿₁ is the mean growth per week and r1jk is the random slope of week, quantifying the individual linear rate of change of engagement over the Spring semester. uₖ is the school random effect and 𝜀ijk is a residual. The week indicator will be coded from 0 to 11 so that the intercept can be interpreted as the initial level of engagement. The parameters of interest in this model are the mean of the individual intercept (i.e, the average initial status); the variance of the intercept; the mean of the slope of week (i.e., the average change in engagement per week); the variance of the slope; and the correlation between intercept and slope.