Learning with Machines: A Randomized Evaluation of AI-Assisted Education in Schools

The intervention provides structured AI-assisted tutoring sessions to ninth-grade students in eight rural middle schools. Students assigned to the treatment group attend a daily two-hour session using an AI chatbot (Doubao) on tablets, following a fixed six-step structure: (1) a short pre-test on the day's topic, (2) AI-generated diagnostic feedback based on the pre-test, (3) an extended period of personalized AI-guided practice in which problem difficulty adapts to student performance, (4) an AI-generated summary of the key concepts covered, and (5) a post-test on the same topic. Students receive a small token incentive, redeemable for a gift, for completing all required steps of each session. Research assistants are present to monitor session integrity and record participation.
Students in the control group continue with their regular classroom instruction and complete the same pre- and post-tests as the treatment group, without access to the AI tutoring sessions, allowing learning gains to be compared across groups. The intervention is delivered over several weeks in the lead-up to the high school entrance examination (zhongkao), with schools entering the study on a staggered schedule."

2026-04-13

2026-06-22

The primary outcome is student performance on the zhongkao (中考), the high school entrance examination, measured as total score and, where available, subject-specific scores (particularly math), drawn from official exam/administrative records matched to each student.
Where actual zhongkao answer sheets or item-level scoring can be obtained, we will additionally conduct a topic-level (question-by-question) analysis: for each exam question, we will code whether it falls within a topic covered during the AI tutoring sessions, and compare treatment-control differences in correctness on covered-topic questions versus non-covered-topic questions. This tests whether any overall treatment effect on total score is concentrated in the specific content areas addressed by the intervention, as opposed to a general test-taking or motivation effect.

This topic-level analysis serves as a within-student, within-exam falsification/mechanism test: since topic coverage varies by session and school (given the staggered rollout, not all schools cover identical topics before the exam), a true content-specific learning effect should show larger treatment-control gaps on covered-topic items than on non-covered-topic items within the same exam and student. This is contingent on obtaining access to item-level scoring or original answer sheets, which is not guaranteed, so it is reported as a planned analysis conditional on data availability, secondary to the total-score comparison

Short term outcomes include session pre/post test score gain, mock zhongkao test scores. Non-cognitive and psychological measures at endline: growth mindset beliefs about intelligence, math self-efficacy/persistence on difficult problems, perceived returns to effort, exam-related stress, and depressive-symptom items (sad, unhappy, hopeless feelings over the past 7 days).
Educational aspirations and post-exam track choice: whether students plan to attend academic high school, vocational school, or enter work, and whether exposure to the program's instructors increased aspirations for further schooling.
AI usage patterns and attitudes among all students (including control): frequency and purpose of independent AI use, beliefs about AI's helpfulness and dependency risk, and peer-to-peer spillover of AI-learned methods (whether students shared or learned techniques from classmates).
Classroom environment spillovers: reported changes in classroom disruption (talking, phone use) during the intervention period, as a proxy for externalities on non-participating peers.
Program satisfaction and take-up among treated students: perceived usefulness of each protocol step, self-reported attendance/compliance, and suggestions for improvement.

These secondary outcomes serve two purposes. First, items 1–3 test plausible mechanisms and shorter-run channels through which AI tutoring could affect the zhongkao: immediate learning gains, shifts in mindset/self-efficacy and psychological wellbeing under an intensified study routine, and changes in students' educational aspirations and track choice. Second, items 4–5 are designed to detect contamination and spillovers that could bias a simple treatment-control comparison — since control students are not prevented from using AI tools on their own, and treated and control students attend the same schools and can share methods or experience classroom-level externalities. Item 6 is intended to characterize implementation fidelity and student-perceived value of specific protocol components (e.g., diagnostic feedback vs. the AI practice loop vs. token incentives), which can help interpret heterogeneity in treatment effects.

This is an individually randomized controlled trial conducted in eight rural middle schools among ninth-grade students preparing for the zhongkao (high school entrance examination). The intervention takes place during the evening self-study period, a block of time with no formal classroom instruction, during which students are otherwise expected to study independently. Within each school, students are randomly assigned to treatment or control, stratified by baseline test score tertile and gender. Students assigned to treatment use this self-study time for a structured two-hour AI-assisted tutoring session with an AI chatbot on tablets, delivered by research staff. Students assigned to control spend this same evening self-study period as they normally would: studying independently, with a teacher available in the room to answer questions if a student asks, but without any structured instruction or AI access. Both groups complete the same periodic assessments. Schools enter the study on a staggered schedule between mid-April and mid-May 2026, so students accumulate varying amounts of exposure time before the exam. The main comparison of interest is between treatment and control students' performance on the zhongkao. A separate, non-randomized comparison sample of students in two urban schools was also surveyed to help interpret rural-urban differences in outcomes and AI access, but is not part of the randomized comparison.

Not available

Randomization done in office by computer (using a random number generator in statistical software), at the individual student level within each school, stratified by baseline academic performance and gender.

Individual student, randomized within school. Randomization was implemented separately within each of the 8 participating rural schools (stratified by baseline academic performance tertile and gender within each school), rather than pooled across schools, so school functions as a stratification block, and the unit of random assignment is the individual student.

No

Not applicable. This is an individually randomized design, not a clustered design. The 8 schools serve as stratification blocks (randomization occurs within each school), not as randomization clusters.

880 ninth-grade students (approximately 410 assigned to treatment and 470 assigned to control) across 8 rural middle schools. This reflects 940 students who completed the baseline survey, of whom 60 declined to participate in the experiment; the remaining 880 were randomized. The treatment/control split is not 50/50 because the number of treatment slots was constrained by the number of tablets purchased for the program (a budget constraint).

410 students in treatment group across 8 schools, stratified by baseline academic performance and gender.

Using the actual randomized sample of N=880 (410 treatment/470 control), two-sided α=0.05, 80% power, R²≈0.40 (baseline academic performance covariate, per Muralidharan, Singh & Ganimian 2019): intent-to-treat MDE ≈ 0.15 SD of the standardized zhongkao score. Accounting for 10-20% non-compliance among treated students, the treatment-on-the-treated MDE is approximately 0.16-0.18 SD.