The effect of AI-enhanced instruction on students’ learning, evidence from a randomized controlled trial

In December 2023, the French Ministry of Education announced the launch of an app in September 2024 for students in grade 10 (first year of high school in France), to support students’ learning in French and Mathematics, as an answer to France's poor results to PISA (Programme for International Student Assessment, led by the OECD).

The app’s primary focus is an adaptive tool based on AI that seeks to adapt the exercises offered to students based on their profiles. Hence, the main goal of this tool is to support French and math teachers in managing the diversity of their classes in terms of skills.

The intervention we study gives access to this app to a large set of high schools. We randomize schools into three groups.

In the control group, students and teachers have access to a simpler version of the app and can access a large library of exercises but teachers have to select the exercises presented to students. Hence, it is comparable to a digital textbook.

In the first treatment group, students and teachers have access to the same catalog of exercises, but also to the AI module that selects the exercises submitted to students, based on an algorithm. This algorithm is based on the theory of the proximal development zone: an exercise should not be too easy, so as not to bore the student, but should not be too difficult, so as not to discourage him or her.

In the second treatment group, students and teachers have access to the large catalog of exercises, to the AI tool, as well as to a tool aimed to promote student collaboration. Teachers can pair students together, so that they have to answer together a set of exercises. In particular, the weakest student proposes an answer, which the stronger pupil must accept or reject. The first student then has the opportunity to correct his or her answer. Hence, is it comparable to peer tutoring.

For all three versions, the scope of the exercises is similar and covers the curriculum in grade 10 in French and Math. The app’s main purpose is to be a remedial tool, helping teachers personalize their instruction to lower achieving students. However, it can be used with all students.

2024-11-04

2025-08-24

The primary outcomes of the study will focus on cognitive skills in French and math, assessed at the end of the academic year. French and math assessments will be adapted from national assessments. We will also study the impact on socio-emotional skills, in particular related to collaboration and self-efficacy, using adaptations of PISA tests. We will also measure the effect on interest in learning as well as desired future specialisation.
Additionally we will focus on teacher’s practices, in particular related to digital tools, collaborative practices and heterogeneity management.
We will study the heterogeneity effects on both primary and secondary outcomes. For students, heterogeneity analysis will include learner profile from baseline data (low / high achiever, low / high motivation), gender, track (general or vocational). For teachers, heterogeneity analysis will include gender, subject (French or math), seniority, ICT profile (past experience with RCT, perceptions on AI).
In subsequent work, we also plan to study outcomes related to academic achievement (in particular, results to the national exam at the end of high school) and specialization (subjects selected during grades 11 and 12, field of study in post-secondary education). Those indicators will be collected from administrative data in the following years.

The intervention will be implemented in schools identified as voluntary by regional authorities in charge of digital education. Regional authorities had 3 months to identify all voluntary high schools. Both vocational and general tracks are included in the study. A small pilot took place the year before, and schools are excluded from the study. Private schools are excluded as well, as too few of them were identified as voluntary.

All public voluntary high schools that were not part of the pilot are included in the study. We randomize at the school level. All grade 10 French and math teachers have the possibility to take part in the study. The baseline survey is administered before treatment begins.

Randomization is done at the school level. Schools were randomly assigned to one of the three groups at the beginning of the year. For technical purposes, randomization was done prior to the baseline questionnaire.

Stratified randomization (performed on R)

Stratification variables:
School social index (2 values: below or above the median)
Remoteness index (2 values: below or above the median)
School type (3 values: general track, vocational track or mixed)

School level

Yes

Around 300 schools

Between 21 000 and 28 000 students

1/3 of the total sample of schools in each treatment arms.

0.18 depending on the final sample size