Experimental evidence on data-driven remedial instruction in the Dominican Republic

In this proof-of-concept, we aim to evaluate an in-school intervention combining the use of a Computer Adaptive Learning (CAL) software with group tutoring as a potentially scalable alternative to accelerate learning recovery among teenagers. Moreover, we aim to generate evidence on whether using CAL-generated data for targeted tutoring and teacher support can lead to better student outcomes. The intervention has two components:

Component #1 introduces the use of CAL as part of schools’ regular instruction, substituting two out of seven weekly hours scheduled for math instruction. The software selected has been adapted to the Dominican Republic’s math curriculum. This arm includes “CAL aides” to support teachers in leveraging the rich source of data the platform provides to assess student learning and adapt their instruction. CAL aides will meet for one hour every two weeks with teachers to support them in accessing and monitoring students’ assessments, tracking their progress, identifying topics that students do not yet master for lesson planning, and using resources offered by the software to ease their workload (e.g., assigning and grading homework and tests).

Component #2 introduces targeted, data-driven (based on CAL software data) small-group tutoring. This component aims to explore complementarities between CAL and more
traditional forms of instruction, such as tutoring, by providing up to three weekly hours of tutoring to high-need students. Students participating in tutoring sessions will have access to the platform (component #1), and tutors will be instructed to use the data to monitor student progress, identify learning gaps, and plan their lessons. These lessons will be implemented during school hours devoted to extracurricular activities.

2023-10-30

2024-06-30

Our pilot includes a data collection strategy to measure final and intermediate students’ and teachers’ outcomes to better understand the mechanisms underlying the results. For this, we will administer surveys among students and teachers at baseline and endline, as well as student assessments. Testing these instruments at this stage will help us to define the best approach to measure our main outcomes in the scale-up phase.
• Student outcomes: We will evaluate students’ learning outcomes in math through standardized assessments (endline survey) and will track progress using the CAL platform’s periodic assessments (time using the platform, number of mastered topics).
• Teacher outcomes: We want to assess whether the software can effectively blend with regular instruction and improve teachers’ productivity. We will focus on measuring teachers’ time use and their practices in and outside the classroom (endline survey, for example “How many hours did you spend at planning lessons/correcting tests and assignments…?, “Indicate how often the following occurred: You talked to a parent about their child's performance/You led a training workshop for your colleagues at your school,” etc.).

Classroom-level assignment, participating classrooms will be randomly assigned to two groups:
1. Treatment. Students and teachers in selected classrooms will receive access to the CAL software and general training on its use (component #1).
2. Control group. The control group will not receive access to the CAL software..

Our design would follow a matched pair randomization design, where classrooms would be matched based on their size. One of each classroom in a pair would be randomly assigned to treatment. In addition, among classes in the treatment arm, we will cross-randomize component #2 (covering 20% of the students in each selected classroom with the lowest math performance).

Each classroom will receive one of three treatment combinations: no intervention (pure control), CAL software without tutoring, and CAL software with tutoring for students with the lowest math performance (bottom 20%).

Not available

Randomization was conducted in the office using Stata.

The unit of randomization is the classroom.

Yes

~40 classrooms

~1300 students attending 9th grade.

~20 classrooms in the control group and ~20 classrooms in the treatment group (CAL treatment). Among the schools in the treatment group, ~10 classrooms will be selected to receive tutoring, which covers 20% of the students with the lowest math performance (within each classroom).