Adopting Computer Assisted Learning (CAL) at Scale: Training and Supporting Teachers and Families to Use CAL Technologies in Puerto Rico Public Schools

The Puerto Rico Department of Education (PRDE) is seeking to introduce teaching reforms in Mathematics to improve educational outcomes in the territory. The study will rigorously evaluate - via a randomized controlled trial - the impact of the ATEMA (Aplicación de la tecnología en la enseñanza de matemáticas) Computer Assisted Learning (CAL) program. The ATEMA program is a teacher training and coaching program designed to support math teachers to use one of the most popular mathematics-oriented CAL educational platforms, Khan Academy (KA). The program will be gradually phased-in and offered at scale to the overall population of approximately 1,630 fourth to eighth-grade math teachers in the PRDE’s approximately 670 primary and middle schools over a three-year period (Academic Years 2021-22 through 2023-24). The overarching objective is to implement a scalable program, designed to function well whether classes are taught online or in person, to improve student achievement and experience in a manner which is free, easy to use, encourages engagement, and which also improves teacher satisfaction and workload.

We will support the PRDE’s development of and test an in-class or virtual program that incorporates KA into in-person or online grades 4-8 math classrooms, particularly as a tool for reinforcing classroom instruction and greater personalization. The training and support of each cohort of teachers will take place over the course of one academic year (approximately 20 hours), and will comprise of three types of sessions: (i) an initial six-hour session before the start of the academic year; (ii) a second session focused on incorporating KA into the class content; and (iii) 18 weekly sessions of 20 to 30 minutes. The program will also include a second treatment component that will establish teacher-parent communication with actionable suggestions, based on KA administrative data, to increase students’ use of and engagement with the platform.

2021-08-01

2024-06-30

The primary outcomes of interest will be student math achievement; specifically, a student level index of standardized test scores as administered by the PRDE. Test scores will be normalized according to the distribution of scores in control group schools in each grade.

Additional secondary outcomes of interest will aid in measuring, quantitatively and qualitatively, student, teacher, and parent experiences with the program via Khan Academy usage, online interviews, and surveys. We will also measure academic performance in other subjects (i.e., Spanish, English, Science) to determine substitution or complementarities in cross-subject study efforts and performance.

The proposed design comprises two stages. Approximately 670 PRDE schools will be stratified by academic level (i.e., primary, middle school), PRDE region, and performance (below or above the median school) at baseline. All schools within each stratum will be randomly assigned to one of three main experimental arms:
1. Cohort 1 – Teaching with Khan (ATEMA) program (AY 2021-22)
2. Cohort 2 – ATEMA program (AY 2022-23)
3. Cohort 3 – Status Quo.

Each of the arms of this stepped-wedge design will be composed of 223-224 schools, with teachers in all schools invited to participate in the ATEMA program by the end of the third year. Eligible teachers in each school will be invited to enroll, and we will engage school principals to encourage their staff to participate. The randomization of the complementary engagement intervention with parents is also at the school level with half of the schools eligible for the ATEMA program (112 schools) receiving this treatment each year.

Not available

The randomization of the first treatment was done by a computer in Stata 11. The randomization was run 100 times following Banerjee et al. (2017). Balance checks were performed on variables including school and teachers characteristics. The code used a max-min p-value criteria, keeping the randomization with the largest minimum p-value among the 45 variables used for balance checks.

For treatments 1 and 2, the unit of randomization is the school.

Yes

There is a total of 671 schools in the sample.

For test scores outcomes, there is an average of 160 students per school. This represents a total of approximately 104,000 students.

For the first treatment, this is a phase-out design with three cohorts of schools. Cohort 1 – Teaching with Khan (ATEMA) program (AY 2021-22); Cohort 2 – ATEMA program (AY 2022-23); and Cohort 3 – Status Quo. There are approximately 223-224 schools per cohort eligible to participate in the program in each academic year. For the second treatment, the randomization is also performed at the school level, with half of the schools in the TwK program receiving the complementary parental engagement intervention. This accounts for a total of 112 schools each year in the TwK program that also receive the parental engagement treatment.