Adopting Computer Assisted Learning (CAL) at Scale: Training and Supporting Teachers and Families to Use CAL Technologies in Puerto Rico Public Schools

Last registered on May 30, 2022


Trial Information

General Information

Adopting Computer Assisted Learning (CAL) at Scale: Training and Supporting Teachers and Families to Use CAL Technologies in Puerto Rico Public Schools
Initial registration date
August 30, 2021

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
August 31, 2021, 1:29 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
May 30, 2022, 2:05 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.


There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

University of Toronto

Other Primary Investigator(s)

PI Affiliation
University of Toronto
PI Affiliation
University of Puerto Rico
PI Affiliation
University of Toronto

Additional Trial Information

In development
Start date
End date
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Educators and policymakers are facing growing disparities in academic achievement between high and low-income students, differences that appear to have increased as a result of the Covid-19 pandemic. A promising cost-effective strategy to address this issue is the use of Computer Assisted Learning (CAL) technologies, which help students progress through topics at their own pace while receiving feedback and support. However, whether CAL interventions can be scaled up effectively remains a first-order question. Our project is a collaboration with the Puerto Rico Department of Education to experimentally evaluate the impact of a large-scale training and coaching program to support mathematics teachers to use Khan Academy, one of the most popular mathematics-oriented CAL educational platforms. The program will be gradually phased-in and offered at scale to the population of fourth to eighth grade Mathematics teachers in the system’s approximately 650 primary and middle schools. We will also test whether complementary technology-enabled behavioral interventions can promote its use via parental involvement. The study will examine the program’s impact on math achievement in elementary and secondary school classrooms, as well as student, teacher, and parent experiences with the program.
External Link(s)

Registration Citation

Bobonis, Gustavo J. et al. 2022. "Adopting Computer Assisted Learning (CAL) at Scale: Training and Supporting Teachers and Families to Use CAL Technologies in Puerto Rico Public Schools." AEA RCT Registry. May 30.
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details


The Puerto Rico Department of Education (PRDE) is seeking to introduce teaching reforms in Mathematics to improve educational outcomes in the territory. The study will rigorously evaluate - via a randomized controlled trial - the impact of the ATEMA (Aplicación de la tecnología en la enseñanza de matemáticas) Computer Assisted Learning (CAL) program. The ATEMA program is a teacher training and coaching program designed to support math teachers to use one of the most popular mathematics-oriented CAL educational platforms, Khan Academy (KA). The program will be gradually phased-in and offered at scale to the overall population of approximately 1,630 fourth to eighth-grade math teachers in the PRDE’s approximately 670 primary and middle schools over a three-year period (Academic Years 2021-22 through 2023-24). The overarching objective is to implement a scalable program, designed to function well whether classes are taught online or in person, to improve student achievement and experience in a manner which is free, easy to use, encourages engagement, and which also improves teacher satisfaction and workload.

We will support the PRDE’s development of and test an in-class or virtual program that incorporates KA into in-person or online grades 4-8 math classrooms, particularly as a tool for reinforcing classroom instruction and greater personalization. The training and support of each cohort of teachers will take place over the course of one academic year (approximately 20 hours), and will comprise of three types of sessions: (i) an initial six-hour session before the start of the academic year; (ii) a second session focused on incorporating KA into the class content; and (iii) 18 weekly sessions of 20 to 30 minutes. The program will also include a second treatment component that will establish teacher-parent communication with actionable suggestions, based on KA administrative data, to increase students’ use of and engagement with the platform.
Intervention Start Date
Intervention End Date

Primary Outcomes

Primary Outcomes (end points)
The primary outcome of interest will be students’ math achievement; specifically, standardized test scores administered by the PRDE. Test scores will be normalized according to the distribution of scores in control group schools in each grade.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Additional secondary outcomes of interest will aid us to measure and estimate possible effects on (a) teacher usage of the KA platform as measured by a binary indicator for participation in a session; the teacher’s set up of the KA account with the class; and the number of ATEMA coaching sessions attended; (b) students’ performance in the KA platform (i.e., the number of skill sets improvement over the year), and the demonstration of student math achievement at proficient or above as defined by PRDE standardized tests (a measure directly relevant to PRDE policymakers); (c) academic performance based on standardized tests in other subjects (i.e., Spanish, English, Science); and (d) data from online interviews and surveys with a subsample of students, teachers, and parents to measure experiences with the program.
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The experimental design comprises two stages. Approximately 670 PRDE schools were stratified by academic level (i.e., primary, middle school), PRDE region, and performance (below or above the median school) at baseline. All the schools in Puerto Rico with students in 4th- to 8th-grade are part of the analysis. All schools within each stratum were randomly assigned to one of five main experimental arms:
● Treatment Arm 1 (TA1): Cohort 1 — ATEMA program (AY 2021-22)
● Treatment Arm 2 (TA2): Cohort 1 — ATEMA program + Parental Engagement Emails (AY 2021-22)
● Treatment Arm 3 (TA3): Cohort 2 — ATEMA program (AY 2022-23)
● Treatment Arm 4 (TA4): Cohort 2 — ATEMA program + Parental Engagement Emails (AY 2022-23)
● Control Group (Control): Cohort 3 – Status Quo
Each of the school cohorts in this stepped-wedge design is composed of 223-224 schools, with approximately 112 schools in each one of TA1-TA4. Eligible teachers in each school will be invited to enroll in ATEMA, and we will engage school principals, regional math administrators, and regional superintendents to encourage their staff to participate. The randomization of the supplementary parental information and engagement intervention was also performed at the school level: parents of students in half of the schools eligible for the ATEMA program (112 schools) in cohorts 1 and 2 have received or will receive the parental engagement treatment each year.
Experimental Design Details
Not available
Randomization Method
The randomization of the first treatment was done by a computer in Stata 11. The randomization was run 100 times following Banerjee et al. (2017). Balance checks were performed on variables including school and teachers characteristics. The code used a max-min p-value criteria, keeping the randomization with the largest minimum p-value among the 45 variables used for balance checks.
Randomization Unit
For all the treatment arms, the unit of randomization is the school.
Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters
There is a total of 671 schools in the sample.
Sample size: planned number of observations
For test scores outcomes, there is an average of 160 students per school. This represents a total of approximately 104,000 students.
Sample size (or number of clusters) by treatment arms
For the first treatment, this is a phase-out design with three cohorts of schools. Cohort 1 – Teaching with Khan (ATEMA) program (AY 2021-22); Cohort 2 – ATEMA program (AY 2022-23); and Cohort 3 – Status Quo. There are approximately 223-224 schools per cohort eligible to participate in the program in each academic year. For the second treatment, the randomization is also performed at the school level, with half of the schools in the ATEMA program receiving the complementary parental engagement intervention. This accounts for a total of 112 schools each year in the ATEMA program that also receive the parental engagement treatment.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Based on earlier CAL research, we aim to be able to detect an ITT minimum effect of increasing mathematical achievement by 0.10s (standard deviations) given a 50% program take-up rate (or 0.20s among takers). The power calculations that we present next are for the comparison between TA1 and the Control group at the end of Year 2 of the study. Data on PRDE student test scores from previous years shows an intra-cluster correlation of 0.12 at the school level (ρ=0.12). Our power calculations consider 80 students and 2.4 Grade 4-8 math teachers per school, on average, with 112 schools in each Treatment Arm and 224 in the Control group. We assume that the outcome variable is standardized within the test-taking population and that after controlling for baseline scores, the residual standard deviation equals 0.9 (sd=0.9). Given this cluster-randomized design, power calculations for ITT effects (power=0.8, α=0.05) indicate that the MDE of comparing TA1 and the control group at the end of year 2 is 0.106s. Our power analysis is conservative as we will use other baseline variables to reduce the outcome's residual variance. We also perform power calculations for the short-term effects of the program. First, we calculate MDE for the comparison between schools in TA1 and the comparison group (the control group, TA3, and TA4) in Year 1 of the program. We use the same parameters for the intra-cluster correlation (ρ=0.12), and the standard deviation of the outcome (sd=0.9). The MDE of this comparison is 0.096s. Second, the MDE for comparing TA3 and the control group two years after the start of the program is 0.106s. Again, this power analysis is conservative as we will use other baseline variables to reduce the outcome's residual variance. Given that our sample includes all primary and middle schools in Puerto Rico, we have statistical power to identify reasonably small MDEs. Furthermore, pooling Year 1 data for students in Cohort 1 and Year 2 data for students in Cohort 2 will allow us to gain significantly greater precision to detect short-term one-year ITT estimates of the interventions.

Institutional Review Boards (IRBs)

IRB Name
University of Puerto Rico
IRB Approval Date
IRB Approval Number
IRB Name
University of Toronto
IRB Approval Date
IRB Approval Number
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information