NEW UPDATE: Completed trials may now upload and register supplementary documents (e.g. null results reports, populated pre-analysis plans, or post-trial results reports) in the Post Trial section under Reports, Papers, & Other Materials.
Solving the 2 Sigma Problem with Khan Academy: A Pilot Study
Initial registration date
July 17, 2020
July 21, 2020 11:40 AM EDT
This section is unavailable to the public. Use the button below
to request access to this information.
University of Toronto
Other Primary Investigator(s)
Brigham Young University
Additional Trial Information
Computer Assisted Learning (CAL) is educational software designed to help students progress through material at their own pace while receiving feedback and advice, similar to the kind that a tutor might provide but on a computer. CAL also makes it easier for teachers to monitor and grade progress. Teachers can assign new topics based on each students’ level of understanding. Despite a large and increasing body of convincing research suggesting large benefits of adopting CAL, many administrators and teachers still shy away. Uncertainty about how to use the software, concerns about subscription costs, and lack of support may explain this reluctance. This pilot aims to develop and test a program that simplifies the adoption of CAL and helps ensure its effective use. The pilot uses Khan Academy (KA), one of the most popular CAL programs, as a tool for Grades 3-8 math teachers to use in class and as graded homework. The program has the potential to save time for teachers, lower costs, and offer a more customizable, enjoyable, and effective learning experience for students. The goal of the pilot is to work out design details for maximizing engagement and simplifying usage. The overall goal is to provide convincing evidence for policy makers, administrators, and teachers to want to adopt the program on a state-wide or national scale.
We propose to pilot a program that provides training and support for teachers to use KA effectively in Grades 3-8 mathematics. We chose mathematics because previous research (cited above) finds CAL can be particularly effective with this subject. We chose Grades 3-8 because KA’s resources for these levels are particularly well developed. The pilot program has the potential to save time for teachers and offer a free, more customizable, enjoyable, and effective learning experience for students. The pilot would help determine best practices for the implementation of a larger randomized controlled trial. Qualitative and quantitative analyses would be used to identify key implementation challenges and solutions, whether classes are provided online or in-person. If a sufficiently large number of classes participate, we will also implement the study as its own randomized controlled trial.
Intervention Start Date
Intervention End Date
Primary Outcomes (end points)
Intermediate outcomes will come from class attendance and grade records. We will also use usage data from Khan Academy. A support letter from them is attached. We will also match students to their standardized math achievement scores from the Utah Performance Assessment System (U-Pass). Qualitative data would be collected on student/teacher/parent experiences and feedback from administrators about the program.
Primary Outcomes (explanation)
Secondary Outcomes (end points)
Secondary Outcomes (explanation)
Randomized at the classroom or grade level. Treatment group receives program in the Fall, 2020, control group receives it in the Spring, 2021.
Experimental Design Details
Randomization done in office by a computer (using STATA).
Class or grade-level.
Was the treatment clustered?
Sample size: planned number of clusters
10 schools, 10 classes each, for a total of 100 classes.
Sample size: planned number of observations
Assuming 20 students per class, target sample size is about 2,000 students.
Sample size (or number of clusters) by treatment arms
50 classes treated, 50 contol.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Hedges and Hedberg (2007) reports intraclass correlation coefficients across schools of about 0.20 without covariates and about 0.15 with covariates. Since our analysis uses clusters within school and grade and includes covariates, we assume an intraclass correlation of 0.15. Further assuming 20 students per class, 76 classes would be needed to attain 80 percent statistical power at a 5 percent significance level. Therefore, we would need about 8 schools, with 10 classrooms per school to have adequate statistical power for conducting an experiment during the pilot. We would need about 5 schools if we instead assume 70 percent statistical power to detect at least a 20 percent standard deviation effect with an intraclass correlation of 0.1.
INSTITUTIONAL REVIEW BOARDS (IRBs)