x

NEW UPDATE: Completed trials may now upload and register supplementary documents (e.g. null results reports, populated pre-analysis plans, or post-trial results reports) in the Post Trial section under Reports, Papers, & Other Materials.
Solving the 2 Sigma Problem with Khan Academy: A Pilot Study
Last registered on July 21, 2020

Pre-Trial

Trial Information
General Information
Title
Solving the 2 Sigma Problem with Khan Academy: A Pilot Study
RCT ID
AEARCTR-0006167
Initial registration date
July 17, 2020
Last updated
July 21, 2020 11:40 AM EDT
Location(s)

This section is unavailable to the public. Use the button below to request access to this information.

Request Information
Primary Investigator
Affiliation
University of Toronto
Other Primary Investigator(s)
PI Affiliation
Brigham Young University
Additional Trial Information
Status
In development
Start date
2020-08-17
End date
2021-08-24
Secondary IDs
Abstract
Computer Assisted Learning (CAL) is educational software designed to help students progress through material at their own pace while receiving feedback and advice, similar to the kind that a tutor might provide but on a computer. CAL also makes it easier for teachers to monitor and grade progress. Teachers can assign new topics based on each students’ level of understanding. Despite a large and increasing body of convincing research suggesting large benefits of adopting CAL, many administrators and teachers still shy away. Uncertainty about how to use the software, concerns about subscription costs, and lack of support may explain this reluctance. This pilot aims to develop and test a program that simplifies the adoption of CAL and helps ensure its effective use. The pilot uses Khan Academy (KA), one of the most popular CAL programs, as a tool for Grades 3-8 math teachers to use in class and as graded homework. The program has the potential to save time for teachers, lower costs, and offer a more customizable, enjoyable, and effective learning experience for students. The goal of the pilot is to work out design details for maximizing engagement and simplifying usage. The overall goal is to provide convincing evidence for policy makers, administrators, and teachers to want to adopt the program on a state-wide or national scale.
External Link(s)
Registration Citation
Citation
Oreopoulos, Philip and Joseph Price. 2020. "Solving the 2 Sigma Problem with Khan Academy: A Pilot Study." AEA RCT Registry. July 21. https://doi.org/10.1257/rct.6167-1.0.
Sponsors & Partners

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information
Experimental Details
Interventions
Intervention(s)
We propose to pilot a program that provides training and support for teachers to use KA effectively in Grades 3-8 mathematics. We chose mathematics because previous research (cited above) finds CAL can be particularly effective with this subject. We chose Grades 3-8 because KA’s resources for these levels are particularly well developed. The pilot program has the potential to save time for teachers and offer a free, more customizable, enjoyable, and effective learning experience for students. The pilot would help determine best practices for the implementation of a larger randomized controlled trial. Qualitative and quantitative analyses would be used to identify key implementation challenges and solutions, whether classes are provided online or in-person. If a sufficiently large number of classes participate, we will also implement the study as its own randomized controlled trial.
Intervention Start Date
2020-08-24
Intervention End Date
2021-06-30
Primary Outcomes
Primary Outcomes (end points)
Intermediate outcomes will come from class attendance and grade records. We will also use usage data from Khan Academy. A support letter from them is attached. We will also match students to their standardized math achievement scores from the Utah Performance Assessment System (U-Pass). Qualitative data would be collected on student/teacher/parent experiences and feedback from administrators about the program.
Primary Outcomes (explanation)
Secondary Outcomes
Secondary Outcomes (end points)
Secondary Outcomes (explanation)
Experimental Design
Experimental Design
Randomized at the classroom or grade level. Treatment group receives program in the Fall, 2020, control group receives it in the Spring, 2021.
Experimental Design Details
Not available
Randomization Method
Randomization done in office by a computer (using STATA).
Randomization Unit
Class or grade-level.
Was the treatment clustered?
Yes
Experiment Characteristics
Sample size: planned number of clusters
10 schools, 10 classes each, for a total of 100 classes.
Sample size: planned number of observations
Assuming 20 students per class, target sample size is about 2,000 students.
Sample size (or number of clusters) by treatment arms
50 classes treated, 50 contol.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Hedges and Hedberg (2007) reports intraclass correlation coefficients across schools of about 0.20 without covariates and about 0.15 with covariates. Since our analysis uses clusters within school and grade and includes covariates, we assume an intraclass correlation of 0.15. Further assuming 20 students per class, 76 classes would be needed to attain 80 percent statistical power at a 5 percent significance level. Therefore, we would need about 8 schools, with 10 classrooms per school to have adequate statistical power for conducting an experiment during the pilot. We would need about 5 schools if we instead assume 70 percent statistical power to detect at least a 20 percent standard deviation effect with an intraclass correlation of 0.1.
Supporting Documents and Materials

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information
IRB
INSTITUTIONAL REVIEW BOARDS (IRBs)
IRB Name
IRB Approval Date
IRB Approval Number