Solving the 2 Sigma Problem with Khan Academy: A Pilot Study

Last registered on July 21, 2020

View Trial History

Pre-Trial

Trial Information

General Information

Title

Solving the 2 Sigma Problem with Khan Academy: A Pilot Study

RCT ID

AEARCTR-0006167

Initial registration date

July 17, 2020

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

July 21, 2020, 11:40 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

United States of America

Region

Utah

Primary Investigator

Name

Philip Oreopoulos

Affiliation

University of Toronto

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Joseph Price

PI Affiliation

Brigham Young University

Contact Investigator

Additional Trial Information

Status

In development

Start date

2020-08-17

End date

2021-08-24

Keywords

Education

Additional Keywords

Khan Academy, Computer Assisted Learning

JEL code(s)

Secondary IDs

Abstract

Computer Assisted Learning (CAL) is educational software designed to help students progress through material at their own pace while receiving feedback and advice, similar to the kind that a tutor might provide but on a computer. CAL also makes it easier for teachers to monitor and grade progress. Teachers can assign new topics based on each students’ level of understanding. Despite a large and increasing body of convincing research suggesting large benefits of adopting CAL, many administrators and teachers still shy away. Uncertainty about how to use the software, concerns about subscription costs, and lack of support may explain this reluctance. This pilot aims to develop and test a program that simplifies the adoption of CAL and helps ensure its effective use. The pilot uses Khan Academy (KA), one of the most popular CAL programs, as a tool for Grades 3-8 math teachers to use in class and as graded homework. The program has the potential to save time for teachers, lower costs, and offer a more customizable, enjoyable, and effective learning experience for students. The goal of the pilot is to work out design details for maximizing engagement and simplifying usage. The overall goal is to provide convincing evidence for policy makers, administrators, and teachers to want to adopt the program on a state-wide or national scale.

External Link(s)

Registration Citation

Citation

Oreopoulos, Philip and Joseph Price. 2020. "Solving the 2 Sigma Problem with Khan Academy: A Pilot Study." AEA RCT Registry. July 21. https://doi.org/10.1257/rct.6167-1.0

Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Experimental Details

Interventions

Intervention(s)

We propose to pilot a program that provides training and support for teachers to use KA effectively in Grades 3-8 mathematics. We chose mathematics because previous research (cited above) finds CAL can be particularly effective with this subject. We chose Grades 3-8 because KA’s resources for these levels are particularly well developed. The pilot program has the potential to save time for teachers and offer a free, more customizable, enjoyable, and effective learning experience for students. The pilot would help determine best practices for the implementation of a larger randomized controlled trial. Qualitative and quantitative analyses would be used to identify key implementation challenges and solutions, whether classes are provided online or in-person. If a sufficiently large number of classes participate, we will also implement the study as its own randomized controlled trial.

Intervention Start Date

2020-08-24

Intervention End Date

2021-06-30

Primary Outcomes

Primary Outcomes (end points)

Intermediate outcomes will come from class attendance and grade records. We will also use usage data from Khan Academy. A support letter from them is attached. We will also match students to their standardized math achievement scores from the Utah Performance Assessment System (U-Pass). Qualitative data would be collected on student/teacher/parent experiences and feedback from administrators about the program.

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

Randomized at the classroom or grade level. Treatment group receives program in the Fall, 2020, control group receives it in the Spring, 2021.

Experimental Design Details

Since our analysis uses clusters within school and grade and includes covariates, we assume an intraclass correlation of 0.15. Further assuming 20 students per class, 76 classes would be needed to attain 80 percent statistical power at a 5 percent significance level. Therefore, we would need about 8 schools, with 10 classrooms per school to have adequate statistical power for conducting an experiment during the pilot. We would need about 5 schools if we instead assume 70 percent statistical power to detect at least a 20 percent standard deviation effect with an intraclass correlation of 0.1.

Randomization Method

Randomization done in office by a computer (using STATA).

Randomization Unit

Class or grade-level.

Was the treatment clustered?

Yes

Experiment Characteristics

Sample size: planned number of clusters

10 schools, 10 classes each, for a total of 100 classes.

Sample size: planned number of observations

Assuming 20 students per class, target sample size is about 2,000 students.

Sample size (or number of clusters) by treatment arms

50 classes treated, 50 contol.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Hedges and Hedberg (2007) reports intraclass correlation coefficients across schools of about 0.20 without covariates and about 0.15 with covariates. Since our analysis uses clusters within school and grade and includes covariates, we assume an intraclass correlation of 0.15. Further assuming 20 students per class, 76 classes would be needed to attain 80 percent statistical power at a 5 percent significance level. Therefore, we would need about 8 schools, with 10 classrooms per school to have adequate statistical power for conducting an experiment during the pilot. We would need about 5 schools if we instead assume 70 percent statistical power to detect at least a 20 percent standard deviation effect with an intraclass correlation of 0.1.

Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

IRB