Solving the 2 Sigma Problem with Khan Academy: A Pilot Study

We propose to pilot a program that provides training and support for teachers to use KA effectively in Grades 3-8 mathematics. We chose mathematics because previous research (cited above) finds CAL can be particularly effective with this subject. We chose Grades 3-8 because KA’s resources for these levels are particularly well developed. The pilot program has the potential to save time for teachers and offer a free, more customizable, enjoyable, and effective learning experience for students. The pilot would help determine best practices for the implementation of a larger randomized controlled trial. Qualitative and quantitative analyses would be used to identify key implementation challenges and solutions, whether classes are provided online or in-person. If a sufficiently large number of classes participate, we will also implement the study as its own randomized controlled trial.

The program uses Khan Academy to facilitate group-level and individual-level instruction. This occurs from assigning students both weekly and end-of-term mastery goals.

With respect to weekly goals, Khoaches will work with teachers and Utah curriculum guidelines to align weekly KA assignments with the material that teachers want to cover at the classroom level. Assignments include instructional videos, exercises and quizzes associated with the unit/material being covered. Exercises will be graded based on being attempted and attaining ‘familiar’, ‘proficient’, or ‘mastered’ achievement levels. Quizzes will be graded based on the percent of questions scored correct. A student can take the quiz as many times as they like, with the final score recorded at the due date.

Teachers also assign students a Mastery Goal, which corresponds to a students’ overall percent proficiency in a set of material. Students can receive the same Mastery Goal, or a customized goal based on remedial or advanced needs. Students are expected to spend about 2 hours a week working on their weekly KA assignments and progress with their mastery goal. Teachers will review individual student progress periodically and adjust student mastery goals accordingly. Students are graded based on percent of goals mastered (corresponding roughly to attaining at least 80 percent of material correct).

The combination of assignment and mastery goals allows students to receive group instruction on core grade level material, individual instruction for struggling students to work on remedial material or spend more time on current material, and individual instruction for more advanced students to work on more challenging material. Teachers review individual student progress periodically and agree with students on a plan for progress, including possibly adjusting mastery goals.

The program allows for flexibility by letting teachers choose KA assignments and mastery goals. Teachers can also choose how they wish to incorporate KA during class, or as extra review. Previous research suggests using Khan as mandatory homework, with the possibility of completing this homework in class or at school, is the most promising approach for leading to a high degree of participation and effectiveness. Teachers can easily monitor progress using the KA dashboard and update their classroom instruction based on satisfactory or unsatisfactory performance on KA assignments.

Some details of the program depend on the state of instruction in the 2020/21 school year. If classes remain online, teachers face additional challenges with keeping students engaged and offering individual feedback. To help address this, Khoaches will offer virtual check-in meetings with individual students not using KA regularly. These check-ins will also facilitate a qualitative analysis of the challenges students face in using KA from home. The number of students targeted for this service and the frequency of the meetings will depend on the total sample of classrooms. The main goal is to better understand design details for helping teachers and students benefit the most from the program.

Another possible add-on for increasing engagement is to send reminders and feedback to students and parents. Past research suggests sending messages about attendance and homework completion can increase achievement. We will work with teachers and school administrators to test an add-on text-based communication program to increase KA engagement.

The sample would be obtained from Utah. In the past, we have worked with schools in the Alpine, Granite, Nebo, Ogden, Provo, Washington, and Weber school districts. At a minimum, we would pilot the study in at least one school with 5-10 classrooms in Grades 3-8, located in a disadvantaged neighborhood. We aim to recruit 10 or more schools with 8-10 classrooms across each school so that we can randomize the program across classrooms.

With principal or superintendent encouragement, teachers would be invited to participate. With a sufficiently large number of classrooms, approximately half would be selected into ‘Group 1’ and the other half into ‘Group 2’. If selected into the study’s ‘Group 1’, volunteer math teachers would complete the 4-hour ‘Khan for Educators Course’ before classes begin and agree to incorporate KA systematically in their class for the first term (until students complete their first assessment). At the end of first term, Group 1 teachers would have the option of whether they want to continue working with Khan or not. Those selected into ‘Group 2’ would have the option of incorporating KA in second term.

Group 1 teachers would also work with assigned coaches (‘Khoaches’) who would offer personal support at the beginning of and during the school year. They will do so remotely via phone, email, or video call unless JPAL and Utah guidelines both deem meeting in person is acceptable.
Khoaches are key to the pilot for helping learn potential challenges to the program’s design and potential solutions. Group 2 teachers will instead receive Khoach support via group meetings or email, which would mimic more closely a scalable program that does not include more intensive personal communication.

Parents of Group 1 and Group 2 would be sent an information sheet about the program with either opt-in or opt-out consent, depending on what our Ethics Review Board and Utah School Districts allow. Both of these approaches that we have used in the past have worked adequately well. Opt-in would also allow for collecting additional survey data and possibly parental text numbers to use for reminders and support.

2020-08-24

2021-06-30

Intermediate outcomes will come from class attendance and grade records. We will also use usage data from Khan Academy. A support letter from them is attached. We will also match students to their standardized math achievement scores from the Utah Performance Assessment System (U-Pass). Qualitative data would be collected on student/teacher/parent experiences and feedback from administrators about the program.

Randomized at the classroom or grade level. Treatment group receives program in the Fall, 2020, control group receives it in the Spring, 2021.

Since our analysis uses clusters within school and grade and includes covariates, we assume an intraclass correlation of 0.15. Further assuming 20 students per class, 76 classes would be needed to attain 80 percent statistical power at a 5 percent significance level. Therefore, we would need about 8 schools, with 10 classrooms per school to have adequate statistical power for conducting an experiment during the pilot. We would need about 5 schools if we instead assume 70 percent statistical power to detect at least a 20 percent standard deviation effect with an intraclass correlation of 0.1.

Randomization done in office by a computer (using STATA).

Class or grade-level.

Yes

10 schools, 10 classes each, for a total of 100 classes.

Assuming 20 students per class, target sample size is about 2,000 students.

50 classes treated, 50 contol.

Hedges and Hedberg (2007) reports intraclass correlation coefficients across schools of about 0.20 without covariates and about 0.15 with covariates. Since our analysis uses clusters within school and grade and includes covariates, we assume an intraclass correlation of 0.15. Further assuming 20 students per class, 76 classes would be needed to attain 80 percent statistical power at a 5 percent significance level. Therefore, we would need about 8 schools, with 10 classrooms per school to have adequate statistical power for conducting an experiment during the pilot. We would need about 5 schools if we instead assume 70 percent statistical power to detect at least a 20 percent standard deviation effect with an intraclass correlation of 0.1.