In-School Support to Teachers for Implementing Computer Assisted Learning Effectively: A Randomized Controlled Trial of Teacher Assistance Across Government Schools In India

Many government schools in India have adopted some form of EdTech tools after encouragement from the State but only a small fraction of teachers and students are able to actually use these tools regularly and effectively due to multiple challenges - 1) Limited digital infrastructure, 2) Teacher shortage, 3) Insufficient digital knowhow across stakeholders, 4) Lack of teacher motivation and 5) Lack of sufficient time to practice with EdTech platform etc.

The project will assign Khoaches to intervention schools to maintain implementation fidelity. Khoaches will be hired on contract basis by Khan Academy and will undergo rigorous training to make sure they are fully equipped to handle the KA program in government schools and provide the necessary support and training to teachers and students. Khoaches will work collaboratively in consultation with the Social Welfare Department and schools to devise best strategies for ensuring adequate practice time (at least 30 minutes to 60 minutes of usage per week). The control schools will also have access to Khan Academy but the program will be driven by teachers themselves (current overall usage rate from this setup is low). Khoaches will visit the assigned school twice a week to supervise and conduct Khan Academy practice sessions. Khoaches will be responsible for - 1) Providing training to teachers and students; 2) Creating and helping teachers and students with their digital accounts (Rostering); 3) Supervising and ensuring Khan Academy sessions are conducted regularly, 4) Helping teachers assign the relevant content & track student performance and 5) Technical troubleshooting. Once teachers become comfortable with the process, they should be able to use and integrate KA with their curriculum with greater ease and effectiveness in subsequent years and the same should reflect in usage data.

Research Question
The purpose of this study is to evaluate the effectiveness of Khan Academy, a Computer Assisted Learning tool that offers personalized learning and high quality state aligned content in vernacular language to students, in improving Math test scores of government school students in India for grades 6 to 8 when supported by trained personnel (Khoaches).

Target Population
The study would be conducted in schools under Social Welfare department in Uttar Pradesh. There are a total of approximately 33,500 students in 105 schools across grades 6 to 12. The study would cover 75 schools out of these.

2024-08-12

2025-05-30

The primary outcomes of interest will be student math achievement. The improvement in student’s math scores will be tracked using the Baseline and Endline assessments. Both Baseline and Endline assessments will be designed by the research team and data collection will be carried out by Khoaches. Assessments would cover grade appropriate skills and competencies up to that point.

Secondary outcomes include teacher satisfaction, and impressions of their students’ learning progress. The study will also measure, quantitatively and qualitatively, student and teacher experiences with the program. We will also collect Khan Academy usage levels, during the year of implementation and in the following year. This will allow us to evaluate whether teachers maintain high dosage KA usage after the program ends in subsequent years.

Research Design
The study would be carried out with government schools under the Social Welfare department in Uttar Pradesh for grades 6 to 8. There are a total of 105 schools under the department out of which 75 schools will be filtered on the basis of factors such as ‘Availability of Internet and Desktops/Tablets’ to ensure sufficient access to digital infrastructure. These 75 schools will be subsequently assigned to the ‘Treatment’ and ‘Control’ category randomly in 1 to 2 ratio leading to 25 Treatment schools and 50 Control schools. The schools in the Treatment category will receive the assistance of Khoaches for grades 6 to 8 from Jul’24 to Feb’25. These Khoaches will visit the schools twice a week as per a pre-determined schedule and supervise Khan Academy sessions. In Control schools, teachers themselves will be responsible for driving the Khan Academy program. This design will allow for credible causal impact evaluation.

Not available

Randomization will be done by office computer.

School leve randomization, with clusters to incude two nearby treated schools and two nearby control schools.

Yes

85 schools

~5000 students

29 treated schools, 56 control schools

School will be the unit of randomization and total 25 school will be selected in the Treatment group. Power calculations follow multi-level clustering design. In each Treatent school, students from grades 6 to 8 will get benefit from the Intervention. Each grade would consist of 35 students on average. Similarly, Control group would consist of 50 schools and students from grades 6 to 8 from each school would take part in the study. The main estimation equation would be Yigs=Tigs+Xic+eigst , where Yigs is a standardized math test score for individual i in grade g from school s, Tigs is a treatment dummy, Xic is a set of individual covariates, including previous achievement, which I anticipate having access to. We aim to randomize at the school level. Hedges and Hedberg (2007) reports intraclass correlation coefficients across schools of about 0.20 without covariates and about 0.15 with covariates. The research study would have an MDE of 0.23 to 0.27 depending on the effective ICC range of 0.1 to 0.15 for comparing the treatment group versus control. This is for the program’s Intent To Treat (ITT) effect. We will also estimate classroom and student participation levels for both treated and control classrooms to further estimate program impacts based on amount of practice time. We expect the takeup rate in the Treatment group to be 75% or more given that supervisors will visit schools twice and schools will have access to digital devices and internet connectivity. The takeup rate in the control group is taken to be 5% or less based on current field implementation.

India Project - PreAnalysis.pdf