Scaling up virtual learning of online learning in high schools

The intervention is provided in two phases according to the educational regimes in Ecuador: Highlands Educational Regime (Students Finish the program in Summer 2020) and Coastal Educational Regime (Students Finish the program in Winter 2020). We have two types of interventions: the training interventions, and then interventions to enhance the take-up of training.


Highlands Educational Regime (Students Finish the program in Summer 2020) includes:
Training interventions.
Schools are randomly assigned to one of the following four treatment groups:
1) Personal initiative + Negotiations training + Role Model videos
2) Personal initiative + Negotiations training + Placebo videos
3) Spanish + Statistics + Role Model videos
4) Spanish + Statistics + Placebo videos

Interventions to enhance the take-up of training.
We are testing different interventions to encourage take-up and usage of the online platform on different levels: Students and teachers. These include on student level:
i) control group
ii) lottery reminder to students - students earn a ticket for a lottery prize for each completed module
iii) Encouragement to students to find a peer in the house to plan and study together
iv) Encouragement to students for those who complete their work
These include on teachers level:
I)control
ii) Role model testimonials/motivational videos to teachers from teachers and students who have already completed the course
iii) Reminder about monitoring and deadline to teachers


Coastal Educational Regime (Students Finish the program in Winter 2020) that includes:
Training interventions.
Schools are randomly assigned to one of the following 6 treatment groups:
1) Personal initiative + Negotiations training + Role Model videos
2) Personal initiative + Negotiations training + Placebo videos
3) Spanish + Statistics + Role Model videos
4) Spanish + Statistics + Placebo videos
3) Spanish + English + Role Model videos
4) Spanish + English + Placebo videos

Interventions to enhance the take-up of training.
We are testing different interventions to encourage take-up and usage of the online platform on the school level:
i) Control
ii)Monitoring (information)
iii)Monitoring (information) + lottery reminder to students - students earn a ticket for a lottery prize for each completed module and performance on platform.
iv)Monitoring (information) + Encouragement to team-up with peers remotely.

2020-05-18

2020-12-31

We have two sets of outcomes: take-up outcomes, and learning outcomes.

Take-up outcomes:
1) Proportion of students who complete at least one module
2) Number of modules completed by students
3) Total days spent active on platform

Learning outcomes:
Test scores of learning based on Spanish and English literacy, knowledge of statistics, science, negotiations and personal initiative.

Course type is randomized at the school level to one of the following in the Highlands Educational Regime:
1) Personal initiative + Negotiations training + Role Model videos
2) Personal initiative + Negotiations training + Placebo videos
3) Spanish + Statistics + Role Model videos
4) Spanish + Statistics + Placebo videos

Take-up interventions are randomized at the individual level or class/teacher level, depending on the intervention.


Course type is randomized at the school level to one of the following in the Coastal Educational Regime:
1) Personal initiative + Negotiations training + Role Model videos
2) Personal initiative + Negotiations training + Placebo videos
3) Spanish + Statistics + Role Model videos
4) Spanish + Statistics + Placebo videos
3) Spanish + English + Role Model videos
4) Spanish + English + Placebo videos

Take-up interventions are cross-randomized on the school level.

Randomization done in office by computer or on the online platform

School for main interventions
Class for some take-up interventions
Student for individual interventions

Yes

Highlands Educational Regime (Students Finish the program in Summer 2020) includes:
416 schools
532 classes
Approximately 24,000 potential students

Coastal Educational Regime (Students Finish the program in Winter 2020) that includes:
598 schools
795 classes
Approximately 28,000 potential students

We will only observe outcomes for students who log in once to the platform - which we anticipate to be over 10,000 students in Highlands Educational Regime and over 11, 000 in Coastal Educational Regime

Highlands Educational Regime (Students Finish the program in Summer 2020) includes:

104 schools in each treatment group
127 to 136 classes in each treatment group

Coastal Educational Regime (Students Finish the program in Winter 2020) that includes:

1) 149 schools Personal initiative + Negotiations training + Role Model videos (~200 classes)
2) 149 schools Personal initiative + Negotiations training + Placebo videos (~200 classes)
3) 75 schools: Spanish + Statistics + Role Model videos (~100 classes)
4) 75 schools: Spanish + Statistics + Placebo videos (~100 classes)
3) 75 schools: Spanish + English + Role Model videos (~100 classes)
4) 75 schools: Spanish + English + Placebo videos (~100 classes)

Pre-Analysis Plan Showing Life Opportunities: Role model effects 16 April 2021

Pre-Analysis Plan Take Take-up Interventions in the National Level (Highlands Regime) 23 July 2020

Pre-analysis Plan Take-up Interventions in the National Level (Coastal Regime) 20 January 2021

Many school systems across the globe turned to online education during the COVID-19 pandemic. This context differs significantly from the prepandemic situation in which massive open online courses attracted large numbers of voluntary learners who struggled with completion. Students who are provided online courses by their high schools also have their behavior determined by actions of their teachers and school system. We conducted experiments to improve participation in online learning before, during, and right after the COVID-19 outbreak, with 1,151 schools covering more than 45,000 students in their final years of high school in Ecuador. These experiments tested light-touch interventions at scale, motivated by behavioral science, and were carried out at three levels: that of the system, teacher, and student. We find the largest impacts come from intervening at the system level. A cheap, online learning management system for centralized monitoring increased participation by 0.21 SD and subject knowledge by 0.13 SD relative to decentralized management. Centralized management is particularly effective for underperforming schools. Teacher-level nudges in the form of benchmarking emails, encouragement messages, and administrative reminders did not improve student participation. There was no significant impact of encouragement messages to students, or in having them plan and team-up with peers. Small financial incentives in the form of lottery prizes for finishing lessons did increase study time, but was less cost-effective, and had no significant impact on knowledge. The results show the difficulty in incentivizing online learning at scale, and a key role for central monitoring.

Asanov I, Asanov AM, Åstebro T, Buenstorf G, Crépon B, McKenzie D, Flores T FP, Mensmann M, Schulte M. (2023) System-, teacher-, and student-level interventions for improving participation in online learning at scale in high schools. Proceedings of the National Academy of Sciences (PNAS). 120(30):e2216686120

https://doi.org/10.1073/pnas.2216686120