Does Nudging Students Decrease Learning Deficits and Dropouts During and After a Pandemic? Experimental Evidence from Covid-19 Responses in Brazil

The intervention has been designed by Instituto Sonho Grande and the Goiás State Secretariat of Education, with the help of Movva (the implementing partner that powers Eduq+). It will take place during the months of June and July/2020, when public high schools will be randomly assigned to have their students receive two messages per week from Eduq+ (an educational nudgebot). 57 schools have been assigned to the treatment group, and 30 to the control group (which receives no intervention). In case the student does not own a phone, messages will be sent to the mobile phone of his/her primary caregiver. The intervention is scheduled to be rolled out on June 9th.

2020-06-09

2020-07-31

We will document the effects of the treatments on the following categories of outcomes for students enrolled in high school (age 15 to 18):
A. Short-term outcomes: probability of logging into the online platform or picking up the material in school, probability of handed in of assignments, as measured by administrative records;
B. Long-term outcomes: attendance, grades, probability of grade repetition and probability of dropout, as measured by administrative records.
Since some students will receive messages on their own mobile phones, while for others it is their caregivers who will be nudged by Eduq+, we will estimate treatment effects within those two subgroups. Power calculations indicate that we could detect treatment effects of at least 1 and 0.9 percentage point for these two subsamples, respectively.
Since there are siblings in the data, we will remove from the main analysis cases when not all siblings are assigned to the same treatment conditions. Depending on how many siblings there are, we also plan to estimate within-family's externalities of the nudges, taking advantage of that sub-sample.
For participants with online access, we hope to get access to daily data, which would allow us to also estimate high-frequency treatment effects through event studies.

The intervention will be evaluated through a randomised control trial with 87 schools in the state of Goiás. Randomization is stratified by gender, grade and phone ownership. 57 schools have been assigned to the treatment group, while 30 have been assigned to control. Students in the treatment group will receive nudges (reminders and encouragement messages), sent twice a week, directly on their mobile phones via text messages (SMS). In case the student does not own a phone, messages will be sent to the mobile phone of his/her primary caregiver. The treatment group consists of 12,056 students, while the control group consists of 6,200 students. Within the sample of 12,056 students assigned to receive nudges, less than half (5,188) own their own mobile phone and will receive messages directly. It is also important to note that not all students in the sample have access to the internet and that those who do not can pick up the printed class material once every week and hand in assignments the following week. For the purpose of this study however, we will be able to measure their outcomes in different ways. In addition, students among the treatment group have been further randomised, in groups of 1,507 students, to receive one of eight variations in message content.
At the end of July, we will be able estimate treatment effects on access to the online platform, and assignment completion, from administrative data provided by the Secretariat. Concretely, we have requested weekly student-level data on log in activity – or face-to-face pick-up of class materials – as well as assignment completion (again, online or offline). For those with online access, we hope to get access to daily data, which would allow us to also estimate high-frequency treatment effects through event studies. Last, after face-to-face classes resume, we will have access to administrative records on student-level attendance, grades, grade repetition and enrollment status.

Randomization done in office by a computer

School-level randomization for treatment vs. control assignment, stratified by gender, grade and phone ownership
Student-level randomization within treatment group for different variations of messages

Yes

87 schools

18256

Control: 30 schools - 6,200 students
Treatment: 57 schools - 12,056 students
Within treatment: 8 groups of 1,507 students

In the case of the full sample, we could detect treatment effects of nudges of at least 0.8 percentage points. Since some students will receive messages on their own mobile phones, while for others it is their caregivers who will be nudged by Eduq+, we will estimate treatment effects within those two subgroups. Power calculations indicate that we could detect treatment effects of at least 1 and 0.9 percentage point for these two subsamples, respectively.

Pre-Analysis Plan update 01/09/2020

Does Nudging Students Decrease Learning Deficits and Dropouts During and After a Pandemic? Experimental Evidence from Covid-19 Responses in Brazil