Helping students learn at home: Experimental evidence from a technology-based Covid-19 response strategy in Indian government schools

In collaboration with a local NGO (“Avanti Fellows”) the Government of Haryana is responding to the Covid-19 crisis while schools remain closed. Under an official government mandate, Avanti Fellows provides remote teaching-learning content, primarily through WhatsApp, video materials, and online quizzes. This home-based learning program (called “Ghar se Padhao”) is expected to continue until regular in-school classes resume.

More specifically, the government’s Covid-19 response strategy requires teachers to provide technology-enabled remote-learning solutions to grade-9 and grade-10 students, in mathematics and science. Teachers form WhatsApp groups with their students, enabling content sharing on a common platform. Avanti Fellows creates and curates digital learning materials, which are shared with district officials, who in turn pass them on to teachers. Teachers are also expected to address students’ questions, review assignments, and provide one-on-one support over the phone. Avanti Fellows also creates a weekly quiz every Saturday, and teachers ask students to participate by submitting responses through a Google Form.

Several features make this intervention noteworthy. First, in stark contrast to other technology solutions, it only requires access to low-end “feature phones”, which are readily available in the great majority of households. In fact, recent survey data (ASER, 2020) suggests 70.7 percent of Haryana’s government secondary students had received remote learning materials from their school in the week prior to the survey, and 92.2 percent of those government-school students did so over WhatsApp. Second, the intervention is directly mapped to the official, state-sanctioned curriculum, and it is delivered in the local language (Hindi). And third, the program is delivered through public teachers, effectively replacing instructional activities that would have otherwise been provided in schools, in-person.

While the program is rolled out state-wide, Avanti Fellows also provides direct support to 160 “encouragement schools”. This direct intervention seeks to contravene two patterns that are common among many educational technology solutions: (a) selective sign-up (where more privileged students are more likely to take up an intervention), and (b) sharp drop-off rates (where, even if students sign on initially, they often do not continuously engage with a given remote-learning platform) (see Reich, 2020). To provide this encouragement, NGO staff join schools’ WhatsApp groups and provide one-on-one support to teachers.

2020-08-01

2021-05-31

The study’s main outcome of interest is student learning, in math and science. A baseline assessment was already administered as paper-based tests (one per grade and subject, when students attended grades 8 and 9), under the same, strict governmental oversight as other central exams (with additional monitoring from the research team). An endline assessment will be administered by J-PAL South Asia, through remote test administration, with a random subsample of students.

Tests are designed by the Principal Investigator; test items are tightly mapped to the official state curriculum, but also include items from up to two years below grade-level. Item-response theory (IRT) will be used to estimate student ability (on a common scale, leveraging common test questions across student grades and assessment rounds). Baseline results provide strong validity evidence for this assessment strategy, with high levels or precision (even in the extreme tails of the ability distribution) and no evidence of floor or ceiling effects.

Program take-up, teaching and learning behaviors, and student attitudes towards math and science will be captured through two rounds of phone-based interviews, with a random subsample of 1,920 students and 480 teachers (one of the rounds is already complete).

We will also monitor implementation fidelity through access to detailed records on the field activities of Avanti Fellows staff. In particular, each staff member records the number of interactions with government-school teachers (along with a unique government-school ID number).

In a cluster-randomized trial, 80 schools receive the state-wide program without additional support (“control group”). Another 160 schools receive the program with additional support, helping teachers and students engage with the technology-enabled learning materials (“encouragement group”). Of these 160 schools, a random 80 had received an in-classroom blended-teaching program, in the previous school year. All 160 intervention schools had also received on-site visits and print materials for grades 9 and 10.

Both last year’s program and this year’s program aim to improve student learning in mathematics and science, in grades 9 and 10. In the group of 80 encouragement schools with last year’s blended learning intervention, this renders one cohort of students with one year of prior exposure to an educational technology program (previously in grade 9, now in grade 10). It also renders one cohort of students with no prior exposure to the blended learning program (previously in grade 8, now in grade 9). In the remaining 80 encouragement schools and in the 80 control schools, none of the students had prior exposure to an educational technology program.

We will estimate the intent-to-treat effect (ITT) of the encouragement treatment on outcome variables at endline (after approximately 10 months of exposure). To do so, we will estimate standard OLS regressions with a single treatment indicator and a vector of baseline controls (selected through a Lasso procedure). Standard errors will be clustered at the school level (cf. Abadie et al., 2017). If the encouragement treatment leads to notable differences in program take-up (i.e., if we observe a strong first stage), we will also report on treatment-on-the treated (ToT) effects, through instrumental variable (IV) estimations.

To identify the potential effect of prior exposure to the blended learning intervention, in secondary analyses, we will also estimate regressions with two treatment indicators (one for each of the 80 treatment-school subgroups, with and without prior exposure).
In our secondary analyses, we will moreover investigate heterogeneous treatment effects by grade, by students’ initial ability level (at baseline), and by student gender. We pre-specify and strictly limit this number of sub-group analyses, to avoid specification searching.

In summary, there are two tests of equal (primary) interest for the study: (1) Whether the encouragement led to increases in student learning; and (2) whether schools' prior exposure to EdTech led to increases in student learning.

Finally, a random subsample of students will be tracked with higher intensity, as overall attrition rates may be at or above 30%. We can then use the random assignment to high-intensity tracking to account for potential attrition bias.

Randomization done in office by a computer.

Schools

Yes

240 schools.

A random subsample of 7,543 students overall, of which 1,920 have been randomly selected for (a) phone surveys and (b) high-intensity tracking.

160 schools in the encouragement group, 80 schools in the control group.

Of the 160 schools: 80 schools in the group with prior ICT treatment exposure, 80 schools with prior non-ICT treatment exposure ("workbook only").

0.136 standard deviations for the encouragement treatment (main question 1), and 0.157 standard deviations for the effect of prior exposure to ICT (main question 2).