Back to History Current Version

Helping students learn at home: Experimental evidence from a technology-based Covid-19 response strategy in Indian government schools

Last registered on July 06, 2021

Pre-Trial

Trial Information

General Information

Title
Helping students learn at home: Experimental evidence from a technology-based Covid-19 response strategy in Indian government schools
RCT ID
AEARCTR-0007915
Initial registration date
July 02, 2021

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
July 06, 2021, 10:43 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
University of California, Irvine

Other Primary Investigator(s)

Additional Trial Information

Status
On going
Start date
2020-08-01
End date
2021-10-31
Secondary IDs
Abstract
This study investigates a state-wide Covid-19 response strategy that leverages a remote-learning solution to teach students while government-run schools remain closed (Haryana's “Ghar se Padhao” program). In a cluster-randomized trial, 80 schools receive the program without additional support (“control group”). Another 160 schools receive the program with additional support, helping teachers and students engage with the technology-enabled learning materials (“encouragement group”). Of these 160 schools, a random 80 had received an in-classroom blended-teaching program, in the previous school year. This research offers causal evidence on the effectiveness of a large-scale effort to avert learning loss, and on potential complementarities of home-based learning solutions with prior exposure to educational technology.
External Link(s)

Registration Citation

Citation
de Barros, Andreas. 2021. "Helping students learn at home: Experimental evidence from a technology-based Covid-19 response strategy in Indian government schools." AEA RCT Registry. July 06. https://doi.org/10.1257/rct.7915-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
In collaboration with a local NGO (“Avanti Fellows”) the Government of Haryana is responding to the Covid-19 crisis while schools remain closed. Under an official government mandate, Avanti Fellows provides remote teaching-learning content, primarily through WhatsApp, video materials, and online quizzes. This home-based learning program (called “Ghar se Padhao”) is expected to continue until regular in-school classes resume.

More specifically, the government’s Covid-19 response strategy requires teachers to provide technology-enabled remote-learning solutions to grade-9 and grade-10 students, in mathematics and science. Teachers form WhatsApp groups with their students, enabling content sharing on a common platform. Avanti Fellows creates and curates digital learning materials, which are shared with district officials, who in turn pass them on to teachers. Teachers are also expected to address students’ questions, review assignments, and provide one-on-one support over the phone. Avanti Fellows also creates a weekly quiz every Saturday, and teachers ask students to participate by submitting responses through a Google Form.

Several features make this intervention noteworthy. First, in stark contrast to other technology solutions, it only requires access to low-end “feature phones”, which are readily available in the great majority of households. In fact, recent survey data (ASER, 2020) suggests 70.7 percent of Haryana’s government secondary students had received remote learning materials from their school in the week prior to the survey, and 92.2 percent of those government-school students did so over WhatsApp. Second, the intervention is directly mapped to the official, state-sanctioned curriculum, and it is delivered in the local language (Hindi). And third, the program is delivered through public teachers, effectively replacing instructional activities that would have otherwise been provided in schools, in-person.

While the program is rolled out state-wide, Avanti Fellows also provides direct support to 160 “encouragement schools”. This direct intervention seeks to contravene two patterns that are common among many educational technology solutions: (a) selective sign-up (where more privileged students are more likely to take up an intervention), and (b) sharp drop-off rates (where, even if students sign on initially, they often do not continuously engage with a given remote-learning platform) (see Reich, 2020). To provide this encouragement, NGO staff join schools’ WhatsApp groups and provide one-on-one support to teachers.
Intervention (Hidden)
Intervention Start Date
2020-08-01
Intervention End Date
2021-05-31

Primary Outcomes

Primary Outcomes (end points)
The study’s main outcome of interest is student learning, in math and science. A baseline assessment was already administered as paper-based tests (one per grade and subject, when students attended grades 8 and 9), under the same, strict governmental oversight as other central exams (with additional monitoring from the research team). An endline assessment will be administered by J-PAL South Asia, through remote test administration, with a random subsample of students.
Primary Outcomes (explanation)
Tests are designed by the Principal Investigator; test items are tightly mapped to the official state curriculum, but also include items from up to two years below grade-level. Item-response theory (IRT) will be used to estimate student ability (on a common scale, leveraging common test questions across student grades and assessment rounds). Baseline results provide strong validity evidence for this assessment strategy, with high levels or precision (even in the extreme tails of the ability distribution) and no evidence of floor or ceiling effects.

Secondary Outcomes

Secondary Outcomes (end points)
Program take-up, teaching and learning behaviors, and student attitudes towards math and science will be captured through two rounds of phone-based interviews, with a random subsample of 1,920 students and 480 teachers (one of the rounds is already complete).

We will also monitor implementation fidelity through access to detailed records on the field activities of Avanti Fellows staff. In particular, each staff member records the number of interactions with government-school teachers (along with a unique government-school ID number).
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
In a cluster-randomized trial, 80 schools receive the state-wide program without additional support (“control group”). Another 160 schools receive the program with additional support, helping teachers and students engage with the technology-enabled learning materials (“encouragement group”). Of these 160 schools, a random 80 had received an in-classroom blended-teaching program, in the previous school year. All 160 intervention schools had also received on-site visits and print materials for grades 9 and 10.

Both last year’s program and this year’s program aim to improve student learning in mathematics and science, in grades 9 and 10. In the group of 80 encouragement schools with last year’s blended learning intervention, this renders one cohort of students with one year of prior exposure to an educational technology program (previously in grade 9, now in grade 10). It also renders one cohort of students with no prior exposure to the blended learning program (previously in grade 8, now in grade 9). In the remaining 80 encouragement schools and in the 80 control schools, none of the students had prior exposure to an educational technology program.
Experimental Design Details
We will estimate the intent-to-treat effect (ITT) of the encouragement treatment on outcome variables at endline (after approximately 10 months of exposure). To do so, we will estimate standard OLS regressions with a single treatment indicator and a vector of baseline controls (selected through a Lasso procedure). Standard errors will be clustered at the school level (cf. Abadie et al., 2017). If the encouragement treatment leads to notable differences in program take-up (i.e., if we observe a strong first stage), we will also report on treatment-on-the treated (ToT) effects, through instrumental variable (IV) estimations.

To identify the potential effect of prior exposure to the blended learning intervention, in secondary analyses, we will also estimate regressions with two treatment indicators (one for each of the 80 treatment-school subgroups, with and without prior exposure).
In our secondary analyses, we will moreover investigate heterogeneous treatment effects by grade, by students’ initial ability level (at baseline), and by student gender. We pre-specify and strictly limit this number of sub-group analyses, to avoid specification searching.

In summary, there are two tests of equal (primary) interest for the study: (1) Whether the encouragement led to increases in student learning; and (2) whether schools' prior exposure to EdTech led to increases in student learning.

Finally, a random subsample of students will be tracked with higher intensity, as overall attrition rates may be at or above 30%. We can then use the random assignment to high-intensity tracking to account for potential attrition bias.
Randomization Method
Randomization done in office by a computer.
Randomization Unit
Schools
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
240 schools.
Sample size: planned number of observations
A random subsample of 7,543 students overall, of which 1,920 have been randomly selected for (a) phone surveys and (b) high-intensity tracking.
Sample size (or number of clusters) by treatment arms
160 schools in the encouragement group, 80 schools in the control group.

Of the 160 schools: 80 schools in the group with prior ICT treatment exposure, 80 schools with prior non-ICT treatment exposure ("workbook only").
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
0.136 standard deviations for the encouragement treatment (main question 1), and 0.157 standard deviations for the effect of prior exposure to ICT (main question 2).
IRB

Institutional Review Boards (IRBs)

IRB Name
MIT COUHES
IRB Approval Date
2021-06-11
IRB Approval Number
2102000327

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials