Do Students Benefit from Blended Instruction? Experimental Evidence from India

240 Government Senior Secondary Schools (GSSS) across 8 districts of Haryana will be equally randomized across two treatment arms and the control group, with 80 schools assigned to each group.

The "ICT group" will receive the full intervention. This includes setting up smart classrooms, provision of digital content to supplement teaching instruction, printed workbooks for practice of students, and capacity building of mathematics and science teachers responsible for teaching class ninth and tenth curriculum.

The "Workbook group"'s program components are equivalent to those administered in the previous group; however, the group does not receive those particular program components related to ICT (i.e., ICT-related infrastructure upgrades or digital content).

The "Control group" continues with "business-as-usual". The schools assigned to the control group will neither receive facilities nor materials. Their teachers will also not undergo the program's teacher training activities.

2018-12-17

2021-01-31

The study's main program outcome of interest is student learning, in math and science. Students in grades nine and ten will be assessed at baseline and through an end-of-year test. Assessments will be administered as paper-based tests (one per grade and subject), under the same, strict governmental oversight as other central exams (with additional monitoring from the research team). Test items are tightly mapped to the official "CBSE/NCERT" school curriculum, but also include items from up to two years below grade-level. Items have been administered in similar contexts previously, in large-scale assessments. From these previous administrations, item response theory (IRT)-based item characteristics are used to maximize each assessments' test information.

Estimates of student ability will be calculated using a standard, three-parameter logistic (3PL) IRT model, with a single guessing parameter. In case a 3PL model does not converge, a 2PL model will be used instead. In doing so, anchor items across grades and test-occasions will allow for the (concurrent) linking of estimates onto one common, continuous ability scale per subject.

1) Student performance on subskills
2) Teaching behaviors and instructional quality
3) Intervention monitoring

1) Student performance on subskills
For mathematics, items are categorized as measures of either one of the following four abilities: Algebra; geometry; number system; and data/statistics. For science, items are categorized as measures of either biology, chemistry, or physics. In summary, this subskill analysis will establish the extent to which the program affected a student's probability of mastering each of these subskills.

In the case of subskills, because of the lower number of items per sub-skill, student performance will not be scored with a continuous measure of ability. Instead, I will distinguish between three categories of mastery, for each of the two subjects: students who have mastered grade-level appropriate material; students who have only mastered material from one and two years below their grade-level; and other students. I will determine students' level of mastery empirically, through a Cognitive Diagnostic Model (CDM).

2) Teaching behaviors and instructional quality
The program's effects on teaching behaviors and on instructional quality will be assessed through two instruments: Classroom observations and student reports. First, the program includes bi-weekly school visits and monthly classroom observations. All visits and observations will be conducted in intervention and control schools. For this purpose, the study developed an instrument to measure the program's effect on the quality of instruction a student receives. Secondly, during school visits, a subset of students will be surveyed on common classroom behaviors. The study explicitly defines either of these data sources as a measure of mechanisms, i.e. not as a main outcome.

Romero et al. employ a Lasso procedure to identify potential mediators (from a large number of variables). I will follow the authors' approach (ibid.) to identify mediating instructional behaviors; yet, the Lasso procedure will be forced to include teachers' use of ICT materials as a predictor. In turn, concerning instructional quality, I pre-specify six dimensions that will be investigated as potential mediators (monitoring of student learning; feedback; maximization of learning time; density of the mathematics / science; clarity of content and lack of errors; richness of the mathematics / science).

3) Intervention monitoring
Sign-in sheets will be used to track teachers' exposure to capacity-building activities. Avanti Fellows will moreover provide data from its software backend, to track teachers' use of videos and digital learning materials. Monitoring data will also be collected in the above-mentioned bi-weekly and monthly visits, through a structured school questionnaire.

Schools are assigned with an equal split across the three groups (for a total of 80 schools per group). To achieve similar control and treatment groups and to improve statistical power, randomization is stratified. Within districts, schools are sorted into randomization strata of three, grouping schools with similar performance on the Haryana Board Exams together.

To rank schools, I use the average school-level Board Exam score for 2018, overall (covering English, Hindi, Mathematics, Sanskrit, Science, and Social Studies). More precisely, I calculate a school's weighted average, using information on the number of student in each of six performance ranges, on the test. If the number of schools for any district ranking is not divisible by three, I randomly assign the remaining schools. In this case, I assign these remaining schools "globally", i.e. across districts. As more than three schools need to be randomized across districts, I once more group schools with similar Board Exam scores together.

Finally, I repeat the above randomization strategy ten times, selecting the randomization with the smallest t-statistic, from comparisons across the treatment and control groups, for a set of select covariates (using Lasso, I select a set of covariates from India's District Information System for Education (DISE), which are predictive of school-level average pass rates for standardized exams, in grades 7 and 8. The selected covariates are: Number of students in grade 7 and 8; percentage female (students); percentage minority (students); percentage OBC (students); number of teachers; percentage female (teachers); percentage graduates (teachers); years in service (school); co-ed (school); school requires minor repairs; wall missing or damaged; ratio computers/students. and the average school-level Board Exam score for 2018.

Randomization done in office by a computer

School

Yes

240 schools

29,280 students in the first cohort, and an additional 12,240 students in the following cohort

80 schools in each of the two treatment groups; 80 schools in the control group

Intent-to-treat effect of 0.158 standard deviations (SD) on student test scores

Revised Pre-analysis Plan 2019-11-26 (detailed table shells and mock figures)

Pre-analysis Plan 2018-12-12