Do Students Benefit from Personalized Learning? Experimental Evidence from India

We will use a computer-assisted learning (CAL) software to randomly assign students to either a "control" (i.e., business-as-usual) or one of three "treatment" (i.e., intervention) conditions. The software is called Mindspark and it was developed by Educational Initiatives (EI), a leading Indian education firm, over a 10-year period. It is currently used by over 400,000 students, it has a database of over 45,000 questions, and it administers over a million questions across its users every day. It can be delivered in school during the school day, before/after school at stand-alone centers, and through a self-guided online platform. It is also platform-agnostic: it can be deployed through computers, tablets, or smartphones, and it can be used both online and offline. A randomized evaluation of the before/after school version of the program in 2015 found that it had a large impact on the math and language achievement of middle school students in Delhi (see Muralidharan et al. 2016). We will use the in-school version of the software which is targeted at Indian government schools. We do not evaluate the impact of the software by itself. Instead, we exploit the existing software to study the impact of personalization on student learning.

2017-07-31

2018-03-02

We will administer paper-based, multiple-choice student assessments of math before (pre-test) and after (post-test) students' interaction with Mindspark. This outcome is our main interest.

Following Muralidharan et al. (2016), we plan to: (a) develop an item map, ensuring that we cover a broad range of topics and domains; (b) draw on publicly-released items from domestic assessments (e.g., Student Learning Survey, Quality Education Study, and the Andhra Pradesh Randomized Studies in Education) and international assessments administered in India and other developing countries (e.g., the Program for International Student Assessment, the Trends in International Mathematics and Science Study, and Young Lives); (c) create different test booklets for adjacent grade combinations; (d) pilot the assessments with a small sample of students who are comparable to those who will participate in this study; and (e) scale and link the results across grades using Item Response Theory. We will also link our assessments to those of the impact evaluation in New Delhi to allow for comparisons across both studies.

To assess mechanisms, we will also collect the data from all the interactions that students have with the Mindspark platform (both its current and adapted versions). These data include: (a) students' initial preparation, as diagnosed by the software on the first session; (b) the set of questions (and grade-based level difficulty) presented to each student by the software; (c) students' response times to each question; and (d) students' answers to each question.

We will randomly assign students within each section to one of four experimental groups: (a) a control group ("not customized"), in which students are provided with activities on the Mindspark platform that matched their enrolled grade level; (b) a "fully customized" treatment group (T1), in which students complete activities that fully adjust to their "diagnosed" grade level (i.e., the way the Mindspark platform currently works); (c) a "partially customized" treatment group (T2), in which students complete activities that adjust to their diagnosed grade level, but in which the degree of customization is limited to only one grade level below or above that level; or (d) a "remedial" treatment group (T3), in which students complete activities that adjust to their diagnosed grade level, but in which the degree of customization is limited to only grades at or below that level.

We will randomly assign students in equal shares to one of the above experimental groups, at the individual level, within each section stratum. Group sizes may differ slightly when the number of students in stratum is not divisible by four. In this case, we randomly assign "left-over" observations (or "misfits") to one of the four groups, within each stratum.

Randomization done in office by a computer (using the -randtreat- package, in Stata).

Individual. See "Experimental Design (Public)" for information on stratification and handling of "misfits" (in case the number of observations in a given stratum is not divisible by four).

No

3,331 students in 14 schools (note that the treatment is assigned at the individual level - i.e., the design is not clustered)

3,331

Equal split of students across four arms (three treatments and one control), with random allocation of "left-over" observations within sections (if the number of students in a given section is odd)

0.121 standard deviations

This is one of the first studies to evaluate the impact of computer-based individualized instruction in a developing country. We randomly assigned 1,528 students in grades 6–8 in 15 “model” public schools in Rajasthan, India who were using a computer-adaptive learning software to: a control group, in which they were only able to access the activities for their enrolled grade level; or a treatment group, in which they were able to access exercises appropriate for their performance level. After 9 months, computer-based individualized instruction had a null average effect on math achievement. However, treatment students with low initial performance outperformed their control counterparts by 0.22 standard deviations. Our results suggest that computer-based individualized instruction is most beneficial for low-performing students.

de Barros, A., & Ganimian, A. J. (2023). Which Students Benefit from Computer-Based Individualized Instruction? Experimental Evidence from Public Schools in India. Journal of Research on Educational Effectiveness, 17(2), 318–343. https://doi.org/10.1080/19345747.2023.2191604

https://doi.org/10.1080/19345747.2023.2191604