|
Field
Last Published
|
Before
February 05, 2025 08:33 AM
|
After
March 07, 2026 07:09 PM
|
|
Field
Intervention (Public)
|
Before
In 2019, the School Education Department, Government of Andhra Pradesh (GoAP) has rolled out a Personalized Adaptive Learning program to students in grades 6 to 9 in 520 schools across 17 districts of Andhra Pradesh. Each of the 520 schools received 30 tablets with the PAL software with Mathematics content loaded on them and were told to setup a dedicated PAL Lab for students. Each student has an individual login through which they access the software. For each Mathematics chapter, the dynamic adaptive software identifies individual student learning gaps through a pre-assessment and provides remedial content including videos and questions. The chapters and content is mapped to the AP state curriculum. School math teachers are instructed to dedicate 2 out of 8 math periods in a week for PAL Lab. Further, the Government hired Field Management Staff (FMS) to help with operations and logistics of the PAL Labs in schools.
In collaboration with the GoAP, the researchers have been conducting 2 research studies (Randomized Controlled Trials) in Government schools. The aim of the first study is to evaluate the impact of the PAL program on student learning outcomes in mathematics. The second study aims to study how the program can be designed to be more effective at scale. These studies will contribute to the evidence base to support the scaling of targeted foundational learning programs in government systems.
|
After
The Government of Andhra Pradesh (GoAP) in India has launched one of the first government-run PAL programs in over 500 schools, with an interest in scaling to more schools. However, the effectiveness of education interventions, including learning through the use of edtech, when conducted at scale can be sensitive to the implementation features (List 2022; Banerjee et al. 2017; Kulik and Fletcher 2016). In collaboration with the GoAP, this project contains 4 studies (Randomized Controlled Trials and other interventions) to achieve the project goals of:
1) Evaluating the impact of the GoAP’s PAL program, and
2) Studying how the program can be designed to be more effective at scale.
Personalized Adaptive Learning (PAL) with edtech fosters remedial learning and mitigates learning gaps in settings where children lag behind grade-level skills (de Barros and Ganimian 2023; Muralidharan, Singh, and Ganimian 2019; Escueta et al. 2017). It can be a cost-effective way to address pandemic-related learning loss and inequities in access to education when investments in edtech have already been made (World Bank 2020).
In 2019, the School Education Department, Government of Andhra Pradesh (GoAP) rolled out one of the first government-run PAL programs for students in grades 6 to 9 in 520 schools across 17 districts of Andhra Pradesh. Each of the 520 schools received 30 tablets with the PAL software with Mathematics content loaded on them and were told to setup a dedicated PAL Lab for students. Each student has an individual login through which they access the software. For each Mathematics chapter, the dynamic adaptive software identifies individual student learning gaps through a pre-assessment and provides remedial content including videos and questions. The chapters and content are mapped to the AP state curriculum. School math teachers are instructed to dedicate 2 out of 8 math periods in a week for PAL Lab. Further, the Government hired Field Management Staff (FMS) to help with operations and logistics of the PAL Labs in schools.
In collaboration with the GoAP, the researchers have been conducting 3 research studies (Randomized Controlled Trials and other studies) in Government schools. The aim of the first study is to evaluate the impact of the PAL program on student learning outcomes in mathematics.
As an extension of the first study, a second phase was conducted to examine the longer-term impacts of the government-run PAL program and the mechanisms underlying its effects. This follow-up was conducted with 40 treatment and 40 control schools from the original RCT to assess the persistence of impacts. In addition, the design includes a non-experimental component drawing on data from 80 additional schools (40 residential schools and 40 schools from the broader group of 500 non-RCT PAL schools that had received the program prior to the RCT). Outcomes in these schools will be compared to those in the original treatment and control schools to better understand how variation in exposure, and implementation context shapes learning outcomes.
The second study aims to study whether varying the intensity of software remediation affects student learning outcomes. In particular, it examines the trade-off between reinforcing remedial concepts—which may enable faster learning over time—and the risk that students remain overly focused on remedial content and progress more slowly. The study also explores whether differences in students’ prerequisite knowledge generate spillover effects at the classroom level within a large-scale personalized adaptive learning program in Andhra Pradesh.
The third study examines how implementation intensity and frontline support shape the effectiveness of a large-scale personalized adaptive learning program, by assessing whether variation in the frequency of Field Management Staff visits influences software usage and sustained engagement in schools.
These studies will contribute to the evidence base to support the scaling of targeted foundational learning programs in government systems.
|
|
Field
Intervention Start Date
|
Before
September 21, 2023
|
After
July 31, 2023
|
|
Field
Intervention End Date
|
Before
March 31, 2026
|
After
July 31, 2026
|
|
Field
Primary Outcomes (Explanation)
|
Before
Math outcomes will be constructed from three sources i) Standardized learning assessments in math at endline, ii) PAL software data and iii) school assessments data provided by the state government
|
After
Math outcomes will be constructed from three sources i) Standardized learning assessments in math at endline, ii) PAL software data available with the software vendor and made available with state permissions and iii) school assessments data provided by the state government
|
|
Field
Experimental Design (Public)
|
Before
We have launched two RCTs to evaluate the PAL program by randomizing at either the student- or school-level.
|
After
We evaluated a large-scale government-run Personalized Adaptive Learning (PAL) program using a combination of randomized controlled trials and A/B tests. We have launched two RCTs to evaluate the PAL program by randomizing at either the student- or school-level, and are currently conducting an A/B test across 524 PAL schools to examine how variation in state-sponsored implementation support affects program usage. As an extension of the first study, we are also conducting a second round of standardized learning assessments to study how learning outcomes vary with PAL exposure across different school types, durations of exposure, and implementation contexts.
We will also examine two methodological considerations related to assessment design. First, we will test whether administering endline assessments on tablets produces differential effects by treatment status. Second, we will assess whether student performance varies based on the language of the test questions (English only versus Telugu only).
|
|
Field
Intervention (Hidden)
|
Before
Study 1 - Impact study (Evaluation of the government-run PAL program):
This study is centered on a randomized control trial where we assigned 120 schools to either the PAL group or the no-PAL (control) group. Schools in the PAL group received 30 tablets each with the PAL software installed. A PAL lab was setup in each of the schools. All students from 6-9th grades in these schools were taken to the PAL lab at least for one period per week. Control schools were not given tablets or the PAL lab and received the status quo activity, which could be practice, free, or class time.
The research team conducted a small-scale dipstick assessment in 6 schools (3 treatment and 3 control) and found suggestive evidence, though imprecise, that students in treated schools scored 0.3 SD (p-value = 0.37) higher than other students. These preliminary observations give us confidence that the program might have an impact on learning.
Study 2 - Remediation study (Varying the intensity of software remediation):
Learning new skills often builds on familiarity with other, prerequisite skills. If a student cannot demonstrate they have prerequisite skills, educators must decide whether the most efficient instructional strategy is to forge ahead or review more basic material. If review, they must also decide how far back to go, and when to press forward. Dwelling on basic concepts could be a valuable investment that produces more rapid learning in the future. But it could also mean that students get stuck going over remedial concepts. This study aims to shed light on this trade-off in the context of the personalized adaptive learning program in schools in Andhra Pradesh.
Students within a school were randomly assigned to conditions that vary the degree to which the software regresses to more basic concepts before returning to more advanced material. These conditions entail modifications to the software on the backend, and so students and teachers were blind to the treatment conditions that students have been assigned to. This study compares the impact of two versions of the software that vary the degree to which the program reviews “foundational” material if students are unable to complete a problem:
“Back to basics” – The software reviews material up to 3-4 grade levels earlier if students are unable to successfully complete a program
“Back to antecedents” – the software will review only directly antecedent material up to 1 grade level earlier.
Our preliminary findings indicate that the group receiving more intensive remediation dedicated more time to remedial practice material and achieved approximately 1 percentage point higher scores (equivalent to 0.2 SD, p-value< 0.001) on software assessments compared to the less intensive remediation group.
|
After
Study 1 - Impact study (Evaluation of the government-run PAL program):
This study is centered on a randomized control trial where we assigned 60 schools to the PAL group or 60 schools to the no-PAL (control) group. Schools in the PAL group received 30 tablets each with the PAL software installed. A PAL lab was setup in each of the schools. All students from 6-9th grades in these schools were taken to the PAL lab at least for one period per week. Control schools were not given tablets or the PAL lab and received the status quo activity, which is business as usual math periods.
The 60 PAL schools were drawn from a sample of the 500+ schools where the PAL program is being implemented at scale by the government. All schools had a non-zero probability of being selected, with selection probabilities varying based on PAL software usage over the past two academic years. In these schools, we conducted endline assessments with students in grades 6–9 using an independently-designed tablet-based learning assessment tool. These learning assessments covered material from grade 2 up to students’ current grade, and were centrally administered and graded.
Prior to the learning assessments, the research team conducted a small-scale dipstick assessment in 6 schools (3 treatment and 3 control) and found suggestive evidence, though imprecise, that students in treated schools scored 0.3 SD (p-value = 0.37) higher than other students. These preliminary observations give us confidence that the program might have an impact on learning.
As an extension of the first study, a second phase was conducted to examine the longer-term impacts of the government-run PAL program and the mechanisms underlying its effects. This follow-up was conducted with 40 treatment and 40 control schools from the original RCT to assess the persistence of impacts.
In addition, the design included a non-experimental component drawing on data from 80 additional schools (40 residential schools and 40 schools from the broader group of 500 non-RCT PAL schools that had been using the software at the time of the original intervention). This component is explicitly not considered identified, and the potential for selection effects to contaminate estimates is acknowledged. The primary motivation for collecting these data is to understand the degree to which usage can serve as a surrogate for learning gains.
The planned analysis builds on findings from Study 1, which showed a largely linear relationship between time spent using the tablet and test performance, with a slope slightly lower than that implied by the 2SLS estimator from the experiment. First, the OLS relationship between usage and test scores will be estimated, conditional on observable characteristics. Second, performance in these additional schools will be compared to outcomes in the original RCT treatment and control schools. Third, the relationship between hardware availability, monitoring intensity, teacher training, and other school characteristics unlikely to be endogenous to usage will be examined to assess the credibility of using these factors as instruments for usage in estimating its impact on test scores.
The study also assessed the extent to which tablet familiarity contributes to observed learning gains, enabling a more direct examination of whether continued exposure to and comfort with the technology help explain sustained effects. This was examined by comparing student performance across assessment modalities (paper-based versus tablet-based). Finally, teacher and headmaster surveys were conducted in all RCT PAL and non-RCT PAL schools included in this phase to document implementation experiences and the ways in which PAL influenced regular classroom instruction. Parallel surveys were administered in non-PAL control schools to understand how remedial learning support for weaker students is provided in the absence of PAL.
Study 2 - Remediation study (Varying the intensity of software remediation):
Learning new skills often builds on familiarity with other, prerequisite skills. If a student cannot demonstrate they have prerequisite skills, educators must decide whether the most efficient instructional strategy is to forge ahead or review more basic material. If review, they must also decide how far back to go, and when to press forward. Dwelling on basic concepts could be a valuable investment that produces more rapid learning in the future. But it could also mean that students get stuck going over remedial concepts. This study aims to shed light on this trade-off in the context of the personalized adaptive learning program in schools in Andhra Pradesh.
Students within a school were randomly assigned to conditions that vary the degree to which the software regresses to more basic concepts before returning to more advanced material. These conditions entail modifications to the software on the backend, and so students and teachers were blind to the treatment conditions that students have been assigned to. This study compares the impact of two versions of the software that vary the degree to which the program reviews “foundational” material if students are unable to complete a problem:
“Back to basics” – The software reviews material up to 3-4 grade levels earlier if students are unable to successfully complete a program
“Back to antecedents” – the software will review only directly antecedent material up to 1 grade level earlier.
Our preliminary findings indicate that the group receiving more intensive remediation dedicated more time to remedial practice material and achieved approximately 1 percentage point higher scores (equivalent to 0.2 SD, p-value< 0.001) on software assessments compared to the less intensive remediation group.
Study 3- Impact of FMS Visits on PAL Usage: The PAL program is implemented at the last mile with support from Field Management Staff (FMS), who provide technical assistance, monitor software usage, and help teachers sustain engagement in schools. FMS are hired and managed by the software provider and are each assigned to 10–15 schools. We conducted an A/B test in 524 schools in the state where the government has implemented the PAL program. The study did not involve any new procedures or data collection from teachers or students. Schools were randomly assigned, within each FMS’s set of schools, to receive either more frequent or less frequent routine FMS visits. Using administrative data on software usage shared by the software provider and a monitoring log of FMS visits designed by the research team and administered by the software provider, we will assess how visit frequency affects PAL usage.
|
|
Field
Secondary Outcomes (End Points)
|
Before
PAL software usage, teacher adoption rates, implementation fidelity, student interest in math and other subjects, student attendance
|
After
PAL software usage, teacher and headmaster adoption rates and perceptions, implementation fidelity, student interest in math and other subjects, student attendance, any spillover on language subjects (english and Telugu), tablets vs paper assessments, one-language math word problems, remedial vs at-grade gains, heterogeneity by gender, grade, baseline ability level.
|
|
Field
Secondary Outcomes (Explanation)
|
Before
|
After
Secondary outcomes for students will be constructed from PAL software data available with the software vendor and made available with state permissions, and surveys conducted with students on interest and use of PAL prior to conducting the learning assessments.
Teacher and Headmaster adoption and perceptions will be captured through teacher-headmaster surveys to understand their experience using PAL and how it has influenced regular classroom teaching. Teacher and headmaster surveys will also be conducted in the non-PAL control schools to understand how remedial and foundational learning for weaker students is supported in the absence of PAL.
|
|
Field
Pi as first author
|
Before
No
|
After
Yes
|