Play to Learn - Improving Foundational Learning with Technology Aided Formative Teaching

Last registered on December 20, 2022

Pre-Trial

Trial Information

General Information

Title
Play to Learn - Improving Foundational Learning with Technology Aided Formative Teaching
RCT ID
AEARCTR-0009648
Initial registration date
September 29, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
October 05, 2022, 11:34 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
December 20, 2022, 8:16 AM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
3ie

Other Primary Investigator(s)

PI Affiliation
Wellesley College
PI Affiliation
University of Virginia
PI Affiliation
University of Virginia

Additional Trial Information

Status
On going
Start date
2022-03-31
End date
2024-04-01
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Computer-aided learning (CAL) can play a vital role in reducing learning gaps in developing countries by providing low-cost access to supplemental learning. While existing studies have focused on the effects of computer-based learning systems on older students, the impact of CAL on younger children during their formative years is still unknown. In our study, we will evaluate the effect of a “formative teaching” CAL model for foundational learning where teachers in grades 1 and 2 will leverage a game-based software, Chimple, to assign supplemental at-home learning activities.

The intervention will be implemented in public schools managed by the Bharti Foundation in Haryana, India. These schools primarily serve students from low-income backgrounds. Hence, the intervention will also illuminate the efficacy of this product for improving outcomes for marginalized populations. As schools return to in-person learning, technology-aided supplemental learning can play an instrumental role in helping teachers address a heterogeneous classroom. In addition to evaluating a teacher-driven model where students are encouraged to continue using the software because their teacher assigns content regularly, we will also randomly assign parents into two groups, with one group receiving low-intensity and the other receiving high-intensity nudges, to study how engagement responds to reminders.
External Link(s)

Registration Citation

Citation
Burga , Ramiro et al. 2022. "Play to Learn - Improving Foundational Learning with Technology Aided Formative Teaching ." AEA RCT Registry. December 20. https://doi.org/10.1257/rct.9648-1.1
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Computed aided learning (CAL) has tremendous potential in bridging learning gaps in developing countries. Bulman and Fairlie (2016) and Escueta et al. (2020) provide comprehensive reviews of this literature. Most current studies focus on summative learning models where student progress is assessed at the end of CAL learning. Muralidharan et al. (2019) test an adaptive student differentiated learning echo system deployed at learning centers and find significant improvements for children in higher grades. In such models, instructors can track the student’s performance and make their pedagogical plans responsive to the progression. The efficacy of such a model for younger children has not been tested thus far.

Our study will leverage the deployment of a free educational app named Chimple, available in local vernacular language (Hindi in our case) geared towards foundational learning (literacy and numeracy) harnessing game-based approaches. Chimple can be installed on low-end smartphones and works offline with its library of 70 unique games - 50 for literacy and 20 for numeracy. The Literacy ladder takes the child from letter recognition to sentence formation and comprehension. The Numeracy ladder takes the child from number sense to basic multiplication. Chimple has a dashboard that tracks the child’s participation and progress. Chimple corresponds to the syllabi for the grade proposed by the government (K-2). These features make it very attractive to marginalized and vulnerable populations as well. The game-based approach circumvents the issue of short attention spans of young children.

XPRIZE and UNESCO have tested Chimple in a pilot evaluation in Tanzania. This program lasted 15 months (December 2017 -March 2019) with 500 children aged 7-11 across 30 villages in the Tanga region of Tanzania. The product was accessed via tablets. Learning outcomes were assessed via EGRA/EGMA test (RTI) and Social-Emotional assessment (UNESCO) as well. Large effects of over 0.5 standard deviations were observed for both literacy and numeracy.

Chimple in India: Implementation in India was initiated in a computer lab setting in 15 schools in Mumbai, Maharashtra, in partnership with the Brihan Mumbai Municipal Corporation, although this has been paused due to COVID. Around 500 children from low-income households accessed Chimple via Door Step School Mumbai community centers. But the efficacy of Chimple in a teacher-driven at-home environment has not been tested.

The intervention will be deployed in 34 schools (the Satya Bharti schools) located in 5 districts in the state of Haryana. Bharti Foundation will spearhead the deployment. Teachers in grades 1 and 2 will integrate Chimple in their online course plans as a supplemental tool. Each week specific topics will be assigned to the students. The teachers will be able to track students’ progress via the software dashboard. Due to COVID surges and inclement weather, schools are often closed. Whatsapp, a messaging tool, is being used as a classroom instructional tool aide primarily for content dissemination. The teachers will ask parents to install the Chimple app on their phones and enable children’s engagement at home. On average, a recently concluded survey indicates that the vast majority of households in Satya Bharti Schools had at least one WhatsApp-enabled phone.
Intervention (Hidden)
Intervention Start Date
2022-07-15
Intervention End Date
2023-04-30

Primary Outcomes

Primary Outcomes (end points)
educational software usage, math and language test scores
Primary Outcomes (explanation)
Educational software usage. We will employ the number of completed activities and time spent in the app by each student to measure software usage. This information will be collected from the Chimple dashboard.
Math and language tests will be similar to those conducted by Bharti Foundation and other organizations such as Pratham.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Our intervention to examine the impact of interactive app-based learning on children's test scores will be deployed in 34 primary schools across 5 districts in Haryana run by Bharti Foundation. Total enrollment across all schools is approximately 1000 students per grade. More than 80% of children enrolled are of disadvantaged castes, and 50% of households earn less than USD$1,300 per year. Approximately 90% of households have access to at least one smartphone that can run Chimple.

Teachers in grades 1 and 2 will integrate Chimple into their course plans as a supplemental tool. Each week teachers will assign homework via Chimple and track students’ progress via the dashboard. The intervention will leverage Whatsapp, a messaging tool, as a platform for teachers to ask parents to install Chimple on their phones. Schools will run a parent information session to highlight the potential value of Chimple for their children’s learning. We will also use these WhatsApp groups to nudge parents to support Chimple assignment completion at varying frequencies.
Experimental Design Details
We will assess the impact of Chimple on children's test scores using a stratified randomized control trial, where a class is the unit of randomization. The sample includes 34 schools with 1 class each in grades 1 and 2, Chimple’s targeted age group. We will use baseline average test scores to group schools in 8 strata (7 groups of 4 schools and one group of six schools). In each stratum, 50% of schools will be randomly chosen to offer the Chimple program in grade 1 (e.g., Schools A and B); the other schools will offer the Chimple program in grade 2 (e.g., Schools C and D). Grade 1 in Schools A and B will serve as a comparison for Grade 1 in Schools C and D, and similarly, Grade 2 in Schools C and D will serve as a comparison for Grade 2 in Schools A and B.

Students in treated classrooms will be randomly assigned to a low and high-intensity WhatsApp group to nudge parents. This randomization will be stratified by classroom and parental access to smartphones (access or no access). In other words, we will randomize the WhatsApp group for each classroom-by-smartphone access cell.
Randomization Method
We will use the Stata software for our randomization exercise. We will sort the schools by the average test score of students in grades 1 and 2, set the seed, and generate a random uniform distribution in the 0-1 interval within each stratum. Schools for which the random variable is below the stratum median, will participate in the Chimple intervention for first graders. The rest of the schools will participate in Chimple for second grade. By design, there will be 17 schools treated in first grade and 17 treated in second grade.
Regarding the randomization of Whatsapp groups, first, students within each classroom-by-smartphone access cell will be sorted by admission date. After setting a seed, a uniform random number will be generated within each cell, and those having values above the median will be assigned to a high-intensity group. The rest will be assigned to the low-intensity group.
Randomization Unit
We will employ a stratified randomized control trial, where a class is the unit of randomization for the Chimple intervention. The Whatsapp randomization will be at the student level.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
Our sample will include 68 classrooms (34 first- and second-grade classrooms).
Sample size: planned number of observations
Our sample will include 68 classrooms, comprising approximately 2000 students in grades 1 and 2.
Sample size (or number of clusters) by treatment arms
34 control classrooms, 34 treated classrooms, half of the students in treated schools high-intensity nudging, and the other half low-intensity nudging.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Using data from past learning assessments conducted by Bharti Foundation and a bootstrapping procedure to account for the within-school randomization, we estimate that the minimal detectable effect (MDE) in the classroom-level randomization ranges from 0.2 to 0.3 SD. That is, we have a reasonable chance (80% power) of statistically detecting a significant difference (5% significance) in test scores between children in treatment classes and children in comparison classes if the true effect of Chimple on test scores is 0.2 - 0.3 SD. An effect of 0.3 SD would be considered a large effect in the education literature; at the same time, the pilot evaluation of Chimple found impacts of approximately 0.5 SD. We used past learning assessments and dashboard data on weekly Chimple engagement (in minutes) to estimate the MDE in the student-level randomization of WhatsApp nudges. It is essential to mention that we used usage information from weeks before we started to nudge families. Different methodologies provide similar estimates. We obtained MDEs in the ranges of 0.14-0.15 SD and 0.19-0.2 SD for test scores and weekly engagement, respectively.
Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
Institutional Review Board for Social and Behavioral Sciences, University of Virginia
IRB Approval Date
2022-08-23
IRB Approval Number
4610
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials