AI, implementation and academic outcomes in poor resource settings: experimental evidence from Ghana

Last registered on December 10, 2024

View Trial History

Pre-Trial

Trial Information

General Information

Title

AI, implementation and academic outcomes in poor resource settings: experimental evidence from Ghana

RCT ID

AEARCTR-0014476

Initial registration date

December 04, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

December 09, 2024, 4:52 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated

December 10, 2024, 12:57 AM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name

Dr. Edward Asiedu

Affiliation

University of Ghana, Legon

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Owen Henkel

PI Affiliation

Rising Academies Network

Contact Investigator

PI Name

Hannah Horne-robinson

PI Affiliation

Rising Academies Network

Contact Investigator

Additional Trial Information

Status

In development

Start date

2025-01-01

End date

2025-12-30

Keywords

Behavior, Education, Post-conflict

Additional Keywords

AI, children, academic outcomes, mathematics, low-cost private schools, government-run public schools, Ghana

JEL code(s)

Secondary IDs

Prior work

This trial is based on or builds upon one or more prior RCTs.

Abstract

Low-and-middle income countries (LMIC) are experiencing a learning crisis, with less than 15% of sub-Saharan African students reaching minimum mathematics proficiency by the end of middle school (UNESCO, 2017). Students are often taught content far above their ability level and lack opportunities for practice and timely feedback. While high-quality one-on-one instruction has long been advocated to address these issues (Bloom 1984; Chi et al., 2001), the low supply and high cost of quality tutors in West Africa, pose significant barriers to access (Bray 2021). Consequently, research is needed to identify affordable and scalable interventions tailored to this context. Leveraging technology-supported learning, such as Intelligent Tutoring Systems (ITS) (Vanlehn, 2011) has shown promise in addressing diverse learning needs in LMICs. However, students in LMICs often don’t have access to home computers and the internet, though mobile phone usage is high (International Telecommunication Union 2022). The University of Ghana and Rising Academies, an educational network based in West Africa, seek to improve learning in primary and junior high schools in Ghana. Rising Academies has developed an AI-driven solution, 'Rori' to help children improve their math performance. Rori is a chat-based math tutor on WhatsApp, which can be accessed on affordable mobile phones. Rori has the potential to generate benefits for students living in poverty who cannot afford extra in-person tutorials. There are many possible avenues to using Rori as it does not require extensive bandwidth or large amounts of costly data, unlike platforms that provide video lessons. Building on encouraging results with students attending Rising Academy schools (Henkel et al., 2024), our proposed pilot will encompass a broader spectrum of schools (low-cost private schools and government-run public schools) to deepen the understanding of implementation challenges and intervention effectiveness. If Rori effectively improves math proficiency in all school types, it could support policy change and the widespread adoption of Rori, contributing to poverty reduction through enhanced educational outcomes. This pilot will also support the planning of a future RCT by helping us understand how to implement Rori in diverse contexts, ultimately contributing to developing scalable and sustainable solutions for improving educational outcomes in LMICs.

External Link(s)

Registration Citation

Citation

Asiedu, Dr. Edward, Owen Henkel and Hannah Horne-robinson. 2024. "AI, implementation and academic outcomes in poor resource settings: experimental evidence from Ghana." AEA RCT Registry. December 10. https://doi.org/10.1257/rct.14476-1.3

Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Experimental Details

Interventions

Intervention(s)

The intervention involves providing mobile phones with Rori embedded to treatment classes. Students in grades 6-9 will participate in this pilot and all participants will receive their regular mathematics instruction. The mobile phones will be stored at the school and students will be supervised in using Rori for at least 1 hour a week.

The central research question guiding our pilot is: "How does the effectiveness of Rori vary based on implementation fidelity?". Building upon two years of research within Rising Academy schools, which has produced some promising results, we now hope to understand what is necessary to scale Rori to other schools. Therefore, this pilot aims to investigate the impact of Rori in government schools, as they are the most common type of schools in Ghana, and other low-cost private schools to compare to Rising schools. The previous evaluation of Rori was conducted with rigorous oversight in controlled environments, where implementation leads ensured adherence through frequent monitoring visits. Before investing in a large-scale RCT of Rori, it is crucial to determine if other schools can adopt Rori without that level of oversight and, if they do so, does it impacts students’ learning. Our pilot will also explore sub-questions, including the influence of school type on Rori usage, assessing the operational effectiveness of device management approaches, and teachers' attitudes and receptiveness towards the intervention. By addressing these questions, we seek to advance understanding of how to effectively implement Rori in a variety of schools, thus informing future efforts to scale up the intervention and assess its impact in a full RCT involving even more schools. We hypothesize that Rori will be effective in improving mathematics learning gains among students in the treatment group. However, we believe implementation fidelity will vary amongst the schools and thus hypothesize the size of learning gains will vary accordingly. Using a mixed-methods approach will help us determine under what conditions students have good learning gains from using Rori, again informing us how to best implement and evaluate Rori in the future.

Intervention Start Date

2025-02-03

Intervention End Date

2025-05-30

Primary Outcomes

Primary Outcomes (end points)

Increase in learning gains

Primary Outcomes (explanation)

The main outcome measure will be learning gains, which will be assessed with adapted items from Trends in International Mathematics and Science Study (TIMSS), an externally validated assessment. We considered other assessments, but the NWEA requires internet and laptops and there are no standard national exams in Ghana with population-level data. All students will be assessed twice in their respective schools, and their baseline scores will be subtracted from their endline scores. The assessment will take between 1 and 1.5 hours.

Secondary Outcomes

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

The sub-questions will have multiple outcome measures including some phones that are retained at the end of the intervention, usage values for each phone, Likert ratings, and qualitative responses from teachers. Teachers will be interviewed to share their experiences and challenges when supervising students using Rori.

Experimental Design

The study includes students from grades 6 to 9 across Rising Academy schools, low-cost private schools, and government schools in the same neighborhood to aid comparison. We recruited 4 low- cost private schools and 5 public schools in addition to the 8 Rising Academy schools, making a total of 17 schools. Given current enrollment estimates, we expect the sample to be around 2531 students. The Rising Academy schools have been previously randomized into control and treatment. For the other schools, to avoid putting an entire school in control, we will randomize classes into treatment and control at the grade level. For example, the control for grade 6 in one school will be grade 6 in another school. Students in the control classes will not be provided with the phones with Rori. Thus, the randomization is at the classroom level within a grade.

Experimental Design Details

Not available

Randomization Method

For government and other private schools, randomization is done by balloting within grades across schools.

For Rising Academy schools, randomization was previously for the pilot at the school level, matching treatment schools geographically with a control school. This randomization will be maintained for operational reasons.

Randomization Unit

Classes

Was the treatment clustered?

Yes

Experiment Characteristics

Sample size: planned number of clusters

17 schools, Four (4) grades (Class 6 to JHS 3), one class per grade
68 classrooms

Sample size: planned number of observations

We plan to assess approximately 2531 students between Grades 6 and 9 across 17 schools. We will observe and document any form of attrition across conditions, classes and schools carefully.

Sample size (or number of clusters) by treatment arms

Treatment ("Rori"): 1260
Control ("No-Rori"): 1271

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

We used data and estimates from a pilot led by some of our research team members based on only Rising Academy schools (an educational network based in Ghana i.e., non-public and partly low-cost schools) in 2023 to inform the parameters (means, standard deviations, and intra-cluster correlation) for our power calculation. The baseline data showed a Math Score of 20.20, with a standard deviation of 8.81 and an intra-cluster correlation (ICC) of 0.09. With our sample of 2,531 students in Grades 9 to 12 across 17 schools and 51 classes, of which 1,260 would be in treatment (49.8% in the treatment arm), which is compared with 1,271 (50.2%) in the comparison group, we are powered to detect an effect size of at least 0.488 standard deviations with power of 0.8 and significance level of 0.05. Thus, our sample is large enough to detect an MDE which is less than one standard deviation. Our calculations are for intention-to-treat effects because we do not expect everybody who receives the treatment to use it consistently. The estimated MDE is within that observed by other researchers. For example, Muralidharan et al. (2019) examining the effect of personalized technology-aided after-school instruction program in India find effects sizes for math between 0.37 and 0.60.

Supporting Documents and Materials