AI, implementation and academic outcomes in poor resource settings: experimental evidence from Ghana

Last registered on December 10, 2024
View Trial History

Pre-Trial

Trial Information

General Information

Title
AI, implementation and academic outcomes in poor resource settings: experimental evidence from Ghana
RCT ID
AEARCTR-0014476
Initial registration date
December 04, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
December 09, 2024, 4:52 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
December 10, 2024, 12:57 AM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country
Ghana
Region
Greater Accra

Primary Investigator

Name
Dr. Edward Asiedu
Affiliation
University of Ghana, Legon
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Owen Henkel
PI Affiliation
Rising Academies Network
Contact Investigator
PI Name
Hannah Horne-robinson
PI Affiliation
Rising Academies Network
Contact Investigator

Additional Trial Information

Status
In development
Start date
2025-01-01
End date
2025-12-30
Keywords
Behavior, Education, Post-conflict
Additional Keywords
AI, children, academic outcomes, mathematics, low-cost private schools, government-run public schools, Ghana
JEL code(s)
Secondary IDs
Prior work
This trial is based on or builds upon one or more prior RCTs.
Abstract
Low-and-middle income countries (LMIC) are experiencing a learning crisis, with less than 15% of sub-Saharan African students reaching minimum mathematics proficiency by the end of middle school (UNESCO, 2017). Students are often taught content far above their ability level and lack opportunities for practice and timely feedback. While high-quality one-on-one instruction has long been advocated to address these issues (Bloom 1984; Chi et al., 2001), the low supply and high cost of quality tutors in West Africa, pose significant barriers to access (Bray 2021). Consequently, research is needed to identify affordable and scalable interventions tailored to this context. Leveraging technology-supported learning, such as Intelligent Tutoring Systems (ITS) (Vanlehn, 2011) has shown promise in addressing diverse learning needs in LMICs. However, students in LMICs often don’t have access to home computers and the internet, though mobile phone usage is high (International Telecommunication Union 2022). The University of Ghana and Rising Academies, an educational network based in West Africa, seek to improve learning in primary and junior high schools in Ghana. Rising Academies has developed an AI-driven solution, 'Rori' to help children improve their math performance. Rori is a chat-based math tutor on WhatsApp, which can be accessed on affordable mobile phones. Rori has the potential to generate benefits for students living in poverty who cannot afford extra in-person tutorials. There are many possible avenues to using Rori as it does not require extensive bandwidth or large amounts of costly data, unlike platforms that provide video lessons. Building on encouraging results with students attending Rising Academy schools (Henkel et al., 2024), our proposed pilot will encompass a broader spectrum of schools (low-cost private schools and government-run public schools) to deepen the understanding of implementation challenges and intervention effectiveness. If Rori effectively improves math proficiency in all school types, it could support policy change and the widespread adoption of Rori, contributing to poverty reduction through enhanced educational outcomes. This pilot will also support the planning of a future RCT by helping us understand how to implement Rori in diverse contexts, ultimately contributing to developing scalable and sustainable solutions for improving educational outcomes in LMICs.
External Link(s)

Registration Citation

Citation
Asiedu, Dr. Edward, Owen Henkel and Hannah Horne-robinson. 2024. "AI, implementation and academic outcomes in poor resource settings: experimental evidence from Ghana." AEA RCT Registry. December 10. https://doi.org/10.1257/rct.14476-1.3
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
The intervention involves providing mobile phones with Rori embedded to treatment classes. Students in grades 6-9 will participate in this pilot and all participants will receive their regular mathematics instruction. The mobile phones will be stored at the school and students will be supervised in using Rori for at least 1 hour a week.

The central research question guiding our pilot is: "How does the effectiveness of Rori vary based on implementation fidelity?". Building upon two years of research within Rising Academy schools, which has produced some promising results, we now hope to understand what is necessary to scale Rori to other schools. Therefore, this pilot aims to investigate the impact of Rori in government schools, as they are the most common type of schools in Ghana, and other low-cost private schools to compare to Rising schools. The previous evaluation of Rori was conducted with rigorous oversight in controlled environments, where implementation leads ensured adherence through frequent monitoring visits. Before investing in a large-scale RCT of Rori, it is crucial to determine if other schools can adopt Rori without that level of oversight and, if they do so, does it impacts students’ learning. Our pilot will also explore sub-questions, including the influence of school type on Rori usage, assessing the operational effectiveness of device management approaches, and teachers' attitudes and receptiveness towards the intervention. By addressing these questions, we seek to advance understanding of how to effectively implement Rori in a variety of schools, thus informing future efforts to scale up the intervention and assess its impact in a full RCT involving even more schools. We hypothesize that Rori will be effective in improving mathematics learning gains among students in the treatment group. However, we believe implementation fidelity will vary amongst the schools and thus hypothesize the size of learning gains will vary accordingly. Using a mixed-methods approach will help us determine under what conditions students have good learning gains from using Rori, again informing us how to best implement and evaluate Rori in the future.
Intervention (Hidden)
Intervention Start Date
2025-02-03
Intervention End Date
2025-05-30

Primary Outcomes

Primary Outcomes (end points)
Increase in learning gains
Primary Outcomes (explanation)
The main outcome measure will be learning gains, which will be assessed with adapted items from Trends in International Mathematics and Science Study (TIMSS), an externally validated assessment. We considered other assessments, but the NWEA requires internet and laptops and there are no standard national exams in Ghana with population-level data. All students will be assessed twice in their respective schools, and their baseline scores will be subtracted from their endline scores. The assessment will take between 1 and 1.5 hours.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)
The sub-questions will have multiple outcome measures including some phones that are retained at the end of the intervention, usage values for each phone, Likert ratings, and qualitative responses from teachers. Teachers will be interviewed to share their experiences and challenges when supervising students using Rori.

Experimental Design

Experimental Design
The study includes students from grades 6 to 9 across Rising Academy schools, low-cost private schools, and government schools in the same neighborhood to aid comparison. We recruited 4 low- cost private schools and 5 public schools in addition to the 8 Rising Academy schools, making a total of 17 schools. Given current enrollment estimates, we expect the sample to be around 2531 students. The Rising Academy schools have been previously randomized into control and treatment. For the other schools, to avoid putting an entire school in control, we will randomize classes into treatment and control at the grade level. For example, the control for grade 6 in one school will be grade 6 in another school. Students in the control classes will not be provided with the phones with Rori. Thus, the randomization is at the classroom level within a grade.
Experimental Design Details
Randomization Method
For government and other private schools, randomization is done by balloting within grades across schools.

For Rising Academy schools, randomization was previously for the pilot at the school level, matching treatment schools geographically with a control school. This randomization will be maintained for operational reasons.
Randomization Unit
Classes
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
17 schools, Four (4) grades (Class 6 to JHS 3), one class per grade
68 classrooms
Sample size: planned number of observations
We plan to assess approximately 2531 students between Grades 6 and 9 across 17 schools. We will observe and document any form of attrition across conditions, classes and schools carefully.
Sample size (or number of clusters) by treatment arms
Treatment ("Rori"): 1260
Control ("No-Rori"): 1271
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We used data and estimates from a pilot led by some of our research team members based on only Rising Academy schools (an educational network based in Ghana i.e., non-public and partly low-cost schools) in 2023 to inform the parameters (means, standard deviations, and intra-cluster correlation) for our power calculation. The baseline data showed a Math Score of 20.20, with a standard deviation of 8.81 and an intra-cluster correlation (ICC) of 0.09. With our sample of 2,531 students in Grades 9 to 12 across 17 schools and 51 classes, of which 1,260 would be in treatment (49.8% in the treatment arm), which is compared with 1,271 (50.2%) in the comparison group, we are powered to detect an effect size of at least 0.488 standard deviations with power of 0.8 and significance level of 0.05. Thus, our sample is large enough to detect an MDE which is less than one standard deviation. Our calculations are for intention-to-treat effects because we do not expect everybody who receives the treatment to use it consistently. The estimated MDE is within that observed by other researchers. For example, Muralidharan et al. (2019) examining the effect of personalized technology-aided after-school instruction program in India find effects sizes for math between 0.37 and 0.60.
Supporting Documents and Materials

Documents

Document Name
Grade 6-7 Assessment
Document Type
survey_instrument
Document Description
File
Grade 6-7 Assessment

MD5: c1dd3783e8b43a897782bdd0415fd0fa

SHA1: 04e33836502808ffd390aed3dfbf46e7617914b4

Uploaded At: November 19, 2024

Document Name
Grade 8 -9 Assessment
Document Type
survey_instrument
Document Description
File
Grade 8 -9 Assessment

MD5: 54f2a08294652f4b0a3b4c38839d9a3c

SHA1: 0e72b6bb1ba1a8f401062a6950f45d82ca34cfbd

Uploaded At: November 19, 2024

IRB

Institutional Review Boards (IRBs)

IRB Name
Noguchi Memorial Institute for Medical Research IRB
IRB Approval Date
2024-10-01
IRB Approval Number
#35/24-25
Analysis Plan

Analysis Plan Documents

AI, implementation and academic outcomes in poor resource settings: experimental evidence from Ghana

MD5: 483c5989c710029ee970a6ef6f59a82e

SHA1: 4e271fc6d3aaf0a060d3587ce1936700da24b475

Uploaded At: December 04, 2024

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials