|
Field
Last Published
|
Before
December 09, 2024 04:52 PM
|
After
December 10, 2024 12:31 AM
|
|
Field
Intervention (Public)
|
Before
Rori is designed to increase access to resources and choices for low-income students as it can be used by students in or outside of school. Rori can improve classroom learning, remediate key skills and concepts, or help with test preparation. It is inspired by successful elements of TaRL (Abdul Latif Jameel Poverty Action Lab) and ITSs (Banerjee, et al., 2016). Rori first assesses students’ ability and then encourages students to work through lessons at their level, each with a brief explanation and a series of scaffolded practice questions. When students struggle, Rori provides first a hint and then a complete solution. Rori will also direct students to more suitable lessons if a lesson is too challenging or too easy. There are over 500 lessons structured around the Global Proficiency Framework, an internationally recognized set of mathematics learning standards. Students can chat with Rori about their mathematics goals and encourage metacognitive skills as it uses various natural language processing (NLP) methods, including specialized language models (LLMs). This also enables Rori to better interpret students’ answer attempts and questions.
|
After
The intervention involves providing mobile phones with Rori embedded to treatment classes. Students in grades 6-9 will participate in this pilot and all participants will receive their regular mathematics instruction. The mobile phones will be stored at the school and students will be supervised in using Rori for at least 1 hour a week.
The central research question guiding our pilot is: "How does the effectiveness of Rori vary based on implementation fidelity?". Building upon two years of research within Rising Academy schools, which has produced some promising results, we now hope to understand what is necessary to scale Rori to other schools. Therefore, this pilot aims to investigate the impact of Rori in government schools, as they are the most common type of schools in Ghana, and other low-cost private schools to compare to Rising schools. The previous evaluation of Rori was conducted with rigorous oversight in controlled environments, where implementation leads ensured adherence through frequent monitoring visits. Before investing in a large-scale RCT of Rori, it is crucial to determine if other schools can adopt Rori without that level of oversight and, if they do so, does it impacts students’ learning. Our pilot will also explore sub-questions, including the influence of school type on Rori usage, assessing the operational effectiveness of device management approaches, and teachers' attitudes and receptiveness towards the intervention. By addressing these questions, we seek to advance understanding of how to effectively implement Rori in a variety of schools, thus informing future efforts to scale up the intervention and assess its impact in a full RCT involving even more schools. We hypothesize that Rori will be effective in improving mathematics learning gains among students in the treatment group. However, we believe implementation fidelity will vary amongst the schools and thus hypothesize the size of learning gains will vary accordingly. Using a mixed-methods approach will help us determine under what conditions students have good learning gains from using Rori, again informing us how to best implement and evaluate Rori in the future.
|
|
Field
Experimental Design (Public)
|
Before
We plan to recruit eight schools, for a total of 24 schools with relevant observable characteristics, such as in similar communities and graduation rates. Students in grades 6-9 will participate in this pilot and all participants will receive their regular math instruction. The classes of the schools will be randomly assigned into two conditions and Rising Academies will provide all treatment classes with phones. Students in the Treatment group will use Rori for an hour a week during this time usually reserved for extracurriculars or study hall. Students in the control group will not receive access to Rori during this time. Our partners will provide clear instructions to the teacher who will be in charge to ensure that the mobile phones are kept in a safe place and children have access to them when it is time for Rori class.
|
After
The study includes students from grades 6 to 9 across Rising Academy schools, low-cost private schools, and government schools in the same neighborhood to aid comparison. We recruited 4 low- cost private schools and 5 public schools in addition to the 8 Rising Academy schools, making a total of 17 schools. Given current enrollment estimates, we expect the sample to be around 2531 students. The Rising Academy schools have been previously randomized into control and treatment. For the other schools, to avoid putting an entire school in control, we will randomize classes into treatment and control at the grade level. For example, the control for grade 6 in one school will be grade 6 in another school. Students in the control classes will not be provided with the phones with Rori. Thus, the randomization is at the classroom level within a grade.
|
|
Field
Randomization Method
|
Before
We randomized in the office amongst 4 people, each individual was assigned a class for each school. whatever one picks
|
After
For government and other private schools, randomization is done by balloting within grades across schools.
For Rising Academy schools, randomization was previously for the pilot at the school level, matching treatment schools geographically with a control school. This randomization will be maintained for operational reasons.
|
|
Field
Randomization Unit
|
Before
Classroom Level Randomization
|
After
Classes
|
|
Field
Planned Number of Clusters
|
Before
24 Schools
|
After
17 schools, Four (4) grades (Class 6 to JHS 3), one class per grade
68 classrooms
|
|
Field
Planned Number of Observations
|
Before
1440 Pupils
|
After
We plan to assess approximately 2531 students between Grades 6 and 9 across 17 schools.
We will observe and document any form of attrition across conditions, classes and schools carefully.
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
12 Schools in Control , 12 Schools in Treatment
|
After
Treatment ("Rori"): 1260
Control ("No-Rori"): 1271
|
|
Field
Power calculation: Minimum Detectable Effect Size for Main Outcomes
|
Before
For our power calculation, we used an Intraclass Correlation Coefficient (ICC) of 0.09, based on findings by Hedges and Hedberg (2007) for low-achievement schools, and a standard error (SE) of 0.01, as per Kelcey et al. (2016) for cluster randomized trials in Sub-Saharan Africa.
|
After
We used data and estimates from a pilot led by some of our research team members based on only Rising Academy schools (an educational network based in Ghana i.e., non-public and partly low-cost schools) in 2023 to inform the parameters (means, standard deviations, and intra-cluster correlation) for our power calculation. The baseline data showed a Math Score of 20.20, with a standard deviation of 8.81 and an intra-cluster correlation (ICC) of 0.09.
With our sample of 2,531 students in Grades 9 to 12 across 17 schools and 51 classes, of which 1,260 would be in treatment (49.8% in the treatment arm), which is compared with 1,271 (50.2%) in the comparison group, we are powered to detect an effect size of at least 0.488 standard deviations with power of 0.8 and significance level of 0.05. Thus, our sample is large enough to detect an MDE which is less than one standard deviation. Our calculations are for intention-to-treat effects because we do not expect everybody who receives the treatment to use it consistently. The estimated MDE is within that observed by other researchers. For example, Muralidharan et al. (2019) examining the effect of personalized technology-aided after-school instruction program in India find effects sizes for math between 0.37 and 0.60.
|
|
Field
Secondary Outcomes (End Points)
|
Before
Poverty Reduction,
|
After
|