Randomized evaluation of markets for impact in the Ghana and Sierra Leone basic education sectors

The governments of Ghana and Sierra Leone have created ‘markets for impact’ by implementing an outcomes funding mechanism in their education sectors. These programs are called the Ghana Education Outcomes Project (GEOP) and the Sierra Leone Education Innovation Challenge (SLEIC). The GEOP has two components: an Accelerated Learning Programme and a Mainstream School Improvement Programme (MSIP). This study is focusing on the MSIP.

Each government identified geographic lots with schools in need of support, awarded a contract per lot to a grantee (service provider) via competitive procurement, set target outcomes (based on literacy and numeracy by gender), and is making year-end payments for outcomes achieved (measured via randomized evaluation). The treatment group in Ghana is 610 schools enrolled in the GEOP MSIP, and in Sierra Leone 325 schools enrolled in the SLEIC. The intended treatment is ‘receipt of a package of support activities from a service provider reimbursed by an outcomes contract’. The design of each mechanism is stable and has been implemented by the respective government. That said, these mechanisms purposefully incentivize innovation. Since providers are paid based on whether students learn, they have substantial freedom to evolve their activities in service of those outcomes. At the bidding stage, most support packages included a combination of teacher professional development, pedagogical training, targeted instruction, and community engagement. The study will purposefully document what each provider is implementing and how this evolves over time.

2022-09-01

2026-07-01

The end points (final outcomes in the theory of change) that we will study as primary outcomes are:

o Low stakes measure of cognitive foundational skills (numeracy and literacy)
o Measure of non-cognitive foundational skills
o Measure of student retention

A key objective of this study is to unpack what providers implemented and how (if at all) these activities contributed to improvements in the above final outcomes. We therefore add a further set of primary outcomes that are further up the causal chain, closer to provider activities:

o Practices to improve learning (combined index)
o Measure of classroom environment (Stallings time-on-task)
o Measure of teacher attendance
o Measure of student time use

Most of our measures are based on standard, validated measurement tools. The exception is our measure of practices to improve learning. Here, we have identified five categories of intervention (activities with head teachers, teachers, districts, parents/caregivers and school management committees). For each category, we will construct a subindex of practices observed/reported. For example, for teachers we will measure (via classroom observation) their use of three specific teaching practices: structured pedagogy, targeted instruction, and SEL pedagogies). We plan to combine these sub-indices into a single index for our primary outcome.

In our secondary, more exploratory analysis, we will study the following outcomes:

o Practices to improve learning (five sub-indices, for: head teachers, teachers, districts, parents/caregivers, SMCs)
o Head teacher and teacher job satisfaction
o Teacher growth mindset

Ghana.
Sampling of schools. An initial sample of schools was drawn by the research team. The sampling frame was the list of basic government-run schools in the World Bank funded Ghana Accountability for Learning Outcomes Project (GALOP) in the six geographic GEOP lots identified by the Ghana Ministry of Education. We randomly sampled an agreed number of schools (sufficient to give the required number of treatment schools, plus an equal number of control schools, and some replacements) using an exclusion restriction based on school GALOP index scores to maximise cross-lot comparability. This created a sample of 1,261 schools located across 27 districts, out of the total 1,626 GALOP schools in the area. We visited these schools in May-July 2022 to conduct a baseline survey. The Independent Verification Agent, KMPG, then drew a slightly smaller sample of 1,220 schools which became the GEOP sample.

Randomisation. There is a single treatment arm (schools receive interventions from GEOP grantees), and a single control arm. KMPG performed the treatment assignment, in private using Stata. The sample was stratified by the 27 districts and, within districts, by GALOP score (above average vs below average score).

Sierra Leone.
Sampling of schools. OPM used the Annual Primary School Census from 2020 as the initial sampling frame for schools. Out of 7,020 schools in Sierra Leone, 3,438 schools were selected based on 8 eligibility criteria: GoSL-approved schools (either government-owned or financially supported by GoSL); pupil enrolment of 100 or more; four of more teachers; single-shift; accessible; Grades 1-6; coed/mixed; not part of the Education Innovation Challenge Programme (a precursor to the SLEIC). The sampling frame was then stratified. Selected schools were divided into five geographical lots and then further stratified within each lot based on: enrolment (3 categories), distance from district HQ (3 categories), and gender parity in enrolment (2 categories). 650 schools were drawn from the 3,438 schools using these lots, and within lots, 13 strata. This gave a total of 65 individual strata across the five lots.

Randomisation. There is a single treatment arm (schools receive interventions from SLEIC grantees), and a single control arm (schools do not receive interventions from SLEIC grantees). OPM performed the treatment assignment, in private using Stata. The method used was using random systematic sampling (with equal probabilities within the stratum). The eligible schools within each stratum were first ordered geographically, in order to provide implicit stratification and a proportional allocation of the schools by district within each lot and stratum. The selected schools within each stratum were then systematically divided into two replicates. One replicate was randomly assigned to the treatment group, and the other to the control group. This ensured that the treatment and control schools are equally representative of the stratum and have the same geographic implicit stratification.

Not available

The randomization was performed by the independent verification agents (KPMG in Ghana and Oxford Policy Management in Sierra Leone). This was done in private, using Stata.

School

Yes

Ghana. There are 1220 schools (clusters). 200 in Lot 1 (3 districts), 200 in Lot 2 (3 districts), 170 in Lot 3 (4 districts), 240 in Lot 4 (6 districts), 240 in Lot 5 (5 districts) and 170 in Lot 6 (6 districts).

Sierra Leone. There are 650 schools (clusters). 130 in Lot 1 (4 districts), 124 in Lot 2 (4 districts), 132 in Lot 3 (2 districts), 130 in Lot 4 (3 districts), and 134 in Lot 5 (3 districts).

Ghana. Measurement will take place in 1220 schools and associated households. We will survey the head teacher in each of the schools (1220 head teacher observations). We will randomly sample four teachers per school to observe their attendance. We will observe four classrooms per school. Our ambition is to observe maths classes for P2, P3, P4 and P5. [This may be adjusted if the school uses multi-grade teaching or if maths classes are not running for some reason.] We will survey each of the four teachers who have had their classrooms observed. [Note that this sample may or may not overlap with the sample drawn to measure attendance.] We will sample 10 students per school (one boy and one girl per grade for P2-P6 inclusive) to assess their cognitive and non-cognitive foundational skills. We aim to survey the main caregiver of each of these 10 students. Sierra Leone. Measurement will take place in 650 schools and associated households. We will survey the head teacher in each of the schools (650 head teacher observations). We will randomly sample four teachers per school to observe their attendance. We will observe four classrooms per school. Our ambition is to observe maths classes for P2, P3, P4 and P5. [This may be adjusted if the school uses multi-grade teaching or if maths classes are not running for some reason.] We will survey each of the four teachers who have had their classrooms observed. [Note that this sample may or may not overlap with the sample drawn to measure attendance.] We will sample 10 students per school (one boy and one girl per grade for P2-P6 inclusive) to assess their cognitive and non-cognitive foundational skills. We aim to survey the main caregiver of each of these 10 students.

Ghana. Clusters by treatment arm, within lot:

Lot 1 100 treatment, 100 control
Lot 2 100 treatment, 100 control
Lot 3 85 treatment, 85 control
Lot 4 120 treatment, 120 control
Lot 5 120 treatment, 120 control
Lot 6 85 treatment, 85 control

Sierra Leone. Clusters by treatment arm, within lot:

Lot 1 65 treatment, 65 control
Lot 2 62 treatment, 62 control
Lot 3 66 treatment, 66 control
Lot 4 65 treatment, 65 control
Lot 5 67 treatment, 67 control

Note that the distinction across lots is important statistically (because randomisation was stratified and strata were created within lots) but also substantively (lots had different grantees who may have adopted a different set of interventions).