Decision-focused evaluation of an NGO-led TaRL model in Morocco

Teaching at the Right-Level (TaRL) is a remedial education approach aimed at enhancing students' basic literacy and numeracy skills. This method involves grouping students by proficiency level and subject rather than by age, as is typically done in the national education system. TaRL incorporates a set of activities and a participatory pedagogical approach, with students attending remedial education sessions outside regular school hours.

The implementing foundation will administer Annual Status of Education Report (ASER) tests, adapted to the Moroccan context, to students in grades 3 to 6. Based on their performance in these tests, students will either receive TaRL support (beneficiaries) or no additional support (non-beneficiaries). ASER is a simple, standardized test that estimates children's educational status and basic learning levels.

The remedial education sessions will be organized as follows:

- 100 hours of TaRL activities (4 hours per week) over 25 weeks (approximately 6 months, or one full semester).
- Each week, there will be 1 session of French (2 hours), 1 session of Arabic (1 hour), and 1 session of Mathematics (1 hour).
- This schedule will provide students with 50 hours of French sessions, 25 hours of Arabic, and 25 hours of Mathematics.

Educators were trained in the TaRL methodology during the second semester of 2023. Concurrently, mentors were trained to supervise and support educators in implementing TaRL activities. The pilot phase began in October 2023 and will conclude, including national school holidays, around June 2024.

2023-11-06

2024-06-02

Scores at the extended ASER test we developed with our partners

Learning outcomes in French, Arabic, and Mathematics will be measured using an extended ASER test, which was constructed with support from the implementing NGO and Pratham. This test is distinct from the one used by the implementing NGO to group students by level.

1) Difference in the implementation cost between the control and treatment interventions.
2) Difference in gender dynamics during remedial education sessions between TaRL and the control intervention.
3) Difference in teachers' implicit gender stereotypes towards students in treatment and control schools.
4) Difference in the trend of average grades given by teachers in treatment and control schools.

2) Gender Dynamics: Gender dynamics will be measured by the frequency of boys' and girls' interventions during the remedial education sessions. An intervention includes raising a hand to speak, speaking when called on by the teacher, speaking without permission, chatting with other students (same or opposite sex), and interrupting (or being interrupted by) another student (same or opposite sex).

3) Teachers' Implicit Gender Stereotypes: We will use two Implicit Association Tests (IATs) to measure teachers' implicit gender stereotypes towards students. The first IAT assesses the stereotype that boys excel in science while girls excel in liberal arts. The second IAT measures the stereotype that girls are more disciplined than boys.

4) Average Grades Given by Teachers: This outcome will be derived from administrative data. For each teacher in our sample schools, we will collect grade data from the past five school years. Using a difference-in-difference method, we will determine if the trend in average grades given by teachers changes due to the implementation of TaRL.

We selected a sample from a pool of 129 schools collaborating with the implementing partner. However, not all schools could participate in the study.

39 schools that implemented TaRL last year as part of a pre-test were excluded from the evaluation. Due to logistical constraints, the implementing partner chose not to implement its usual program in the provinces of Inezgane and Khmisset. Only one school in Sidi Slimane remained eligible for inclusion in the sample, as it was not part of the pre-test list submitted by the implementing partner.

To conduct impact analyses by province, a mix of schools implementing TaRL and the traditional remedial education program was necessary. Unfortunately, this arrangement was not feasible in Inezgane, Khmisset, and Sidi Slimane, leading to their exclusion from the sampling frame. Consequently, the final sample frame consisted of 89 schools.

From these 89 schools, a subset of 60 schools was selected for the impact evaluation. Each selected school was randomly allocated to either the TaRL group or the usual remediation group.

Schools were selected based on a stratified randomization procedure to ensure balance along outcomes of interest and representation of subgroups. Stratification was done by province, and within each school, students were randomly drawn in each stratum (gender and school grade) proportionately to the Sanady student population.

Due to implementation issues in the province of Taroudant, this province was excluded from our sample (8 schools). An additional school was removed because the implementing partner was unable to work there. The final sample comprises 51 schools with an average of 22 students per school. All teachers conducting remedial education in each school will take the Implicit Association Test, with an expected average of 3 teachers per school.

Randomization was conducted in office by a computer.

Two-stage cluster random sampling:
First stage is schools ;
Second stage is students

Yes

51 schools

1,140 pupils Approximately 153 teachers for the Implicit Association Tests

26 schools control, 25 TaRL schools

To calculate power we use 80% power and 5% alpha. For ASER score, we assume an ICC of 0.19 based on baseline data and a correlation between baseline and endline scores of 0.6 which leads to a MDE of 0.1 s.d.