Reading Books and Literacy Training Program in Indonesia

The intervention comprises three components: (1) the provision of a curated collection of reading books tailored to various reading proficiency levels, (2) a four-day teacher training session, and (3) the establishment of mentored learning groups across participating schools.

1. Book Provision
Each treatment school will receive 600 books, featuring 200 distinct titles that equally span five distinct reading levels. These books are designed to match the diverse literacy levels of students.

2. Teacher Training
Two teachers from each treatment school, specifically those responsible for literacy and numeracy instruction, will participate in a four-day training session led by local facilitators. The first two days of training focus on literacy, equipping teachers with strategies for managing high-quality reading materials and fostering reading habits among students. The remaining two days shift focus to numeracy, which, while secondary to this study's primary emphasis on literacy, may still influence numeracy scores in the national assessment. The final session of the training discusses the plan to share the knowledge gained during the training with other teachers and staff at participants' respective schools.
While the training includes a numeracy component, the study's primary focus remains on literacy, consistent with the previous phase of the program. Numeracy has only recently been added to the program. In the analysis, we will also examine whether the training has any impact on numeracy outcomes.

3. Learning Groups
Post-training, treatment schools will be grouped based on their geographic proximity. These groups will be invited to a one-time in-person sharing session, where schools exchange best practices for utilizing the provided books and enhancing students’ literacy skills. The local facilitator for the teacher training will provide assistance/mentoring during this session.

2024-05-01

2024-12-20

Literacy scores

We will use the student-level literacy scores based on the annual national assessment for grade 5 students.

Numeracy test scores, students’ reading habits, and literacy teaching practices in schools.

We aim to understand the mechanism behind the results by examining additional outcomes. First, we will investigate whether the intervention affects other subjects by analyzing student-level numeracy scores from the National Assessment. Second, we will assess the program's impact on students' reading habits by collecting data on the frequency of their reading at home. Finally, we will evaluate whether the program changes how teachers teach literacy. The primary data for these analyses will come from the National Assessment, complemented by a phone survey conducted in a subsample of 440 schools.

The impact evaluation is designed with a Randomized Evaluation across 1,000 primary schools (504 treatment and 496 control) with low literacy performance, located in 36 districts across 10 provinces in Indonesia. The treatment schools will receive a package of intervention, including reading books, training, and routine sharing sessions, while the control schools will not receive any intervention in 2024. The program is limited to schools that had at least 200 students to optimize the benefits of the distributed books.

The study focuses on level 2 primary schools, that is, those where less than 50 percent of students have achieved the minimum reading literacy competence. We focus on this group because all level 1 schools have already received the intervention, and the Ministry has identified level 2 schools as having the greatest need for further literacy improvement. For the study, we further restricted ourselves to schools that participated in the national assessment as the scores from the assessment would be our main outcomes.

The baseline for the study includes data from the National Assessment conducted in October 2023, which provides an estimate of literacy levels for grade 5 students, as well as a phone survey conducted between February and May 2024 in a random subsample of 440 schools. The phone survey collects information from the headmaster and one teacher per school. The distribution of books and teacher training occurs from May to early September 2024.

In October 2024, we will conduct a qualitative study in 20 treatment schools to further explore how the intervention has influenced literacy education practices. The program's impact will be assessed using data from the National Assessments in October 2024 and 2025. Additionally, a final phone survey in early 2025 will evaluate changes in the availability and use of reading books in the schools that participated in the initial phone survey.

Not available

Randomization was done in the office using Stata.

School

Yes

- 1000 schools for the admistrative data analysis
- 440 schools for the phone survey data analysis

For the administrative data analysis, the National Assessment includes a maximum of 30 students per school, resulting in up to 30,000 observations. For the phone survey, we plan to survey two respondents per school—the headmaster and a teacher—yielding a total of up to 880 respondents.

The number of clusters per treatment arm includes 504 treatment schools and 440 control schools. From these, we selected a subsample of 220 treatment schools and 220 control schools for the phone survey.

The power calculations for this study are based on the public use data distributed for an evaluation of the teacher certification program in Indonesia, as detailed in the study Double for Nothing? Experimental Evidence on an Unconditional Teacher Salary Increase in Indonesia by Joppe de Ree and others (The Quarterly Journal of Economics, Volume 133, Issue 2, May 2018, Pages 993–1039). We use Table A6, which presents the results for language for primary schools as the base specification. De Ree et. al (2018) conducted their study using a representative sample of 10 districts across Indonesia and tested almost all grades. Our calculations were performed in Stata, using the estimation files provided by De Ree et. al (2018) and J-PAL’s programs for power calculations. We assume a 100 percent take up rate. Previous evaluation studies conducted by the Ministry indicate that there is no record of schools not participating in the training. There may be an issue with book delivery. Although all schools signed for receipt, a survey of PSKP (policy evaluation unit of the Ministry) indicated many respondents were not aware that new books had arrived. During the study we will monitor and encourage timely book delivery. With randomization at the school level, no controls, 0.8 power, the minimum sample required for an effect size of 0.1 is 451 schools per group. For an effect size of 0.2, this would be reduced to 113. Including the controls that were included in the aforementioned study (which include strata dummies and baseline test scores) the required sample size drops to 208 for 0.1 std dev. effect size and 58 for 0.2 std dev effect size. These calculations are based on the assumption of 20 students tested per school. We use the 0.1 standard deviation as our effect size for the literacy score. This is a program that mostly provides inputs (reading books) and only has a light training component (4 days centralized training by 2 delegates from each school). Comparable programs found negligible to very minimum impact on reading skills. In the Indian context, the training of librarians on library operations and on educational literacy activities demonstrated minimal impact on language scores (Borkum et al, 2012). Meanwhile the provision of additional textbooks in Kenya had limited overall influence, primarily benefiting higher-scoring students with test scores increasing by 0.14 to 0.22 standard deviations, but such initiative resulted in positive changes in teacher behavior (Glewwe et al, 2009; Sabarwal et al, 2014). Conversely, in the Philippines, a more intensive literacy program yielded noteworthy improvements, increasing reading scores by 0.13 standard deviations (Abeberese, 2014). For our study, we will not have access to student level baseline tests. We will have access to 2022 and 2023 individual national assessment data in grade 5, which will give us an indication of the baseline literacy level in the school. Our final study sample is randomized at the school level and contains 504 treatment and 496 control schools. It is spread out over 10 provinces. It is thus sufficient to detect an effect of 0.1 std dev improvement in literacy scores. As we will be able to condition on strata fixed effects and literacy scores of the school in the baseline national assessments, we think we overpowered the 0.1 standard deviations effect size.