A randomized textbooks for homework experiment in primary education in South Kivu, DRC

This specific project aims at increasing the effectiveness of usage of existing textbooks within primary schools in South Kivu by encouraging teachers and students to use the textbooks in the classrooms and in particular at home for homework. To be more specific, the project will implement and evaluate an intervention that will integrate textbooks into the homework routine giving students the opportunity (and expectation) to bring home textbooks after class. Diagnostic inquiry suggested that textbooks in South Kivu were sometimes used in the classroom but never for homework due to concern for loss or damage. At the same time teacher focus groups suggested that using textbooks for homework could make a difference for learning.

The intervention will take place in a randomly picked sample of 45 out of the 90 primary schools in which the dutch NGO Cordaid is currently running a results based financing program. The method follows the convention for RCTs with a baseline, the intervention and an end line. The base line and end line will be conducted in all 90 schools, but the intervention only in a random sample of 45 schools stratified across two different school districts.

For the intervention, all students should bring home textbooks twice a week. One day the focus is the French language, and students bring home French language textbooks. The other day the focus is math and the students bring home math textbooks. To verify that this procedure is followed, all students and the teacher sign in and out textbooks at the end of the day (sign out) and the morning (sign in) with initials. This is also a check that books are not going missing, and if so, who is responsible. In addition to this procedure there will be a weekly quiz on the homework based on the textbooks. The conduct of the quiz will also be recorded on a separate column in the sign in and out sheets. To stimulate the students and teachers to abide by the intervention, posters explaining the purpose of the intervention and encouraging students to take their homework seriously will be developed and put up in the classroom.

In addition to intrinsic interest in learning and doing well, students will be incentivized to take the intervention seriously through a reward system divided into two parts. The first part is a non-material individual reward system based on a publicly announced (posters in the classroom) star system. A student earns a star each week that the student has:
• Taken home and returned in good condition her/his textbooks according to the assignments.
• Taken part in the weekly homework quiz.
Similar public star systems have been found effective in other contexts. Note that the individual stars are not associated with any material benefits. The incentives work only through mechanisms of social status.

The second part is a material group reward in case the class as a whole performs well enough. We have decided on a group rather than individual material reward system partly for practical purposes, but also because we want to encourage kids to take a joint and collaborative approach towards reaching the goals. Classes that fulfill the following conditions are eligible for a reward:
- The ratio of actual stars to possible stars (if all class students got stars every week) should be at least 75 %.
- No book has been lost or badly damaged
The reward consists of handouts of school material such as notebooks and pens and pencils. If a book is lost, the whole class will be encouraged to help find it. As mentioned above, a lost or damaged book also means that no material reward is handed out.

2016-12-08

2018-05-31

Student learning: Student results on French and math tests designed by research team and conducted at baseline and end line for all 90 schools. Student results on TENAFEP (national test) at the end of grade 6.

We will also analyze the impact of treatment on two variables reflecting student aspirations, whether they sign up to take the national test TENAFEP (a prerequisite to continue to secondary school) and whether they aspire to continue in secondary school. For the subset of students for which we have parent surveys, similar aspirations will be analyzed among the parents, and also parental engagement and satisfaction with the school. We will also look at some outcomes that can only be measured at the school level. In particular we will analyze how the intervention has affected the use of textbooks in the classroom, teachers’ perceptions about the effectiveness of textbooks for learning, and teacher efficacy. These questions will be analyzed using the teacher surveys and well established instruments to gage efficacy.

The experiment is set up as a randomized controlled trial with a single treatment arm. A sample of 90 schools is randomly selected into 45 treatment and 45 control schools, stratified across two school districts (Walungu 1 and Shabunda 1).

A baseline will be conducted through tablet surveys of headmasters, 5th grade teachers, 12 randomly selected 5th grade parents and all 5th grade students. In addition to that we have also developed and tested a mathematics and French language test for the 5th grade students to get a baseline of their knowledge. Our trained enumerators will conduct the headmaster, teacher and student surveys (and student test), and local community based organizations with experience from doing household surveys in the specific field settings will be responsible for the parent surveys.

The intervention (described above) will then be implemented in a standardized way in all the 45 treatment schools, immediately following the baseline. Progress will be monitored with regularity.

End line: The intervention is planned to run until June 2018. At that point we will conduct an end line surveying the same subjects (we are collecting panel data) basically following the format of the base line to help us analyze if the textbook intervention has made any difference for student learning, parent and student satisfaction and the active use of textbooks in the teaching. For this we will use our own designed tests and survey instruments, but we can also look at data on retention rates between grades 5 and 6. We also have an opportunity to look at national test score results in grade 6. We will also conduct testing of teacher factual knowledge to analyze if impact is conditional on teacher knowledge.

Done in office by a computer, stratified based on district (Shabunda and Walungu).

The primary unit of randomization is at the level of the school.
In addition we will randomly draw a set of 12 parent households for each 5th grade class to interview. These are drawn from a class list stratified according to gender composition in the class.

Yes

90 schools, 45 treatment and 45 control.

All 5th grade students in the schools, at baseline 4982. There will be attrition between base line and end line, so final panel sample will be smaller. 1 teacher per class, estimated to be 90 teachers. 1 head master per school, 90 head masters. 12 parents per class, estimated to be 1 080.

Just one treatment and symmetry in size between treatment and control groups, so based on baseline we expect:

2 490 students.
45 teachers.
45 head masters.
540 parents.

Final numbers may be lower due to attrition.

Using a very conservative rate of attrition, a power calculation based on 2700 students across 45 treatment and 45 control schools with an unconditional intra-school correlation of 0.40 and requiring 80% power of detecting a significant difference at the two-sided 10% level suggests a minimum detectable difference of 0.39 z-score.