Promoting Early Grade Reading & Numeracy in Tanzania

Treatment Group 1, Capitation Grant (70 schools): In this group the capitation grant (CG) program, as dictated by the Tanzanian Ministry of Education and Vocational Training, is implemented at each school. Each school receives a grant of 10,000Tsh/student, delivered in two tranches, one in March and one in July. This treatment focuses on making the existing policy work by channeling funds in full and more effectively to primary schools, and to test the effects of basic information provision about the grants. The evaluation will seek to measure the extent to which the funds reach schools, the level of citizen engagement on the use of funds, and ultimately the impact of funds and information on improving learning outcomes. In 2013, the average CG distributed to schools was 7,646,429Tsh.

Treatment Group 2, Cash on Delivery (70 schools): In this group teachers of Kiswahili, English and Math in Grades 1, 2, and 3 are eligible to receive 5,000Tsh per student that passes a given subject test at the end of the year, for a total of 15,000Tsh possible per student. Teachers are not penalized for students who do not pass.

Treatment Group 3, Combination (70 schools): Schools in this treatment group are given both the Capitation Grant and Cash on Delivery intervention.

All interventions were implemented directly by Twaweza and it’s District Partners, with money given through the CG and COD interventions coming also from Twaweza. Within each intervention, information describing the intervention was distributed to schools and the communities via school and community meetings in early-2013. The District Partners then followed up with additional school visits in July and August to answer any questions regarding the program. All students in Grades 1, 2, and 3 in schools in Treatment Groups 2 and 3 were tested in Kiswahili, English and Math at the end of the school year to determine teacher incentive payments. Tanzanian education professionals, following a similar structure as the Uwezo annual learning assessment, developed the subject tests for Grades 1, 2, and 3. The same schedule will be followed in 2014.

2013-03-07

2015-02-28

The main outcome variables are students test scores, as a proxy for student learning. Assuming we are able to establish that the treatments had an impact in students test scores, secondary outcome variables would try to establish spillover effects, the mechanisms behind the main treatment effects and how the treatment effects vary across student, household, teacher and school characteristics.

A sample of 30 students per school (10 students from Grades 1, 2, and 3) are to be tested in all treatment schools and in 140 control schools. Students will be tested before the intervention begins, at the end of the first year and at the end of the second year. The information of these test will be used to calculate standardized test scores and compare achievement for children across treatment groups. We also collect detailed student information (e.g. age and gender); detailed school information (e.g. facilities, management practices and head teacher characteristics), detailed teacher information (e.g. education, age, experience and self reported time use), and detailed household information (e.g. parents engagement in child's education, parents own education, household composition, and assets owned by the household).

The information from our household, teacher and schools survey can be used to identify the mechanisms through which the treatment affects test scores. For example, we can look at the change in learning outcomes in non-incentivized subjects; how teachers spend their time in school; how schools spent the funds of the Capitation Grants (e.g. text books, scholarships, meals, etc); whether schools increased the hours taught in the incentivized subjects; and whether household become more or less engaged in the child's education after the intervention. Using the base line survey we can study how the treatment effects differ across student, household, teacher and school characteristics.

The main outcomes variables will be standardized test scores. First, we will construct a standardized test score for each subject in each grade, by subtracting the mean and dividing by the standard deviation of the test scores in the control group. Once we have subject-grade standardized test scores, we will add these up across grades and the re-normalize (dividing by the standard deviation of the test scores in the control group); this will yield subject standardized test score.

For some analysis we will also aggregate test scores across subjects by summing them and the re-normalizing (dividing by the standard deviation of the test scores in the control group).

This evaluation was conducted in 350 schools in 10 randomly selected districts across Tanzania. All government primary schools were eligible in each of the 10 districts, but 35 schools were randomly selected from each district to be part of the evaluation. The probability that a school was chosen was proportional to the number of students enrolled in the school. In each sampled district, 14 schools were assigned to the control group and 7 schools were assigned to each treatment group, CG, COD and Combination.

The aims of the three treatment groups are as follows:
1. Capitation Grant (CG): Test the impact of providing capitation grants (while providing the community with information on these grants) on improving basic literacy and numeracy.
2. Cash on Delivery (COD): Test the impact of incentivizing teachers to achieve previously set, absolute levels of learning among students in Grades 1, 2, and 3.
3. Combination (both CG and COD): Test the impact of COD "on top of" existing programs and budgets, from the CG intervention, in creating an incentive to make better use of those resources.

Randomization done in office using Stata.

Random sampling: District level. Random treatment assignment: School level

Yes

350 schools in 10 districts (35 schools per district)

350 schools; 14,000 students (40 per school); 3,300 teachers (8-12 per school); 5,250 households (15 per school)

70 schools Treatment 1 (Capitation Grant)
70 schools Treatment 2 (Cash on Delivery)
70 schools Treatment 3 (Combination, capitation grant and cash on delivery)
140 schools Control (no treatment)

We use Optimal Design Software for all power calculations. We originally assumed an intra-cluster correlation of 0.1 (that is the intra-cluster correlation for value added) and that 30% of the variation can be explained by baseline test scores and other covariates (such as age, gender, school and teacher characteristics) and district fix effects. Our main outcomes (the effect of capitation grants and cash on delivery) have a total of 280 clusters (140 controls schools and 140 treatment schools - 70 with the pure treatment and 70 with the combination treatment). In other words to estimate the effect of the capitation grant we have 140 control schools and 140 schools that recieve the capitation grant, 70 from Treatment 1 and 70 from Treatment 3. Similarly, we have 140 schools that receive Cash on Delivery. To estimate the interaction between the two main treatments we only have 70 treatment schools (Treatment 3) and 140 controls schools. With 280 clusters (i.e. to estimate the effect of the Capitation Grant or the Cash on Delivery) and a significance level of 5% we have: minimum detectable effect size of 0.2 with power of 99.9%, minimum detectable effect size of 0.15 with power of 97%, and minimum detectable effect size of 0.1 with power of 75%. With 210 clusters (i.e. to estimate the effect of the interaction between the Capitation Grant and Cash on Delivery) and a significance level of 5% we have: minimum detectable effect size of 0.2 with power of 99.5%, minimum detectable effect size of 0.15 with power of 92%, and minimum detectable effect size of 0.1 with power of 62.5%. We are aware that the numbers for the interaction assume symmetry and so the true numbers will be slightly lower, but the difference won't be first order. In practice, based on the first year data, we have that the intra-cluster correlation is 0.15 for Kiswahili, 0.06 for English and 0.14 for Math. The proportion of the variation can be explained by baseline test scores and other covariates (such as age, gender, school and teacher characteristics) and district fix effects. is 40% for Kiswahili, 36% for English and 37% for Math. Using the most conservative estimates (0.15 intra-cluster correlation and 36% of the variance explained by baseline characteristics) we have the following numbers: With 280 clusters (i.e. to estimate the effect of the Capitation Grant or the Cash on Delivery) and a significance level of 5% we have: minimum detectable effect size of 0.2 with power of 99%, minimum detectable effect size of 0.15 with power of 94%, and minimum detectable effect size of 0.1 with power of 65%. With 210 clusters (i.e. to estimate the effect of the interaction between the Capitation Grant and Cash on Delivery) and a significance level of 5% we have: minimum detectable effect size of 0.2 with power of 98.5%, minimum detectable effect size of 0.15 with power of 86%, and minimum detectable effect size of 0.1 with power of 53%.

Pre-Analysis Plan

We present results from a large-scale randomized experiment across 350 schools in Tanzania that studied the impact of providing schools with (i) unconditional grants, (ii) teacher incentives based on student performance, and (iii) both of the above. After two years, we find (i) no impact on student test scores from providing school grants, (ii) some evidence of positive effects from teacher incentives, and (iii) significant positive effects from providing both programs. Most important, we find strong evidence of complementarities between the programs, with the effect of joint provision being significantly greater than the sum of the individual effects. Our results suggest that combining spending on school inputs (the default policy) with improved teacher incentives could substantially increase the cost-effectiveness of public spending on education.

Isaac Mbiti, Karthik Muralidharan, Mauricio Romero, Youdi Schipper, Constantine Manda, Rakesh Rajani, Inputs, Incentives, and Complementarities in Education: Experimental Evidence from Tanzania, The Quarterly Journal of Economics, Volume 134, Issue 3, August 2019, Pages 1627–1673, https://doi.org/10.1093/qje/qjz010

https://academic.oup.com/qje/article/134/3/1627/5479257