Experimental Design
Sampling Strategy and Methods
This study uses cluster randomization at the health post level (HP) to identify intervention impact. Both the control and treatment groups received the standard Ethiopian Ministry of Health procedures for health promotion activities and the immunization schedule. The treatment group additionally received a behaviorally-informed feedback and non-monetary rewards intervention.
The intervention occurred in Arsi and East Shewa zones of the Oromia region of Ethiopia. For the evaluation, 90 Health Posts (HPs) were selected and 30 random sample of households from each HP catchment cluster were included in the household survey. In total, 2,760 households were surveyed (Figure 1). Recent Household Census of HP catchments conducted by the Health Extension Program of the Regional Government was used as a sampling frame to randomly select 30 sample households in each cluster. All households with a child or children less than 23 months, were included in the random selection of survey households. To capture the same cohort of children targeted at the baseline, our endline survey will cover all children under 4 years old in the sampled households.
The impact evaluation sample consists of 2,700 households, with equal numbers (1350 households) selected into treatment group (Health Posts with HEW Outreach Movement prompts and Stamp System) and selected out of the outreach movement prompts and stamp system (control groups). The behaviorally informed interventions included in the evaluation took place in 45 health post catchment areas/clusters in Arsi and East Shewa Zones of Oromia Regional State, Ethiopia. The other 45 randomly selected health post catchment areas remained controls for the study duration.
Project Area Selection Justification
The primary consideration in selecting Arsi and East Shewa Zones of Oromia for this pilot intervention were: 1) ensuring that the zones have a large number of districts with higher dropout rates and health post catchment areas have some level of mobile phone coverages, and 2) the zones are geographically accessible to ensure the health post sample size is sufficient to rigorously evaluate the impact of the intervention. The overall selection frame was thus based on dropout rates, feasibility and pilot operational considerations. Among the two zones, 8 woredas (districts) were selected based on the highest dropout rates. When a woreda was selected, then all Health Centers and the Health Post attached to that health center were automatically eligible for the selection. Households from within these Health Posts were randomly sampled for inclusion in the baseline and endline surveys. There are on average two HEWs in the selected 129 Health Posts. On average, selected distrcits (woredas) have higher dropout rates than other districts in the region.
Power Calculations
The Oromia region in Ethiopia has low levels of immunization coverage. The 2011 Ethiopia Demographic and Health Survey (Ethiopia Central Statistics Agency, 2011) calculated the current full immunization rate for the region at 12%, thus we used that as our baseline, as opposed to the national rate of 24% full immunization coverage. Rates for households in the lowest three wealth quintiles nationally are less than 20% (EDHS 2012). For these sample size calculations, we assumed a significance level (an alpha) of 0.05 and a power level (a beta) of 80%.
As we are employed a cluster-level design, we also estimated the Intracluster correlation (ICC). Using the Ethiopia DHS data, we estimated the ICC to be between 0.21 and 0.25 and compared it to published values in the literature. Banerjee et al. (2010) found an ICC of 0.21 related to community immunization in India while Blanton et al. (2007) estimated the average ICC for full immunization rates from 48 nationally-representative DHS surveys around the world to be 0.21. To be consistent with the literature, we used 0.21 for our calculations, and performed a sensitivity analysis to ensure that does not significantly affect the sample size.
Power calculations were completed using 3ie sample size minimum-detectable-effect calculator with the parameters above. It estimated a minimum detectable effect of 8.6 percent. Thus we would expect to be able to detect an increase in full immunization rate as small as 8.6 percent and a reduction in vaccine dropout by 8.6 percent – that is for example, we expect to detect an effect in the reduction of dropout as low as from its current 13.5% to 12.3%. Related studies in immunization detected similar of higher effect sizes. Banerjee et al. (2010) detected an impact of more than 50% in immunization with a non-monetary incentive given to households while Ryman et al. (2011) used a traditional training intervention to detect an increase of 30% in full immunization coverage in India.
Data Collection Instrument and Field Work Procedures
The questionnaire was designed and tested to capture background information on household composition, basic demographics, education, socio-economic and health status. However, the main focus was placed on information regarding health seeking behavior and immunization levels. The questionnaire was translated to Oromifa, the commonly used language in the study areas.
Prior to the baseline survey, enumerators were given a week-long training on the baseline survey module. Supervisors were deployed to follow up interviews and to ensure the quality of data. The field team were provided with the lists of health post clusters and corresponding households samples. Thirty respondent were interviewed per cluster in both the control and treatment groups. Data from each interview was scrutinized by the survey supervisor and for a second time by the field manager.