Experimental Design Details
For the analysis at the facility level, this evaluation uses a cluster randomized experimental design to estimate the causal impact of the interventions, where the clusters are markets of health facilities, and the cluster size is the market size, or number of health facilities per cluster. All local markets of health facilities in the three counties–Kakamega, Kilifi, and Meru–will be randomized into treatment and control groups, taking into consideration the size of the market. At the market level, the design is an stratified random design, with stratification at the market size.
The three counties were chosen after an extensive consultation with representatives of the Kenyan county health executives, based on prior analysis of health markets in all Kenyan counties. Specifically, we constructed health markets in each county and highlighted the counties that contained markets with diverse sizes. In some counties like Turkana, there were few markets with more than one health facility (typically in the arid Northern counties) and evaluations from these counties would not be predictive of performance in denser health markets. In other counties, such as Mombasa, most health facilities are in markets with a large number of other health facilities. In such counties, the predictive power of the evaluation for smaller markets is poor. Ten counties satisfied our technical criteria of health markets with diverse sizes. The analysis and counties were discussed with the county health executives, who decided the ultimate study areas.
The choice of a cluster-randomized experimental design stems from the nature of the intervention and the relevance of capturing the effect of the intervention at the market level. Particularly, the treatment applied to one health facility is thought to have potential spillovers on geographically proximate health facilities through consumer demand. Therefore, the cluster-randomized design was selected because of the potential spillover at the market-level, and the policy-relevance of impacts at this level in terms of quality of care and patient safety, quantity and prices.
The markets are defined at this stage by geographic delimitations. Based on preliminary data from the baseline from around 8,500 patients surveyed we find that around 75% of the respondents live 4 km or closer from the health facility they visited. Using this distance, we apply a clustering algorithm called the K-means algorithm that identifies markets of health facilities by the nearest center, conditional on the K-distance to the center. Our algorithm stops creating new clusters when all facilities are within the predefined distance (4 km) from their respective market center. Based on this exercise, we estimate there are 273 such markets in the three counties selected, with diverse market size ranging from singletons to markets with 2, 3, 4, 5 or more health facilities.
In total, we have 16 strata. For each county, we have 5 strata by market size for markets with 1, 2, 3, 4-10, and 11+ health facilities. In addition, we worry that extreme-value markets may not be balanced for analysis at the health facility level therefore a stratification of extreme values was added for Meru. Therefore, we have an additional stratum for market size 34 or more health facilities (3 markets) in Meru for a total of 16 strata. This is important because, for our analysis at the health facility level, imbalances in the market size (e.g., that all big markets end up only in one treatment arm or in the control group) will affect that level of analysis. We have 87 markets in the control group (411 facilities), 90 markets in the treatment group 1 (388 facilities), and 96 markets in the treatment group 2 with scorecards (473 facilities).