Experimental Design
The ideal evaluation design would consist of randomizing the access to the sewage system. This would assure comparability between households with and without access, and lead to a robust impact estimation of the intervention. Nevertheless, operative, logistical, and ethical reasons make this design unfeasible in this context.
On the other hand, this project can be considered as a natural experiment due to some of its particular characteristics, although available information as of now is not completely certain on this matter. The selection criteria used to determine the intervention places and coverage zones were associated with geographical and spatial features related to the San Miguelito basins. Therefore, the absence of differences in health and community welfare dimensions between treated and control households would not be surprising. If this were the case, a comparison of both household groups before and after the intervention would lead to reliable impact estimators.
Nonetheless, both groups could differ in observable and unobservable characteristics that could affect outcomes in health and welfare, which would hinder the identification of causal effects. Balance testing on observable characteristics using information produced by the baseline will help determine whether or not these differences exist. The presence of balance would ratify the existence of a natural experiment.
On the contrary, if balance between both groups does not exist, quasi-experimental techniques should be employed for impact calculation. In this particular case, the combination of matching and difference-in-differences methods would be suggested. In this scenario, the information collected in the baseline is useful to determine which households in the control group possess characteristics more closely related to those of the treatment group. Among the different matching techniques available, Propensity Matching Score, Balance-Sample Size Frontier Matching, and Genetic Matching are the most distinguished. Besides, the nature of this intervention makes it reasonable to use other impact evaluation techniques, such as a Geographic Regression Discontinuity or Instrumental Variables. Once household matching is accomplished, a difference-in-differences model would compare variations in outcomes on both groups.
In addition, taking into account possible spillover effects, an alternative control group was established using households located in the Panama North District, which is located far away from San Miguelito but shares similar sociodemographic characteristics. The same difference-in-differences and matching approach would be used to compare outcomes in both control groups to quantify the spillover effect.