Primary Outcomes (explanation)
We estimated the impacts of the Strengthening Civic Participation program for six outcome indices: awareness of local government meetings, familiarity with local government officials, knowledge about local government affairs, access to district government information, citizen influence on government, and satisfaction with local services. We constructed these outcome indices by grouping together survey questions related to the same underlying outcome using factor analysis. We did this for several reasons. First, estimating the impacts by comparing the treatment and the control groups on several survey questions is likely to result in one or more statistically significant impacts by chance when there is actually no impact. In other words, we are more likely to incorrectly reject the null hypothesis, when considering a series of hypothesis tests—a problem commonly known as the multiple comparison problem (Benjamini and Hochberg, 1995). Second, since factor analysis assumes that the observed variables are influenced by a few underlying variables or factors that are unobserved, constructing those underlying latent variables (the unobserved factors) can provide useful information about processes or behavior of the population of interest. Third, grouping survey questions into a few outcome indices helps in examining impacts in a tractable manner.
For each of the six outcome domains, we conducted factor analysis using the survey questions or observed variables that are most relevant for that domain in three steps:
(1) We used the principal-component factor method to obtain the factor solutions. For each outcome domain, we found that only one underlying factor explained the variation in the responses to the included survey questions.
(2) We then used orthogonal rotation to rotate the factor loadings and estimated factor scores using the regression method, which estimates a factor as a weighted sum of the included observed variables.
(3) Finally, we converted each of the six estimated factors to binary variables to interpret the impact estimates better. In particular, if a survey respondent’s factor score was above the mean score for the full survey sample in that year, the binary variable was coded as 1, otherwise it was coded as 0. Thus, the impact estimates compare the percentage of citizens with an above-average factor score in the treatment districts with the percentage in control districts.