NEW UPDATE: Completed trials may now upload and register supplementary documents (e.g. null results reports, populated pre-analysis plans, or post-trial results reports) in the Post Trial section under Reports, Papers, & Other Materials.
Understanding the Impacts of a Civic Participation Initiative in Rwanda: Experimental Findings
Initial registration date
Not yet registered
April 15, 2014 11:47 AM EDT
Mathematica Policy Research
Other Primary Investigator(s)
Additional Trial Information
The Millennium Challenge Corporation sponsored the Rwanda Threshold Program (RTP) to help the Government of Rwanda improve its performance on governance indicators related to citizens’ political rights and civil liberties. Mathematica Policy Research conducted an evaluation of the RTP’s Strengthening Civic Participation (SCP) component, an initiative with two focus areas: (1) supporting the efforts of civil society organizations to advocate for local issues and (2) training local government officials to increase responsiveness to the concerns and priorities of citizens. The evaluation uses a stratified random assignment design, whereby districts within each of Rwanda’s five provinces were paired based on district population and economic characteristics and then randomly assigned to either a treatment or a control group. We designed a household survey to collect data for evaluating SCP impacts, and administered this survey at baseline before the program (in early 2011) and one year after the start of the program (2012). We drew nationally representative samples of approximately 10,000 households in each data collection round. The baseline and follow-up surveys collected data on respondents’ civic participation levels, including awareness and perceptions of local government performance, responsiveness, and accountability. Specifically, respondents were asked about their awareness of local government meetings, familiarity with local government officials, perceived citizen influence, and knowledge and access to information about local government affairs. Respondents were also asked about their overall satisfaction with government services related to water infrastructure, local road conditions, waste collection, public schools, and health clinics. We use this household survey data to analyze program impacts.
Nichols-Barrer, Ira. 2014. "Understanding the Impacts of a Civic Participation Initiative in Rwanda: Experimental Findings." AEA RCT Registry. April 15.
The Strengthening Civic Participation (SCP) program was an initiative with two focus areas: (1) supporting the efforts of civil society organizations to advocate for local issues through grant support and technical assistance and (2) supporting the efforts of local government officials to increase responsiveness to the concerns and priorities of citizens through grant support and technical assistance.
Intervention Start Date
Intervention End Date
Primary Outcomes (end points)
The baseline and follow-up surveys collected data on respondents’ civic participation levels, including awareness and perceptions of local government performance, responsiveness, and accountability. Specifically, respondents were asked about their awareness of local government meetings, familiarity with local government officials, perceived citizen influence, and knowledge and access to information about local government affairs. Respondents were also asked about their overall satisfaction with government services related to water infrastructure, local road conditions, waste collection, public schools, and health clinics.
Primary Outcomes (explanation)
We estimated the impacts of the Strengthening Civic Participation program for six outcome indices: awareness of local government meetings, familiarity with local government officials, knowledge about local government affairs, access to district government information, citizen influence on government, and satisfaction with local services. We constructed these outcome indices by grouping together survey questions related to the same underlying outcome using factor analysis. We did this for several reasons. First, estimating the impacts by comparing the treatment and the control groups on several survey questions is likely to result in one or more statistically significant impacts by chance when there is actually no impact. In other words, we are more likely to incorrectly reject the null hypothesis, when considering a series of hypothesis tests—a problem commonly known as the multiple comparison problem (Benjamini and Hochberg, 1995). Second, since factor analysis assumes that the observed variables are influenced by a few underlying variables or factors that are unobserved, constructing those underlying latent variables (the unobserved factors) can provide useful information about processes or behavior of the population of interest. Third, grouping survey questions into a few outcome indices helps in examining impacts in a tractable manner.
For each of the six outcome domains, we conducted factor analysis using the survey questions or observed variables that are most relevant for that domain in three steps: (1) We used the principal-component factor method to obtain the factor solutions. For each outcome domain, we found that only one underlying factor explained the variation in the responses to the included survey questions. (2) We then used orthogonal rotation to rotate the factor loadings and estimated factor scores using the regression method, which estimates a factor as a weighted sum of the included observed variables. (3) Finally, we converted each of the six estimated factors to binary variables to interpret the impact estimates better. In particular, if a survey respondent’s factor score was above the mean score for the full survey sample in that year, the binary variable was coded as 1, otherwise it was coded as 0. Thus, the impact estimates compare the percentage of citizens with an above-average factor score in the treatment districts with the percentage in control districts.
Secondary Outcomes (end points)
Secondary Outcomes (explanation)
The activities under the Strengthening Civic Participation component of the RTP were planned to be implemented in two separate, year-long phases, with 15 districts receiving activities in each phase. We took advantage of this plan by randomly dividing Rwanda’s 30 districts to either phase I or phase II. To do so, districts within each province were first matched in pairs or groups of three to ensure the best possible match between the two district groups. The districts were matched on the following five district characteristics:
• Population change between 2002 and 2006
• Population density
• Common Development Fund (CDF) appropriation amounts for FY 2008 (as a proxy for poverty levels)
• Share of district spending obtained through local revenues in FY 2008
• District expenditure per capita on good governance and social affairs These matched district pairs or triplets were then used as randomization blocks. Specifically, after these blocks were created a public lottery was conducted in the presence of district officials to randomly assign districts within each block to the treatment group (districts that would receive program activities in phase I in 2011), or the control group (districts that were to receive program activities in phase II in 2012). The 15 control districts never received any program activities because the RTP was not extended beyond 2011.
Experimental Design Details
Clustered random assignment occurred at the district level.
Was the treatment clustered?
Sample size: planned number of clusters
Sample size: planned number of observations
Sample size (or number of clusters) by treatment arms
15 districts control, 15 districts treatment
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
INSTITUTIONAL REVIEW BOARDS (IRBs)
National Institute of Statistics Rwanda
IRB Approval Date
IRB Approval Number
Post Trial Information
Is the intervention completed?
Intervention Completion Date
December 31, 2011, 12:00 AM +00:00
Is data collection complete?
Data Collection Completion Date
November 30, 2013, 12:00 AM +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
Was attrition correlated with treatment status?
Final Sample Size: Total Number of Observations
Final Sample Size (or Number of Clusters) by Treatment Arms
15 treatment districts, 15 control districts