Do At-Scale Social Audits Improve Service Delivery of a Large Public Works Program?

Last registered on December 06, 2021

Pre-Trial

Trial Information

General Information

Title
Do At-Scale Social Audits Improve Service Delivery of a Large Public Works Program?
RCT ID
AEARCTR-0008640
Initial registration date
December 03, 2021
Last updated
December 06, 2021, 10:09 AM EST

Locations

Region

Primary Investigator

Affiliation
University of Southern California

Other Primary Investigator(s)

PI Affiliation
University of Southern California
PI Affiliation
Yale University
PI Affiliation
Yale University

Additional Trial Information

Status
On going
Start date
2019-04-01
End date
2022-06-30
Secondary IDs
Prior work
This trial is based on or builds upon one or more prior RCTs.
Abstract
We study the recent rapid expansion of Bihar's social audits initiative as a measure to improve access and delivery of social protection under India's national workfare scheme. We undertake a causal evaluation utilizing administrative data from a randomized controlled trial across the state. Initial analysis using the administrative data finds that audits lead to a significant, sustained reduction in work as reported in administrative data, a result that could reflect some combination of reduced corruption --- a goal of audits, or declines in actual work --- an unintended consequence. Looking ahead, we will collect additional survey data needed to differentiate between these possibilities and provide further direction for program improvement.
External Link(s)

Registration Citation

Citation
Moore, Charity et al. 2021. "Do At-Scale Social Audits Improve Service Delivery of a Large Public Works Program?." AEA RCT Registry. December 06. https://doi.org/10.1257/rct.8640-1.0
Experimental Details

Interventions

Intervention(s)
We study the impact of "social audits" of one of India's largest social protection programs, the Mahatma Gandhi National Rural Employment Guarantee Scheme (MGNREGS), in the Indian state of Bihar. MGNREGS guarantees at least one hundred days of unskilled manual labor to every rural household that demands work; the program is self-targeted and aims to provide income support to the rural population. Social audits of MGNREGS combine community-led monitoring with external oversight: an external team of mostly female auditors reviews the program's implementation, and the audit culminates in a public meeting where local residents discuss the results of the audits and hold local officials accountable. To identify the causal impact of social audits on MGNREGS program performance, we collaborated with the state agency responsible for implementing social audits to conduct a randomized controlled trial (RCT) that leveraged the state's ambitious audit expansion plan in the 2019-2020 fiscal year (FY).
Intervention Start Date
2019-04-01
Intervention End Date
2020-03-31

Primary Outcomes

Primary Outcomes (end points)
Through this research, we aim to identify the causal effect of a policy innovation designed to improve accountability under MGNREGS, social audits, on program outcomes, including program expenditure, work provision, and program leakage.
Our analysis will focus on the following primary outcomes:
Access to MGNREGS entitlements */**
Demand for work under MGNREGS */**
Incidence of false work in official work records *
Outcomes marked by a single asterisk (*) will be measured using survey data. Outcomes marked by a double asterisk (**) will be measured using administrative data.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Our analysis will focus on the following secondary outcomes:
Awareness of MGNREGS entitlements *
Experience of misconduct in the program *
Interest in participating in the program *
Outcomes marked by a single asterisk (*) will be measured using survey data. Outcomes marked by a double asterisk (**) will be measured using administrative data.
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
To identify the causal impact of social audits on MGNREGS program performance, we collaborated with Bihar's Social Audits Society to implement an RCT that leveraged the state's ambitious audit expansion plan in FY 2019-2020.

1) First, the state set targets for the number of panchayats (the lowest level of administration in rural India, typically comprising 4-5 villages) it wanted to audit in each block.
2) Then, we randomly selected the panchayats to be audited in blocks where less than 100% of locations were targeted for an audit.
3) Thus, our randomization was stratified on the block, with different probabilities of treatment driven by the implementation agency’s audit target for each block.
4) The experimental sample includes 3,437 panchayats across 178 blocks. Of these, 2,178 panchayats were assigned to be audited (treatment), while the remaining 1,259 panchayats served as the control group.

The aim of our endline data collection is to assess whether administrative data-based reductions in work reflect reduced corruption or reductions in actual work. To understand whether workers may be discouraged from seeking work under the program due to audits, we will also incorporate a salience experiment into the endline. In this add-on survey experiment, we will inform a randomly selected subset of survey respondents in audit GPs of social audit findings. We will only conduct the audit salience experiment within our main RCT treatment GPs, randomizing at the individual level, and then collect data on interest in work and willingness to officially register work requests across all study locations.
Experimental Design Details
Not available
Randomization Method
Randomization was conducted in the office via a Stata program.
Randomization Unit
Panchayat level
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
Analysis from Administrative Data: 3,437 clusters (panchayats)
Analysis from Survey Data: 274 clusters (panchayats)
Sample size: planned number of observations
Analysis from Administrative Data: 3,437 panchayats across 178 blocks. 2,178 panchayats in the treatment group (873 actually audited) and 1,259 panchayats in the control group. Analysis from Survey data: 5,480 respondents from 274 panchayats from 19 blocks
Sample size (or number of clusters) by treatment arms
Analysis from Administrative Data: Of the 3,437 panchayats in the experimental sample, 2,178 panchayats were assigned to be audited (treatment), while the remaining 1,259 panchayats served as the control group.
Analysis from Survey Data: Of the 274 panchayats in the endline survey sample, half will be in the treatment group, that is, they were assigned to be audited in 2019-20. The remaining half will be drawn from the control group.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We will use two sources to measure outcomes: (1) administrative data on MGNREGS and (2) survey data from respondents. For the latter group, for every respondent we survey who did not work in MGNREGS in the past two years, we will select three who did. In each panchayat, we will randomly sample 20 workers who appear on MGNREGS work rolls from 2019-20 and 2020-21, as well as 10 citizens who appear on voter rolls. We anticipate that we will be able to locate and interview 75% of the MGNREGS workers and 50% of individuals on the voter rolls, leaving us with a survey sample of 4,110 MGNREGS workers and 1,370 registered voters in each panchayat. These respondents will be equally split across treatment and control groups. We followed convention in the social sciences for power calculations, using a significance level (probability of Type I error) of 0.05 and power (probability of avoiding a Type II error) of 0.8. To estimate statistical power, we utilized data collected from a process monitoring survey conducted for social audits in 2019-20 across 13 districts in Bihar to get an estimate for the inter-cluster correlation (ICC) for one of the primary outcomes of interest (dummy that the respondent worked in MGNREGS in the past fiscal year); we take the ICC to be 0.04. Given the sample size of 274 GPs, with a targeted sample size of 15 respondents who had worked for MGNREGS, we reach a standardized minimum detectable effect (SMDE) of 0.109 units to detect real work rates among individuals listed as having worked on MGNREGS payrolls. The standardized treatment effect on the person-days worked calculated from administrative data for this sample comes very close to this SMDE.
IRB

Institutional Review Boards (IRBs)

IRB Name
Yale University
IRB Approval Date
2020-09-14
IRB Approval Number
2000027920
IRB Name
Institute of Financial Management and Research (IFMR) — local IRB in India
IRB Approval Date
2020-09-30
IRB Approval Number
IRB00007107
Analysis Plan

Analysis Plan Documents

SA_IE_PAP.pdf

MD5: 884a101dca45c96243d358e43dd752a8

SHA1: 4ae163a44a8df2e3b699e9362ae2dfe6e7e5d491

Uploaded At: December 03, 2021