Experimental Design
Overview:
The study uses a randomized controlled trial to measure the impact of the water demand management intervention. The rollout of the intervention has been randomized across a total study window of 132 villages in the Indian states of Andhra Pradesh and Rajasthan; 50 were randomly chosen as control; the rest were treated. Within Rajasthan the treatment status was stratified on whether the presidency of the Panchayat (local council) was reserved for a woman. Since reservation at this level is randomized within Rajasthan (Chattopadhyay and Duflo, 2004), a stratified design has more power to measure heterogeneous effects based on the gender of the Panchayat president.
Context:
Since mid-2014, villages in the treatment group have been receiving the water management intervention while the control villages serve as a valid counterfactual for estimating the effect of the intervention. Villages in five districts in the semi-arid regions of Rajasthan and Andrah Pradesh received the intervention. The Rajasthani districts of Bhilwara, Pratapgarh, and Udaipur are separated by many miles from Anantapur and Chittoor in Andhra Pradesh. Nevertheless, all five are similar in being historically drier than India's other districts, and for having especially suffered from the effects of climate change.
The field partner – FES – is an Indian NGO working to strengthen rural institutions and natural resource governance for improved livelihood outcomes. Since 2013, FES is implementing a water demand management intervention that is defined by three core components with a focus on institutional strengthening. The first step is to create committees in each village with the power to make and enforce rules about the use of water. The committees can set rules about which crops can be grown when rains are bad—for example, rice requires far more water than lentils—and how much water the farmers can draw from common pools. The second step is to give farmers the means to comply with these rules by showing them how they can use less water. FES will train farmers to grow new crops and to plant varieties of traditional crops that require less water. Finally, FES will train the community to leverage existing government programs to improve their access to water. The village committees will be trained to build irrigation infrastructure using labor paid for by the National Rural Employment Guarantee (NREGA) Scheme, a federal make-work program.
Design:
Baseline household data was collected in 2014/2015 from roughly 1700 respondents prior to the treatment implementation. Within each village roughly 11 households were interviewed in a stratified random sample. Stratification was used to ensure a sufficient household sample size from each of four groups: male-headed households with little or no land (less than 1 hectare), male-headed households with small plots (1 to 2 hectares), male-headed households with large plots (more than 2 hectares), and female-headed households. A village-level survey is scheduled for March 2016 to explore the mechanisms linking the program to the outcomes of interest.
The next step, for which funding is being sought from IGC, is to collect a midline panel household survey in the months of October through December of 2016. Funding for this midline survey is our primary request. Since the baseline survey, all of the villages have received at least one year of treatment. The same questionnaire will be administered to the original respondents from the baseline household survey. To better reveal the mechanisms through which the intervention works, funding is also being sought to collect another round of village level surveys with community leaders from March through May of 2017 (scheduled to be one year after the 2016 village level surveys). Eventually the study hopes to collect a third round of household and village level data to test longer term program effects, but funding for the final round of data is beyond the scope of this application.
Finally, in addition to the surveys, FES will also collect qualitative data to give a more complete picture of the intervention. Within each treated village, FES will collect and translate the terms of references that establish the village committees. FES will collect the bylaws they enact and the minutes of the meetings they hold. It will also collect the record of all decisions made by the village as a whole, as well as the crop water budgets the village creates.
Data Collection:
The impact of the intervention will be measured using two primary survey instruments – a household and village survey. The household instrument asks households about their income, land, crops grown, water used, participation in government, perceptions of climate change, and whether they were aware of rules governing the use of water.
The village survey will be collected by interviewing several key informants in each village—officials and local volunteers—as well as by holding focus groups. The village survey will record the water and forest resources of the village, the precise rules governing the use of each, details about local politicians and officials, and the average participation of households in village government.
Responses to the surveys will be recorded electronically on tablet computers, which have been programmed using Open Data Kit (ODK). From June through August the investigators and activity manager, together with FES, will program Hindi and Telagu versions of the survey. In early September an experienced field manager from Cloudburst will travel to Rajasthan and Andhra Pradesh to help FES project managers hire and train survey enumerators. From October to December the enumerators will find the original households surveyed at baseline and interview them for the household survey. Every day a senior research assistant will run simple statistical checks on the quality of the data and guide real-time corrections in the field. These tests are possible because ODK allows the data to be uploaded to a central server as it is collected. These teams will return to the field in March of 2017 to collect the village survey. The research assistants will then prepare the data for analysis.
Analysis:
Since the intervention is randomized, comparing the treatment and control groups will give an unbiased estimate of its effect. The main outcomes of interest will come from the household survey. These will include total income, crops planted, water used, and awareness of rules for water use. The investigators will test for average effects as well as differential effects on female-headed households, landless households, and large landholders. The investigators will run multivariate regressions of the outcomes of interest on an indicator for whether the village was assigned to treatment. The study will maximize power by controlling for baseline values of the outcome and indicators for belonging to each of the clusters on which we stratified the survey. Power calculations run before the intervention suggested controlling for the baseline value yielded more power than household fixed-effects. The standard errors will allow for arbitrary correlation in the error terms within a village.
This regression will yield an "intent-to-treat" estimate, which captures the average effect of planning to run the intervention in a cluster. This effect will differ from the average effect of actually receiving the intervention if, for example, FES is unable to finish the intervention in all treatment clusters. Average effect can be approximated by using instrumental variables, instrumenting for the presence of FES with the indicator for whether the cluster was assigned to treatment. The evaluation will also test whether the potential benefits of the intervention are shared evenly across households with different amounts of land and household heads that are male versus female. The household survey was stratified on these characteristics to maximize the power of these tests (see above).
Since the intervention has three components, the village survey will be used to disentangle the effects of each. The village survey asks detailed questions about the types of rules that govern common resources and the year in which the rules came into effect. It also asks about which projects were planned and built through the National Rural Employment Guarantee Scheme. These data will enable a test of whether treated villages are actually passing rules and building infrastructure. Together with data on rainfall, the study can also test whether the new rules and infrastructure are mitigating the effects of low rainfall. The texts of the bylaws will also allow the analysis to gauge what types of rules are passed and may suggest which types of rules are most effective.
Finally, the evaluation can explore whether the gender of the village council president changes the effectiveness of the intervention. India reserves a fraction of Panchayat presidencies for women. Within Rajasthan the seats reserved are randomized. As noted above, the design stratifies the intervention to exploit this random variation. Prior literature suggests that female leaders have different priorities than men, and simply changing the gender of the Panchayat president can change the policies of the village council (Chatopadhyay and Duflo, 2004). The study will test whether a female leader makes the intervention more or less effective.