Experimental Design Details
(1) Impact evaluation:
To assess the welfare impact of the reform from in-kind to electronic delivery of food assistance benefits, we are working with the GoI to conduct a randomized evaluation to compare the delivery of social assistance using electronic methods (i.e. BPNT) to the status quo of in-kind rice assistance (i.e. Raskin). This randomized evaluation is only possible due to the government’s plan to roll out BPNT in phases, transitioning only some districts in 2018. To facilitate this evaluation, the Indonesian government asked the research team for assistance to randomly select districts to be treated in the 2018 expansion of BPNT from a list of districts that were potentially ready for the reform, under the condition that the total number of beneficiaries in the selected districts amounted to around 8.3 million (roughly equivalent to the budgetary allocation for 2018 for the reform process for this experimental sample).
The experimental sample, thus, involves a total of 105 districts, with 42 districts receiving BPNT serving as our treatment group and 63 receiving districts receiving Raskin serving as our control group. Due to logistical constraints, the GoI additionally decided to implement this expansion in four phases. We additionally randomized which districts were assigned to which phase within our experimental sample.
Note that in addition to our experimental districts, there are two other types of non-experimental reform districts. First, there are the original 44 cities that were part of the GoI’s 2017 pilot. Second, the GoI also selected certain districts to receive the reforms in 2018 regardless of the experimental design of this study. While these districts are not part of our experimental sample, we will collect data on them when possible and will descriptively understand the reform process in these districts.
(2) Agent ratio experiment:
In this study, we will test whether changes to the agent to beneficiary ratio change the effectiveness of the BPNT program. For example, we will assess whether increasing agent coverage in a district makes it easier for households to access the BPNT benefit, lowers the price of the rice beneficiaries receive, improves rice quality, etc.
As with the first experimental component, our unit of analysis is the district. Working with the Government of Indonesia, we identified 216 districts where the BPNT reforms have been carried out. These 216 districts include both districts in our experimental sample for Experiment 1 and districts the non-experimental sample (i.e. the 44 pilot cities from 2017, districts designated to reform in 2018 regardless of experimental design, etc.). We then worked with the GoI to randomly assign the districts into two experimental groups of equal size (Group A and Group B). In Group A, the Government of Indonesia is requiring banks to fulfill a stricter set of agent coverage requirements that are specified by the government. These Group A requirements are: 1) an agent to beneficiary ratio of at least 1: 250 and 2) a minimum of 2 agents per village. In the remaining districts (Group B), we loosened the agent coverage criteria. The banks in Group B are required to fulfill the first criterion only, which is the agent to beneficiary ratio of at least 1: 250.
By varying the requirements that the banks need to fulfill in different districts, we will be able to (1) answer the policy question posed by the government of whether the additional criterion of a minimum 2 agents per village is necessary and (2) shed light on the optimal ratio that should be considered by the government as a future standard to ensure policy success. The banks have agreed to participate in the study, and we are closely coordinating with them on the study implementation process, particularly on the sharing of updated administrative data on agents.
To introduce the agent coverage criteria for this experiment, our GoI partners convened initial discussions with the banks. Through close coordination with the GoI, we have also sent enforcement letters highlighting the agent coverage requirements to the 216 districts in the agent ratio experiment. Furthermore, we have conducted a round of phone surveys with local government representatives and local bank officials in both experimental areas (278 districts across Experiments 1 and 2). These phone surveys serve the dual purpose of reinforcing the agent coverage requirements in Experiment 2 (through survey scripts that reference the applicable criteria) and providing valuable feedback on the implementation of both experiments, as well as the on-the-ground challenges associated with this historic reform.