High frequency monitoring in India’s Public Distribution System

2023-10-09

2024-11-04

Our first primary outcome of interest is an index of the dashboard indicators listed below, weighted by recipient preferences. Weights for this index will be estimated based on data collected in midline or a later steady-state round.

Our intervention involves sharing a dashboard with the marketing inspectors (MIs, explained below in the intervention section). The indicators on this dashboard will also serve as primary outcomes of interest. Specifically, these include:
- Percentage of entitlement received according to phone surveys
- Percentage of beneficiaries receiving more than 80% of their entitlement according to phone surveys
- Percentage of entitlement received according to ePoS data
- Percentage of beneficiaries receiving more than 80% of their entitlement according to ePoS data
- Percentage of transactions that take one trip to receive all their ration for a given month
- Percentage of transactions with no authentication problems
- Overall beneficiary rating of the PDS experience
- Percentage of beneficiaries receiving more than the share of entitlements received by the 20th percentile of the control distribution of beneficiaries, according to phone surveys [this measure will not be in the dashboard to begin with, but we intend to use this as an alternative if there are substantial shifts in the control distribution over time]

At the outset, we are designing this study with an explicit focus to improve outcomes on the left tail of the distribution of the beneficiary population and FPSs. We list our approach to test these below:
- Quantile regression using the baseline and endline distributions of FPS performance, specified at quartiles
- Overall treatment effects on the following outcome:
(0.8E_i - Y_i) where Y_i is the survey data report of how much grain a beneficiary received, and E_i is their entitlement
- The average change in the FPS-level share of beneficiaries that receive >= 80% of their entitlement [this is a beta test for the second phase of the study, details below in the intervention section]

Separately, we are also interested in testing the effects on how MIs and FPS owners value their roles relative to outside options. We do not have measures for these yet, but will specify them prior to the midline or endline, which is when we intend to collect them.

We will test heterogenous treatment effects on the following dimensions:
- Baseline measures of pro-sociality among MIs. Note that heterogeneity on this dimension is not obvious. More pro-social MIs may value incentives more or less than less pro-social ones, and the differential marginal value of information to these groups is also not clear ex-ante. We will perform this test with the following variables:
-- Using a vignette-style survey elicitation in the baseline, we have MIs’ ranks of various job attributes (mission, intrinsic motivation, status, career motivation, and signaling). We use the first principal component of these measures, and split the observations on above and below the median of the first eigenvector. This binary will be used as a variable to test heterogeneous treatment effects on.
-- For comparisons with measures used in this literature, we also specify a binary variable for whether an MIs’ WTP for a “mission-oriented” job is above or (weakly) below INR 25,000. This WTP was also elicited through a vignette-style elicitation at baseline.
- Baseline accuracy of MIs’ beliefs about the implementation quality in their jurisdiction. We intend to use this heterogeneity to distinguish between the information and incentive channels of our intervention. Specifically, the intervention has both an information component (by telling MI’s how they are doing), and an incentive component (by letting their supervisors and senior officials know how they are performing). When MI’s have accurate beliefs on their jurisdiction, the main channel of intervention impact will be the incentive channel; when they do not, both channels will matter.
- Some additional measures that we we will also explore heterogeneous treatment effects on, but consider as substantially less important than the ones above:
-- Baseline number of FPSs under MI jurisdiction (above/below median)
-- Baseline share of GPs vs private FPSs under MI jurisdiction (above/below median)
-- Baseline share of rural vs urban FPSs under MI jurisdiction (above/below median)
-- MIs’ confidence in their FPS ratings (above/below median)

Regression specifications:
We will report ITT estimates, which compare average outcomes in treatment and control areas. Our primary outcomes are defined at the beneficiary level, which is the unit at which we will analyze them. Regressions will include fixed effects at the level of the randomization stratum and will be estimated using inverse sampling probabilities as weights. Standard errors will be clustered at the unit of randomization (MI).
We will analyze data monthly as well as pooled across the entire duration of the study, with month fixed effects in the latter specification:
Y_itfmsd = \alpha + \beta_t * treatment_itfmsd + \gamma*X_ifmsd + \epsilson_itfmsd
Y_itfmsd = \alpha + \beta * treatment_itfmsd + \gamma*X_ifmsd + \phi_t + \epsilson_itfmsd
Where i is the individual, f is the FPS, m is the MI, s is the stratum and d is the district. phi_t is a month fixed effect and X_ifmsd is a vector of baseline characteristics of the household that we observe in the administrative data (household size). Standard errors will be clustered at the MI level. We will also conduct randomization inference as a robustness check.
In addition, we will also report quantile regressions, with quartiles specified based on both baseline and endline performance indicators at the FPS level.
For the outcomes at the MI and FPS level regs:
Y_jtsd = \alpha + \beta * treatment_jtmsd + \gamma*X_ifmsd + \phi_t + \epsilson_jtsd
Y_jtsd = \alpha + \beta * treatment_jtsd + \gamma*X_jsd + \phi_t + \epsilson_jtsd
Where j is the MI or FPS owner, s is the stratum and d is the district. phi_t is a month fixed effect and X_ifmsd is a vector of baseline characteristics of the household that we observe in the administrative data (household size). Standard errors will be clustered at the MI level for the FPS level regressions. We will also conduct randomization inference as a robustness check.

- We will test the effect on MI effort, measured the following ways:
-- We will test MIs’ effort as reported by MIs themselves, through the number of FPSs visited in the past month, the number of surprise visits, the number of scheduled visits, self-reported hours reported on visiting FPSs and beneficiaries, and self-reported hours reported on grievance redressal. These measures will be collected at midline and endline.
-- We will also test MIs’ effort as reported by FPSs. Specifically, we will test their ability to identify the MI, the number of interactions with their MI, whether the MI made a surprise visit, the number and frequency of surprise visits, the number and type of problems the MI resolved. These measures will be collected at midline and endline.
-- Additionally, we will test MI effort in the bottom quartile of FPSs by performance.
- We will also test whether the intervention has an affect on a measure of the accuracy of MIs’ beliefs on the quality of PDS implementation in their blocks.
- Separately, we will also test the effect on their confidence in these ranks.

- This intervention is being implemented in the Public Distribution System (PDS) in Odisha, India. The PDS is an arterial public welfare program, delivering massively subsidized grains to poor households across the country through fair price shops (FPSs) that are typically operated by third-party non-state actors. This program has historically faced chronic implementation issues, including the pilfering and leakage of grains away from intended beneficiaries. In this context, we are designing a high frequency monitoring system wherein we do large-scale short surveys with a representative population of beneficiaries on a monthly basis. This information is delivered to last-mile government officials who are tasked with maintaining the implementation quality. These agents are the marketing inspectors (MIs). Each MI is responsible for approximately 25-40 FPSs in their jurisdiction.
- In the first phase of the study which is expected to last six months, half the MIs in the study area will be allocated to the treatment group and the other half to the control group
- MIs in the treatment group will receive a personalized dashboard with monthly statistics on the aggregate quality of service delivery in their jurisdiction, and these indicators will be used by their supervisors and state-level bureaucrats to incentivize their performance
- These incentives will be delivered through the inclusion of these statistics in their personnel files, as well as structured meetings with state-level superior officers where their performance is discussed
- In the second phase of the study, we will cross-randomize an additional intervention, i.e. half the MIs in both the treatment and control groups from the first phase will be allocated to treatment and control conditions in the second stage
- The intervention in the second phase will entail sharing dashboards with MIs depicting data at a more granular level than the first phase; in the second phase, report cards will detail FPS-level performance for all the FPSs in an MIs’ jurisdiction
- The two phases of the study are designed to delineate the effect of more detailed and granular information relative to the effect of incentives, which are held constant across the duration of the experiment
[Note that we will specify additional tests and outcomes before beginning the second phase of the study.]

Not available

- Conducted using a computer algorithm
- The randomization protocol is detailed as follows:
-- We stratify by district; where the strata have fewer than 8 observations, we group smaller strata together to meet this threshold
-- Within strata, we rank MIs based on the proportion of entitlement received by beneficiaries according to survey data, which is a baseline measure of their performance
-- We assign treatment status in groups of four, with treatment status for both phases allocated randomly within these groups; the four treatment statuses are CC, CT, TC, TT, denoting the control and treatment status over the two phases
-- Singleton observations are pooled across multiple strata and allocated treatment in groups of four as far as possible
--- 3 or fewer residual singletons are then allocated to one of the treatment groups each with equal probability, if needed

The unit of randomization is the Marketing Inspector (MI)

No

150

Approximately 65,000 observations per month for 16 months, totaling approximately 1,040,000

First stage: approximately 33,000 per arm per month
Second stage: approximately 16,500 per arm per month