Impact Evaluation of the Pilot Forecast-based Financing in Nepal

Last registered on November 02, 2022

Pre-Trial

Trial Information

General Information

Title
Impact Evaluation of the Pilot Forecast-based Financing in Nepal
RCT ID
AEARCTR-0009903
Initial registration date
November 02, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
November 02, 2022, 5:35 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Primary Investigator

Affiliation
World Food Programme

Other Primary Investigator(s)

PI Affiliation
UN World Food Programme
PI Affiliation
The World Bank (DIME)

Additional Trial Information

Status
On going
Start date
2022-10-07
End date
2023-12-31
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Forecast-based Financing (FbF) enables anticipatory actions for disaster mitigation at the community and government level using credible seasonal and weather forecasts. These forecasts are linked to predetermined contingency plans, actors, and funding instruments used to reduce the humanitarian caseload in the critical window between a forecast and an extreme weather event. The hypothesis is that an anticipatory approach can lead to a more effective and efficient response and help affected populations avoid negative coping strategies such as skipping meals, reducing portions, and eating lower-quality food (WFP, 2019). This study will employ a clustered randomized controlled trial (RCT) design to assess the impact of FbF in Nepal's two most vulnerable basins prone to floods (Karnali and Koshi, or a single basin depending on trigger activation). The key objective of this evaluation is to measure the causal effects of providing anticipatory cash assistance to flood-prone households (based on projections prior to an expected flood peak) on food security, coping, and livelihood outcomes, compared to a more "traditional" humanitarian response, which comes in the form of providing assistance after a flood peak has been confirmed.
External Link(s)

Registration Citation

Citation
Christian, Paul J , Felipe Alexander Dunsch and Jonas Heirman. 2022. "Impact Evaluation of the Pilot Forecast-based Financing in Nepal." AEA RCT Registry. November 02. https://doi.org/10.1257/rct.9903-1.0
Experimental Details

Interventions

Intervention(s)
There is momentum within the international humanitarian sector to move from a more responsive approach to anticipatory actions. This means planning for disasters by putting response plans and funding in place before the worst impacts materialize.

One such approach is Forecast-based Financing (FbF) or anticipatory action (AA) for severe floods and droughts. FbF enables anticipatory actions for disaster mitigation at the community and government level using credible seasonal and weather forecasts. These forecasts are linked to predetermined contingency plans, actors, and funding instruments used to reduce the humanitarian caseload in the critical window between a forecast and an extreme weather event. The hypothesis is that an anticipatory approach can lead to a more effective and efficient response and can help affected populations to avoid negative coping strategies such as skipping meals, reducing portion size, and eating lower quality food (WFP, 2019). The World Food Programme (WFP) has started to implement innovative programmes to reduce losses and damages in people’s livelihoods who face increasing climate extremes to support countries in the mitigation and management of climate risks.

WFP Nepal has been implementing the Forecast Based Financing (FbF) since 2015 which has enabled the identification of key risks, challenges, and corresponding mitigation/ prevention measures to ensure timely implementation of the proposed anticipatory intervention. The impact evaluation in Nepal focuses on anticipatory action encompassing a range of activities, for example, the preparedness actions at the national and regional level, the distribution of warning messages, as well as direct food assistance to vulnerable households. To specifically investigate the impacts of the important feature of providing cash assistance at 3 different points in time, the impact evaluation will employ a randomized controlled trial (RCT) methodology in the Karnali and Koshi basins.

If both basins activate, villages are randomly allocated to 3 treatment arms, with households on the pre-defined beneficiary list receiving NRS 15,000 (117.9 USD) unconditionally.

Within the group of villages forecast to be flood affected and therefore eligible for transfers, the impact evaluation will randomize the timing and the conditions under which a smaller subset of households receive transfers.

If only one basin activates the FbF programme, the IE will be reduced to 2 treatment arms (removing treatment 2). As of 31 October 2022, there is an activation only in the Karnali basin (West Nepal).

Treatment 1: For households in the “forecast based financing” group, a single transfer of NRS 15,000 ($117.9) will be initiated ca. 1-3 days before the projected flood peak.

Treatment 2: The “expedited post-flood transfer” group will not receive the forecast-based transfer but will receive a transfer of NRS 15,000 ($117.9 dollars) within two weeks following the actual flood peak if their geographical location was indeed flooded.

Treatment 3: The “traditional response transfer” group will not receive the forecast-based transfer but will receive a transfer of at least NRS 15,000 ($117.9) within one -two months of the actual flood peak, the normal timeline in which WFP is normally able to responds to disasters (given the time it requires to get resources, mobilize partners and register beneficiaries).

Given the fact that there are many more beneficiaries than what will be covered by the study, and the fact that it is not clear where the action will be triggered, the trial does not use a baseline. There will be multiple rounds of follow up data collections.

Acknowledging that some years monsoon flooding in Nepal is intense and surpasses the ability of communities to cope, creates widespread humanitarian needs with longer term development consequences. Further, it is often possible to predict severe monsoon flooding events and a targeted anticipatory approach can lead to a more timely, effective, efficient solution to respond to and ultimately reduce humanitarian needs. The goal of the anticipatory action response is a more effective, timely and humanitarian response for beneficiaries in anticipation of severe monsoon flooding in highly vulnerable districts of Nepal. In line with best practice, the pilot combines the pre-agreed components of- forecast and triggers; anticipatory actions, and finance (cash transfers to the beneficiaries). Given the sparse evidence in this domain, the objective of this impact evaluation is to assess the impact of anticipatory action on household resilience and generate the necessary evidence to inform humanitarian response.
Intervention Start Date
2022-10-08
Intervention End Date
2023-03-31

Primary Outcomes

Primary Outcomes (end points)
The main objective of the impact evaluation study is to answer the following research question:
“What is the impact of providing anticipatory humanitarian assistance in the form of an unconditional cash transfer ahead of a severe flooding event (based on pre-defined forecast triggers) on food security, coping, and livelihood outcomes, as compared to a more "traditional" humanitarian response of providing post-flood assistance?”

The first round of data collection is a high frequency round, with a shorter survey duration focusing only on main outcomes of interest to be collected prior to the 2nd group receiving the post-shock transfers. The key outcomes for the first round will be grouped in the following modules:
- Food Security (FCS)
- Business, agriculture, and wage incomes and profits
- Coping Strategies (reduced version of the lCSI and rCSI)
- Psychological well-being (Cantril’s Ladder & PHQ4)
- Verification: Early warning message received, questions on CERF cash transfers received by HHs

The second and third round of data collection will be a longer survey, and will focus on all primary outcomes, after the 2nd group received transfers:
- Food Security (FCS, FIES)
- Business, agriculture, and wage incomes and profits
- Coping Strategies incl. the lCSI, rCSI, and questions on loan behavior
- Financial Outcomes (Consumption, and expenditure on food and non-food items)
- Shocks
- Flood preparedness
- Loss and Damage module (Loss or damage to assets e.g. livestock, crops, equipment, etc.)
- EWS (Early warning messaging), and CERF specific questions (including questions on spillovers)
- Psychological well-being (Cantril’s Ladder & PHQ4)
Primary Outcomes (explanation)
All outcomes will be measured at the community, household and/or individual levels, through the construction of indexes, based on the mentioned modules:

Business, agriculture, and wage incomes and profits will be asked directly to the household head

Food Security will be measured at the household level using the Food Consumption Score (FCS) and the Food Insecurity Experience Scale (FIES). The FCS is a score calculated using the frequency of consumption of different food groups consumed by a household during the 7 days before the survey. The FIES is experience-based measures of household or individual food security.

Coping Strategies will be measured at the household level asking questions related to shocks experienced by the household and livelihood coping strategies used; and implementing a reduced Coping Strategy Index (rCSI) scale and the lCSI (reduced in the first round of surveys). More specifically, this module will assess the strategies used by the household to cope with- livelihood loss, food shortages, psychological distress, and financial distress.

Financial Outcomes will be measured at the household level asking questions related to loans/borrowing and Savings; and implementing the Economic Capacity to Meet Essential Needs (ECMEN) scale.

Psychological well-being- this will be measured using Cantril’s ladder and the PHQ4 tool.

Asset Loss and Damage will be measured at the household level asking questions related to the household’s ability to protect loss and damage in the face of severe flood. The module will assess the assets owned by the household, and whether any loss or damage occurred during or after the floods.

The shocks module includes questions around the flood impact (severity of damage), other and other experienced (non-flood related) shocks.

Flood Preparedness will be measured at the household and community levels asking questions related to the household and community’s preparedness and the ability to reduce the impact of the flood. For instance, the module will ask questions about preparedness measures taken by the household before, during, or after the floods (for example, storing food, elevating the house).


Secondary Outcomes

Secondary Outcomes (end points)
In addition to measuring outcomes on households, a key interest is assessing the precision of the activation trigger. Given that the triggers are based on forecasts, and therefore by definition imprecise, it will be very important to be able to quantify the accuracy of current triggers to inform future policymaking in the space of forecast-based financing. Through the re-verification process in this impact evaluation, we aim to re-check whether the areas that were forecasted to be flooded, are actually flooded or not. If not, then the cash transfers can be re-directed to the most flood affected areas.
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
To specifically investigate the impacts of the important feature of providing cash assistance at 3 different points in time, the impact evaluation will employ a randomized controlled trial (RCT) methodology.

If both basins activate, households are randomly allocated to 3 treatment arms, receiving NRS 15,000 (117.9 USD). If only one basin activates, one treatment arm will be dropped due to statistical power concerns (Treatment 2).

Within the group of villages forecast to be flood affected and therefore eligible for transfers, the impact evaluation will randomize the timing and the conditions under which a smaller subset of households receive transfers.

Treatment 1: For households in the “forecast based financing” group, a single transfer of NRS 15,000 ($117.9) will be initiated ca. 1-3 days before the projected flood peak.

Treatment 2: The “expedited post-flood transfer” group will not receive the forecast-based transfer but will receive a transfer of NRS 15,000 ($117.9 dollars) within two weeks following the actual flood peak if their geographical location was indeed flooded.

Treatment 3: The “traditional response transfer” group will not receive the forecast-based transfer but will receive a transfer of at least NRS 15,000 ($117.9) within one -two months of the actual flood peak, the normal timeline in which WFP is normally able to responds to disasters (given the time it requires to get resources, mobilize partners and register beneficiaries).

There is no dedicated baseline data collection and data will be collected through in-person surveys during at least 3 waves after the projected flood peak. Outcome data collection starts on 3 November 2022 (first round of data collection).
Experimental Design Details
Randomization Method
Randomization will be done in office by a computer, using STATA
Randomization Unit
To identify the causal impact of providing anticipatory action, the evaluation will employ a randomized controlled trial (RCT) design with the unit of observation being the household, but the randomization will be at the settlement/village level blocked by district.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
140 villages (70 and 70 per arm) in the Karnali basin only (2 arm trial). In case the Eastern basin activates too, there will be more villages (Information as of 2 November 2022).
Sample size: planned number of observations
We have 2321 HHs in the survey sample, 260 Replacement HHs, and 317 HHs that are not a part of the survey sample. This is only in the Karnali basin.
Sample size (or number of clusters) by treatment arms
1499 households receiving anticipatory action/FbF support, 1484 households receiving post-shock support. Of those, 2569 are sampled for inclusion in the impact evaluation study. Of these we have 2321 HHs in the survey sample, 260 Replacement HHs, and 317 HHs that are not a part of the survey sample.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
The basis for power calculations are Nepal FANSEP impact evaluation baseline and midline data, collected in 2021 and 2022. The results table from an evaluation of anticipatory transfers in Bangladesh (Pople et al , 2021) were used to inform the choice of minimum detectable effects (MDE) and outcomes. Based on the overlap between outcomes reported in Pople et al and the FANSEP impact evaluation data available to the PI team, we chose to report power for the three following indicators:   FIES score   Binary for whether children in household consumed 3 meals per day   Binary for whether woman in household consumed protein in the last day   Intra-cluster correlation and autocorrelation for above outcomes was calculated in the Nepal data. The sampsi command in STATA was used to conduct the power calculations analysis, the following table represents the relationship between minimum detectable effect (MDE) and statistical power. The resulting power calculations suggest that our power is between 65% and 96% to detect impacts of 0.1 standard treatment effects and power is between 80% and 100% to detect impacts of 0.15 standard treatment effects based on the size of effects of forecast based transfers reported for analogous outcomes in Pople et al. For comparison, Pople et al, report effects of .1 standard deviations or more for costly borrowing index and live satisfaction index, and report an effect of .09 standard deviations for days that protein was consumed in the previous 7 days. We expect that scenarios 4, 5 and 6 below are most plausible, since the intracluster correlation is calculated as the average of 3 outcomes of interest.    SAMPLING PROCEDURE    The sampling procedure will rely on beneficiary selection data collected by WFP Nepal country office in July 2022. They will identify households that are eligible for transfers based on their internal scoring cut-off. The research team will randomly select the required number of villages from villages ranked as most vulnerable in WFP’s flood vulnerability ranking. Each village must have on average 30 transfer eligible households, with the minimum village size being 10 transfer eligible households and maximum being 60. Within each sampled village transfer eligible households will be randomly selected proportional to the village size. Transfer ineligible households will be excluded from the sample. On average 20 households will be sampled per village with an additional 10 provided as replacements.
Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
Government of Nepal, NHRC (Nepal Health Research Council)
IRB Approval Date
2022-02-04
IRB Approval Number
Ref No. 1963

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials