Impact Evaluation of the Pilot Forecast-based Financing in Nepal

There is momentum within the international humanitarian sector to move from a more responsive approach to anticipatory actions. This means planning for disasters by putting response plans and funding in place before the worst impacts materialize.

One such approach is Forecast-based Financing (FbF) or anticipatory action (AA) for severe floods and droughts. FbF enables anticipatory actions for disaster mitigation at the community and government level using credible seasonal and weather forecasts. These forecasts are linked to predetermined contingency plans, actors, and funding instruments used to reduce the humanitarian caseload in the critical window between a forecast and an extreme weather event. The hypothesis is that an anticipatory approach can lead to a more effective and efficient response and can help affected populations to avoid negative coping strategies such as skipping meals, reducing portion size, and eating lower quality food (WFP, 2019). The World Food Programme (WFP) has started to implement innovative programmes to reduce losses and damages in people’s livelihoods who face increasing climate extremes to support countries in the mitigation and management of climate risks.

WFP Nepal has been implementing the Forecast Based Financing (FbF) since 2015 which has enabled the identification of key risks, challenges, and corresponding mitigation/ prevention measures to ensure timely implementation of the proposed anticipatory intervention. The impact evaluation in Nepal focuses on anticipatory action encompassing a range of activities, for example, the preparedness actions at the national and regional level, the distribution of warning messages, as well as direct food assistance to vulnerable households. To specifically investigate the impacts of the important feature of providing cash assistance at 3 different points in time, the impact evaluation will employ a randomized controlled trial (RCT) methodology in the Karnali and Koshi basins.

If both basins activate, villages are randomly allocated to 3 treatment arms, with households on the pre-defined beneficiary list receiving NRS 15,000 (117.9 USD) unconditionally.

Within the group of villages forecast to be flood affected and therefore eligible for transfers, the impact evaluation will randomize the timing and the conditions under which a smaller subset of households receive transfers.

If only one basin activates the FbF programme, the IE will be reduced to 2 treatment arms (removing treatment 2). As of 31 October 2022, there is an activation only in the Karnali basin (West Nepal).

Treatment 1: For households in the “forecast based financing” group, a single transfer of NRS 15,000 ($117.9) will be initiated ca. 1-3 days before the projected flood peak.

Treatment 2: The “expedited post-flood transfer” group will not receive the forecast-based transfer but will receive a transfer of NRS 15,000 ($117.9 dollars) within two weeks following the actual flood peak if their geographical location was indeed flooded.

Treatment 3: The “traditional response transfer” group will not receive the forecast-based transfer but will receive a transfer of at least NRS 15,000 ($117.9) within one -two months of the actual flood peak, the normal timeline in which WFP is normally able to responds to disasters (given the time it requires to get resources, mobilize partners and register beneficiaries).

Given the fact that there are many more beneficiaries than what will be covered by the study, and the fact that it is not clear where the action will be triggered, the trial does not use a baseline. There will be multiple rounds of follow up data collections.

Acknowledging that some years monsoon flooding in Nepal is intense and surpasses the ability of communities to cope, creates widespread humanitarian needs with longer term development consequences. Further, it is often possible to predict severe monsoon flooding events and a targeted anticipatory approach can lead to a more timely, effective, efficient solution to respond to and ultimately reduce humanitarian needs. The goal of the anticipatory action response is a more effective, timely and humanitarian response for beneficiaries in anticipation of severe monsoon flooding in highly vulnerable districts of Nepal. In line with best practice, the pilot combines the pre-agreed components of- forecast and triggers; anticipatory actions, and finance (cash transfers to the beneficiaries). Given the sparse evidence in this domain, the objective of this impact evaluation is to assess the impact of anticipatory action on household resilience and generate the necessary evidence to inform humanitarian response.

The pilot in 2022 is a scale-up of anticipatory humanitarian action in response to possible severe monsoon flooding across Kankai basin (Jhapa), Koshi basin (Sunsari and Saptari districts) and Karnali, and West Rapti basin (Kailali, Bardiya and Banke districts).

Project Targeting
The pilot projects rely on a multi-stage targeting approach:
• Regional: Regional targeting is based on an overlay of flood risk (using Sentinel-1 satellite imagery for 2017, 2019, and 2020) and socio-economic vulnerability measures include poverty, food poverty, housing condition, marginalized groups, stunting, and wasting. Additionally, consultation with humanitarian organizations with extensive field-level experience on historical floods and flood response, engagement with local governments, and seasonal monsoon forecast are taken into consideration for the final selection of municipalities. for 2017, 2019, and 2020) and socio-economic vulnerability measures include poverty, food poverty, housing condition, marginalized groups, stunting, and wasting. Additionally, consultation with humanitarian organizations with extensive field-level experience on historical floods and flood response, engagement with local governments, and seasonal monsoon forecast are taken into consideration for the final selection of municipalities.

• Municipal: The list of municipalities was refined by focusing on those areas that could be covered under either of the Global Flood Awareness System (GloFAS)-based trigger mechanisms (taking into account feasibility). The priority list includes 9 municipalities covered by the trigger mechanism for the East and 6 municipalities covered by the trigger mechanism for the West.

• Household-level: In coordination with local government at municipal level, households-at-risk are identified based on combination of hazard maps based on historical flood events, exposure to floods, socio-economic vulnerability of households. Vulnerability factors include:
o the building construction type
o the household’s ability to survive on their own land production for up to 6 months, households
o the household’s dependence on agriculture and/or casual labour
o the household’s education level
o the household’s social protection status

The existing database of historical flood response will be explored at implementing Palikas. This further involves field level verification and further registration of beneficiaries in consultation with Local Palikas household targeting criteria. Endorsement of cash guidelines followed by household level orientation towards general preparedness and anticipatory actions will be continued in safeguarding lives and livelihoods of communities. Similarly, linkages and orientation among National/District/ Local Emergency Operation Centers, Palikas, remittance agents, cooperating partners and financial service providers will be setup for efficient and effective cash distribution with follow up of COVID-19 safety guidelines.

Beneficiary eligibility
For the 2022 monsoon flood season, WFP will target 26,000 families in 5 districts across two river basins who are socioeconomically poor and vulnerable to flood impacts. With the Central Emergency Response Fund (CERF) support, households-at-risk will be reached through WFP programming, beneficiaries will receive a one-off unconditional cash transfer of NRS 15,000 via remittance agents before or after the flooding event.

Triggers & Thresholds
Several global and national flood forecasts are available for Nepal. For this project, two forecasts were selected based on thorough study by the United Nations Office for the Coordination of Humanitarian Affairs (UN OCHA) and the UN Resident Coordinator Office (RCO), and previous experience triggering anticipatory action for monsoon floods by WFP.
A trigger should be based on a set of criteria to help answer the questions on when and where to act before the disaster. It should give an indication when a hazard becomes an out-of-the-ordinary (or severe) shock and impact crosses a certain threshold for the exposed vulnerable community.
The development of a two-step trigger system for the Nepal Anticipatory Action (AA) pilot is distributed across two major river basins in Nepal, as multiple river basins exist and they lack strong correlation due to complex topography. This pilot therefore relies on separate systems: one for the Karnali, Babai, and West Rapti Basins (in the West of Nepal) and one for the Koshi Basin and Saptakoshi Watershed (in the East of Nepal).
The Nepal pilot will rely on a two-step trigger system for each basin:
● Stage I: A readiness trigger based on the Global Flood Awareness System (GloFAS) forecast, that will provide approx. 7 days’ lead time.
● Stage II: An action trigger based on government systems and the Global Flood Awareness System (GloFAS) forecast, that will provide anywhere from 3 hours to 3 days’ lead.

Readiness Trigger Action Trigger
GloFAS 7-day forecast predicts 50% probability of reaching 1-in-2 year return period Flood warning bulletin issued by the Government of Nepal’s Department of Hydrology and Meteorology (DHM)
+
GloFAS 3-day forecast predicts 50% probability of reaching 1-in-2 year return period OR Water level reaches government-defined danger level
(3-hour warning only)

The action trigger relies on 2 components – DHM flood bulleting on the one hand and GloFAS 3-day forecast or government-based observational data on the other – in order to minimize the risk of false alarms.

Messaging
In addition, WFP will disseminate last mile early warnings: The dissemination of the last-mile early warning messages will continue at the important public places in the community. Different mediums i.e., the loudspeaker, megaphone by volunteers, community radio, etc. will be utilized to disseminate the warning messages as applicable. Flood vulnerable people will receive flood warning messages in advance of the flood peak, which will alert them to take necessary measures to protect themselves from flood risks.
The main objective of the Impact Evaluation (IE) is to answer the following evaluation question:
What is the impact of providing anticipatory humanitarian assistance in the form of an unconditional cash transfer ahead of a severe flooding event (based on pre-defined forecast triggers) on food security, coping, and livelihood outcomes?
Examining the effect of timing of Forecast-Based Financing (FbF) transfers on key well-being indicators will support programme teams to time FbF interventions in a way that is most beneficial to the affected populations. The targeting of anticipatory action (“at risk”) is therefore distinct and different from targeting for humanitarian response which is determined by existing need (“in need”; OCHA, 2021).

2022-10-08

2023-03-31

The main objective of the impact evaluation study is to answer the following research question:
“What is the impact of providing anticipatory humanitarian assistance in the form of an unconditional cash transfer ahead of a severe flooding event (based on pre-defined forecast triggers) on food security, coping, and livelihood outcomes, as compared to a more "traditional" humanitarian response of providing post-flood assistance?”

The first round of data collection is a high frequency round, with a shorter survey duration focusing only on main outcomes of interest to be collected prior to the 2nd group receiving the post-shock transfers. The key outcomes for the first round will be grouped in the following modules:
- Food Security (FCS)
- Business, agriculture, and wage incomes and profits
- Coping Strategies (reduced version of the lCSI and rCSI)
- Psychological well-being (Cantril’s Ladder & PHQ4)
- Verification: Early warning message received, questions on CERF cash transfers received by HHs

The second and third round of data collection will be a longer survey, and will focus on all primary outcomes, after the 2nd group received transfers:
- Food Security (FCS, FIES)
- Business, agriculture, and wage incomes and profits
- Coping Strategies incl. the lCSI, rCSI, and questions on loan behavior
- Financial Outcomes (Consumption, and expenditure on food and non-food items)
- Shocks
- Flood preparedness
- Loss and Damage module (Loss or damage to assets e.g. livestock, crops, equipment, etc.)
- EWS (Early warning messaging), and CERF specific questions (including questions on spillovers)
- Psychological well-being (Cantril’s Ladder & PHQ4)

All outcomes will be measured at the community, household and/or individual levels, through the construction of indexes, based on the mentioned modules:

Business, agriculture, and wage incomes and profits will be asked directly to the household head

Food Security will be measured at the household level using the Food Consumption Score (FCS) and the Food Insecurity Experience Scale (FIES). The FCS is a score calculated using the frequency of consumption of different food groups consumed by a household during the 7 days before the survey. The FIES is experience-based measures of household or individual food security.

Coping Strategies will be measured at the household level asking questions related to shocks experienced by the household and livelihood coping strategies used; and implementing a reduced Coping Strategy Index (rCSI) scale and the lCSI (reduced in the first round of surveys). More specifically, this module will assess the strategies used by the household to cope with- livelihood loss, food shortages, psychological distress, and financial distress.

Financial Outcomes will be measured at the household level asking questions related to loans/borrowing and Savings; and implementing the Economic Capacity to Meet Essential Needs (ECMEN) scale.

Psychological well-being- this will be measured using Cantril’s ladder and the PHQ4 tool.

Asset Loss and Damage will be measured at the household level asking questions related to the household’s ability to protect loss and damage in the face of severe flood. The module will assess the assets owned by the household, and whether any loss or damage occurred during or after the floods.

The shocks module includes questions around the flood impact (severity of damage), other and other experienced (non-flood related) shocks.

Flood Preparedness will be measured at the household and community levels asking questions related to the household and community’s preparedness and the ability to reduce the impact of the flood. For instance, the module will ask questions about preparedness measures taken by the household before, during, or after the floods (for example, storing food, elevating the house).

In addition to measuring outcomes on households, a key interest is assessing the precision of the activation trigger. Given that the triggers are based on forecasts, and therefore by definition imprecise, it will be very important to be able to quantify the accuracy of current triggers to inform future policymaking in the space of forecast-based financing. Through the re-verification process in this impact evaluation, we aim to re-check whether the areas that were forecasted to be flooded, are actually flooded or not. If not, then the cash transfers can be re-directed to the most flood affected areas.

To specifically investigate the impacts of the important feature of providing cash assistance at 3 different points in time, the impact evaluation will employ a randomized controlled trial (RCT) methodology.

If both basins activate, households are randomly allocated to 3 treatment arms, receiving NRS 15,000 (117.9 USD). If only one basin activates, one treatment arm will be dropped due to statistical power concerns (Treatment 2).

Within the group of villages forecast to be flood affected and therefore eligible for transfers, the impact evaluation will randomize the timing and the conditions under which a smaller subset of households receive transfers.

Treatment 1: For households in the “forecast based financing” group, a single transfer of NRS 15,000 ($117.9) will be initiated ca. 1-3 days before the projected flood peak.

Treatment 2: The “expedited post-flood transfer” group will not receive the forecast-based transfer but will receive a transfer of NRS 15,000 ($117.9 dollars) within two weeks following the actual flood peak if their geographical location was indeed flooded.

Treatment 3: The “traditional response transfer” group will not receive the forecast-based transfer but will receive a transfer of at least NRS 15,000 ($117.9) within one -two months of the actual flood peak, the normal timeline in which WFP is normally able to responds to disasters (given the time it requires to get resources, mobilize partners and register beneficiaries).

There is no dedicated baseline data collection and data will be collected through in-person surveys during at least 3 waves after the projected flood peak. Outcome data collection starts on 3 November 2022 (first round of data collection).

Randomization will be done in office by a computer, using STATA

To identify the causal impact of providing anticipatory action, the evaluation will employ a randomized controlled trial (RCT) design with the unit of observation being the household, but the randomization will be at the settlement/village level blocked by district.

Yes

140 villages (70 and 70 per arm) in the Karnali basin only (2 arm trial). In case the Eastern basin activates too, there will be more villages (Information as of 2 November 2022).

We have 2321 HHs in the survey sample, 260 Replacement HHs, and 317 HHs that are not a part of the survey sample. This is only in the Karnali basin.

1499 households receiving anticipatory action/FbF support, 1484 households receiving post-shock support. Of those, 2569 are sampled for inclusion in the impact evaluation study. Of these we have 2321 HHs in the survey sample, 260 Replacement HHs, and 317 HHs that are not a part of the survey sample.

The basis for power calculations are Nepal FANSEP impact evaluation baseline and midline data, collected in 2021 and 2022. The results table from an evaluation of anticipatory transfers in Bangladesh (Pople et al , 2021) were used to inform the choice of minimum detectable effects (MDE) and outcomes. Based on the overlap between outcomes reported in Pople et al and the FANSEP impact evaluation data available to the PI team, we chose to report power for the three following indicators:   FIES score   Binary for whether children in household consumed 3 meals per day   Binary for whether woman in household consumed protein in the last day   Intra-cluster correlation and autocorrelation for above outcomes was calculated in the Nepal data. The sampsi command in STATA was used to conduct the power calculations analysis, the following table represents the relationship between minimum detectable effect (MDE) and statistical power. The resulting power calculations suggest that our power is between 65% and 96% to detect impacts of 0.1 standard treatment effects and power is between 80% and 100% to detect impacts of 0.15 standard treatment effects based on the size of effects of forecast based transfers reported for analogous outcomes in Pople et al. For comparison, Pople et al, report effects of .1 standard deviations or more for costly borrowing index and live satisfaction index, and report an effect of .09 standard deviations for days that protein was consumed in the previous 7 days. We expect that scenarios 4, 5 and 6 below are most plausible, since the intracluster correlation is calculated as the average of 3 outcomes of interest.    SAMPLING PROCEDURE    The sampling procedure will rely on beneficiary selection data collected by WFP Nepal country office in July 2022. They will identify households that are eligible for transfers based on their internal scoring cut-off. The research team will randomly select the required number of villages from villages ranked as most vulnerable in WFP’s flood vulnerability ranking. Each village must have on average 30 transfer eligible households, with the minimum village size being 10 transfer eligible households and maximum being 60. Within each sampled village transfer eligible households will be randomly selected proportional to the village size. Transfer ineligible households will be excluded from the sample. On average 20 households will be sampled per village with an additional 10 provided as replacements.