Impact Evaluation of Farmer Field Schools and eVouchers in Mozambique

The IEs described below will leverage the forthcoming Farmer Field School (FFS) and eVoucher programs which will be implemented by FAO in 10 districts in Nampula and Zambézia provinces in northern Mozambique under the EU’s PROMOVE-Agribiz program.

PART A – FARMER FIELD SCHOOL
The FFS is a group-based learning approach, which was developed by FAO and first implemented in Indonesia in 1989. Since then, the methodology has been applied in over 90 countries in Africa, Asia and South America (FAO, 2018) and adopted by various NGOs. The FFS methodology represents a departure from the traditional extension service as it is customarily structured in Mozambique and most sub-Saharan African countries. Instead of receiving top-down generalized recommendations or specific inputs, farmers are directly involved in group discussions and hands-on learning activities, which focus on identifying local agricultural challenges and proposing context-feasible solutions. The FFS includes the establishment of a “learning plot” in order to test and show the productivity gains from a new agricultural input or practices in comparison with traditional practices. FFS meets once a week and is generally composed of a group of around 25-30 smallholder farmers. The activities run throughout an entire agricultural campaign, i.e., covering both primary and secondary cropping seasons. Although this IE focuses on agriculture production systems, the same FFS methodology can also be applied to livestock and fisheries production.

To assess the ability of the FFS methodology to successful transfer knowledge and increase adoption of improved inputs, we have worked with FAO to define a core curriculum of practices and accompanying learning goals. The sustainable land management (SLM) practices that will be promoted include row planting, mulching, intercropping, rotation, zero-tillage, and contour farming. Other related activities, which are considered as complementary to SLM techniques, are the production of organic or fermented fertilizer (“Biol”), composting and green manure. The learning plot will also be used to show the returns to the modern inputs that are covered by the eVoucher intervention that will be rolled-out by FAO in parallel to the FFS. They include improved seeds, inorganic fertilizer, post-harvest pesticides for seed conservation and agricultural services. The FFS methodology is based on a participatory process to define the topics to be discussed, but these core practices will be covered in all FFSs in a relatively standardized way and form the basis of the measurement strategy to evaluate impact in knowledge and adoption.

PART B – EVOUCHER
Like the previous experiences, the eVoucher will have multiple values and levels of farmer co-payment, which will be defined by the project parties, to target both smallholder and emerging farmers. The subsidies will allow producers to purchase different packages of certified seeds, inorganic fertilizers and/or post-harvest insecticides at a discounted price from agrodealers and their retailers.

Subsidies were provided in the form of nominal voucher coupons and redeemable by the intended beneficiary named in the certificate at authorized private agricultural dealers/outlets and retailers. Subsidized crops will be selected according to local demand, agro-ecological conditions and market opportunities. Crops that will be prioritized include maize, beans, vegetables and cashew.

Participating agrodealers (and their associated retailers) will be selected by FAO through a detailed selection process prior to the start of the project. Only agrodealers who present logistical, managerial and financial capacity to store and distribute large quantities of inputs, as well as their market presence and development plans to expand coverage, will be selected. Farmers that meet the project eligibility criteria will be identified and registered for the eVoucher by the eVoucher Service Provider, under the guidance and supervision of FAO, and district-level government officials and extension agents. Priority will be given to farmers that participate in the Farmer Field School program also being implemented by FAO under PROMOVE-Agribiz.

2020-11-01

2024-05-31

PART A - FFS
- Total Revenue per hectare cultivated (Farmer level)
- Total Revenue from crop harvest net input costs (Farmer level)
- Total income received from crop sales net input costs (Farmer level

PART B - EVOUCHER
- Total Revenue per hectare cultivated (Farmer level)
- Total Revenue from crop harvest net input costs (Farmer level)
- Total income received from crop sales net input costs (Farmer level)
- Total sales (Retailer level)
- Total sales net of costs (Retailer level)

PART A - FFS
- Average number of sessions attended per month (Farmer level)
- Average number of sessions led per month (Extension Agent level)
- Knowledge of promoted practices (Farmer level)
- Knowledge of promoted practices (Extension Agent level)
- Adoption of promoted practices (Farmer level)

PART B - EVOUCHER
- Share of cultivated land using improved inputs (Farmer level)
- Total spending on improved inputs (Farmer level)
- Revealed and stated willingness to pay for input package (Farmer level)
- Distance to nearest location where inputs can be bought (Farmer level)
- Unit price of inputs (Farmer level)
- Availability of inputs (Retailer level)

EA SAMPLE AND EXPERIMENTAL DESIGN

97 FAO Extension Agents ("EAs"), all EAs under the FFS and eVoucher programs with at least 6 communities under coverage, comprise our experimental sample of EAs. 52 of the 97 EAs have communities that were eligible to be part of the eVoucher in the first year as they were within range of a program agrodealer. From the third year the program will cover all communities. The EAs were randomized, first, into high/low saturation eVoucher status if the EA was covered by an agrodealer in the first year of the eVoucher, stratified by district. Second, the FFS facilitator selection protocol was randomized, stratified by i) EA gender, ii) eVoucher EA eligibility in the first year, and iii) eVoucher saturation status. Resulting in:

13 Low eVoucher Saturation x Status Quo FFS Facilitator Selection
14 High eVoucher Saturation x Status Quo FFS Facilitator Selection
22 Not-eligible eVoucher Saturation x Status Quo FFS Facilitator Selection

13 Low eVoucher Saturation x Female FFS Facilitator Promotion
12 High eVoucher Saturation x Female FFS Facilitator Promotion
23 Not-eligible eVoucher Saturation x Female FFS Facilitator Promotion

Under Low eVoucher Saturation, 2 communities covered by the the EA were randomly selected as eVoucher communities (see "COMMUNITY SAMPLE AND EXPERIMENTAL DESIGN"). Under High eVoucher Saturation, *all but 2* (so at least 4) communities covered by the EA were randomly selected as eVoucher communities. Under Not-eligible eVoucher Saturation, all communities covered by the EA could not have received the eVoucher at the start of the program.

Under Female FFS Facilitator Promotion, EAs are strongly nudged by FAO to select a female facilitator for the FFS at the end of the initial season, whereas under Status Quo FFS Facilitator Selection the facilitator selection is under the full discretion of the EA.

COMMUNITY SAMPLE AND EXPERIMENTAL DESIGN

In each EA, communities were randomly assigned to
1 2020/2021 FFS x No eVoucher (97 total)
1 2022/2023 FFS x No eVoucher (97 total)
1 2020/2021 FFS x eVoucher (97 total)
1 2022/2023 FFS x eVoucher (97 total)
Remainder 2021/2022 FFS x eVoucher Overflow

The first 4 groups comprise our primary experimental sample; a community listing, including elicitation of interest in participation in a FFS and identification of two potential facilitators (with at least one woman), was conducted in these 4 communities before community treatment assignment was revealed.

Under 2020/2021 FFS, communities are assigned to receive a Farmer Field School in the 2020/2021 primary cropping season. Other communities receive a Farmer Field School in their assigned season, with business-as-usual extension continuing otherwise.

Under No eVoucher, communities are assigned to not receive eVouchers. Under eVoucher, selected households (see "HOUSEHOLD SAMPLE AND EXPERIMENTAL SAMPLE DESIGN") in communities are assigned to receive eVouchers. Under eVoucher Overflow, communities are assigned to receive eVouchers if the EA is a High eVoucher Saturation EA, and are otherwise assigned to not receive eVouchers. eVoucher implementation was able to begin in the 2020/2021 Primary Cropping Season for 56 EAs, and in the 2022/2023 Primary Cropping Season for the remaining 41 EAs.

HOUSEHOLD SAMPLE AND EXPERIMENTAL DESIGN

In each eVoucher Community, eVoucher-eligible households were randomly assigned, stratified on FFS-interest, to
16 Priority eVoucher List (12 FFS-interested, 4 not FFS-interested)
4 Default eVoucher (3 FFS-interested, 1 not FFS-interested)
4 Small and medium package discount (3 FFS-interested, 1 not FFS-interested)
4 Large package discount (3 FFS-interested, 1 not FFS-interested)
4 All package discount (3 FFS-interested, 1 not FFS-interested)
8 De-prioritized eVoucher List (6 FFS-interested, 2 not FFS-interested)
Remainder Intermediate Priority eVoucher List

Households in the eVoucher Control communities were also randomized into shadow eVoucher household treatment statuses in corresponding numbers to the eVoucher Treatment communities presented above.

Our household survey sample is comprised of a randomly sampled 12 (stratified on eVoucher treatment assignment and FFS-interest) of the first 24 households.

During eVoucher Registration, EAs were instructed to register all Priority eVoucher households first, and to then register Intermediate Priority eVoucher households until the community reached their registration quota. This registration quota was smaller than the number of Priority and Intermediate Priority households, so we do not expect De-prioritized households to be registered for the eVoucher. Priority eVoucher households therefore correspond to "Treated" households, while De-prioritized eVoucher households correspond to "Control" households.

Priority eVoucher households were further randomized into 4 groups: Default eVoucher (eVoucher entitles the household to purchase any of the FAO packages at standard rates), Small and medium package discount (household receives a 300 MZN discount on the small and medium eVoucher packages), Large package discount (household receives a 1200 MZN discount on the large eVoucher packages), and All package discount (household receives both sets of discounts).

In the 2021/22 primary cropping season, 896 households over 112 communities were targeted for implementation, determined by those communities that a) had eVoucher coverage and b) were eVoucher Treatment communities. Two of the four households in each Priority eVoucher household subgroup were randomly selected for implementation. The total target sample was:
224 Default eVoucher
224 Small and medium package discount
224 Large package discount
224 All package discount

In the 2022/23 primary cropping season, 3,040 households over 192 communities were targeted for implementation, determined by those communities that a) were eVoucher Treatment communities, and b) the community listing was performed correctly (2 cases dropped).
760 Default eVoucher
760 Small and medium package discount
760 Large package discount
760 All package discount

The randomization was carried out used the randtreat command in Stata on a computer.

PART A - EXTENSION AGENT
The randomization of eVoucher Saturation and Female FFS Facilitator Promotion is done at the extension agent level.

PART B - FFS
The randomization of FFS treatment assignment is done at the community level (stratified by extension agent coverage) while the unit of observation, at which surveys are administered, is the household.

PART C - EVOUCHER
The first eVoucher community randomization is done at the community level (stratified by extension agent). The second layer of eVoucher household randomization is done at the household level.

Yes

PART A - EXTENSION AGENT
The sample consists of 97 extension agents (cluster)

PART B - FFS
The sample consists of 388 communities (cluster)

PART C - EVOUCHER
Year 1 and 2: 224 communities (cluster)
Year 3: 388 communities (cluster)

PART A - EXTENSION AGENT 4,656 households PART B - FFS 4,656 households PART C - EVOUCHER 4,656 households

Community randomization:
- 97 FFS T and eVoucher T (combined)
- 97 FFS T and eVoucher C (FFS only)
- 97 FFS C and eVoucher T (eVoucher only)
- 97 FFS C and eVoucher C (pure control)

Across-community treatment assignment Based on Carter et al. (2013), we assume the impact of being assigned to receiving a voucher (regardless of the type/size of the package) on the probability of taking up the voucher to be 0.22 percentage points (pp). We start by estimating the minimum number of clusters we would need to detect this effect on take-up. Using a power of 0.8, an alpha of 0.05, and an intra-cluster correlation (ICC) of 0.1, the sample size per arm required under cluster randomization is 12 communities (i.e., 22 in total). If the impact on take up is half of the one observed in Carter et al. (2013), i.e., 0.11 pp, the total number of clusters needed would increase to 50. The second outcome in the theory of change is the adoption of the high-yielding inputs, such as certified seeds and chemical fertilizer. We use the intent-to-treat (ITT) effect during the subsidized time period found by Carter et al. (2019) observed, which is equal to 0.16pp for both ma ‘fertilizer on maize’ and ‘improved maize seeds’. Assuming the same parameters as above, we would need to have a sample of 190 clusters, evenly split between treatment and control, to detect the underlying minimum detectable effect (MDE). Finally, we run power calculations for productivity estimates. In particular, we consider baseline revenue yields from a recent survey with more than 1,000 smallholders DIME performed in the project provinces – mean maize revenue per hectare (MZN/ha) is 5,400, with standard deviation of 6,732 – and the ITT effect from Carter et al. (2019) – i.e., a 0.19% yield increase for subsidy-recipient households – to benchmark the MDE. With a baseline and one follow up survey, assuming a correlation coefficient between measures in the two phases of 0.35, 328 communities will give us enough power to detect a change in such outcome. Within-community treatment assignment We now focus on the within community variation generated by the different price schedule offered to beneficiary farmers: for each 12 farmers in a treated community, 4 are assigned a voucher which entitles them to redeem a small package and 4 a large package of inputs. A scenario with 28 clusters yields a minimum detectable difference in take-up of 0.21 pp. Doubling (tripling) the number of clusters to 56 (84) would allow us to detect a difference of 0.15 (0.12) pp.