Impact evaluation of an agroforestry tree-seedling distribution program in Rwanda

The intervention consists of providing farmers with agroforestry tree seedlings of mainly grevillea species (though in combination with other species depending on the agroecological zone) in an “opt in” setting. In 2023, around 90% of the program will be designed as a decentralized nursery model, in which one single nursery will be set up in a central location in a cell (admin 4 level in Rwanda). A nursery manager from the community will nurture the seeds until the seedling stage and will be paid a unit sum for each healthy tree seedling produced. Around three months after planting, farmers of each cell will come to the nursery to pick up the seedlings for free.
Since the tree distribution program already has nearly country-wide coverage, there are no suitable control areas outside of the program areas. Only regions that are particularly hard to reach or otherwise differ significantly from the existing program areas have not yet been reached by the program. Thus, we have decided to use cells from within the existing program areas as controls. To create a valid counterfactual the program will be modified in control cells by providing farmers with fruit (avocado) rather than agroforestry tree seedlings. With this design the impact estimates will capture the incremental effect of delivering the tree program on agroforestry outcomes for one additional year.

2023-10-01

2025-01-01

1) Agroforestry Tree count survey
• Total trees planted (both 1AF and non-1AF species)
• Total trees survived (both 1AF and non-1AF species)
• Trees planted by species (1AF species)
• Trees survived by species (1AF species)

- Tree count survey (benchmark to monitoring and tree value model)
The tree count survey will comprise a visual tree count of newly planted trees and a module on household characteristics.
The objective of the visual tree count is to estimate the number of trees planted, by species and plot, focusing specifically on all trees planted in the most recent planting season (Season 2023A). The visual tree count will be conducted by an enumerator together with the respondent. At endline all surviving trees planted during previous years planting season will be counted. To increase the accuracy of the tree count, the survey will be conducted before the following planting season has started. Tree numbers by species will be recorded and photos will be used to back-check tree species identification. Our understanding is that the average farmer has two plots in addition to planting trees around their homestead – but that a farmer could have as many as 5 plots. The goal is to visit and count trees on all of the farmer’s plots. Where that is not possible, for example because the farmer is not willing to visit all plots with the enumerator, we will record self-reported numbers of planted trees by species.
The outcomes measured from the tree planting surveys will be used to construct the following metrics:
- Tree survival rates
Measured as the proportion of seedlings planted that are still alive at endline, pooling both treatment and control group observations. This effect might be further split up by tree species and survival of tree seedlings provided by the program and seedlings from other sources as far as sample size allows.
Survival Rate=(surviving trees_(T+C))/(planted trees_(T+C) )

-Difference in total trees planted
Defined as the difference between the treatment and control in total trees planted. Trees planted will be measured as all tree seedlings planted in soil counted during the planting survey.
∆ trees planted_total=trees planted〗_(total,T)-trees planted〗_(total,C)


-Incremental trees
Defined as the difference between the treatment and control in total trees planted. Trees planted will be measured as all tree seedlings planted in soil counted during the planting survey.
∆ trees planted_total=trees planted〗_(total,T)-trees planted〗_(total,C)

-Difference in trees planted by species
The difference in the total number of trees planted per species. This measure is similar to the total planted tree measure, however, it is disaggregated by species. The aim is to understand whether farmer plant trees provided through in addition or instead of the trees they would have planted in the absence of the program.
∆ trees planted_S=trees planted〗_ST - trees planted〗_SC

2) Agroforestry Tree usage and value survey (8 species)
• 1AF programme take-up
• Tree use cases
• Timing and frequency of tree uses
• Prices of tree products
• Volume and prices of tree products sold
• Direct financial and opportunity costs of maintaining mature trees
• Timing and frequency of costs for maintaining mature trees

3) Agroforestry Tree price survey (4 species)
• Prices of tree products
• Volume of sales for tree products
• Average cut age of tree (by species)
• Average value of tree cut sales
• Characteristics impacting the value of trees (by species)
• Tree value (Net present value)

- 1AF programme take-up
The 1AF tree program is an opt-in treatment programme, which means while everyone in a specific cell is eligible to receive the seedlings and farmers are free to choose whether they will do so. We thus investigate the determinants of programme take up, that is whether people show-up to pick-up trees at distribution day by estimating the probability to pick-up trees conditional on a wide range of covariates.

-Tree usage and value survey (inputs for tree value model)
For the tree usage and value survey we will select eight species of trees, each part of four groups of similar trees. Three of those four groups have two species each, which will each be included in the value model, and one group has three species, of which two will be included in the model and used to extrapolate the value of the third species. We will select farmers/trees to interview using a stratified sampling approach where we stratify mature trees by species and age, to construct an accurate image of the life span of trees of different species. We will select, insofar as possible, two mature trees (each of a different species) per respondent. Farmers will be asked about how they have used each of the selected trees in the past 6 months, what volume of different products they derived from each tree, how frequently they can harvest a similar amount from each tree, and whether they are planning to cut down each tree within the next year, or its remaining lifetime. We will also collect information on the direct financial and opportunity costs of maintaining mature trees, as well as the opportunity cost of land from planting trees rather than crops. Households will also be asked about the typical prices they pay or receive for tree products when purchasing or buying them from local markets or traders. Finally, we will record the age of each selected tree and measure its stem circumference to help establish a relationship between age/circumference and tree value.

-Tree price survey (inputs for tree value model)
The tree price survey with tree product vendors and tree traders will focus on current market prices for various tree products from different species, as well as prices for fully harvested trees by species. We will ask vendors and traders to report prices by species as well as other characteristics that might affect prices such as tree age or circumference. Collecting these data alongside both the planting and survival survey will ensure that we have data that includes enough seasonal variation, as we expect tree usage and product values may vary by season.

- Tree value
The value of trees will be estimated by subtracting tree costs from its benefits, we will estimate tree values for species grevillea, alnus, cedrella, markhamia and jacaranda specifically for a period of 20 years .
Tree costs will be a function of tree related costs and frequency and timing of cost incurred expressed in USD.
Before planting, tree costs are: the costs of the seedling, the transportation costs for seedlings, the cost of planting trees (opportunity costs of time planting trees expressed as in local daily agricultural minimum wages).
Costs from one year after planting until being cut down: These costs contain the maintenance costs, input costs (e.g. fertilizer), and the opportunity costs of land that is ‘displaced’ by trees, the costs of harvesting tree products (opportunity costs of time expressed in local daily agricultural minimum wages) and transportation costs for products or full trees to a market or tree trader.
Tree benefits are a function of tree revenues generated through selling full grown trees (full cut, such as timber) or revenue derived from selling tree products over the life course of a tree (such as firewood, mulch, bean poles and other poles) to tree traders. We add costs saved by using trees and tree products domestically as benefits. We estimate tree benefits in USD for each relevant specifies.

See Table 2 in Annex document for details on surveys

This study is a cluster randomized controlled trial, which is considered the gold standard to measure programme effectiveness. 1AF's nurseries are operating across Rwanda, so we have randomized clusters (nurseries that operate at the cell level) divided into control cells and treatment cells. We have randomized 30 cells into the control group and another 30 cells in the treatment group in 3 provinces: South (Amajyepfo), East (Iburasirazuba), and West (Iburengerazuba). We have implemented a stratified randomization by first stratifying cells into 4 artificial agro-ecological zones based on 1AF's programme implementation and the prevalence of tree species. These are regions where Grevillea trees (in the Northwest and Southwest as separate strata), lightwood ornamental trees, or lightwood medicinal trees are most prevalent. Rutsiro, Gicumbi, and Nyarugenge districts and cells adjacent to international borders and cells at the border with Rutsiro and Nyarugenge districts were excluded.
All program elements will remain the same in both treatment and control cells with one exception in the control group: instead of agroforestry tree seedlings, people will receive avocado trees in the control cells in case they decide to participate and collect tree seedlings on the tree distribution days.
We have implemented a stratified randomization. First, cells were stratified into 4 artificial agro-ecological zones. This was based on 1AF’s programme implementation and on the prevalence of tree species. These are regions where Grevillea trees (in the Northwest and Southwest as separate strata) lightwood ornamental trees, or lightwood medicinal trees are most prevalent. Subsequently, we have excluded Rutsiro, Gicumbi and Nyarugenge districts, excluded cells adjacent to international borders and cells at the border with Rutsiro and Nyarugenge districts. This is because in some of these districts 1AF programme implementation slightly differs -there are no nurseries present - and to avoid possible spillover effects to and from other nearby nurseries. In each stratum we have randomized cells into a treatment (distribution and pick-up of tree seedlings) and a control group (distribution and pick-up of avocado trees).
One Acre’s fund programme is currently active all across Rwanda. It is unrealistic to stop or alter implementation (distributing avocado trees) in all potential 1897 cells. Therefore, we have randomized 30 cells into control, where 1AF will distribute avocado trees. Note that all other programme elements remain the same, that is: nurseries and farmers in control cells will be notified that they are to receive avocado trees (instead of tree seedlings). And the nurseries will only receive avocado trees very shortly before the distribution day.
We immediately combine randomization and sampling strategy at the cell level. This means that 1AF will thus only alter implementation (distributing avocado trees) at nurseries in cells randomized into the control group who are also selected into the sample for surveying. The remaining treatment cells will have tree seedling distribution as normal. This results in 30 treatment cells, and 30 control cells. In each cell, we will sample 2 village to conduct household level surveys among both people who pick-up seedlings or avocado trees (compliers) and those who do not pick-up seedlings or avocado trees (non-compliers).
See Annex (Figure 3) for schematic representation of randomization strategy.

Not available

Stratified randomization by computer

Cell (admin 4)

Yes

59 clusters (cells) in total. 29 on treatment and 30 on control arms.

4,130 households in treatment and control cells. 2,100 in control and 2,030 in treatment clusters. 60 villages (2 villages in each cell) 35 households per village (17 compliers and 18 non compliers)

Planned Number of Clusters
59 clusters (cells) in total. 29 on treatment and 30 on control arms.

The MDE for our main outcomes (assuming 80% power): -Tree survival rate MDE is 4% -Incremental trees survived MDE is 2.4 trees -Tree substitution MDE is 0.08 trees -Tree substation value (in USD) is 2.60 USD See Annex (Table 2) for details on calculations.