Uganda Coffee Agronomy Training Impact Evaluation

The Uganda Coffee Agronomy Training (UCAT) aims to train 60,000 smallholder coffee growers in Uganda on recommended coffee practices, with the goal of increasing yields and improving livelihoods. Under the project, two training implementers, Hanns R. Neumann Stiftung (HRNS) and TechnoServe, provide in-person agronomy training in two regions of Uganda over a period of four years through three two-year cohorts. Each implementer covers a distinct geographical area, and each implementer-cohort covers a subset of this area. This evaluation covers the second cohort of the program.
Both the HRNS and TechnoServe programs deliver material to farmers through monthly training sessions. In the HRNS program, these are delivered by volunteers who receive three days of training per quarter, and compensation for travel expenses and time spent training farmers. HRNS training sessions cover 1 to 4 topics each, according to an annual calendar. Material for these topics is drawn from a training manual published jointly by the Uganda Coffee Development Authority (UCDA) and the Ugandan Ministry of Agriculture, Animal Industry and Fisheries (MAAIF). All field extensionists have a copy of a large, laminated, spiral-bound copy of the UCDA/MAAIF manual, which they are encouraged to use as a visual aid during training.
Trainers employed by TechnoServe receive two days of instruction prior to their delivery of each session. TechnoServe trainings follow a step-by-step lesson plan that covers one or two related topics each month, and includes visual aids, demonstrations of practices, and group exercises. These materials are developed by TechnoServe and adapted to local conditions in the districts where they operate. TechnoServe training materials are not publicly available but cover the same topics as the training manual used by HRNS, typically in greater detail.
A mobile phone-based extension service, implemented by Precision Development (PxD), is provided to a subset of Cohort 2 farmers. This service is implemented both as a stand-alone intervention for farmers in villages where in-person training is not offered, and as a reinforcement of training offered by HRNS and TechnoServe. Recorded voice calls are sent to farmers, and farmers may also call a toll-free number to record questions. An agronomist then records answers to these questions, which are subsequently sent back to the number from which the question originated. Stand-alone messages and reinforcement for the HRNS intervention are drawn from the training manual mentioned above, and TechnoServe provides content for the reinforcement messages sent to farmers in their program areas.

2019-02-03

2022-08-02

- Coffee agronomy practices
- Coffee yield per tree

- Coffee practices will be assessed based on enumerator observation and rules agreed by the in-person training implementers. A yield-based index, constructed as a linear combination of binary practice indicators, with weights corresponding to the median expected effect of each practice on yield as assessed by 15 coffee agronomists surveyed, will be the primary practice for both in-person training and the mobile phone-based intervention. Practice definitions and weights are described in Appendix 1 of the pre-analysis plan.

- The inverse hyperbolic sine (IHS) of coffee yield per tree will be a primary outcome for the in-person training intervention only. The mean number of green cherries per tree from the pre-harvest yield measurement (calculated using the weight of green cherries per tree per cherry size category and the per-cherry weight per size category) for each farmer will be multiplied by the mean weight of ripe cherries collected at the peak of harvest from each farmer to estimate the yield of ripe cherry per tree. If ripe cherry weight is not available for a given farmer, the median weight of ripe cherries for farmers in the same village will be used. If fewer than 5 observations of the ripe cherry weight are available for a village, the median weight for farmers in villages assigned to the same treatment within the same cluster will be used.

For both interventions:
- Additional coffee practice variables
- Coffee agronomy knowledge

For in-person training only:
- Gross coffee profit
- Household labor days devoted to coffee
- Non-coffee household income
- Income controlled by women
- Food crops

Secondary outcomes for both interventions:

- Additional coffee practice outcomes:
(a) Individual binary coffee practices, with significance thresholds adjusted for multiple hypothesis tests across these using the Romano-Wolf stepdown procedure.
(b) Indices of these practices classified along the following three dimensions:
o high/low monetary cost, based on the median cash outlay for non-labor inputs on each practice per acres of land under coffee (as defined in point 4a below) over the past year, respectively, among all households using the practice at follow-up
o high/low labor cost, based on the median number of household labor days spent on each practice per acres of land under coffee (as defined in point 4a below) over the past year, respectively, among all households using the practice at follow-up
o high/low complexity, based on expert opinion as described in Appendix 1
- Practices for which the median cash outlay or household labor input as defined above is above the median across practices will be classified as high-cost on that dimension, and those for which this cost / input is below the median will be classified as low-cost.
- Indices for each category (high monetary cost, low monetary cost, high labor cost, low labor cost, high complexity, low complexity) will be constructed using inverse covariance weights as proposed by Anderson (2008) as implemented via Stata’s swindex command.
- P-values for these six practice indices will be corrected for multiple hypothesis tests using the Romano-Wolf stepdown procedure.

- Coffee knowledge: an index of coffee knowledge will be a secondary outcome for both the in-person training and mobile phone-based interventions. Knowledge questions shown in Appendix 2 will be aggregated first by the practice to which they relate based on item response theory (IRT), as the predicted latent train using Stata’s irt 1pl (one-parameter logistic model). These will then be aggregated using the same yield-based weights as for the yield-based practice index.
Additional coffee knowledge outcomes will include:
a) Practice-specific knowledge indices as described above, with significance thresholds adjusted for multiple hypothesis tests across these using the Romano-Wolf step-down procedure.
b) Impacts on coffee practices classified along the three cost, labor, and complexity dimensions described above (aggregating by applying swindex to the practice-level knowledge scores described above), corrected for multiple hypothesis tests using the Romano-Wolf procedure.
c) An index constructed from the practice-specific indices using Anderson’s method

Secondary outcomes for in-person training only:

- IHS of gross coffee profit constructed as total coffee revenue over the past 12 months, minus the cost of inputs (pesticides, chemical fertilizer, manure, compost, mulching material), hired labor, and marketing expenses.

- Person-days of household members and other unpaid workers applied to coffee over the past year, constructed as the sum of household person-days applied per practice over the past year.

- Non-coffee household income:
a) IHS of household income from sources other than coffee, calculated as the sum of revenue minus costs (excluding the value of household labor) associated with each of the following sources:
Sale of crops aside from coffee
Sale of livestock products (eggs, milk)
Sale of small livestock (including goats, sheep, pigs, poultry; excluding cattle, which are expected)
Non-farm business
Wages from casual labor
Salaries
Other income
b) Qualitative change in non-coffee sources over the past 4 years, calculated as the sum of a set of variables per non-coffee income source in 6a. defined as = 1 if income from the source has increased over the past 4 years or is new since 4 years ago, = 0 if income from the source has remained constant, and = -1 if income from the source has decreased or if this was a source of income 4 years ago and longer is. For this measure, each crop sold, each type of livestock product, type of livestock, business, and individual’s casual wages will be counted as a separate source.
P-values for the two non-coffee outcomes will be corrected for multiple hypothesis tests using the Romano-Wolf stepdown procedure, and an index of the two will be constructed using Stata’s swindex command.

- Income controlled by women
a) IHS of household income from sources other than coffee (constructed following the method for household non-coffee income) that are controlled primarily by women.
b) Qualitative change in gross household income from sources other than coffee (constructed following the method for qualitative change in household income) that are controlled primarily by women.
c) A measure of women’s control over coffee income constructed as follows:
∑_jSUM(w_j)
Where w_j is a variable representing the female head’s role in the decision to spend coffee income on item j, equal to 1 if the female head primarily made the decision, 0.5 if the decision was made jointly, and 0 if the the male head primarily made the decision.
P-values for incomes in this category will be corrected for multiple hypothesis tests using the Romano-Wolf stepdown procedure, and an index of the three will be constructed using Stata’s swindex command.

- Food crops
a) Number of crops grown for own consumption
b) Qualitative change in crops grown for own consumption over the past 4 years, calculated as the sum of a set of variables per crop, defined as = 1 if the amount of the crop grown has increased over the past 4 years or the crop is newly grown since 4 years ago, = 0 if the amount of that crop grown has remained constant, and = -1 if the amount of the crop grown has decreased since 4 years ago or if the crop was grown 4 years ago and longer is.
P-values for outcomes in this group will be corrected for multiple hypothesis tests using the Romano-Wolf stepdown procedure, and an index of the two will be constructed using Stata’s swindex command.

Population and sample

In September and October of 2018 in the region where HRNS provides training, and May of 2019, in the region where TechnoServe provides training, each of the coffee agronomy training implementers recorded the name of each village containing at least 20 coffee-growing households, the name of the administrative units (parish, subcounty) within which the village was located, and the GPS coordinates of the approximate center of the village. Based on the GPS coordinates, 360 villages per region were selected that were at least 1.9 KM (HRNS region) or 1.4 KM (TechnoServe region) apart from one another.
In all 720 study villages, a training session on coffee harvest and post-harvest practices was conducted prior to the initiation of the training program to be evaluated. These training sessions were run by the same organization which would later implement coffee agronomy training in that area. Attendance of these sessions was taken at the household level, and the attendees constitute the sampling frame for the impact evaluation.

Village-level randomization to in-person coffee agronomy training program

The 720 study villages were stratified into 180 geographically defined clusters of 4 villages each. Within each cluster, 2 villages were randomly assigned to receive in-person agronomy training by the training implementer operating in that region. This resulted in 360 villages (180 per implementer region) assigned to receive in-person agronomy training, and 360 nearby villages assigned to serve as controls.

Baseline survey

The baseline survey was administered to households randomly selected from among harvest training attendees. The target number of participants was 12 in treatment villages and 18 in control villages. The larger sample in control villages was included to allow for testing of a stand-alone mobile extension service within these villages. No treatment-specific project activities occurred before the baseline survey was completed.

Assignment to mobile extension treatments

Assignment to the two mobile extension treatments (standalone and reinforcement) was conducted after the baseline survey. A subset of farmers in 120 of the 180 villages per region where no in-person agronomy training was offered were assigned to the standalone mobile extension treatment. These villages were selected as follows. First, in each of the geographical clusters used to stratify assignment of in-person training, one of the two villages in which no in-person training would be offered (90 villages per region) was randomly assigned to the standalone mobile extension treatment. The remaining 90 control villages in each region were then grouped into terciles by a practice index constructed as the mean adoption rate of recommended coffee agronomy practices related to yield that were either observed or self-reported in all baseline surveys. 10 villages per tercile were then randomly assigned to the standalone mobile extension treatment.

In the HRNS implementation area, the 6 farmers assigned to the limited survey group were automatically assigned to the standalone treatment group in mobile extension villages. The village-level randomization described above was repeated in the HRNS sample until p-values of tests for equality of means of the practice index across each of the following comparisons was > 0.6, based on a simple regression with standard errors clustered at the village level.
1. standalone mobile extension versus spillover (farmers in the same villages but not assigned to the standalone intervention, i.e., those to whom either the observational, standard, or standard + gender survey was administered)
2. standalone mobile extension versus pure control (farmers in villages where no farmers were assigned to the mobile extension intervention)

In the TNS implementation area, the index of practices was found to be imbalanced between those farmers to whom the limited survey had been administered, versus others. In this region, therefore, farmer assignment to the mobile extension treatment was randomly re-assigned. This procedure (beginning with the random selection of one village from each cluster) was repeated until the p-value of means of the practice index across each of the comparisons above was > 0.6.

Farmers in In-person training villages were stratified by village and then randomly assigned to receive mobile reinforcement. Balance on the index of practices as above was confirmed at p > 0.6.

Follow-up data

Physical measurement of coffee yield will be conducted among all baseline farmers in in-person training treatment villages, and all baseline farmers in in-person training control villages (including mobile extension villages) who were not assigned to the mobile extension intervention.

Pre-harvest yield measurement

Immediately preceding the fourth main coffee harvest after the start of training, yields will be measured by harvesting all green cherry from three trees along the longest diagonal transect of the coffee plot where the farmer indicated at baseline that new practices would be implemented first.

To identify these trees, the enumerator will walk from the corner of the plot nearest to the farmer’s home, to the opposite corner, and record the number of steps used to cross the plot. At three points along this transect, equidistant from the edges of the plot and each other, the nearest tree will be selected for yield measurement. To avoid purposive selection of trees with certain characteristics by enumerators (for example, those with few cherries or low branches), trees will be selected and marked during an initial visit and harvested by a separate team the next day. Enumerators on both teams will take photographs of each selected tree to enable visual back-checks. The harvesting team will take photos of trees both before and after harvest.

At two of the three sampling points (the order of which will be randomly pre-assigned), enumerators on the harvesting team will mark out a 10 by 10-meter square and count the number of productive coffee trees, newly planted coffee trees, and any other trees within it, to estimate tree density and the extent of new coffee planted within existing coffee fields.

Endline survey

Survey data on coffee expenses, labor input, income from other sources, changes in income from each source over the past three years, and land under coffee, will be collected. Farmers will be asked a series of knowledge questions reflecting the content of the agronomy training provided by the implementers. Coffee practices will be observed in-field using the same module as used in the Observational Survey administered to a subset of farmers at baseline. Farmers will be asked about their attendance of coffee agronomy training over the past two years and asked to find their names on attendance lists from their own village (for those who live in treatment villages) and the nearest other three villages where training was offered (for all farmers, including those who live in treatment and control villages).

Ripe cherry sampling

At the height of the fourth main coffee harvest after the start of training, a sample of ripe cherries will be collected from at least 10 trees randomly selected along a vertical transect of the same coffee plot from which green cherry was harvested prior to the start of the harvest.

Randomization done in office by a computer

In-person training was randomized at the village level
Mobile phone based training was randomized at the individual level

Yes

720 villages

Originally planned numbers: - 8640 farmers for evaluation of in-person training - 4320 farmers for evaluation of mobile reinforcement intervention (vs in-person training with no mobile reinforcement) - 4320 farmers for evaluation of standalone mobile agronomy intervention (vs farmers within the same village not assigned to receive messages) Due to low attendance of pre-intervention training sessions, we have fewer observations (baseline sample): - 8002 farmers for evaluation of in-person training - 4123 farmers for evaluation of mobile reinforcement training - XXXX farmers for evaluation of standalone mobile agronomy messages

The sample sizes below reflect the number of farmers in the baseline sample:

In-person training
- 4123 farmers in 359 villages where in-person training is offered
- 3879 farmers in 357 villages where in-person training is not offered

Mobile-reinforcement
- 2067 farmers in 359 in-person training treatment villages assigned to receive reinforcement messages by phone

Stand-alone mobile agronomy messages
- 1312 farmers in 238 control villages assigned to receive coffee agronomy messages by phone

The study as originally planned was powered to detect a 17% increase in coffee yield among farmers who completed training with power of 0.8. This estimate assumes a difference in attendance of agronomy training of 41.4% across treatment and control villages, based on a pilot study. We assume a standard deviation of log yield per tree of 0.89 and intra-cluster correlation of yield across villages of 0.12. Power to detect an effect of the mobile phone treatments on an index of coffee practices (assuming a standard deviation of 0.0936): Mobile reinforcement: 0.0145 percentage points Mobile standalone (vs. within-village controls): 0.0087 Mobile standalone (vs. within-village controls): 0.0087 Estimates have not been updated to reflect the lower sample size arising from restricting analysis to farmers who attended a pre-intervention training session.