Large Scale Pilot Test of low-cost soil tests to promote sustainable farming practices among smallholder farms in Indonesia

The intervention consists of providing soil nutrient management training and conducting on-the-field manual soil tests to encourage smallholder farmers to adopt a balanced fertilizer application behaviour and to increase the use of organic inputs.

2022-09-01

2022-10-31

We are interested in studying the effect of treatment on several outcomes of interest. All outcomes will be measured in the endline survey which will take place between the end of May and mid-July 2023. We differentiate between three families of outcomes: (1) Adoption, (2) profits and yields and (3) knowledge and perception.
Regarding adoption, profits and yields we will primarily focus on rice. For adoption outcomes we will additionally also consider non-rice plots.

Family 1: Adoption

1. Outcome: Index of organic fertilizer application
This outcome will be a count variable from 0 to 4 for the number of organic fertilizer types the farmers applied on his/her rice fields during the last season.
The following organic fertilizers/inputs will be considered:
• Applied fermented manure (binary variable) =1 if the respondent applied fermented manure
• Returned plant residues to the soil (binary variable) =1 if respondent returned rice plant residues and did not burn the remaining part
• Applied green manure (binary variable) = 1 if the respondent applied green manure other than rice plant residues
• Applied other organic fertilizers or non-pesticide organic inputs (binary variable) = 1 if the respondent applied other organic fertilizers As sub-outcomes, we will further consider the individual organic fertilizers.




2. Self-produced organic fertilizers
To explore whether a potential increase in organic fertilizer application is driven by increased self-production, we will further look at self-produced fermented manure and self-produced organic fertilizer.
Outcome i: Self-produced fermented manure
i. Applied self-produced fermented manure (binary variable) =1 if respondent applied fermented manure that his/her household fermented themselves.
Outcome ii: Self-produced other organic fertilizer
Applied self-produced other organic fertilizer/input (binary variable) =1 if respondent applied other organic fertilizers/inputs such as liquid organic fertilizers, MOL, PGPR or compost


3. Outcome: Lime application
During the training, the trainers explained the importance of the optimal ph level and also that lime (quick results) and manure (longer term) application can help to increase the ph level.
• Lime application (binary variable) = 1 if respondent applied lime on his/her rice field during the last planting season.




4. Outcome: Macro-nutrient application in tons/per ha from chemical fertilizer

Chemical fertilizer application quantity of N, P and K in tons/ha (continuous variables): We will estimate these variables based on the reported quantity of different fertilizer types applied during the last planting season for rice. While we expect that we can identify the majority of fertilizers, in rare cases, farmers may report a fertilizer type for which we cannot find a reference in the local agricultural shops. Additionally, some farmers may have forgotten which type they used. If this is the case, enumerators will ask to take a picture of the package if it is still available. Yet, a small error in the variable may remain. We will convert the reported quantity to the measurement tons/ha based on the reported land size. This variable will be top-coded at the 95th percentile of the overall distribution.



5. Outcome: Share of farmers that over-apply macro-nutrients from chemical fertilizers
This outcome is closely related to the previous outcome, yet focuses more specifically on overapplication.

i. Chemical fertilizer overapplication Nitrogen (binary variable) =1 if respondent applies more than 120 % of the recommended Nitrogen quantity (in tons/ha). The recommendations are sourced from the Ministry of Agriculture.

ii. Chemical fertilizer overapplication Phosphorus (binary variable) =1 if respondent applies more than 120 % of the recommended Phosphorus quantity (in tons/ha). The recommendations are sourced from the Ministry of Agriculture.

iii. Chemical fertilizer overapplication Potassium (binary variable) =1 if respondent applies more than 120 % of the recommended Potassium quantity (in tons/ha). The recommendations are sourced from the Ministry of Agriculture.


6. Outcome: Index of fertilizer application pattern in accordance with training
This outcome will be a count variable from 0 to 4 where higher numbers indicate that the farmers’ application pattern is more in line with the recommendations from the training.
We use an index to asses farmers’ fertilizer timing. The index is based on the following variables:
i. Early Phosphorus Application (binary variable) = 1
ii. Split Nitrogen Application (binary variable) = 1
iii. Early/Medium Potassium Application (binary variable) = 1
iv. No late Nitrogen Application (binary variable) = 1



7. Outcome: Use of LCC during last season

Use of LCC during last season (binary variable) = 1 if farmer reported having used the LCC during the last season




Family 2: Yields and profits

1. Outcome: Rice yields in tons/ha
Through improved timing of the fertilization as well as through a more balanced quantity of nutrient application, the training, especially the soil test, may have had a positive effect on yields.

i. Rice yields in tons/ha (continuous variable): We will convert the reported quantity to the measurement tons/ha based on the reported land size. This variable will be top-coded at the 95th percentile of the overall distribution.

2. Outcome: Profits

i. Profits in 1000 IDR/ha (continuous variable): A high share of the harvest in our sample is self-consumed. We will convert this share into income based on average local prices. Similarly, we will estimate family labor costs based on the average local wages of agricultural workers. This variable will be top-coded at the 5th and 95th percentile of the overall distribution.



Family 3: Knowledge and perception
1. Outcome: Knowledge score on soil and nutrient properties
This outcome will be a count variable from 0 to 6 for the number of correct answers to 6 knowledge questions. The questions are either multiple-choice or open-ended. The following knowledge questions will be considered:
i. What is the optimal pH level for rice? (open-ended question, answers between 5.5 and 7 will be coded as correct)
ii. Which of the following nutrients is the main determinant of a rice crop’s greenness? a) P, b) N, c) K, d) don’t know
iii. Which of the following nutrients is the main determinant of root length? a) P, b) N, c) K, d) don’t know
iv. Which of the following nutrients is the main determinant of grain density? a) P, b) N, c) K, d) don’t know
v. The application of which of the following macro-nutrients inhibits plant maturity/prolongs the time until flowering? a) P, b) N, c) K, d) don’t know
vi. Which of the following is a sign of too much N? a) yellow leaves, b) stems are weak and easy to collapse, c) stunted plant growth, d) don’t know

2. Outcome: Knowledge score on fertilizer management
This outcome will be a count variable from 0 to 5 for the number of correct answers to 5 knowledge questions on fertilizer management. The following questions will be considered:
i. Consider two plots. At the time of the second fertilization, the rice plants on plot 1 have a light green color similar to number 3 in the picture. The rice plants on plot 2 have a dark green color similar to number 5 in the picture. How much urea do the plots need?
a) Plot 1 needs more urea b) Plot 2 needs more urea c) The plots need the same amount of urea d) I don’t know

ii. Do you know what input to use when the ph level is too low? (open-ended question, answers lime and manure will be coded as correct)

iii. In the growing season, when should the application of Urea be stopped (in HST=days after planting)?

iv. What is the optimal time to apply fertilizers that contain Phosphorus such as TSP (or NPK)?

v. If farmers want to increase the share of organic matter in their soil, which of the following inputs should they apply? Multiple answers are possible. a) Manure, b) Urea, c) rice residues, d) NPK, e) don’t know

Secondary outcomes will allow us to explore the mechanism of how the training worked, in particular, if the training changed farmers’ understanding and perception of soil tests and soil quality. It also helps us explore if the farmers used digital tools to adopt farming practices.

1. Outcome: Understanding of PUTS
• Asked to respondents in T2
i. Do farmers remember how to collect soil samples?
ii. Do farmers remember which soil properties can be analyzed using the PUTS?

2. Outcome: Perception of training and PUTS
• Asked to respondents in T2
i. After the training, do farmers feel confident to use the soil test by themselves?
ii. Did you use the soil test again after the training?
iii. If (ii) yes, when did you use it?

3. Outcome: Perception of soil (asked to all respondents)
• Soil quality criteria used: We will investigate whether farmers provide different criteria to assess soil health depending on the treatment group
• Self-perceived soil fertility: We will investigate whether farmers self-assess their soil differently depending on the treatment group  soil quality is measured using a Likert scale (1= very low, 2=low, 3=medium, 4=good, 5=very good)

4. Outcome: Digital extension use (asked to T1 and T2)
• Did you log into the Lentera Desa Application after the training? Did you watch any videos?
• After the training, did you log in again and use the “kalkulator” (a tool to calculate fertilizer recommendations based on soil test results and land size)?

We applied a multi-stage sampling design to sample 1,104 respondents from 69 villages, i.e. 16 respondents per village. In the first stage, we dropped unsuitable villages from the village database for the three districts - Sleman, Kulon Progo and Bantul. In the second stage, we randomly sampled one sub-village from each of the remaining villages (sub-village, or Dusun in Bahasa, is an administrative subdivision of a village in Indonesia). A team of enumerators then visited each sub-village to obtain more detailed information on farmers’ cultivation focus and demography from the sub-village head and the farmer group head. Information sessions about digital agricultural resources of a duration of about 2 hours were then provided in all the sampled sub-villages.
In the third stage, we randomly sampled respondents among the attendants of the information session. The three sampling criteria for respondents were as follows: (1) access to smartphone ownership, (2) rice cultivation of at least one plot and (3) younger than 65 years (70 years in case of farmer group head). In some information sessions, there were fewer than 16 suitable respondents. In such cases, we asked the farmer group head to provide us with contact details of further farmers who did not attend the information session.
The treatment was randomized at the village level and consisted of training on sustainable farming methods and the use of manual soil tests. Our RCT design consists of three groups - (1) control group, (2) treatment group 1 and (3) treatment group 2. The treatment arms are exclusive, allowing us to evaluate the effectiveness and cost-effectiveness of different extension intensities.
Treatment group 1 was offered a short training, similar in structure and length to a typical training from extension officers (around 1 day). The training consisted of topics including the importance of soil nutrient management, the role of organic matter in the physical structure of soil, use of leaf colour chart, the principles of LEISA (low-external input sustainable agriculture) and digital resources that can provide farmers additional information on sustainable farming practices. It also included a practical session on how to prepare organic inputs. The training was jointly designed with the Agricultural Faculty of Gadjah Mada University (UGM) and trainers from P4S. P4S are self-help agricultural and rural training groups that are owned and managed by farmers. They exist in most districts in Indonesia and receive financial resources from the local government.
Treatment group 2 was offered a two-day training. The first day comprised the same training that was also offered to farmers in treatment group 1, whereas the second day focused on soil testing. The soil tests used were manual soil tests from the Indonesian Soil Research Institute (ISRI). These soil tests, which have been extensively tested and validated, help in testing the level of Nitrogen, Phosphorus, Potassium (NPK), Ph, and Organic Matter in the soil. The second day also focused on materials specifically designed to assist farmers’ decision-making after soil testing. This included training on how to use the online soil test calculator and how to complement the soil test results with the recommendations given during day 1 on sustainable soil fertility management. The farmers received IDR 50,000 (around 3.5 USD) for each day of training attendance to cover any transport costs and to compensate them for a part of potentially foregone earnings.
To avoid any biased nominations and biased answers during the interviews, none of the treatment groups were informed about the upcoming training before the survey was completed in that village. Based on our survey data, we observe that the share of respondents who attended info-sessions was similar across the two treatment groups and the control group (T2 – 58.96%, T1 – 58.42% and Control – 57.61%).

Randomization done in office by a computer.

The treatment was randomized at the village level.

Yes

69 villages.

1104

23 clusters per treatement arm.

The baseline data was also used to revise the power calculations as it provides key information on the means, standard deviations and intra-cluster correlation of key outcome variables. Our power calculations are for intention-to-treat effects. The MDEs are based on a 95% confidence interval and a power of 80%. The baseline data shows a mean usage of 170 kg Nitrogen per hectare, with a standard deviation of 129 kg and an intra-cluster correlation (ICC) of 0.154. Given our cluster size of 16 individuals per cluster and 69 clusters in total, we are powered to detect a minimum effect of 48.4 kg, or 0.375 standard deviations. With regard to yields per hectare, the baseline data show a mean yield of 4.64 tons per hectare, with a standard deviation of 2.57 tons and an Inter-Cluster Correlation of 0.174. The minimum detectable effect size is 1 ton, or 0.392 standard deviations. The reported minimum detectable effect sizes refer to a simple comparison, i.e. a simple regression without controls. In our analysis, we will be able to increase precision by including various controls, including the baseline outcome, respondents’ education and wealth.