School Based Programming Impact Evaluation - Jordan

The current National School Feeding Programme (NSFP) in Jordan distributes date-filled bars or high-protein biscuits to 456,000 students in 2,314 schools around the country, targeting poverty-pocket areas defined by the government.
In 2022 the World Food Programme (WFP) Country Office launched a new healthy meal intervention in collaboration with the Government of Jordan, also referred to as the community-based kitchen model. This new community-based kitchen model aims to deliver healthy meals using kitchen facilities run by community-based organisations (CBOs). 11 community kitchens were operationalized in the directorates of South Ghour, Ramtha, Mafraq, Northeast Badeya, Bsera, and Tafeela, Madaba, and Shouth Shooneh. The community-based kitchen model delivers a meal composed of a cheese pastry, one fruit and one vegetable over four days, followed by one day of date bars per week (while the NSFP provides date-filled bars or high-protein biscuits for five days a week). The community-based kitchen model seeks to provide 20-30% of children's daily caloric requirement while also improving children's dietary diversity by adding rarely consumed food groups in Jordan (fresh fruits and vegetables). Greater children’s dietary diversity can also be linked to positive effects on learning and attention span outcomes.
Under the community-based kitchen model, meals will be prepared in a kitchen facility run by a community-based organisation (CBO). Each kitchen is expected to employ 25-45 female workers, creating income and employment opportunities for women in deprived areas and potentially improving women's empowerment and bargaining position within the household.

2022-09-05

2023-06-30

Service delivery, Children’s outcomes, Worker and worker household outcomes

Service delivery (food) examines the outcomes related to the delivery of school meals (food delivery, food quality)
Children’s outcomes examine the outcomes of educational progression, attention span, standardized testing, cognitive ability, and dietary diversity
Worker and worker household outcomes examine the outcomes related to worker and household income and expenditure, dietary diversity, time-use, gender norms and attitudes, and psychosocial outcomes

The IE is planned to include 4,730 students in 473 schools and 530 women who have the opportunity to work in the 8 community kitchens. It will extend for 2 semesters starting in September 2022 until the end of the 2022/2023 academic year. The evaluation will incorporate two randomized assignments, one of schools to the two different school-feeding models, and one of eligible worker candidates to job offers.

First, to investigate the impacts on primary-school students the impact evaluation will compare children’s outcomes from 142 randomly selected schools that will receive meals through the community-based kitchen model and will compare it with the outcomes from children from 331 school schools randomly selected to continue receiving the date bars/high protein biscuit model as the current model. The 473 schools participating in the IE are selected within a 45-minute driving distance from the 8 newly established community kitchens which have never received healthy meals. (The different size of the treatment arms is due to the limited production capacity of the kitchens.)
Second, to investigate the impacts on kitchen workers of being hired at the community-based kitchens, the impact evaluation will compare outcomes across two randomised groups of eligible workers: 215 women who received a job offer to work in the kitchen and 315 women who did not receive a job offer to work in the kitchen. The 530 workers participating in the IE are female Jordanian and Syrian workers who apply and are eligible to work in one of the 8 community-based kitchens. In each kitchen, an open call was made to interested women to apply. All applications were scored against a set of defined criteria.

The sampling and randomization were conducted using Stata software. The randomized assignment of schools to one of the two feeding models was stratified at the kitchen catchment area level, the randomization of kitchen workers into receiving a job offer was stratified at the kitchen and worker experience level.

Schools, Workers

Yes

473 schools in 456 school clusters. A school cluster is composed by schools that are geographically proximate to each other (the distance between them is 50 meters or less).

4730 students, 530 workers

- Treatment: 1420 students in 142 schools, 215 workers in 8 kitchens
- Control: 3310 students in 331 schools, 315 workers in 8 kitchens

Component 1: Comparison of school feeding models We implemented power calculations for two main outcomes (among others), students' dietary diversity (using data from the Jordan National Nutrition Survey 2019) and for Math test scores (using Jordan PISA data for 2018). Power was set to 0.8 and the significance level to 0.05. The number of students included per cluster is equal to 10 and was chosen as the maximum number that would fit the budget envelope. Note that our power calculations are conservative since they do not consider efficiency improvements obtained by using control variables for which we currently do not have data for. With 318 treatment and 138 control school clusters, and 10 students per cluster interviewed, the minimum detectable effect (MDE) for child dietary diversity is 0.31 additional food groups, from a mean of 4.79 food groups out of 10 measured. The community-based kitchen model provides three food groups (dairy, fruit, and vegetable) that the date bar/high protein biscuit model does not provide. Since only 65 % consume dairy, 32 % consume fruit, and 56 % consume vegetables among the children in the lowest wealth quintile of the settled Jordanian population, increasing consumption of these three food groups to 90 % on average (i.e., accounting for absences, or not all students receiving meals which could preclude 100 %) would yield an expected effect of 1.17 food groups. The expected effect is thus larger than the MDE. On the other hand, we may not be fully powered to detect impacts on the math score as measured by the PISA test, with a MDE of 0.16 SD, at least without efficiency improvements. Most other evaluations of school-feeding programs that compare the introduction of school feeding versus no school feeding – rather than different modalities – have found smaller impacts on math scores (e.g., 0.09 SD over 5 years in Chakraborty and Jayaraman (2019), 0.147 SD over 2 years in Aurino et al. (2018)). Component 2: Employment in community-based kitchens Randomisation for the comparison of workers will be done at the worker level based on the list of eligible workers provided by the implementing partners. The randomisation will be stratified by kitchen and the experience of workers in commercial kitchens. Each hired and non-hired worker will be surveyed for baseline and endline surveys, and for high-frequency surveys in between. With this given sample size for the randomised trial, we implemented power calculations for two main outcomes (among others), household income (using mVAM data from the WFP Jordan Country Office for poverty pockets) and for food security measured by the reduced coping strategy index (rCSI, using WFP Jordan Country Office community assessment data). The power was set to 0.8 and the significance level to 0.05. Note that these power calculations are conservative since they do not consider efficiency improvements obtained by using control variables for which we currently do not have data (e.g., stratification variables, baseline outcomes). With 315 individuals in the treatment and 215 individuals in the control group, we are powered to detect effects of 0.25 SD for a simple comparison of means at endline. For household income, this translates into an MDE of 270 USD and of (minus) 4.23 for the rCSI. The MDE of 270 USD compares to the expected monthly income from employment in the community-based kitchens of about 250 USD. The expected effect may therefore be smaller than this MDE. Therefore, in addition to baseline and endline surveys, we plan four shorter high-frequency surveys to be administered every two months, and whose repeated measurements of the same main outcomes will improve power. We do not have data on autocorrelation of these outcomes in Jordan, and therefore assume a medium autocorrelation of 0.3 for the six measurements. All outcomes will be included in the baseline and endline, and (to keep high-frequency surveys short) all outcomes will be included in only two of the four high-frequency rounds, for a total of four measurements per outcome. With these repeated measurements, the MDE reduces to about 0.17 SD, which translates into an increase of 180 USD for household income, and 2.82 for the rCSI.