The primary research question for the RCT was whether giving students solar lights would shift their study habits and, more importantly, improve academic performance. We prioritized undertaking the research in a rural location similar to other places in Sub-Saharan Africa where solar lanterns had successfully been sold at scale, while at the same time accounting for the risk of “contamination” through participants’ exposure to solar lanterns outside the research context. In addition, we had to recruit participants that could plausibly be motivated to use solar lanterns in their studies in order to improve academic performance. And we needed to introduce the solar lights in a manner similar to what such participants might encounter outside of a research setting (typically the sale of a light by a social enterprise that highlights its potential educational benefits).
We focused the research on students in grades 7 through 9—the last three grades of primary school in Zambia—for several reasons. First, children in earlier grades would likely have been too young to be able to answer the questions in our surveys. Second, our scoping research revealed that lower grades were generally not assigned much homework, making it less likely that improved lighting would influence studies and performance. Third, school officials pointed out that it is mostly grade 7 and beyond when children drop out of school altogether, so interventions that might improve performance and encourage ongoing enrollment might be particularly well targeted to those grades. Fourth, children in grades 7 and 9 take standardized national examinations. By all accounts, students across Zambia—as well as their parents and teachers—are well aware of the importance of these tests and take them seriously. We could safely assume that children would be quite motivated to use all tools at their disposal—including, potentially, solar lanterns—to improve performance. Moreover, the fact that the national examinations are standardized and graded equally between different schools and classrooms makes them an ideal way to measure academic performance outcomes in an RCT.
We carried out the RCT in 12 government-run primary schools randomly selected from a master file of all schools in Zimba District. Over 1,400 children in grades 7, 8 and 9 completed in-school surveys at the start of the school year in February 2016, as well as during national examinations season in October 2016. Not all children attend school every day but 80% of children who filled out the October survey also completed the earlier one in February. Through these surveys, we collected data on study habits and other factors that might influence educational outcomes. The student surveys took about one hour to complete. Following the first survey, a different team of researchers, who otherwise had no role in the study, conducted lotteries that served as the delivery mechanism for the RCT’s “treatments”, including the giveaway of over 200 solar lanterns to randomly selected children. These lotteries took place at the start of the second school term in May 2016, were designed to isolate the impact of solar lighting itself.
Although we collected the national examination scores directly from education officials, we nevertheless asked students to complete detailed surveys, for several reasons. First, we were interested in detecting impacts of solar lanterns on certain study habits irrespective of examination performance. In other words, the times of day that children study, study locations, study partners, and types of lights used for nighttime studies were additional outcomes of interest in our RCT. Second, having this additional data enabled us to control for additional background variables that could allow us to obtain more precise estimates of the impacts of solar lights on examination scores and study habits.
Third, the surveys were key to our additional research objective of examining the mechanisms through which solar lights are introduced and used. Simply handing out lights does not provide information about whether and how students actually use them nor how study habits relate to examination scores. Moreover, making the surveys broad in scope was itself a tool through which we blinded participants to the study’s goals. We did this in order to not have students feel like there were “right answers” when it came to reporting study habits, the use of solar lanterns and, most importantly, the relationship between the two.
Finally, the rich survey dataset covering students’ daily lives is what enabled us to gain important insights into the broader educational environment into which solar lanterns are deployed. That way we could address not just whether we detected academic impacts of solar lanterns but also why we might observe the relevant results. Ultimately, this broader examination of the relationships between household energy access, poverty, and children’s academic opportunities is key to understanding how improved household lighting might translate to improved educational outcomes.
TREATMENT IMPLEMENTATION DETAILS:
The priority for our experiment’s implementation was ensuring that any given student within a grade at a particular school had an equal and random chance of being “treated”. To do that, we conducted a series of 36 lotteries – one for each grade level at all 12 schools. Since not all children attend school every day, only those that both took the baseline survey and were present on the day of the lottery several months later were eligible to participate. Therefore, children that missed school during either of the two surveys or the lottery are left out of the RCT analysis.
We took precautions to ensure that the lotteries were not perceived as solar lantern giveaways. Instead, the goal was for school officials, teachers, students, parents, and even some of our own researchers to perceive the lottery as an exercise intended to thank children for participating in a general study of Zambian schooling. We therefore also gave away three other prizes or “treatments”: backpacks, battery-powered alarm clocks, and soap. The “control” students received a candy. There was no general emphasis on the lanterns; they were just one of several prizes that students were eligible to win thanks to completing broad surveys about their daily lives.
An advantage of the multiple prize design was that it enabled us to isolate the solar-powered lighting attribute of our target intervention and thereby control for any income effects that might have been triggered by the receipt of a solar lantern. The other prizes we handed out were worth approximately the same as the sales price of a solar lantern and, in some cases, could also be considered helpful for education. We could therefore study the impact of receiving a lighting product, distinct from the impact of receiving something that is worth approximately USD 10 and that could potentially be monetized and repurposed. An additional advantage was that the other three prizes were familiar items that children would have been aware are valuable and not normally given away. They were therefore useful to signal the value of solar lanterns to children that might not have previously been exposed to solar lights (or may possibly have viewed them as free goods that charities hand out). We note that we did not see evidence that the prizes were monetized, repurposed, traded or otherwise not used for their intended purpose. During the endline survey, 93% of backpack recipients and 87% of both solar lantern and clock recipients reported still owning the prizes they won in the lottery. Importantly, very few children reported having sold or given away their prize. We therefore believe that we were successful in implementing our research design with the goal of studying the impact of having received a product with solar-powered lighting attributes.
This approach also enabled us to deliver the lights in an educational setting and encourage students to use them, but at the same time hopefully avoid giving cues about any particular impacts we “wanted,” which could have resulted in data bias (or even potentially favorable treatment by teachers towards certain students). Another benefit of awarding multiple prizes was that it enabled us to have a consistent approach with the three “pure control” schools where no students received lights. By also awarding prizes to those schools’ students, we avoided the political and practical risks of control schools being perceived as different from the 9 “treated” ones.
Nevertheless, because solar lanterns were likely not as familiar to the children as the other prizes, we did take limited additional measures when handing them out. Lantern winners received an “information card” – a brief, easy-to-understand sheet (printed on high-quality cardstock) that consisted of instructions on proper use, emphasized that the lantern could be helpful for studying, and provided a number to call in case it stopped working (see Appendix 6). Research staff also demonstrated how to use the light and delivered the same messages from the information card verbally when giving a child a lantern. In this regard, we mimicked what a vendor might do when selling a solar lantern while still not drawing too much attention to the lights somehow being more special than the other prizes. Finally, we sent one research team member back to the schools on four different occasions to check whether students who had won a light still owned it and were using it. This was presented to participants as a routine part of warranty support for the lanterns by their distributor.