Experimenting with Labs

The Laboratory Efficiency Assessment Framework (LEAF) is an environmental accreditation scheme designed to reduce the environmental impact of university research labs. It is key to achieve Imperial College London’s target of becoming a sustainable net-zero carbon institution by 2040. This paper analyses its effect on saved costs, including energy, and carbon emissions leveraging a randomized controlled trial (RCT), the gold standard methodology on the Maryland Scale of impact evaluation. By randomizing the LEAF programme roll-out to a select number of laboratory buildings, we isolate its causal treatment effect from potential confounders. Further, we analyze whether a top down vis-a-vis a bottom-up implementation is more effective in increasing uptake. By providing rigorous evidence on the program’s average treatment effect of 3,000 pounds total cost and 8,5 tons of CO2 savings per lab space, this study seeks to inform potential replications in other higher education institutions in Great Britain and beyond.

2022-09-30

2022-12-31

The data collected using qualtrics measures a number of variables which are aggregated to the two primary outcome variables, namely total cost and carbon savings per lab.

The variables include figures such as the number of waste bags collected per week and their average weight, the number of fume cupboards, their hours of daily operation and number of days used during the week, their face velocity, aperture and percentage time that the sash opens; the number of safety cabinets, their power demands measured in watts, their average hourly usage per day and week, and whether they are ducted; the number of PC monitors, their level of brightness and time until sleep mode sets in; the number of cold storage units, their age, their operating temperature, the average ambient temperature, whether their filters are clean and how often their doors are opened, etc. Further, as no two labs are the same and the type of equipment varies between labs and natural science disciplines, the categorization also includes an additional bespoke category where other potentially very energy-intensive laboratory equipment can be added, such as ovens or lasers for example.

The primary outcomes are calculated as difference between endline and baseline values for each building.

The LEAF program/data entry portal is calculating carbon and cost savings for each of seven categories (Waste, Fume Cupboards, Safety Cabinets, IT, Cold Storage, Equipment, Water). Given the complexity of the underlying parameters involved, please enquire the pre-specified, fixed formulas. Please note that they are the intellectual property of Martin Farley at University College London, the inventor of LEAF.

A secondary outcome variable that we measure is the electricity consumption at the building level. To do so, we clean the raw electricity consumption data compiled by the Sigma-software. We reduce its frequency from half-hourly measurements each day to monthly data. We compute compound annual growth rates from month to month.
In the light of error-prone measurements of faulty transformer-meters, we filter out values that exceed growth rates of 100 percent from one data-point to the next. Further, we aggregate the values from individual transformers to their respective buildings by reconstructing a system-inherent algorithm that assigns individual grid-transformers to buildings within Imperial College.

Unfortunately, we can only measure electricity consumption at the buildings level, as installing further sub-meters would be prohibitively costly (exceeding 1500 pounds per meter). At the point of the LEAF rollout we still decided against a roll out with POWBALL smart-meters that can measure electricity consumption at the plug-level. The plugs shut off at times of peak electricity prices, which could potentially disrupt complicated experiments conducted within the laboratories.

We randomize at the buildings level in order to minimize potential spillovers. Thus the buildings can be considered as randomization clusters.

Sample Size: As you can see from the randomization table in the annex, there are 23 buildings, which comprise a total of 1460 laboratory spaces associated with them.

We use the STATA randtreat-command to carry out the randomization into three treatment arms of equal size, namely two treated and one control.

You can find the resulting randomization in the annex.

We re-randomized increasing the integer seed until we obtained a randomization satisfying two conditions: (1) to be balanced, and (2) to not include any department into both of the two treated arms to implement clean treatments.

(1) Balance is verified with the balancing table with no stars. This indicates no significant difference between any of the treatments, hence balance. Primarily, the number of lab spaces is broadly comparable across the three treatment arms.

(2) The second condition results from our experimental design, specifically the exogenously varied rollout either via top-down instruction from the heads of department or via a bottom-up approach where laboratories can individually sign up themselves. Therefore, departments must not form part in both top-down and bottom-up treatments.

We obtained this desirable result setting the seed at seven, after iteratively increasing the integer value. The second condition precluded us from applying more elaborate approaches such as the minimum-maximum t-statistic out of 1000 draws described for example in Bruhn $&$ McKenzy (2009). We randomly assign the two misfits globally, that is among all three treatment arms.

We randomize at the buildings level in order to minimize potential spillovers. Thus the buildings can be considered as randomization clusters.

Yes

23

Roughly 1500 lab spaces in 23 buildings

LEAF programme rollout:
Bottom-Up 8 buildings, 417 laboratory spaces
Control 8 buildings, 476 laboratory spaces
Top-Down 7 buildings, 587 laboratory spaces

We are powered to detect a difference in means of 0.83 percent between the treatment and control groups with 23 buildings participating in total and three groups (two treatment with eight buildings each, one control with seven buildings). The power calculations \footnote{Power is computed with the G*Power application. The assumptions are: one tailed test; t tests family; statistical test: correlation point biserial model. (https://www.psychologie.hhu.de/arbeitsgruppen/allgemeine-psychologie-und-arbeitspsychologie/gpower)}. are based from the data obtained from a small-scale pilot LEAF rollout in 2020, in which 21 labs from the Natural Sciences, Engineering and Medicine faculties at Imperial College London participated. The combined monetary savings per lab totalled 5,296 pounds on average, with a corresponding standard deviation of 6319, resulting in an effect size of 0,83. The underlying assumptions here are an $\alpha$ parameter of $0.05$ and a $\beta$ of $0.2$, thus resulting in a high-power of $80$ percent. This strong effect size is desirable as it will allow us to detect LEAF's effect even in a small sample of treated laboratory units. The total savings are the sum of savings via waste reduction, more efficient use of fume cupboards, safety cabinets, IT, cold storage units, kits and other sources. The figures in the pilot were self-reported using a cost-savings calculator, for which the data was inserted using the Qualtrics software. Corresponding figures were also computed for carbon emissions reductions based on self-reported savings and implied carbon emissions factors.