# Experimenting with Labs

Last registered on November 17, 2022

## Pre-Trial

Trial Information

### General Information

Title
Experimenting with Labs
RCT ID
AEARCTR-0009105
Initial registration date
November 11, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
November 17, 2022, 3:40 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Region

### Primary Investigator

Affiliation
DIW Berlin / Berlin School of Economics / LSE Grantham

### Other Primary Investigator(s)

PI Affiliation
Imperial College London

Status
In development
Start date
2022-09-30
End date
2022-12-31
Secondary IDs
/
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
The Laboratory Efficiency Assessment Framework (LEAF) is an environmental accreditation scheme designed to reduce the environmental impact of university research labs. It is key to achieve Imperial College London’s target of becoming a sustainable net-zero carbon institution by 2040. This paper analyses its effect on saved costs, including energy, and carbon emissions leveraging a randomized controlled trial (RCT), the gold standard methodology on the Maryland Scale of impact evaluation. By randomizing the LEAF programme roll-out to a select number of laboratory buildings, we isolate its causal treatment effect from potential confounders. Further, we analyze whether a top down vis-a-vis a bottom-up implementation is more effective in increasing uptake. By providing rigorous evidence on the program’s average treatment effect of 3,000 pounds total cost and 8,5 tons of CO2 savings per lab space, this study seeks to inform potential replications in other higher education institutions in Great Britain and beyond.

### Registration Citation

Citation
Handler, Nils and Ralf Martin. 2022. "Experimenting with Labs." AEA RCT Registry. November 17. https://doi.org/10.1257/rct.9105-1.0

There is information in this trial unavailable to the public. Use the button below to request access.

Experimental Details

### Interventions

Intervention(s)
The Laboratory Efficiency Assessment Framework (LEAF) is an environmental accreditation scheme designed to reduce the environmental impact of university research labs. It is key to achieve Imperial College London’s target of becoming a sustainable net-zero carbon institution by 2040. This paper analyses its effect on saved costs, including energy, and carbon emissions leveraging a randomized controlled trial (RCT), the gold standard methodology on the Maryland Scale of impact evaluation. By randomizing the LEAF programme roll-out to a select number of laboratory buildings, we isolate its causal treatment effect from potential confounders. Further, we analyze whether a top down vis-a-vis a bottom-up implementation is more effective in increasing uptake. By providing rigorous evidence on the program’s average treatment effect of 3,000 pounds total cost and 8,5 tons of CO2 savings per lab space, this study seeks to inform potential replications in other higher education institutions in Great Britain and beyond.
Intervention Start Date
2022-09-30
Intervention End Date
2022-12-31

### Primary Outcomes

Primary Outcomes (end points)
The data collected using qualtrics measures a number of variables which are aggregated to the two primary outcome variables, namely total cost and carbon savings per lab.

The variables include figures such as the number of waste bags collected per week and their average weight, the number of fume cupboards, their hours of daily operation and number of days used during the week, their face velocity, aperture and percentage time that the sash opens; the number of safety cabinets, their power demands measured in watts, their average hourly usage per day and week, and whether they are ducted; the number of PC monitors, their level of brightness and time until sleep mode sets in; the number of cold storage units, their age, their operating temperature, the average ambient temperature, whether their filters are clean and how often their doors are opened, etc. Further, as no two labs are the same and the type of equipment varies between labs and natural science disciplines, the categorization also includes an additional bespoke category where other potentially very energy-intensive laboratory equipment can be added, such as ovens or lasers for example.

The primary outcomes are calculated as difference between endline and baseline values for each building.
Primary Outcomes (explanation)
The LEAF program/data entry portal is calculating carbon and cost savings for each of seven categories (Waste, Fume Cupboards, Safety Cabinets, IT, Cold Storage, Equipment, Water). Given the complexity of the underlying parameters involved, please enquire the pre-specified, fixed formulas. Please note that they are the intellectual property of Martin Farley at University College London, the inventor of LEAF.

### Secondary Outcomes

Secondary Outcomes (end points)
A secondary outcome variable that we measure is the electricity consumption at the building level. To do so, we clean the raw electricity consumption data compiled by the Sigma-software. We reduce its frequency from half-hourly measurements each day to monthly data. We compute compound annual growth rates from month to month.
In the light of error-prone measurements of faulty transformer-meters, we filter out values that exceed growth rates of 100 percent from one data-point to the next. Further, we aggregate the values from individual transformers to their respective buildings by reconstructing a system-inherent algorithm that assigns individual grid-transformers to buildings within Imperial College.
Secondary Outcomes (explanation)
Unfortunately, we can only measure electricity consumption at the buildings level, as installing further sub-meters would be prohibitively costly (exceeding 1500 pounds per meter). At the point of the LEAF rollout we still decided against a roll out with POWBALL smart-meters that can measure electricity consumption at the plug-level. The plugs shut off at times of peak electricity prices, which could potentially disrupt complicated experiments conducted within the laboratories.

### Experimental Design

Experimental Design
We randomize at the buildings level in order to minimize potential spillovers. Thus the buildings can be considered as randomization clusters.

Sample Size: As you can see from the randomization table in the annex, there are 23 buildings, which comprise a total of 1460 laboratory spaces associated with them.

We use the STATA randtreat-command to carry out the randomization into three treatment arms of equal size, namely two treated and one control.
Experimental Design Details
You can find the resulting randomization in the annex.
Randomization Method
We re-randomized increasing the integer seed until we obtained a randomization satisfying two conditions: (1) to be balanced, and (2) to not include any department into both of the two treated arms to implement clean treatments.

(1) Balance is verified with the balancing table with no stars. This indicates no significant difference between any of the treatments, hence balance. Primarily, the number of lab spaces is broadly comparable across the three treatment arms.

(2) The second condition results from our experimental design, specifically the exogenously varied rollout either via top-down instruction from the heads of department or via a bottom-up approach where laboratories can individually sign up themselves. Therefore, departments must not form part in both top-down and bottom-up treatments.

We obtained this desirable result setting the seed at seven, after iteratively increasing the integer value. The second condition precluded us from applying more elaborate approaches such as the minimum-maximum t-statistic out of 1000 draws described for example in Bruhn $&$ McKenzy (2009). We randomly assign the two misfits globally, that is among all three treatment arms.
Randomization Unit
We randomize at the buildings level in order to minimize potential spillovers. Thus the buildings can be considered as randomization clusters.
Was the treatment clustered?
Yes

### Experiment Characteristics

Sample size: planned number of clusters
23
Sample size: planned number of observations
Roughly 1500 lab spaces in 23 buildings
Sample size (or number of clusters) by treatment arms
LEAF programme rollout:
Bottom-Up 8 buildings, 417 laboratory spaces
Control 8 buildings, 476 laboratory spaces
Top-Down 7 buildings, 587 laboratory spaces
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We are powered to detect a difference in means of 0.83 percent between the treatment and control groups with 23 buildings participating in total and three groups (two treatment with eight buildings each, one control with seven buildings). The power calculations \footnote{Power is computed with the G*Power application. The assumptions are: one tailed test; t tests family; statistical test: correlation point biserial model. (https://www.psychologie.hhu.de/arbeitsgruppen/allgemeine-psychologie-und-arbeitspsychologie/gpower)}. are based from the data obtained from a small-scale pilot LEAF rollout in 2020, in which 21 labs from the Natural Sciences, Engineering and Medicine faculties at Imperial College London participated. The combined monetary savings per lab totalled 5,296 pounds on average, with a corresponding standard deviation of 6319, resulting in an effect size of 0,83. The underlying assumptions here are an $\alpha$ parameter of $0.05$ and a $\beta$ of $0.2$, thus resulting in a high-power of $80$ percent. This strong effect size is desirable as it will allow us to detect LEAF's effect even in a small sample of treated laboratory units. The total savings are the sum of savings via waste reduction, more efficient use of fume cupboards, safety cabinets, IT, cold storage units, kits and other sources. The figures in the pilot were self-reported using a cost-savings calculator, for which the data was inserted using the Qualtrics software. Corresponding figures were also computed for carbon emissions reductions based on self-reported savings and implied carbon emissions factors.
IRB

### Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

## Post-Trial

Post Trial Information

### Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

### Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

### Data Publication

Is public data available?
No

### Program Files

Program Files
Reports, Papers & Other Materials