Experimental Design
We are using a stepped-wedge design with randomized recruitment and treatment. Details are below.
Step 1: Sampling and recruitment
Excluding the pilot sites, the sampling frame consists of 1,124 facilities. After establishing this sampling frame, we randomly assigned facilities into two groups:
FREE group: 70 percent of facilities were o ered free Lightapp services
DISCOUNT group: 30 percent of facilities were o ered a 25 percent discount for Lightapp services
For both FREE and DISCOUNT groups, we fully subsidize the one-time installation of Lightapp
software devices and any subsequent maintenance or repair costs incurred during the duration of the
project.
We also cross-randomized the timing in which Lightapp's sales team will contact each facility to o ffer
its services. We randomly sorted the facilities in our sampling frame into an ordered list, and divided
facilities into 50 sequential tranches according to this randomized ordering. Each week, we released an
additional tranche of facilities to Lightapp's sales team. Information included company names, addresses,
and contact information. The release date initiated a rolling 30- or 40-day recruitment window, during
which Lightapp was encouraged to contact potential program participants and convert them into Lightapp
customers. After this tranche-speci fic recruitment window closed, Lightapp agreed not to recruit any
facility in the tranche that had not signed a contract for the remainder of the experiment.
We made three adjustments to the experimental design near the beginning of the recruitment phase.
We expanded the recruitment window from 30 days to 40 days after discovering that this change would
give more facilities sufficient time to gain internal approval and sign contracts with Lightapp. We also
increased the tranche size from 12 to 14 facilities due to the unexpectedly large number of facilities
Lightapp deemed ineligible. We maintained the same randomization order for determining the facilities in
each tranche. At the same time, we removed water and sewage treatment facilities from the experiment.
These facilities initially comprised approximately 20 percent of our sampling frame, yet they were very unlikely to have compressed air onsite. When we removed water
and sewage treatment facilities, we preserved the randomization order among all other facilities. We
had released eight tranches of facility names to Lightapp when we introduced these three experimental
design changes. We will perform sensitivity analysis on key results excluding facilities recruited prior to
instituting these changes.
Following the approach by Hussey and Hughes (2006) for power calculations in stepped wedge designs,
we pre-specifi ed a target sample size of 100 Lightapp participants. Due to funding constraints, we made
this target a ceiling, so recruitment would end after 100 facilities signed participation contracts. Since
Lightapp successfully recruited 100 customers before we released all facilities in the sampling frame, we
hit this ceiling and stopped recruitment. At this time, there were 117 uncontacted facilities remaining.
Step 2: Installation and Validation
After the end of the recruitment window, Lightapp and the other installers have 80 days to install the
EMS software in the new participating facilities. This process requires connecting the Lightapp system
to air compressors. A group of installation subcontractors perform these installations. If the lead installer, Ingersoll
Rand, deems an installation more expensive than authorized in their contract, Lightapp or another
subcontractor, Compressor IQ, performs the installation. At the time of registering this pre-analysis
plan, two individuals from Ingersoll Rand had overseen every installation, ensuring that installations
were performed correctly and consistently.
After installation is complete, Lightapp validates the data by comparing readings from the newly
installed meters to the PG&E or SCE electricity meter readings. Lightapp explores and addresses any
data discrepancies.
Step 3: Baseline data collection
Following meter installation and validation, each facility in the project enters into a baseline data
collection period of no less than 90 days. During this time, no one at the participating facility has
access to the Lightapp software platform or receives any performance reports or alerts. This establishes
a "business as usual" compressed air energy usage baseline for each facility. Note that we will test for
the presence of Hawthorne eff ects.
To ensure that the facilities begin the treatment phase (Step 4) in the established randomization
order, we occasionally extend baseline periods for some facilities. If one facility experiences a delay in
treatment period commencement, Lightapp withholds data from all facilities later in the randomization
order. We plan to look for the existence of heterogeneous impacts with respect
to baseline period length.
Step 4: Treatment period data collection
After this baseline period is over, facilities begin receiving real-time Lightapp data. This treatment
period lasts 12 months. During this time, facility personnel have access to all EMS data, scheduled reports,
and noti cations with tips and recommended actions for improving compressor eciency. Noti cations
include performance summaries and alerts for energy use data anomalies that cannot be explained by
changes in production. Participants can also compare their Compressed Air System Efficiency (CAS)
scores, which reflect the energy required to produce 100 cubic feet per minute (CFM), to other sites in
the project. All data are anonymized, but users can sort based on the general industry. Companies with
multiple plants that have Lightapp access can directly compare detailed data across these sites. We limit
the number of facilities that can begin the treatment period to three facilities per week.