NEW UPDATE: Completed trials may now upload and register supplementary documents (e.g. null results reports, populated pre-analysis plans, or post-trial results reports) in the Post Trial section under Reports, Papers, & Other Materials.
Decomposing the Wedge: Mechanisms driving the gap between projected and realized returns from energy efficiency investments
Initial registration date
May 14, 2018
May 16, 2018 12:34 PM EDT
University of Illinois, Urbana-Champaign
Other Primary Investigator(s)
University of Illinois
Additional Trial Information
We will evaluate the effects of performance-based bonus pay on the quality of work done by contractors to improve building shell tightness in the Weatherization Assistance Program, the largest residential energy efficiency program in the United States. In almost all WAP jobs, the measures contractors are asked to perform are chosen based on engineering calculations and quality control is done through an inspection process. Because they are not compensated directly on the energy performance of the home, contractors lack an incentive to seek out and execute any additional cost effective measures that they find on site, such as air sealing. We will evaluate whether performance-based bonus pay can result in cost effective improvements in the quality and quantity of air-sealing performed by contractors. We will also examine the effects of performance incentives on work done by bonus-eligible contractors that is excluded from the bonus payment.
Christensen, Peter, Paul Francisco and Erica Myers. 2018. "Decomposing the Wedge: Mechanisms driving the gap between projected and realized returns from energy efficiency investments." AEA RCT Registry. May 16.
In this study we will evaluate the impact of financial incentives to contractors on air sealing results and overall energy efficiency improvements. This will be conducted in the state of Illinois due to strong connections with the state weatherization program. In Illinois, a formula is used that estimates the amount of air sealing expected based on a pre-retrofit air leakage test; the leakier the home, the greater the expected air leakage reductions. The housing sample will be jobs that are done outside of Chicago, which represents about half of the state weatherization program. Chicago-based jobs will be excluded because the agency that runs the Chicago program already compensates their contractors for air sealing based on measured airtightness outcomes, whereas the rest of the state compensates contractors using pre-set sums that are based on the degree of expected air leakage reductions. In this study some of these non-Chicago homes will be eligible for an additional incentive to contractors based on air leakage reduction (the “treatment” groups) and others will not (the “control” group which will only be eligible for the pre-set compensation).
Our intervention consists of bonus payments that are made to contractors who achieve CFM50 readings below the minimum required by the state, which is 10% above the “target” that is based on the starting value. (The 10% allowance by the state takes into account that some homes have characteristics that make achieving target unusually difficult.) All contractors are made aware of the bonus program through a series of program meetings preceding the program year and the magnitude of incentive payments are clearly printed on the work order that each contractor receives at the outset of a job. Our intervention consists of two different bonus regimes: a “high” payment ($1.00) and a “low” payment ($.40) per CFM50 reduction reductions beyond 10% above target. The control jobs will not be eligible for bonus payments. The selection of air sealing as the basis for our study stems from its importance in delivered energy savings. Air sealing is deemed to be cost-effective in all homes, with the assumption that air sealing is permanent and therefore provides savings throughout the life of the home. As such, every home receives air sealing. Additionally, air leakage sites can vary widely from house to house, and so there is more potential for the attention and skill of the contractor to have an impact on identifying and properly sealing the leaks. Third, the metric for assessing airtightness (a blower door test) has a finer resolution than metrics for other common measures such as insulation, providing an opportunity to compensate based on a wide range of possible outcomes. The blower door test assesses the volumetric flow rate of air from the home when the home is depressurized by a set amount (50 Pascals [Pa]). The leakier the home, the more air needs to be moved to cause that amount of depressurization. The unit of measurement from the blower door test is cubic feet per minute at 50 Pa (CFM50). It is standard practice to conduct blower door measurements before and after a home is weatherized. These measurements are already required for WAP homes. Lower CFM50 values indicate that the home is better air-sealed, and thus well suited to retain heat.
Intervention Start Date
Intervention End Date
Primary Outcomes (end points)
CFM50 reductions, Reductions in energy consumption as assessed on utility bills, Quantity and type of findings and deficiencies tabulated from post-weatherization inspection reports
Primary Outcomes (explanation)
Secondary Outcomes (end points)
CFM50 reductions at concurrent but ineligible jobs, CFM50 reductions as a function of time after intervention (learning)
Secondary Outcomes (explanation)
Our experiment offers bonus payments to contractors who achieve CFM50 readings below the minimum required by the state, which is 10% above the “target” that is based on the starting value. (The 10% allowance by the state takes into account that some homes have characteristics that make achieving target unusually difficult.) We will have two different treatment arms, one with a “high” payment ($1.00) and one with a “low” payment ($.40) per CFM50 reduction reductions beyond 10% above target. The control jobs will not be eligible for bonus payments.
In addition to the data from WeatherWorks and utilities, we will also be tracking information directly from agencies on the inspector reports. Each month, a representative from each weatherization agency will be asked to report on the number of contractor call-backs, and what the call-backs were for. We will be merging the information with data in WeatherWorks based on the job number.
Experimental Design Details
When a new job is initiated in the WeatherWorks system, it will automatically be assigned to the “high” payment treatment, “low” payment treatment, or control. The overall aim of the project is to have group populations of 25%, 25%, 50%, respectively. Randomization is done through custom software built into the program that generates the work orders. The state of Illinois expects to weatherize approximately 1500 homes outside of Cook County in program year 2018. 1200 of these will be done on single family homes for which we have utility data. Therefore, we expected roughly 300 jobs for each of our 2 treatments and 600 for the control group. If a job is assigned to one of the two treatment groups, there will be a message printed on the front of the work order indicating that it is eligible for a $1.00 or $.40/ CFM50 bonus, how that bonus will be calculated, and how to receive payment.
Was the treatment clustered?
Sample size: planned number of clusters
Sample size: planned number of observations
Sample size (or number of clusters) by treatment arms
300 “high” payment treatment, 300 “low” payment treatment, 600 control
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We utilize prior utility and air sealing data from the Weatherization program to identify the sample size requirements for a statistical power. In order to address autocorrelation in monthly energy consumption data, we simulate the minimum detectable effect for our experiment, clustering our standard errors at the house-level. The 3 tables below display the results of these simulations for: (1) Total electricity consumption (electric + gas), gas consumption, and CFM Reductions. All tests presented below utilize 1,000 simulated regressions and assume 90% test power, and a total sample size of 1200 (note: 90% test power provides more conservative estimates than standard power 80% power values used). Each row represents a number of treated jobs. Each column represents a percentage point effect. The numbers in the cells are the p-value assuming our level of power. We note that there have been multiple changes in the funding restrictions and air sealing targets in the weatherization program in the past few years, so these power calculations may not represent the variance in these outcomes in 2017-2018.
Our power calculations suggest that for a sample of 600 treated and 600 control jobs, the MDE for gas and electricity consumption is 4-5 percentage points, the MDE for gas consumption is 3-4 percentage points, and the MDE for CFM reductions is 6-7 percentage points. For a sample of 300 treated (Low/High Groups) and 600 control jobs, the MDE for gas and electricity consumption is 5-6 percentage points, the MDE for gas consumption is 4-5 percentage points, and the MDE for CFM reductions is 7-8 percentage points.
INSTITUTIONAL REVIEW BOARDS (IRBs)
University of Illinois at Urbana-Champaign Institutional Review Board (IRB)
IRB Approval Date
IRB Approval Number
Post Trial Information
Is the intervention completed?
Is data collection complete?