We implemented our study with a medium-sized utility company (“the Utility”) in the north- eastern United States and a third-party contractor (the “Implementer”) that implements standard home energy reports. During the winter of 2015, the Implementer mailed four home energy reports on an opt-out basis to 10,000 randomly-selected Utility residential natural gas consumers. The Utility plans to continue these same customers on a home energy report program in winter 2016. In the same envelope as the final report of 2015, we included an incentive-compatible multiple price list (MPL) survey that allows us to measure each household’s willingness-to-pay (WTP) for the next winter of home energy reports and construct a demand curve for continued reports.
Aside from simply measuring the demand curve, our design also involved several useful features. First, out MPL questions were structured such that they allow recipients to reveal positive or negative WTP. This is important because a small number of consumers opt out of home energy reports, implying that they dislike them enough to bear the time costs of opting out. Second, to test the channels through which the reports might affect willingness-to-pay, we randomly assigned households to three different survey versions. One cued the reports’ social comparison feature, the second cued the environmental benefits of energy conservation, and the the third was a control.
Third, we carefully designed the survey to be able to address non-response. This is important because the intervention is delivered on an opt-out basis, meaning that we are interested in welfare effects across the entire 10,000-household population receiving the reports, but many recipients will not respond to our MPL survey. The direction and magnitude of non-response bias is an empirical question: non-responders might be more likely to have zero or negative WTP (this is very likely the case for people who do not open the home energy reports and thus never see the survey), but some might have positive WTP but a high cost of time for survey response. To address this, we carried out “intensive follow-up” (DiNardo, McCrary, and Sanbonmatsu 2006) by re-surveying a randomly-selected 2/3 of the original survey population. Comparing average WTP in the base and intensive follow-up groups suggests the direction of non-response bias, and the ratio of the WTP difference to the change in response rates suggests the magnitude. Thus, we can evaluate welfare effects within the sample of respondents and extrapolate (under functional form assumptions) to the entire treated population.
With the basic empirical results in hand, we can carry out a full welfare evaluation of continuing the home energy report program. The welfare effects of continuing the program are the sum of effects on consumer welfare (the area under the demand curve), minus the cost of mailing the reports, plus the reduction in uninternalized energy use externalities. We compare this to the more traditional cost effectiveness metric, which compares the program implementation cost to the social cost of energy. By ignoring consumer welfare effects, the traditional metric could generate very different results.
One potential criticism of our approach might be that it “takes revealed preference too seriously”: if one motivation for the program is that consumers are poorly informed about energy use, why should we assume that their valuations are well-informed? Conceptually, it is important to re- member that WTP is assessed after recipients are experienced with the intervention, so they should be capable of valuing it. Our empirical data also allow us to provide additional evidence. One specific concern might be that consumers know the costs they incur in conserving energy, but they do not know the benefits because it is difficult to infer financial gains of particular energy conservation actions. (“How much did we save last month because we turned the lights off more?”) To test this, we elicit beliefs over the average energy cost savings induced by the intervention and compare that to the true empirical estimates. A second and more conceptually challenging concern is that consumers might tend to be optimistically biased about their energy use compared to neighbors, and willingness to pay is low because the intervention reveals the truth. As in Brunnermeier and Parker (2005) and Oster, Shoulson, and Dorsey (2013), there is then some question about whether welfare analysis should respect or ignore a reduction in consumer surplus from eliminating optimism bias. To calibrate robustness checks in the welfare analysis, we ask report recipients whether they were initially positively or negatively surprised by the social comparison information, and we can adjust welfare estimates to account for correlation between low WTP and initial overoptimism.