All Models Are Wrong, but Is Mine Useful? Evaluating Behavioral Explanations and Interventions for the Energy Efficiency Gap

The intervention involves randomization into treatments that represent different combinations of information about the energy efficiency of the smart power strip, a discount, and feedback. This information includes information about the energy cost savings and reduced emissions associated with a smart power strip. In the discount treatment, prices for the smart power strip are framed as discounted, but the final prices offered are the same in all treatments. Personalized feedback is provided to individuals on the potential deviation of their smart power strip demand from a model-based prediction of their optimal demand after being provided information. Feedback involves framing the power strip as a bundle of goods that enable cost savings and emissions reduction consistent with the theoretical basis for the prediction. A rough, illustrative example of this feedback is as follows: According to the conclusion that follows from the statements summarizing your previous responses, your RWTP should be $X, or $Y higher. $X is the amount you were willing to pay for a carbon offset that reduces your emissions by the same amount the smart power strip does.

The guidance of Banerjee et al. (2020) for developing parsimonious pre-analysis plans was followed in this pre-registration.

2024-11-20

2024-12-31

1. Demand for the smart power strip. This is operationalized as the relative willingness-to-pay for the smart power strip (the energy-efficient good) when the alternative is a traditional power strip.
2. Deviation of smart power strip demand from predicted demand. Demand is predicted by an economic model of behavior based on Lancaster’s characteristics model (1966). It uses respondents’ beliefs and preferences as inputs. In this model, a product is viewed as a bundle of characteristics, and preferences for the product are linearly related to preferences for product characteristics (additive preferences).

1. Preferences for information treatment (self-reported ranking of preferred information treatments)
2. Preferences for feedback (self-reported)

The experiment is administered via an online survey to a sample of respondents who reside in the United States. Primary outcomes include demand for an energy-efficient good and behavioral deviations from theoretical predictions of demand. Treatments involve providing information about energy efficiency, financial incentives, and model-augmented feedback on respondent decision-making. This feedback uses respondents’ stated preferences, beliefs, and previous choices as inputs. See the complete list of treatments in "Sample Size" section.

Treatments are administered within- and between-subjects. Three measures of the primary outcomes are collected for each respondent: a baseline measure before information, after information, and after feedback.

[Research Question 1a] Are individuals making decisions consistent with their own stated preferences when provided with information about an energy-efficient good?
• Analysis 1: Test for deviations from predicted demand given respondents’ stated preferences by comparing incentivized smart power strip demand with predicted demand after combined information is provided (conditions 1 and 2 above).

• [Research Question 1b] What respondent characteristics are associated with behavioral deviations from theoretical predictions?
o Analysis: Identify predictors of deviations from predicted demand after combined info (conditions 1 and 2) is provided by examining the relationship between the type of deviation — a function of the magnitude and sign — and the following covariates:
 - baseline demand,
 - *measure of pro-environmental behavior/preferences,
 - *hesitance in paying above the market price for a power strip,
 - *agreement with the theoretical basis for predicted demand: whether respondents see the power strip as a means to reduce emissions or save money,
 - reference price for a traditional power strip,
 - valuing simplicity and convenience in power strip,
 - likelihood of using the smart power strip’s features,
 - attention to information,
 - focus on the total price of one power strip or discount instead of the relative price between power strips,
 - deliberativeness, and
 -numeracy.
Cluster analysis will be used to identify deviation types. A joint hypothesis test will be conducted on the relationship between deviation type and this set of covariates. Individual hypothesis tests with the asterisked covariates above will also be conducted.

[Research Question 2a] Can personalized feedback on an individual’s deviation from predicted optimal behavior reduce inconsistencies with their stated preferences and increase demand for energy efficiency?
• Analysis 1: Compare deviations from predicted demand between respondents in the combined info treatment with feedback and those in the combined info treatment without feedback (conditions 1 and 2).
• Analysis 2: Compare smart power strip demand between respondents in the combined info treatment with feedback and those in the combined info treatment without feedback (conditions 1 and 2 above)

• [Research Question 2b] Is there demand for this type of feedback?
o Analysis: Compare stated preferences for feedback (all respondents).

[Research Question 3a] Is imperfect information about the cost savings or environmental impact associated with energy use a key behavioral barrier to increased demand for energy-efficient technology? This question assesses the instrumental value of providing information about the private or public benefits of energy-efficient purchases.
• Analysis 1: Compare smart power strip demand between respondents in each of the following information treatments and those in no info: cost info alone, emissions info alone, combined info (conditions 2 through 5 above)
• Analysis 2: Compare smart power strip demand between respondents in combined info treatment and those in cost info alone (conditions 2 and 6 above)

• [Research Question 3b] Is there demand for information about energy efficiency? This question assesses the intrinsic value of information about the private or public benefits of energy-efficient purchases.
o Analysis: Compare stated preferences (self-reported ranking) for different types of information (all respondents).

[Research Question 4] Do financial incentives for energy-efficient goods decrease the effectiveness of information?
• Analysis 1: Compare smart power strip demand between respondents in the combined info treatment with a discount and those in the combined info treatment without a discount (conditions 2 and 6 above)
• Analysis 2: Compare deviations between predicted demand between respondents in the combined info treatment with a discount and those in the combined info treatment without a discount (conditions 2 and 6 above)

Inference criteria:
Distributional differences in the primary outcomes in confirmatory analyses will be tested using non-parametric or semi-parametric methods. This approach is motivated by the expectation that assumptions of standard parametric tests will be violated, complicating the interpretation of treatment effects. Specifically, substantial heterogeneity in treatment responses (e.g. negative, positive, and zero effects), heteroskedasticity, and non-normality are expected due to likely multi-modality in the outcome variable. Multiple hypothesis testing procedures will be applied to confirmatory analyses. All analyses specified are confirmatory, unless listed otherwise.

Results of parametric means tests, along with other summary statistics, will be presented descriptively. However, given the rationale for using non-parametric or semi-parametric methods above, estimates of the average treatment effect are not expected to be particularly informative.

Covariate adjustment:
Statistical models for confirmatory analyses will be adjusted by covariates predictive of treatment effects to improve precision and address potential covariate imbalances across experimental conditions due to attrition or chance. If significant covariate imbalances are detected across conditions, residualized models will also be considered for model parsimony. Unadjusted analyses will be used as robustness checks.

Data-driven variable selection methods will be used to select the covariates used for adjustment. Questions in this survey experiment were designed and included based on theory of what would be predictive of treatment effects. Therefore, candidate variables for the variable selection method will include variables created from these questions, along with metadata, timing data, location data, and other measures of comprehension, engagement, and attention. Contingent on the number of variables not being substantially larger than the number of variables identified by the variable selection method, the variables under the “High Priority” heading below will be forced to be included in the model for theoretical reasons if feasible (e.g. the “amelioration set” in post-double selection lasso; see Cilliers, Elashmawy, and McKenzie 2024). At an absolute minimum, the baseline outcome will be included.

High Priority
• Baseline outcome
• Reference price for a traditional power strip and certainty of reference price
• Measure(s) of pro-environmental behavior/preferences
• Key variables used directly in the theoretical model to calculate predicted demand:
o Likelihood of using the smart power strip’s features
o Attributes of a power strip selected as factoring into decision-making.

When the number of candidate variables does not affect the likelihood that a variable is selected (Cilliers, Elashmawy, and McKenzie 2024), candidate variables for the variable selection method will include other variables beyond the High Priority variables. Otherwise, theory will determine which variables are candidates for selection. When a more theoretically important variable (variable with a higher priority level) is collinear with a less theoretically important variable, the more theoretically important variable will be included in place of the less theoretically important one as a candidate. Models with different sets of candidate variables will be presented in a “populated analysis plan” or an appendix (see Banerjee et al. 2020).

Only measures of these covariates orthogonal to treatment will be included if there is evidence that treatment influenced participants’ answers to the questions. When it is not possible to obtain orthogonal measures, on an exploratory basis, correlations will be examined between the covariates or potential transformations of them and individual changes in demand after treatment (information, information and incentives, feedback).

Exploratory analyses:
Bunching at certain points in the distribution of the outcome variable will be explored as evidence of anchoring to numbers presented in the information treatments — in terms of total and relative price — and respondents’ reference price.

Fit between the theoretical model used to predict demand and the data will be examined by looking at how well respondents’ demand for product characteristics (e.g. emissions impact) predicts their demand for the whole product (smart power strip demand).

Chat GPT will be used to analyze open-ended responses. The most important open-ended question asks people to explain their potential inconsistencies or decision-making in this study.

Heterogeneity Analyses:
Heterogeneous treatment effects will be examined on an exploratory basis. These examinations will involve identifying predictors of deviations from anticipated demand and predictors of responsiveness to behavioral interventions (information, information in the presence of incentives, and feedback). Respondents will be classified into different types when possible. Discrete choice models will be additionally used to examine heterogeneity in treatment effects based on price and the size of the discount on the smart power strip.

References:
1. Banerjee, Abhijit, Esther Duflo, Amy Finkelstein, Lawrence F. Katz, Benjamin A. Olken, and Anja Sautmann. 2020. "In Praise of Moderation: Suggestions for the Scope and Use of Pre-Analysis Plans for RCTs in Economics." National Bureau of Economic Research Working Paper Series, no. 26993. April 2020. https://doi.org/10.3386/w26993.
2. Cilliers, Jacobus, Nour Ahmed Mohamed Abdelwahab Elashmawy, and David J. McKenzie. 2024. Using Post-Double Selection Lasso in Field Experiments. Policy Research Working Paper; Impact Evaluation Series; Prosperity. Washington, D.C.: World Bank Group. http://documents.worldbank.org/curated/en/099721209262431433/IDU19a34656a19be714ca21ac1f1c8c30a8b071d.
3. Lancaster, Kelvin J. 1966. "A New Approach to Consumer Theory." Journal of Political Economy 74 (2): 132–57. https://doi.org/10.1086/259131.

Simple randomization is conducted at the individual level by a computer.

Individual

No

N/A

~2423 respondents are planned to be recruited by Qualtrics in one wave of data collection. Exclusion criteria: Respondents must be at least 18 years of age to be eligible for the study. They must also complete the survey to be included in the study sample. Respondents will be excluded from the sample if they take longer than two hours to complete the thirty-minute study, fail attention checks, or for other data quality issues, such as providing gibberish or ambiguous responses, invalid email addresses, or invalid mail addresses. Data scrubs by Qualtrics also include checks for straightlining, speeding, bot detection, and duplicate responses. Data will be analyzed with each of the following included and excluded: influential and high leverage points, respondents who indicated they were confused or experienced technical errors, and respondents with censored valuations for the smart power strip or its attributes.

Experimental conditions and sample size per condition:
1. Combined information + no feedback: ~500 people. “Combined information” refers to providing information about both the energy cost savings and the reduced emissions associated with use of the smart power strip.
2. Combined information + feedback: ~500 people.
3. No information + feedback: ~350 people
4. Cost savings information + feedback: ~350 people
5. Emissions impact information + feedback: ~350 people
6. Combined information + discount + feedback: ~350 people

Note that the questions and flow of the experiment in the first two treatment arms listed above are the same until feedback is provided to respondents. This implies that there are ~1000 people (500 + 500) to estimate the effect of combined information. The sample sizes for the first two treatment arms are larger than those of the other arms because they are used to test research questions of most interest.