Crowdflex utilisation payments mini-trial 1 invitation randomised encouragement pre-analysis plan

Last registered on May 13, 2024
View Trial History

Pre-Trial

Trial Information

General Information

Title
Crowdflex utilisation payments mini-trial 1 invitation randomised encouragement pre-analysis plan
RCT ID
AEARCTR-0013541
Initial registration date
May 07, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
May 13, 2024, 12:23 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name
Andrew Schein
Affiliation
Centre for Net Zero
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Robert Metcalfe
PI Affiliation
USC, Centre for Net Zero
Contact Investigator
PI Name
Joe Grainger
PI Affiliation
Centre for Net Zero
Contact Investigator
PI Name
Louise Bernard
PI Affiliation
Centre for Net Zero
Contact Investigator

Additional Trial Information

Status
In development
Start date
2024-05-02
End date
2025-12-31
Keywords
Environment & Energy
Additional Keywords
flexibility
JEL code(s)
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
will undertake a series of mini-trials as part of CrowdFlex Beta, a UKRI and Ofgem funded programme of work. The trials will encourage residential energy customers to turn down or up their energy consumption during flexibility events, as well as making their appliances available to Flexibility Service Providers to remotely control for flexibility both inside and outside of specific events. The first mini-trial will take place from May 2nd to July 2024. This mini-trial will involve a Randomised Control Trial (RCT).
Randomisation will involve two steps:
Invitation to the trial (Randomised Encouragement Design – the subject of this pre-analysis plan):
Invitation randomisation: We will randomise invitations from the pool of eligible customers for the mini-trial. The pool of eligible customers will be refreshed before each mini-trial. The participants for the later mini-trial will be a mix of the previous mini-trial participants, customers from the first eligible pool and customers from the refreshed pool.
Within the list of customers that have signed up for the minitrials (Randomised Control Trial):
Wait-list randomization: We will waitlist some customers who opt into the CrowdFlex programme, leaving them as a ‘control’ group until the final event of the first mini-trial.
Individual-level randomisation within a mini-trial into 5 treatment arms: low, medium, high, low to high and finally high to low incentives during flexibility events
The analysis of the RCT design is the subject of a formal trial protocol as part of CrowdFlex Beta and is registered at [https://www.socialscienceregistry.org/trials/13539]. This pre-analysis plan focuses on the randomised encouragement design.
External Link(s)

Registration Citation

Citation
Bernard, Louise et al. 2024. "Crowdflex utilisation payments mini-trial 1 invitation randomised encouragement pre-analysis plan." AEA RCT Registry. May 13. https://doi.org/10.1257/rct.13541-1.0
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
See pre-analysis plan.
Intervention Start Date
2024-05-02
Intervention End Date
2025-12-31

Primary Outcomes

Primary Outcomes (end points)
Electricity consumption ( (kWh) per half-hour) during the event window
Primary Outcomes (explanation)
Our primary outcome of interest is customers’ electricity consumption during the event window. This will primarily be measured using customers’ smart meters, which record electricity consumption in 30-minute intervals, though asset level data may also be utilised where available.

Secondary Outcomes

Secondary Outcomes (end points)
Electricity consumption 24 hrs before/after the event window (event window specified in the event timetable)
Secondary Outcomes (explanation)
Our secondary outcome of interest is customers’ electricity consumption outside the event window (24 hours before and after the event). This is to understand whether customers shift their energy consumption to half hours outside the event window versus whether the energy demand is created or destroyed. This will also be measured using smart meters and asset data where available.

Experimental Design

Experimental Design
See pre-analysis plan.
Experimental Design Details
Not available
Randomization Method
Randomization done in office by a computer.
Randomization Unit
Household
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
1.46m households
Sample size: planned number of observations
As discussed in https://www.socialscienceregistry.org/trials/13539, we cluster half-hourly consumption observations within the household. There are 31 events comprising 107 half-hours.
Sample size (or number of clusters) by treatment arms
1.46m households
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Depending on the take-up rate, we will have a treatment group (customers that received the invitation email) of 192,000 to 480,000 customers, and a control group of 1,268,000 to 980,000 customers corresponding to a take-up rate of 25% to 10% respectively. We assume that there is no sign-up among non-invited customers. We use data from a random sample of 16,000 OE smart meter domestic customers to estimate the Minimum Detectable Effect Size for our trial. To do so, we simulate the impact of a placebo treatment (being invited to join CrowdFlex) on consumption data for peak hours over 30 events. We use the standard errors from our placebo regressions and predict their change for higher sample sizes. We then use the rule of 2.8 to estimate the effect we could capture to have a power of 80% and 5% significance level. We find that the MDES on sign-up invitation (the ITT) would be between 0.0006 kWh (if the sign-up rate is 25%, resulting in 192,000 invited customers) or 0.0004 kWh (if the sign-up rate is 10%, resulting in 480,000 invited customers). Estimates from Saving Sessions 2022-2023 (an implementation of the Demand Flexibility Service by Octopus Energy), from Jacob at al. (2023) Table AT.4 indicate potential effect sizes of: -0.1 kWh per half-hour from signing up -0.15 kWh per half-hour from opting in We anticipate a smaller effect as incentives are smaller, closer to 0.08 kWh per sign-up. Assuming this 0.08 kWh effect per sign-up, the ITT effect, diluted by non-sign-up, would be: If 10%: 10% * 0.08 = 0.008 kWh per invitation If 25%: 25% * 0.08 = 0.02 kWh per invitation. Our MDES estimates are comfortably below these expected ITT effects.
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information