Energy Demand Flexibility at Scale: Late Invitations

See analysis plan.

Centre for Net Zero (CNZ), in partnership with Octopus Energy, will undertake a series of field trials during Octopus Energy’s second Saving Sessions (winter 2023 - 2024). The trials will rigorously evaluate how and why the program itself works to reduce customers’ energy consumption during hours of high grid constraint.

This sub-trial is part of a larger randomized encouragement design centered around saving sessions 2. In the main trial, we randomized 2.7 millions customers into 2 groups: 2.5m customers where invited to join Octoplus, the membership programme that allows to sign up for saving sessions and being rewarded when reducing electricity consumption during events, 120k customers in the control group that did not receive the invitation but were still free to participate. See: https://www.socialscienceregistry.org/trials/12848.

In this sub-trial, we perform an additional analysis by inviting part of the control group to receive an invitation. Specifically, e are randomizing 40k of the 120k control to receive a first communication email inviting them to sign up to Octoplus, sent the week commencing 26 February 2024. We used a simple randomization for this subtrial.

2024-02-26

2024-09-30

Sign-Up to Octoplus
Electricity Consumption in kWh per half-hour

Sign-Up to Octoplus: We will analyze whether the customer has signed up to Octoplus by each Saving Session (exact Session dates and times will be determined during winter 2023-2024 by National Grid ESO and are not yet known). We will compare sign-up rates between the main treatment group, the late invitation group and the rest of the control group.
Electricity Consumption in kWh per half-hour: We will analyze the electricity consumption (kWh per half-hour) of participating customers during and around Saving Sessions.

See analysis plan.

The main goal of this subtrial is to analyze how flexibility changes over time. The novelty and excitement around initial participation by consumers may wear off over time, leading to a weakened level of flexibility. Or, adoption of new habits and technologies and learning effects may cause higher flexibility over time. Which of these effects dominates is of great policy relevance. The late invitation group should exhibit less fatigue than the early invited group.

The details of the three groups are the following:
1. Control group: 79,999 of the 119,999 customers in the original control group that did not receive an invitation to join Octoplus and will stay uncontacted.
2. Late invitation group: 40,000 customers of the 119,999 customers in the original control group to receive a late invitation. The goal is to estimate the impact of the first event and subsequent events while removing time specific effects.
3. Original treatment group: the original 2,571,287 customers that received the first invitation to join Octoplus.

As of 26 February 2024, 46.3% of the original treatment group have signed up for Octoplus, as have 29.9% of the original control group. We also remove any customers with an open complaint or that have left Octopus. These exclusions mean that out of the 40,000 randomly selected customers, only 27,344 actually receive the late invitation.

Our Intent to Treat analysis will consider the late invitation group to be the group of approximately 40,000 customers, regardless of whether they had yet signed up to Octoplus.

Simple randomisation (using R).

Customer

Yes

There are approximately 5-10 Saving Sessions we expect to happen between the sending of the late invitation, on or just after 21 February, and the end of the 2023-2024 Saving Sessions, which will occur in March or April 2024. The exact number (and length of each event) depends on the needs of Great Britain's Electricity System Operator.

Assuming 5 events of 1 hour in length, there would be
1. Control group: 79,999 * 2 * 5 = 799,990 customer*half-hours during Saving Sessions
2. Late invitation group: 40,000 * 2 * 5 = 400,000 customer*half-hours during Saving Sessions
3. Original treatment group: 2,571,287 * 2 * 5 = 25,712,870 customer*half-hours during Saving Sessions

As stated above: 1. Control group: 79,999 of the 119,999 customers in the original control group that did not receive an invitation to join Octoplus and will stay uncontacted. 2. Late invitation group: 40,000 customers of the 119,999 customers in the original control group to receive a late invitation. The goal is to estimate the impact of the first event and subsequent events while removing time specific effects. 3. Original treatment group: the original 2,571,287 customers that received the first invitation to join Octoplus.

See above regarding Planned Number of Clusters and Planned Number of Observations.

We use a random sample of 8,298 Octopus Energy smart meter customers to produce the standard errors on the impact of the variable “Saving Session” of 10 “placebo” Saving Sessions. We then use the rule of 2.8 to estimate the Minimum Detectable Effect Size by multiplying the standard errors by 2.8, giving us an estimate of what effect size we could detect for a 5% significance level and 80% power. We then predict the standard errors for a larger sample where each group has 40,000 customers (note that this is slightly conservative, as one of our groups has much more than 40,000 customers). ITT For a late invitation group of 40,000 customers, we find a MDES of 0.003 kWh ~ 0.1% of half hourly consumption for our sample during the winter evening peak (5pm to 6pm in winter 2022-2023). We expect an average invitation treatment effect of 0.1 kWh per half-hour lower consumption during Saving Sessions among customers signed up to Octoplus based on values from Jacob et al (2023) Table AT.4. This average is based on what the authors note appears to be a slight decay in the ITT effect over time. We expect the late invitation group to reach similar levels of sign-up to Octoplus as the original treatment group; in other words, we expect both to have similar “compliance” with the encouragement to sign up to Octoplus. Assuming a 0.07 kWh difference in demand between those signed up in the original treatment group versus those signed up in the late invitation group (decay effect) and a difference of 15 percentage points in signup to Octoplus due to the late invitation (compliance) compared to signup to Octoplus in the control group (~30% vs ~45%), we have an expected ITT of 0.15 * 0.07 = 0.0105 kWh. For the comparison between the late invited and the rest of the control group, we expect a similar treatment effect of on average 0.1 kWh (invitation effect) and a difference of 15 percentage points in signup due to the late invitation (compliance); thus an expected ITT of 0.015 kWh. We should be well powered for both comparisons.