Energy Demand Flexibility at Scale: Sign Up Chaser Emails

See analysis plan.

This trial, led by the Centre for Net Zero (CNZ), in partnership with Octopus Energy, builds on a larger randomized encouragement design nested in Octopus Energy’s second Saving Sessions (winter 2023 - 2024) program. In the main trial, we randomized customers into a treatment group that got invited to join Octoplus – a membership that enables customers to participate in flexibility programmes like Saving Sessions – and a control group that did not receive the invitation. See: https://www.socialscienceregistry.org/trials/12848.

More precisely, a total of 2.7 million eligible participants were initially categorized into two groups: those who were invited to participate in Octopus Energy's new flexibility program, Octoplus, launched in October 2023 (randomly selected treatment group ~ about 2.5m), and those who were not (randomly selected control group ~ 119,999 customers).
In this phase of the study, we expanded our investigation by introducing an extra layer of randomization within the primary treatment cohort. Specifically, we selected an additional subset of 70,000 individuals from the main treatment group and designated them as the holdout group, which meant they would not get a subsequent "chaser" email if they failed to sign up to Octoplus following their initial invite and were opted in to marketing emails. Conversely, the rest of the participants in the treatment group who were opted in to marketing emails, approximately 680k customers, were set to receive the chaser email after their initial saving sessions. (Note that many customers in our treatment group were not opted in to marketing emails. While they received the initial Octoplus sign-up encouragement regardless of marketing email opt-in, customers who had not opted in to marketing emails did not receive the “chaser”.)

The subtrial effectively created two distinct groups for analysis:
1. The hold out group that received the initial invitation but not the chaser email, numbering 70,000 customers, with an anticipated medium sign-up rate.
2. The larger segment of the treatment group, the chaser group, of approximately 680k customers, who received both the initial invitation and the chaser email, predicted to have the highest sign-up rate.

As of January 17th, the date the chaser was sent out, the proportion of the main treatment group that was signed up to OctoPlus is 43%. This means that while the chaser holdout group number is approximately 70,000 customers, the number of customers from this group who have not signed up and thus would have received the chaser is only approximately 40,000. However, our ITT analysis will consider the chaser holdout group to be the group of approximately 70,000 customers who have opted in to marketing emails (and thus were eligible to receive the chaser, if they hadn’t yet signed up to Octoplus), regardless of whether they had yet signed up to Octoplus.

Note: Even though the email was sent out on January 17th, before the pre-registration of this analysis, we haven’t retrieved nor analyzed the data yet.

2024-01-17

2024-09-30

Sign-Up to Octoplus
Electricity Consumption in kWh per half-hour

Sign-Up to Octoplus: We will analyze whether the customers in the hold out group (invitation but no chaser) and chaser group (invitation and chaser email) have signed up to Octoplus by each Saving Session (exact Session dates and times will be determined during winter 2023-2024 by National Grid ESO and are not yet known).
Electricity Consumption in kWh per half-hour: We will analyze the electricity consumption (kWh per half-hour) of participating customers during and around Saving Sessions in the hold out group (invitation but no chaser) and chaser group (invitation and chaser email).

See analysis plan.

ITT: The ITT (Intent to Treat) estimator will be used to calculate the impact of receiving the chaser email (hold out versus chaser).

LATE on Octoplus sign-up: Therefore, we will also estimate the LATE (Local Average Treatment Effect) to capture the causal impact of receiving the extra email for compliers (those who would not have signed up to Octoplus, but did because of the email encouraging them to do so when the chaser email was sent) on their electricity consumption during (and around) Saving Sessions compared to the customers in the hold out group.

See analysis plan for details.

Simple randomisation (using R).

Customer

Yes

680k customers received the chaser (treatment group).
70k customers did not (control group)

There are approximately 10-20 Saving Sessions we expect to happen between the sending of the chaser, on 17 January 2024, and the end of the 2023-2024 Saving Sessions, which will occur in March or April 2024. The exact number (and length of each event) depends on the needs of Great Britain's Electricity System Operator. Assuming 10 events of 1 hour in length, there would be 10*2*680k = 13,600,000 customer*half-hours during Saving Sessions in the treatment (chased) group; and 10*2*70k = 1,400,000 customer*half-hours during Saving Sessions in the control group.

See above regarding Planned Number of Clusters and Planned Number of Observations.

We use a random sample of 8,298 Octopus Energy smart meter customers to produce the standard errors on the impact of the variable “Saving Session” of 10 “placebo” Saving Sessions. We then use the rule of 2.8 to estimate the Minimum Detectable Effect Size by multiplying the standard errors by 2.8, giving us an estimate of what effect size we could detect for a 5% significance level and 80% power. We then predict the standard errors for a larger sample where each group has 70,000 customers (note that this is slightly conservative, as one of our groups has much more than 70,000 customers). We find a MDES for the ITT of 0.003 kWh. This is ~ 0.7% of half hourly consumption for our sample at peak hour (5pm to 6pm) in winter 2022-2023. We expect an average treatment effect of 0.1 kWh for sign-up through the season based on Jacob et al (2023) Table AT.4. We also expect the chaser group to increase signup by 10 percentage points. Assuming a 10 percentage point higher sign-up to Octoplus between groups, we have an expected treatment effect of 0.1 (additional sign-up from the chaser) * 0.1 (kWh per signup) = 0.01 kWh ITT effect. In this case, we are powered to detect the ITT effect, as 0.01 > MDES of 0.003 kWh. We do the same exercise for the LATE analysis: we do a placebo 2SLS regressions of a placebo invitation on sign-up. We obtain a MDES of 0.014 kWh (about 4% of half hourly consumption). In this case, we are not powered to 80% to detect this LATE, as 0.014 kWh is greater than the 0.1 kWh per half-hour effect we expect from signing up to Octoplus.