LLMs and e-mail marketing

Last registered on December 21, 2023

View Trial History

Pre-Trial

Trial Information

General Information

Title

LLMs and e-mail marketing

RCT ID

AEARCTR-0012739

Initial registration date

December 20, 2023

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

December 21, 2023, 8:06 AM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country

United States of America

Region

national

Primary Investigator

Name

Jean-Pierre Dube

Affiliation

University of Chicago Booth School of Business

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Ningyin X

PI Affiliation

University of Chicago Booth School of Business

Contact Investigator

Additional Trial Information

Status

In development

Start date

2023-12-31

End date

2024-01-29

Keywords

Other

Additional Keywords

consumer, demand, marketing, large language models, experiment

JEL code(s)

Secondary IDs

L00, C9

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

We aim to test the effectiveness of large language models (LLM) in producing “newsletter creative” (NC) compared to human writers. We collaborate with Wine Access (WA), an online wine retailer, to run a randomized controlled experiment (RCT) in a 2-week period so respondents will get/not get NC generated from different sources:
1. Control group (size = 500): The respondent receives no WA newsletter at all.
2. Test 1 (size = 9,000): The respondent receives daily WA newsletters created by the human writer team.
3. Test 2 (size = 9,000): The respondent receives daily WA newsletters created by the LLM.
4. Test 3 (size = 9,000): The respondent receives daily WA newsletters created by the human team with access to and can tune the LLM.
At the start of the experiment, each subject will be randomly assigned to one of the experimental cells above and will remain in that cell throughout the 2-week duration of the experiment. Each cell will have a human fact-checker that’s independent of the human writer team so the newsletter has accurate information. The effectiveness of NC can be assessed based on comparisons of multiple outcome variables such as site visits, revenues, and profits.

External Link(s)

Registration Citation

Citation

Dube, Jean-Pierre and Ningyin X. 2023. "LLMs and e-mail marketing." AEA RCT Registry. December 21. https://doi.org/10.1257/rct.12739-1.0

Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Experimental Details

Interventions

Intervention(s)

The randomized controlled experiment (RCT) will run for a 2-week period with the following cells in parallel:
1. Control group (size = 500): The respondent receives no WA newsletter at all.
2. Test 1 (size = 9,000): The respondent receives daily WA newsletters created by the human writer team.
3. Test 2 (size = 9,000): The respondent receives daily WA newsletters created by the LLM.
4. Test 3 (size = 9,000): The respondent receives daily WA newsletters created by the human team with access to and can tune the LLM.

Intervention Start Date

2024-01-08

Intervention End Date

2024-01-22

Primary Outcomes

Primary Outcomes (end points)

Purchases and profits

Primary Outcomes (explanation)

We are interested in whether an LLM can produce e-mail creatives that are at least as effective (gross and net of costs) than the creatives produced by a salaried team of writers.

Secondary Outcomes

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

We will randomly assign approximately 27,000 customers to 4 experimental cells:
1. no intervention (control)
2. base intervention (creative designed by human team)
3. LLM Intervention (creative designed by LLM)
4. hybrid intervention (creative designed by human team after receiving creative from the LLM)

Experimental Design Details

Randomization Method

random number generator

Randomization Unit

individual customer identification number

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

N/A

Sample size: planned number of observations

approximately 27,000 customers

Sample size (or number of clusters) by treatment arms

500 in control
9,000 in each of the intervention arms

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

using historic data, the average customer clicking on ads generates $8.11 in profit, whereas the average customer not clicking on ads generates only $5.46. To ensure 10% power and significance, this requires 459 respondents in the control and test group to detect the treatment effect relative to "no intervention." We then plan to allocate the entire remainder of the customer base evenly across the intervention arms.

Supporting Documents and Materials

IRB