How much do we gain from personalization?

Last registered on October 17, 2023

Pre-Trial

Trial Information

General Information

Title
How much do we gain from personalization?
RCT ID
AEARCTR-0012190
Initial registration date
October 16, 2023

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
October 17, 2023, 1:55 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
ITAM

Other Primary Investigator(s)

PI Affiliation
Bocconi University
PI Affiliation
IDB
PI Affiliation
Cornell University
PI Affiliation
Cornell University

Additional Trial Information

Status
In development
Start date
2023-10-16
End date
2023-12-31
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
This paper studies the tradeoffs of personalizing treatments based on heterogeneity analysis. We do so in the setting of a learning platform (Conecta Ideas) for primary school students in Peru. In the first phase, we randomize students' parents who have previously downloaded the app into 16 treatment groups and a control group. The parents in each group receive a combination of 2 messages out of 4 possible messages through an app notification. These messages seek increase the use of the learning up by students. to At the end of this phase, we conduct two analyses to determine the "best" message. First, we choose the best combination of messages based on their (estimated) average treatment effect. Second, we use machine learning techniques (a la Athey & Wager, 2018) to estimate the best message for each person based on (pre-treatment) observable characteristics. In the second phase, we run an experiment in which parents are randomly assigned to a treatment group in which they all receive the "best overall" message (based on the estimated ATE), a treatment group in which each parent is assigned the "best personalized" message based on the heterogeneity analysis and a control group. This allows us to compare the gains (if any) of personalization. Intuitively, the tradeoff in phase 2 is that the "best overall" message is estimated with more precision but may be suboptimal for some, while the "best personalized" is "optimal" among the set of messages for each individual, but this "optimality" is estimated with less precision.

External Link(s)

Registration Citation

Citation
AULAGNON, RAPHAELLE et al. 2023. "How much do we gain from personalization?." AEA RCT Registry. October 17. https://doi.org/10.1257/rct.12190-1.0
Experimental Details

Interventions

Intervention(s)
This paper studies the tradeoffs of personalizing treatments based on heterogeneity analysis. We do so in the setting of a learning platform (Conecta Ideas) in Peru.

In the first phase, we randomize parents who have previously downloaded the app into 16 groups or a pure control group. These groups have all possible combinations of 16 messages to encourage platform use. Parents receive the messages in their group once times between October 12 and October 13, 2023. We then observe whether they connect and use the platform or not in the next couple of weeks.

In the second phase, we run an experiment in which parents are assigned to a treatment group in which they all receive the "best overall" message (based on the estimated ATE) or a treatment group in which each parent is assigned the "best personalized" message based on the heterogeneity analysis of the first phase. Parents can also be randomly assigned to a control group. This will take place between October 30 and November 3, 2023.
Intervention (Hidden)
Intervention Start Date
2023-10-16
Intervention End Date
2023-11-30

Primary Outcomes

Primary Outcomes (end points)
Login to the plataform, and the amount of time spent in it
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Our sample consists of 100,000 parents.

80,000 of these are assigned to the first phase of the experiment and are evenly divided among 16 groups.

We then re-randomize all parents (including the 20,000 not involved in phase 1) into three treatment groups. XXX of these parents will be assigned to the "best overall" treatment, XXX to the "best personalized" treatment, and XX to the control.

The treatment in both cases will be stratified by whether the parents connected in the previous week or not
Experimental Design Details
Randomization Method
Randomization is done in Stata
Randomization Unit
The treatment is randomized at the individual level
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
NA
Sample size: planned number of observations
100,000
Sample size (or number of clusters) by treatment arms
In the first phase, there are 5,000 parents in each of the 16 possible groups.

In the second phase, there are 30,000 parents in each of the 2 treatment groups (best overall, and best personalized), and 40,000 in the control group.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We estimate that we are able to detect a treatment effect (or a difference in treatments) of XX percentage points in phase 2. This is with power 80% and size 5%.
IRB

Institutional Review Boards (IRBs)

IRB Name
Using Machine Learning to test the effeciveness of personalized messages to improve homework completion on a learning app
IRB Approval Date
2023-10-05
IRB Approval Number
0147998

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials