How much do we gain from personalization?

This paper studies the tradeoffs of personalizing treatments based on heterogeneity analysis. We do so in the setting of a learning platform (Conecta Ideas) for primary school students in Peru. In the first phase, we randomize students' parents who have previously downloaded the app into 16 treatment groups and a control group. The parents in each group receive a combination of 2 messages out of 4 possible messages through an app notification. These messages seek increase the use of the learning up by students. to At the end of this phase, we conduct two analyses to determine the "best" message. First, we choose the best combination of messages based on their (estimated) average treatment effect. Second, we use machine learning techniques (a la Athey & Wager, 2018) to estimate the best message for each person based on (pre-treatment) observable characteristics. In the second phase, we run an experiment in which parents are randomly assigned to a treatment group in which they all receive the "best overall" message (based on the estimated ATE), a treatment group in which each parent is assigned the "best personalized" message based on the heterogeneity analysis and a control group. This allows us to compare the gains (if any) of personalization. Intuitively, the tradeoff in phase 2 is that the "best overall" message is estimated with more precision but may be suboptimal for some, while the "best personalized" is "optimal" among the set of messages for each individual, but this "optimality" is estimated with less precision.