Text messaging for parental engagement in student learning

Stage 1: The goal is to choose one of four treatments (message type) for implementation in the RCT in Stage 2. In the 5 weeks prior to the winter break 2020 (Nov 2-Dec 07 weeks), we will conduct an experiment with five waves using adaptive treatment assignment. Each treatment consists of a set of text messages to be sent to parents weekly in order to encourage the use of the partner organization's proprietary Whatsapp-based quiz platform. The four treatments consist of four "types" of messages, sent each week to the randomly selected treatment groups:

1. Motivation framing: a set of 10 messages focusing on different motivations for learning.
2. Testimonial framing: a set of 10 messages focusing on the value of the quiz app.
3. One specifi c quiz: proposing 1 quiz tailored to what the student covered in class that
week/term.
4. Many quizzes: proposing several quizzes tailored to what the student covered in class that
week/term.

The outcome is a binary variable indicating "success". It is equal to 1 if at least one quiz was completed within 30 hours from the text message, equal to 0 if no quiz was completed from the phone number contacted.

Stage 2: A treatment group will receive the message type chosen above on a weekly basis for the duration of the school term beginning in January 2021.

Based on the estimated quiz uptake rates in Stage 1, the quiz usage rates prior to the experiment, and grade distributions from prior years, we will calculate the minimal average increase in assignment scores we would need to observe for each child who has adopted
the quizzes, in order to see a significant treatment effect using traditional frequentist inference for treatment and control group. If this increase is unrealistic, the partner organization may decide not to conduct the trial.

2020-11-02

2020-11-30

Stage 1: indicator whether at least 1 practice quiz was taken 30 hours after text message was sent. Stage 2: grade-specific normalized midterm and endterm exam scores.

We will form three strata: primary 1 and 2, primary 3-5, and primary 6.

First stage: For each phone number, one child associated with this number is selected randomly, generating a sample of 4705 children.

We stratify the sample and randomly select an equal share of each stratum for each of the 5 experimental waves. We then randomly assign children in each stratum and wave to the four treatment arms. In the fi rst wave, assignment to each treatment arm is equal (aside
from "misfi ts" arising from indivisibilities in the total numbers). In all following waves, we use exploration sampling to choose the assignment share to each treatment arm within each stratum.

Second stage: Using the full sample of 6066 children (as of Nov 2, 2020), families will be strati fied by the primary grade level of their child or children, and randomly assigned in equal proportions to the treatment and control group. If for any child in a given family, the most successful treatment type is treatment 3 or 4 (speci fic quizzes), the parent will receive one message for each child in that treatment type. The message will include the name of the child to attribute the correct quizzes to each child. If any children in the same family are assigned messages of type 1 or 2 according to Stage 1 results, the family will receive one message for all those treatment arms. Messages will be randomized according to the proportion of children in each treatment arm (to avoid duplicate messaging with similar contents).

For example, suppose a family has 5 children, 2 in primaries 1 and 2, 2 in primaries 4 and 5, and 1 in primary 6. Primaries 1-2 is assigned treatment arm 3, primaries 3-5 are assigned arm 2, and primary 6 is assigned arm 1. Then this family will receive 3 messages: one for each child in the quiz messaging arm (primaries 1 and 2) and one randomly selected message that is with 2/3 probability treatment arm 2, and 1/3 probability treatment arm 1.

Assignment according to the predetermined assignment shares (equal shares, or as determined by exploration sampling algorithm) is carried out using the randtreat command in Stata, which carries out factorial assignment within stratum with tie-breaking.

Parent/phone number

Yes

4,705 phone numbers

First stage: 4,705 phone numbers (parents) Second stage: 6,066 children

First stage: in each wave, in expectation 235 phone number are assigned to each of the four messaging treatment arms. However, assignment shares will vary after the first stage according to exploration sampling assignment shares. Each wave will consist of approximately 941 parents in total receiving messages.

Second stage: the sample will be split evenly between treatment and control apartment from any misfits that arise from stratification, i.e., we aim to assign 2,353 families to each treatment and control, with balanced allocation of observations.

We will update this registration with power calculations for the second stage.

PAP-Nigeria-11-2-2020.pdf