Text messaging for parental engagement in student learning

Last registered on May 06, 2021

Pre-Trial

Trial Information

General Information

Title
Text messaging for parental engagement in student learning
RCT ID
AEARCTR-0006701
Initial registration date
November 02, 2020
Last updated
May 06, 2021, 4:30 PM EDT

Locations

Region

Primary Investigator

Affiliation
World Bank

Other Primary Investigator(s)

Additional Trial Information

Status
Completed
Start date
2020-11-02
End date
2020-11-30
Secondary IDs
Abstract
This experiment will use exploration sampling to test the effects of weekly text messaging to parents' phones as a means to increase student use of a phone quiz platform and ultimately learning outcomes.

The study will be implemented by a partner organization that operates private schools in Lagos, Nigeria, among approximately 4705 students in Primaries 1-6. The study will proceed in two stages. In the first stage, we will use adaptive treatment assignment with exploration sampling and stratification to choose one out of four "types" of text messages. Designed by the partner, the four message "types" include motivational framing, testimonial framing, and varying numbers of practice quizzes tailored to what the student recently covered in class. The text message type chosen in the first stage is the one that promises to lead to the greatest increase in use by students of the partner's quiz platform. In the second stage, we will use a standard RCT with a treatment group that receives the message type chosen in the first stage, and a control group that receives no messaging, to test whether quiz use leads to sustained increases in exam scores.
External Link(s)

Registration Citation

Citation
Sautmann, Anja. 2021. "Text messaging for parental engagement in student learning." AEA RCT Registry. May 06. https://doi.org/10.1257/rct.6701-1.1
Experimental Details

Interventions

Intervention(s)
Stage 1: The goal is to choose one of four treatments (message type) for implementation in the RCT in Stage 2. In the 5 weeks prior to the winter break 2020 (Nov 2-Dec 07 weeks), we will conduct an experiment with five waves using adaptive treatment assignment. Each treatment consists of a set of text messages to be sent to parents weekly in order to encourage the use of the partner organization's proprietary Whatsapp-based quiz platform. The four treatments consist of four "types" of messages, sent each week to the randomly selected treatment groups:

1. Motivation framing: a set of 10 messages focusing on different motivations for learning.
2. Testimonial framing: a set of 10 messages focusing on the value of the quiz app.
3. One specifi c quiz: proposing 1 quiz tailored to what the student covered in class that
week/term.
4. Many quizzes: proposing several quizzes tailored to what the student covered in class that
week/term.

The outcome is a binary variable indicating "success". It is equal to 1 if at least one quiz was completed within 30 hours from the text message, equal to 0 if no quiz was completed from the phone number contacted.

Stage 2: A treatment group will receive the message type chosen above on a weekly basis for the duration of the school term beginning in January 2021.

Based on the estimated quiz uptake rates in Stage 1, the quiz usage rates prior to the experiment, and grade distributions from prior years, we will calculate the minimal average increase in assignment scores we would need to observe for each child who has adopted
the quizzes, in order to see a significant treatment effect using traditional frequentist inference for treatment and control group. If this increase is unrealistic, the partner organization may decide not to conduct the trial.
Intervention Start Date
2020-11-02
Intervention End Date
2020-11-30

Primary Outcomes

Primary Outcomes (end points)
Stage 1: indicator whether at least 1 practice quiz was taken 30 hours after text message was sent. Stage 2: grade-specific normalized midterm and endterm exam scores.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We will form three strata: primary 1 and 2, primary 3-5, and primary 6.

First stage: For each phone number, one child associated with this number is selected randomly, generating a sample of 4705 children.

We stratify the sample and randomly select an equal share of each stratum for each of the 5 experimental waves. We then randomly assign children in each stratum and wave to the four treatment arms. In the fi rst wave, assignment to each treatment arm is equal (aside
from "misfi ts" arising from indivisibilities in the total numbers). In all following waves, we use exploration sampling to choose the assignment share to each treatment arm within each stratum.

Second stage: Using the full sample of 6066 children (as of Nov 2, 2020), families will be strati fied by the primary grade level of their child or children, and randomly assigned in equal proportions to the treatment and control group. If for any child in a given family, the most successful treatment type is treatment 3 or 4 (speci fic quizzes), the parent will receive one message for each child in that treatment type. The message will include the name of the child to attribute the correct quizzes to each child. If any children in the same family are assigned messages of type 1 or 2 according to Stage 1 results, the family will receive one message for all those treatment arms. Messages will be randomized according to the proportion of children in each treatment arm (to avoid duplicate messaging with similar contents).

For example, suppose a family has 5 children, 2 in primaries 1 and 2, 2 in primaries 4 and 5, and 1 in primary 6. Primaries 1-2 is assigned treatment arm 3, primaries 3-5 are assigned arm 2, and primary 6 is assigned arm 1. Then this family will receive 3 messages: one for each child in the quiz messaging arm (primaries 1 and 2) and one randomly selected message that is with 2/3 probability treatment arm 2, and 1/3 probability treatment arm 1.
Experimental Design Details
Randomization Method
Assignment according to the predetermined assignment shares (equal shares, or as determined by exploration sampling algorithm) is carried out using the randtreat command in Stata, which carries out factorial assignment within stratum with tie-breaking.
Randomization Unit
Parent/phone number
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
4,705 phone numbers
Sample size: planned number of observations
First stage: 4,705 phone numbers (parents) Second stage: 6,066 children
Sample size (or number of clusters) by treatment arms
First stage: in each wave, in expectation 235 phone number are assigned to each of the four messaging treatment arms. However, assignment shares will vary after the first stage according to exploration sampling assignment shares. Each wave will consist of approximately 941 parents in total receiving messages.

Second stage: the sample will be split evenly between treatment and control apartment from any misfits that arise from stratification, i.e., we aim to assign 2,353 families to each treatment and control, with balanced allocation of observations.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
We will update this registration with power calculations for the second stage.
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number
Analysis Plan

Analysis Plan Documents

PAP-Nigeria-11-2-2020.pdf

MD5: 15d72889f154b316ab8eb878ccdcbdf0

SHA1: caf9cfea91a8e0999486348750c0d9d7e3f367b6

Uploaded At: November 02, 2020

Post-Trial

Post Trial Information

Study Withdrawal

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
No
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
No
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials