Impact Evaluation of TipsByText: A Parent-Oriented Language Program for Preschool Children

TipsByText is a text-messaging program that aims to help parents support their children’s language development in the preschool years. For eight months, parents receive three weekly text messages with tips for fun and simple language-stimulating games and activities they can do with their children. The text messages are formulated in non-complex language and the tone in the texts is short, direct, and with a call for immediate action in daily routines, e.g., bath time. The curriculum is aimed at parents of three- to five-year-old children, as children in Denmark typically enter preschool around the age of three and stay there until they start primary school (kindergarten class) in the year they turn six.

The intervention is an adapted version of an American parent education program, Ready4K, developed by Susanna Loeb and Benjamin York of Stanford University. The intervention is designed around three behavioral barriers to involved parenting: First, some parents have insufficient information about the impact their child-directed speech has on their child’s language development. To overcome this barrier, each Monday parents receive a “FACT” text designed to generate knowledge by highlighting the importance of particular skills. Second, parenting is a complex task and the cognitive load of parenting can result in a low prioritization of active language stimulation. To help alleviate this barrier, each Wednesday the parents receive a “TIP” text, which contains a specific and creative exercise on how to incorporate a language-stimulating element into a daily routine that is already taking place. Third, to motivate parents and keep them engaged in the language support of their children, each Friday the parents receive a “GROWTH” text, which provides encouragement and a deeper insight of the benefits of the exercise given on Wednesday.

The Danish adaptation of the intervention builds on the same principles as the American version but is specific to the Danish set-up and culture. In particular, the Danish curriculum focuses on three components of language development, which are a priority for language development in the public childcare sector. These components are:
• Formal language (vocabulary: understanding and producing words)
• Early literacy (letters, reading direction, etc.)
• Use of language (conversation, dialogue, narrativity, taking turns, and initiatives).
The adaptation was facilitated by the ROCKWOOL Foundation Interventions Unit (RFI) and was carried out in collaboration with Susanna Loeb and a diverse group of researchers and stakeholders who represent a wide knowledge of early language acquisition, child psychology, child development, and parent education programs and/or early interventions in Denmark. This group consisted of Pia Thomson, who is a prominent Danish language researcher, local municipalities, practitioners, and parent groups.

2020-01-24

2020-09-04

The primary outcome of this study will be children’s language skills as measured by the Danish language assessment tool, “Sprogvurdering 3-6.” The language assessment tool is designed to measure the language skills of children in preschool (ages 3 to 6) and in the first year of primary school (ages 5 or 6). The tool was developed as part of a federal requirement that all public schools perform language assessment on all children the year they start primary school (kindergarten class), and all public preschools perform language assessment on all three-year-old children who are suspected of lagging behind in language development. In practice, many preschools use this test more widely than required as a tool to identify the children who lag behind in their language development and therefore need extra language stimulation. For example, in the year before the intervention, the collaborating preschools assessed the language of 81% of 3-year-olds, 12% of 4-year-olds, and 53% of 5-year-old children, on average, using the tool “Sprogvurdering 3-6.” As part of the RCT, the partnering preschools will increase the scope of their assessments for three consecutive years to include all preschool children included in the trial as well as change the timing of the assessments to fit with the timing of the trial.

Specifically, the first round of endline data collection will be initiated immediately after the end of the intervention in the fall of 2020. The second and third rounds of endline data collection will be initiated in the fall of 2021 and 2021, respectively.

We plan to conduct the following analyses:
1. Causal effect of the intervention on language skills in the short run:
In a linear regression framework, we will compare the language skills measured immediately after the end of the intervention of children in the treatment group to those in the control group. Because the main outcome is a composite measure, we will also conduct reverse regression using machine learning to identify whether there is any function of the outcomes that predicts treatment assignment (Ludwig, Mullainathan, and Spiess 2019).

2. Subgroup analysis of the effect of the intervention on language skills in the short run:
We hypothesize that the intervention benefits three groups in particular: children who score below the median in the baseline language assessment, children of parents without a university degree, and children of immigrants. We will explore heterogeneous treatment effects across the distribution of baseline language skills and parental education levels. Our sample does not include enough children of immigrants to provide power to detect a plausible effect size difference for children of immigrants, but the analysis will include explorative subgroup analyses. Furthermore, using a machine-learning framework, we will explore heterogeneous treatment effects in children’s language skills in relation to a variety of demographic characteristics, such as parental education, income, and whether the parents are immigrants, to allow these characteristics to interact with each other in a way that the linear model cannot capture.

3. Causal effect of the intervention on language skills in the medium run:
We will repeat exercises 1. and 2. for the endline data collected one and two years, respectively, after the end of the intervention.

The language assessment “Sprogvurdering 3-6” measures the child’s language skills in four to seven dimensions depending on the age of the child. There are eight different dimensions in total: language comprehension, receptive vocabulary, rhyming, written language awareness, letter recognition, word detection, sound discrimination, and communication strategies. Each subtest (except communication strategies) consists of 12-25 questions of increasing difficulty. The questions are asked by the teacher/language specialist and answered by the child, and each answer is registered by the adult supervising the test as either “correct,” “incorrect,” or “not answered.” The dimension “communication strategies” consists of 15 questions directed at the teacher/language specialist about the child’s general use of language and each question has four possible answers, “never,” “rarely,” “often,” “always.”

In the preschools, the assessment is performed on a 1:1 basis by a teacher or language specialist, and in the first year of primary school the assessment is a class-based test performed by the teacher. Children in primary school who receive a low score on the class-based test are also assessed on a 1:1 basis with the teacher or a language specialist. There are five different versions of the test determined by the child’s age and whether the test is performed in the preschool or in school. The older the child, the more subtests the child answers and the more difficult the questions are within each subtest. For example, a 3-year-old child in preschool answers four subtests and answers the easiest 12-25 questions within each subtest. A 5-year-old child in preschool answers seven subtests and answers the most challenging questions within each subtest. Thus, even though a 3-year-old child and a 5-year-old child are assessed in four overlapping dimensions, they answer very few of the same questions.

Because the test material depends on the age of the child, we cannot make direct comparisons between children of different ages. Instead, all cross-material analyses will include material-level fixed effects to ensure that children are only compared to other children having completed the same version of the test.

Within each test material, we construct a composite measure of language development in the following way: First, for each child we calculate the score within each subtest as the number of correct answers in that subtest. We choose to focus only on the questions answered correctly because children are more likely to skip the more difficult questions. In effect, we treat incorrect and missing answers equally. For the dimension “communication strategies” we calculate the average score on the four-point scale across the 15 questions. Second, we standardize the scores for each subtest within each test material because not all subtests use the same scale. Third, for each child we average the standardized scores across subtests to get a single score. Finally, we standardize the scores within each material again to ensure that the outcome has a mean of zero and a standard deviation of 1 within each test material.

We plan to conduct sensitivity analyses for different ways of dealing with unanswered questions, including analyzing whether treatment affects the number of questions answered. Furthermore, we plan to conduct a reverse regression of treatment status based on all elements in the composite measure of language development.

As the primary outcome measure is a composite measure of the different subtests in the language assessment “Sprogvurdering 3-6,” the individual subtests will be considered secondary outcomes. These outcomes will be the percentage correctly answered questions in the subtests language comprehension, receptive vocabulary, rhyming, written language awareness, letter recognition, word detection, sound discrimination, as well as the total score in communication strategies. In addition, as the composite measure treats missing answers as incorrect, the total number of missing answers across subtests will be considered a secondary measure. As missing answers are typically a result of the child’s inattention (either because the individual question is too difficult or because the test is abandoned due to lack of focus) this measure is a proxy for the child’s engagement in the language assessment.

We plan to conduct the following analyses:
1. Causal effect of the intervention on each subtest:
In a linear regression framework, we will compare the percentage correct answers in each subtest in the language assessment for children in the treatment group to those in the control group, while controlling for multiple hypothesis testing. This test will allow us to evaluate whether the intervention affected the different dimensions of language development differently.

2. Causal effect of the intervention on the proportion of answered questions
Similarly, in a linear regression framework, we will compare the percentage of missing answers across all subtests in the language assessment for children in the treatment group to those in the control group. In case we find no main effect on the composite measure of language development, this test will allow us to evaluate whether the intervention has affected the children’s engagement in the language assessment.

We employ a family-level randomized controlled trial with an opt-out design for parents of children aged three-to-five in five partnering municipalities in Denmark: Hoeje-Taastrup, Ikast-Brande, Langeland, Lejre, and Middelfart.

Recruitment: Each of the five municipalities provided us with a list of preschools that would participate in the evaluation; it was at the municipalities’ discretion to determine which preschools would participate, and the municipalities included between 67% and 100% of public preschools.

Sample: In total, 3,804 children born in or later than year 2016 (age 3 at the start of the intervention) were enrolled in the 70 participating preschools on October 1, 2019; these children constitute our population. Children in the population were included in the trial if two conditions were met on January 1st, 2020: First, that their parents had not opted out of the trial, second, that we had received at least one parental phone number from the collaborating municipalities. In total, 3,498 children in 3,181 families were included in the trial. At the start of the intervention their age distribution was as follows: 1,024 3-year-old children, 1,106 4-year-old children, 1,219 5-year-old children, and 149 6-year-old children.

Timing: Baseline language assessments were performed by the collaborating preschools in the Fall of 2019. Randomization was conducted in January 2020. On January 24, 2020, parents in the treatment condition received the first text message. Children in the control condition received treatment as usual. The last text message will be sent out on September 5, 2020.

Endline data: In September 2020, the municipalities will start collecting the first round of child-level endline data on language development (using “Sprogvurdering 3-6”). The data collection is scheduled to be complete by the end of 2020, at which time we will receive the data. In September-November 2021 and 2022, respectively, the municipalities will collect the second and third rounds of child-level endline data on language development (also using “Sprogvurdering 3-6”).

Information provided to parents: All parents in the participating preschools received the same informational pamphlet about the intervention in October 2019 before any data was collected. The pamphlet describes the intervention, the evaluation (including randomization), the data collection, and the research team. The information was delivered in printed form to the parents in the wardrobe of each preschool and was uploaded to most preschools’ intranet. The material was translated into eight languages; all parents received the printed version in Danish, and for each child the personnel in the preschool decided whether to attach a translated version. The material provided the parents with a link to a website that gave additional information, and with a phone number/email they could contact if they wished to opt out of the trial. In total, parents of 52 children opted out of the trial before randomization was conducted.
Spillover/contamination risk: To minimize spillover and contamination of the intervention to the control group, we aimed to minimize the probability that the preschool staff knew the allocation of children into the treatment or control group. We used three strategies to achieve this. First, the preschool staff was given no information about the allocation. Second, the preschool staff was explicitly instructed not to initiate conversations with the parents about the intervention. Third, to minimize the risk of parents engaging the preschool staff in conversation about the intervention, the informational pamphlet does not mention the preschool. Instead, the sender of the informational pamphlet is the municipality and the ROCKWOOL Foundation Interventions Unit.

Randomization of treatment status was done by a computer using the randtreat command in Stata 15.1. Families were randomized into three different groups: Treatment group, Control group, and Interview group. First, 119 families were drawn at random (stratified by municipality and whether the youngest child’s baseline language assessment was above or below the median) to be interviewed about their experience with the intervention and were removed from the RCT sample. Second, the remaining families were randomized 50/50 into treatment and control group stratified by three variables: 1. preschool (70 strata), 2. whether the youngest child scored above/below the median within their age group in the baseline language assessment or has a missing baseline language assessment (3 strata), and 3. expected language assessment material at endline (4 strata). In total there were 840 strata, and misfits were allocated using the global-method in the randtreat command, where misfits are assigned in equal proportions to treatment and control. All analyses will control for fixed effects in the variables that the randomization stratified on: daycare center, above/below median language development within age group at baseline, and age-group at endline.

Randomization of treatment status was done at the family level, which means that all parents to one child receive the same treatment. To have the cleanest treatment possible, children were deemed to be in the same family if they shared a parental phone number, which means that stepsiblings/stepparents were also deemed to belong to the same family. To ensure that all children within a family were assigned the same treatment status, older siblings were removed from the dataset before randomizing.

Yes

3,181 families

3,498 children

1,591 families (1,752 children) in the treatment group and 1,590 families (1,746 children) in the control group.

With a sample of 3,181 families, we will have 80% power to detect an effect of 0.1 standard deviation with a 5% alpha level under the assumption of an attrition rate of 20% and an R^2 of 0.3.