TalkingPoints Evaluation

Last registered on October 11, 2023

Pre-Trial

Trial Information

General Information

Title
TalkingPoints Evaluation
RCT ID
AEARCTR-0004814
Initial registration date
October 03, 2019

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
October 04, 2019, 11:54 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
October 11, 2023, 3:48 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
The University of Chicago

Other Primary Investigator(s)

PI Affiliation
The University of Chicago

Additional Trial Information

Status
Withdrawn
Start date
2020-09-07
End date
2021-06-11
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
We evaluate an education technology platform that is designed to increase parent engagement with the goal of improving student engagement in the short run and academic outcomes in the long run. TalkingPoints has two components. TalkingPoints Multilingual Messaging facilitates communication between teachers and parents by providing an easy-to-use platform that promotes two-way translated messages in over 30 languages. TalkingPoints Coach provides parents tips for communicating with teachers and other information about their children’s schooling. Messages between parents and teachers come directly from TalkingPoints. Both local and federal law requires schools to provide programing intended to promote parental engagement in their children’s education. We assess the effectiveness of TalkingPoints in a school-based randomized field experiment over one school year in 50–65 schools using objective measures of parent communication with schools, engagement with children and children’s executive functioning.
External Link(s)

Registration Citation

Citation
Kalil, Ariel and Michelle Park Michelini. 2023. "TalkingPoints Evaluation." AEA RCT Registry. October 11. https://doi.org/10.1257/rct.4814-3.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
The Evaluation
The Sample. The evaluation will be a school-based randomized controlled trial that will be implemented in grade three. We focus on one grade to improve power. We selected third grade because it is the point in schooling when academics become more rigorous and consequential in public schools. Developmentally and institutionally, students are expected to have acclimated to the classroom routine and have the executive functioning skills for staying on task and completing homework by third grade, which are elements required for our outcome measures described below. The evaluation will include 50-60 schools in the US. One third-grade classroom in each school will participate in the evaluation. Classes are expected to include about 25 students. There is an oversample of Title I schools and schools with a large English Second Language population.
The Treatment. Students in one third-grade classroom in both treatment and control schools will be enrolled in TalkingPoints for this evaluation. Teachers in treatment classrooms will have access to TalkingPoints while teachers in control classrooms will not. Control group teachers will be subject to “business as usual” in terms of communicating with parents. Prior to randomization, teachers in both treatment and control classrooms will register their students in TalkingPoints by entering demographic into the TalkingPoints registration system. These data will be used to test balance across the treatment and control groups and as covariates in the data analysis. We also plan to collect student attendance data from the school or district.
We will consent teachers and students based on the policies of participating schools. Once students in all participating classrooms are registered and parents have consented, schools will be randomized into either the treatment or the control group. Following randomization, treatment schools will receive TalkingPoints at no cost for the 2020–2021 school year. Control group schools will not receive TalkingPoints in 2020–2021 but will receive it at no cost in the 2021–2022 school year. TalkingPoints will become available to treatment schools in October 2020. The treatment will continue until the end of the school year. All student assessments and parent and teacher surveys will be completed in May 2021.
Intervention (Hidden)
Intervention Start Date
2020-10-05
Intervention End Date
2021-05-28

Primary Outcomes

Primary Outcomes (end points)
We have three outcome variables of interest, which align with the end goals of TalkingPoints: 1) communication between teachers and parents; 2) parent engagement at home with their child's schooling; and 3) student executive functioning skills, which are a precursor to academic and life success.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
We will survey parents and teachers to get qualitative feedback on their experience using TalkingPoints.
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Communication between Teachers and Parents. As an objective measure of parent-teacher communication, we will use simulated communications to test whether treatment parents are more likely than control parents to communicate with schools. If TalkingPoints increases the chance that a parent will communicate with the school, then when TalkingPoints sends a message to both treatment and control parents asking for a response, parents in the treatment group ought to be more likely than parents in the control group to respond. For example, a simulated message might be, “Did your child have any math homework this week?” The parent would be asked to reply by clicking yes or no. The measure of communication will be the number of replies.
Parent Engagement in Their Child’s Schooling. We cannot observe parent engagement with their child at home and parent reports of their own engagement are likely to be biased, so we will create simulated engagement activities. One of the main ways that schools hope parents will engage their children is by helping them complete their homework on time. To measure parental engagement we will develop simulated homework assignments. Teachers will distribute and collect these simulated assignments as regular classroom assignments. The number of returned homework assignments will be a measure of parent engagement in their child’s schooling, since engaged parents are more likely to monitor their child’s homework.
We will use the simulated assignments to measure the extent to which parents engage directly with their child by having some simulated assignments require parental engagement. We will compare the average number of simulated assignments turned in by control and treatment students. We also will look at differences in the trajectory of the number of simulated assignments turned in over time. We expect that if parents become more engaged with their child’s schooling due to participation in TalkingPoints, the number of assignments turned in will increase for treatment parents but not control parents over time or at least over the first few months, after which the difference may level off.
Child’s Executive Functioning Skill. Executive function is a set of cognitive processes that are required for the control of behavior and include working memory, cognitive flexibility, and inhibitory control. These processes taken as a unitary construct control one’s ability to focus, stay on task when faced with distraction, exercise self control, set and accomplish goals, problem solve, remain flexible during times of transition or change, and manage the daily demands of life. Parent involvement in executive function training and development in children is hypothesized to have a positive impact, and TalkingPoints is a natural experiment to test this theory. We will compare the standardized score differential between pre and post-test periods for treatment and control students.
Parent Survey. We will collect qualitative feedback from parents about TalkingPoints usage and the evaluation at the end of the intervention via text (through TalkingPoints) with incentives.
Teacher Survey. At the end of the intervention we will implement a short survey of both treatment and control teachers to ask about the social, economic, and skill composition of their classrooms. We will also ask treatment teachers about their use of TalkingPoints, although we will have objective data on the fidelity of their use from data collected in TalkingPoints.
Experimental Design Details
Randomization Method
Randomization will be completed using statistical software such as Stata or R.
Randomization Unit
Randomization will be at the school level.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
50-60 schools with one participating 3rd grade classroom each.
Sample size: planned number of observations
1,500 students.
Sample size (or number of clusters) by treatment arms
750 control and 750 treatment students
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
To estimate power, we assume that we will have 60 schools each with one third-grade classroom with 25 students (for a total of 1,500 students). Half of the schools will be assigned to treatment. We assume an intraclass correlation of 0.2 and set power to 80% and the significance level to p=.05. If TalkingPoints is effective, more parents in the treatment group than in the control group will respond to the simulated text messages, more students will return the simulated homework assignments, and students’ scores will improve more on the measures of executive functioning skill. Under these assumptions we would be able to detect an effect when the treatment group returns twice as many texts or assignments even if the control group returns only two texts or assignments out of twelve. We tested the sensitivity of the results to ICC=0.4 instead of ICC=0.2, 50 rather than 60 schools, and when the difference between the treatment and control group drops to 50% and 33%. In the most conservative estimate with 50 schools and treatment parents returning 33% more texts and assignments the control group would have to return 7 out of 12 texts or assignments for us to detect a MDES=0.25. We could find no evidence on the proportion of students who return homework assignments. Informal interviews with teachers suggest that around 80% of advantaged students return a homework assignment, but for disadvantaged students the number can drop to well below half. Using 80% and 40% if half of students are advantaged and half disadvantaged in both the treatment and control group, then students would turn in 60% of assignments, or about 7 of 12 assignments absent an intervention. Using the power estimate for 60 schools ICC=0.2 and assuming that treatment students turn in 33% more of 12 assignments than control students, we would be able to detect this effect. Research suggests that unprompted parents will likely return only around 10% of text messages. We use the same assumptions to estimate power for the measures of executive functioning skill. With 60 schools, we would be able to detect an effect if the treatment group scores twice as many points as the control group as long as the control group answers 17% of the answers correctly. When the difference between the treatment and control drops to 50%, we would be able to detect the treatment effect as long as the control group scores at least 33% more. Furthermore, we have a good set of covariates and for the measures of executive functioning skill we will have pretest scores, both of which will further increase power. We will also condition on background variables to reduce the intraschool correlation further. Finally, we will estimate heterogeneity effects, because TalkingPoints expects effects to be greater for students in less advantaged schools and in families whose primary language is one other than English.
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Chicago SBS-IRB
IRB Approval Date
2019-04-29
IRB Approval Number
IRB19-0592

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
No
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials