TalkingPoints Evaluation

The Evaluation
The Sample. The evaluation will be a school-based randomized controlled trial that will be implemented in grade three. We focus on one grade to improve power. We selected third grade because it is the point in schooling when academics become more rigorous and consequential in public schools. Developmentally and institutionally, students are expected to have acclimated to the classroom routine and have the executive functioning skills for staying on task and completing homework by third grade, which are elements required for our outcome measures described below. The evaluation will include 50-60 schools in the US. One third-grade classroom in each school will participate in the evaluation. Classes are expected to include about 25 students. There is an oversample of Title I schools and schools with a large English Second Language population.
The Treatment. Students in one third-grade classroom in both treatment and control schools will be enrolled in TalkingPoints for this evaluation. Teachers in treatment classrooms will have access to TalkingPoints while teachers in control classrooms will not. Control group teachers will be subject to “business as usual” in terms of communicating with parents. Prior to randomization, teachers in both treatment and control classrooms will register their students in TalkingPoints by entering demographic into the TalkingPoints registration system. These data will be used to test balance across the treatment and control groups and as covariates in the data analysis. We also plan to collect student attendance data from the school or district.
We will consent teachers and students based on the policies of participating schools. Once students in all participating classrooms are registered and parents have consented, schools will be randomized into either the treatment or the control group. Following randomization, treatment schools will receive TalkingPoints at no cost for the 2020–2021 school year. Control group schools will not receive TalkingPoints in 2020–2021 but will receive it at no cost in the 2021–2022 school year. TalkingPoints will become available to treatment schools in October 2020. The treatment will continue until the end of the school year. All student assessments and parent and teacher surveys will be completed in May 2021.

2020-10-05

2021-05-28

We have three outcome variables of interest, which align with the end goals of TalkingPoints: 1) communication between teachers and parents; 2) parent engagement at home with their child's schooling; and 3) student executive functioning skills, which are a precursor to academic and life success.

We will survey parents and teachers to get qualitative feedback on their experience using TalkingPoints.

Communication between Teachers and Parents. As an objective measure of parent-teacher communication, we will use simulated communications to test whether treatment parents are more likely than control parents to communicate with schools. If TalkingPoints increases the chance that a parent will communicate with the school, then when TalkingPoints sends a message to both treatment and control parents asking for a response, parents in the treatment group ought to be more likely than parents in the control group to respond. For example, a simulated message might be, “Did your child have any math homework this week?” The parent would be asked to reply by clicking yes or no. The measure of communication will be the number of replies.
Parent Engagement in Their Child’s Schooling. We cannot observe parent engagement with their child at home and parent reports of their own engagement are likely to be biased, so we will create simulated engagement activities. One of the main ways that schools hope parents will engage their children is by helping them complete their homework on time. To measure parental engagement we will develop simulated homework assignments. Teachers will distribute and collect these simulated assignments as regular classroom assignments. The number of returned homework assignments will be a measure of parent engagement in their child’s schooling, since engaged parents are more likely to monitor their child’s homework.
We will use the simulated assignments to measure the extent to which parents engage directly with their child by having some simulated assignments require parental engagement. We will compare the average number of simulated assignments turned in by control and treatment students. We also will look at differences in the trajectory of the number of simulated assignments turned in over time. We expect that if parents become more engaged with their child’s schooling due to participation in TalkingPoints, the number of assignments turned in will increase for treatment parents but not control parents over time or at least over the first few months, after which the difference may level off.
Child’s Executive Functioning Skill. Executive function is a set of cognitive processes that are required for the control of behavior and include working memory, cognitive flexibility, and inhibitory control. These processes taken as a unitary construct control one’s ability to focus, stay on task when faced with distraction, exercise self control, set and accomplish goals, problem solve, remain flexible during times of transition or change, and manage the daily demands of life. Parent involvement in executive function training and development in children is hypothesized to have a positive impact, and TalkingPoints is a natural experiment to test this theory. We will compare the standardized score differential between pre and post-test periods for treatment and control students.
Parent Survey. We will collect qualitative feedback from parents about TalkingPoints usage and the evaluation at the end of the intervention via text (through TalkingPoints) with incentives.
Teacher Survey. At the end of the intervention we will implement a short survey of both treatment and control teachers to ask about the social, economic, and skill composition of their classrooms. We will also ask treatment teachers about their use of TalkingPoints, although we will have objective data on the fidelity of their use from data collected in TalkingPoints.

Randomization will be completed using statistical software such as Stata or R.

Randomization will be at the school level.

No

50-60 schools with one participating 3rd grade classroom each.

1,500 students.

750 control and 750 treatment students

To estimate power, we assume that we will have 60 schools each with one third-grade classroom with 25 students (for a total of 1,500 students). Half of the schools will be assigned to treatment. We assume an intraclass correlation of 0.2 and set power to 80% and the significance level to p=.05. If TalkingPoints is effective, more parents in the treatment group than in the control group will respond to the simulated text messages, more students will return the simulated homework assignments, and students’ scores will improve more on the measures of executive functioning skill. Under these assumptions we would be able to detect an effect when the treatment group returns twice as many texts or assignments even if the control group returns only two texts or assignments out of twelve. We tested the sensitivity of the results to ICC=0.4 instead of ICC=0.2, 50 rather than 60 schools, and when the difference between the treatment and control group drops to 50% and 33%. In the most conservative estimate with 50 schools and treatment parents returning 33% more texts and assignments the control group would have to return 7 out of 12 texts or assignments for us to detect a MDES=0.25. We could find no evidence on the proportion of students who return homework assignments. Informal interviews with teachers suggest that around 80% of advantaged students return a homework assignment, but for disadvantaged students the number can drop to well below half. Using 80% and 40% if half of students are advantaged and half disadvantaged in both the treatment and control group, then students would turn in 60% of assignments, or about 7 of 12 assignments absent an intervention. Using the power estimate for 60 schools ICC=0.2 and assuming that treatment students turn in 33% more of 12 assignments than control students, we would be able to detect this effect. Research suggests that unprompted parents will likely return only around 10% of text messages. We use the same assumptions to estimate power for the measures of executive functioning skill. With 60 schools, we would be able to detect an effect if the treatment group scores twice as many points as the control group as long as the control group answers 17% of the answers correctly. When the difference between the treatment and control drops to 50%, we would be able to detect the treatment effect as long as the control group scores at least 33% more. Furthermore, we have a good set of covariates and for the measures of executive functioning skill we will have pretest scores, both of which will further increase power. We will also condition on background variables to reduce the intraschool correlation further. Finally, we will estimate heterogeneity effects, because TalkingPoints expects effects to be greater for students in less advantaged schools and in families whose primary language is one other than English.