Experimental evaluation of the integration of Sesame Workshop’s Watch, Play, Learn and the Semillas de Apego intervention to foster maternal mental health and early childhood development among conflict-affected communities in Colombia

The SA program is implemented through 15-weekly sessions among groups of 16 participants, who are mothers or primary caregivers of children 3 to 5. The implementation of the SA program is based on a community-based model, where each implementation group is led by 2 non-professional community agents, also known as facilitators, who have previously participated in a three-week intensive training or have been participants of the SA program in the past. Overall, there are 8 facilitators, arranged into 4 dyads. In each municipality, one dyad will lead the group assigned to the standard model while the other dyad will lead the sessions of the groups assigned to the enhanced program. There will be 10 groups in the WPL intervention and 10 groups in the control arm, with 160 participants in each group.
Two to three weeks before the program started, community meetings were conducted to share the information on the program, its objectives, structure, and eligibility criteria, and to invite caregivers to participate. To facilitate participation and promote adherence to the program, we held during the day and time convened with the participants, provided incentives, including a 10 USD voucher for the participants every time they attend 3 sessions and a monthly phone plan to those participants that remain in the program.
Only the treatment group received the WPL content. The WPL content was conceptually integrated into each session starting from session 7, which is when SA sessions’ objectives emphasize understanding early childhood development milestones and healthy child-parent relationships to promote Early Childhood Development (ECD) (weeks 8-12) and building support networks and child-rearing teams (weeks 12-15). The education team at SW and the technical team at Universidad de los Andes organized the WPL videos and visual aids into five groups. Group 1 introduces big emotions; Group 2 addresses fear and dealing with challenging situations; Group 3 helps caregivers relate the child's behavior to big emotions and identify alternative strategies to deal with these emotions; Group 4 involves child-rearing teams in identifying the child's emotions and strategies; and Group 5 reinforces breathing techniques. At the end of each session, facilitators linked the discussions that emerged during the session to the WPL videos.
The WPL content was delivered through social media (Facebook and WhatsApp). On FB, the contents were disseminated on private groups within the SA general page, one private group per intervention group. Private groups allow control of the enrollment of caregivers, which is key for the validity of the experimental evaluation, and in particular to prevent contamination to caregivers in the control group. In the private groups, caregivers in the treatment arms received private posts with WPL contents after each weekly session is completed in addition to the standard SA contents. Additionally, they received messages through private group chats and had the chance to interact with other members in the same group. Participants in the control group only received SA content via group chats and the public posts on the SA FB general page. Since the success of this strategy depended on access to internet, participants in both groups received pre-paid phone plans.
On WhatsApp, participants in the treatment arm received a message with a WPL trailer for that week’s videos and a link inviting to watch the videos in the corresponding FB group. Additionally, by the end of each week, the participants in the treatment arm received a full WPL video, in case they were not able to join FB for any particular reason. In this sense, WhatsApp allow for greater reach as it gives access to those participants without FB access. Facebook’s large advantage is that it provides data analytics to understand the participant’s access and use of the WPL content.

2023-04-10

2023-07-28

We identify five different constructs as primary outcomes for evaluation. We will conduct two rounds of post-intervention data collection that will take place immediately after implementation and four (4) months after the 15-week program has concluded. This will enable us to observe the short and medium-run impacts of the intervention. The second post-intervention will be conditioned to the results of access and use of the Sesame Workshop content in the first post-intervention evaluation.
The five primary outcomes and the corresponding hypothesis that guide this evaluation are explained below. For each outcome, we will pool data from the following psychometric scales and construct a composite index for each family of outcomes, following Kling et al. (2020) based on the standardized inverse-covariance weighted average of the different items in each dimension. This estimation method reduces measurement error in any individual scale or assessment and reduces the dimensions of the analysis, which increases the statistical power. For children’s outcomes, we will also standardize by age. The primary outcomes are the following:
1. Primary caregiver’s mental health: Participation in the intervention arm of the evaluation will generate a reduction in the number of emotional symptoms reported by caregivers one month after the intervention has been completed (first post-intervention assessment) and noticeably four months after the end of the intervention (second post-intervention assessment). This outcome will be measured with five sub-scales of an adapted version of the Symptom Checklist-90-R (Derogatis, 1994). This self-reported measure asks caregivers to report on several indicators of emotional well-being and mental health experienced during the last 30 days.
2. Child rearing practices: Participation in the intervention arm of the evaluation will generate an increment in the number and type of reported child-rearing practices implemented by caregivers right after the end of the intervention (first post-intervention assessment). These effects will persist four months after the end of the intervention (second post-intervention assessment). Outcomes will be measured with a questionnaire in which caregivers report whether they engaged in any of the six stimulating activities with their children in the previous week: (1) reading stories or looking at books with images; (2) telling stories; (3) singing songs; (4) playing with child; (5) taking the child outside; and (6) spending time in physical activities with child. Additionally, they report the type of disciplines used during child rearing practice. To analyze this construct as a composite measure we will compute a summary score ranging from zero (no engagement in any activity) to six (engagement in the six activities), as previously done in several studies (e.g., Bornstein & Putnick, 2012; Cabrera et al., 2011; Jeong et al., 2017; Jeong et al.,2016; Sunet al., 2016) and also weigh in the corresponding score of the disciplines.
3. Healthy child-parent emotional bonds: Participation in the intervention arm of the evaluation will lower the stress in the child parent relationship (first post-intervention assessment) and will generate a positive effect in the quality of the attachment relationship between caregiver and children four months after the end of the intervention (second post-intervention assessment). Outcome will be measured with three different instruments: the parenting stress-index (PSI, Hasket et al, 2006), the Being a Mother (BaM, Matthey, 2011) and the child parent relationship (CPR, Discroll and Pianta, 2011). The PSI is a measure focused on three major domains of stress: child characteristics, parent characteristics and situational/demographic life stress. Additionally, during the BaM questionnaire, participants will be asked to answer 13 items on child and adult experience and relationship closeness, which they will respond to using a 4-point Likert scale. Finally, CPR asks the caregiver 30 items which capture the quality of the child-parent relationship. We will compute a summary score for the three scales.
4. Children’s mental health: Participation in the intervention arm of the evaluation will generate a positive effect in the caregivers’ report of child mental health in the medium run, or four months after the end of the intervention (second post-intervention assessment). Outcome will be measured with an adapted instrument of the Trauma Symptom Checklist for Young Children (TSCYC, Briere, 2005) to describe child-levels of trauma and abuse-related symptomatology.
5. Child’s emotional development: Participation in the intervention arm of the evaluation will generate a positive effect in children’s’ cognitive and socioemotional development in the medium run, or four months after the end of the intervention (second post-intervention assessment). Measured with the Strengths and Difficulties Questionnaire (ASQ-SE). We will focus our report on a composite score of the five different subscales within this questionnaire.

Based on dynamic auto-complementarity in early childhood development and parental inputs, we expect the WPL and SA integration to have multiplying effects, with the two channels reinforcing each other. As caregivers are better able to respond to children's progress, investments should become more productive improving emotional coregulatory process between caregiver and child and better social and emotional outcomes for the child. This should then map out into caregiver’s perceptions of the effectiveness of their inputs and strategies and further promote their capacity and willingness to respond to children’s development, thereby creating virtuous cycles and reinforcing effects.

The secondary outcomes of the evaluation are mainly exploratory and consist of analyzing each individual scale and subscales. We will calculate individual continuous z scores and risk thresholds for each subscale and scale. We will adjust the signs of each scale so that positive scores denote improvements and standardize the scales according to the standard deviation of the control group by wave. The secondary outcomes in each primary outcome are:
1. Primary caregiver’s mental health: We focus on the following Symptom Checklist-90-R’s sub-scales: (i) Anxiety, (ii) Depression, (iii) Phobic Anxiety, (iv) Hostility and (v) Sensitivity.
2. Child rearing practices: We will focus on three subscales: the six stimulating activities, the negative disciplines and the positive disciplines.
3. Healthy child-parent emotional bonds: Since there are three different instruments for this outcome, we have three different types of subscales. We will analyze the individual sub-scales for the following domains of the PSI: parental distress, parent-child dysfunctional interaction, and difficult child. BaM will provide insights into the following subscales: emotional closeness, child experience and adult experience. CPR will assess the subscales: emotional closeness, conflict resolution and dependence.
4. Children’s mental health: The subscales of the Trauma Symptom Checklist for Young Children (TSCYC, Briere, 2005) are: anger/agression, anxiety, depression, dissociation, sex concerns, post-traumatic stress arousal, post-traumatic stress avoidance, post-traumatic stress intrusion and post-traumatic stress total.
5. Child’s emotional development: We will assess each child on five different SDQ subscales including emotional symptoms, conduct problems, hyperactivity and inattention, peer relationship problems and prosocial behavior.

The experimental evaluation of the enhanced model will be conducted as a Cluster-Randomized Control Trial (C-RCT). The eligible population for the study is all families in Tumaco and Jamundí, Colombia whose children’s ages range between 3 to 5 years of age.
Random assignment to the treatment was conducted at the group level (n=20). The groups were assigned by neighborhoods to minimize possible spillovers and to facilitate the program’s implementation. We could not cluster our random assignment at the facilitator level (randomizing which pair of facilitators has the WPL content), because we would have few clusters, and this hurts our power. For this reason, we will randomize across groups, meaning that certain facilitators will introduce the WPL in some groups. Participants are, thus, blind to treatment prior to the implementation.
Originally, 10 groups were going to be assigned to receive the enhanced model and the other 10 groups would have been assigned to the control arm. However, once the implementation started, only 9 groups were selected to the treatment arm due to geographical and contextual violence reasons. Additionally, the groups were expected to have 16 participants, but they actually have on average 13 participants, totaling 227 participants in the pilot.

Randomization using Stata commands

Intervention groups (n=20)

Yes

20 intervention groups

320 participants

10 intervention groups each into treatment and control arms; 160 participants per treatment arm.

With 20 intervention groups overall and 160 participants in each treatment arm, we estimated 80% of power to detect effect size of 0.4 standard deviations at minimum. See Figure 1. The minimal detectable effect size (MDES) is based on estimates a 0.05 probability of type I error, assigning 10 groups and half of the individual-level sample (160) to the treatment, and an intra-class correlation coefficient (ICC) of 0.03. Graph 1: Power analysis for planned scenario