Unlocking the Value of Social and Emotional Learning Interventions: Maximizing Benefits and Measuring the Opportunity Cost

We propose to implement the RCT in Mexico through the NGO Jóvenes Ayudando a Niñas y Niños A.C. JANN is an organization that manages a free tutoring program that connects about K-12 students, with grades ranging from 1st to 9th, with university students that provide Math tutoring online. Tutors and students meet twice weekly for one hour from 12 to 24 weeks. We are concerned with measuring the impact of a social and emotional learning intervention on math learning. The intervention consists of 8 modules of social and emotional learning material based on RULER (Hoffmann, Brackett, Bailey Willner,2020).


In a previous RCT we documented learning gains in math associated with receiving the additional SEL curriculum.
This RCT follows up and aims to answer two policy relevant research questions.

What is the opportunity cost of devoting instruction time to SEL rather than other subjects, for example Math. The intervention comes at the cost of sacrificing instructional time that would have been devoted to math in the absence of the intervention. Do the benefits brought by SEL outweigh the costs?
What channel is driving the effects in learning derived from the SEL intervention? This can help understand the optimal delivery of SEL content to obtain the maximum benefits derived from SEL.We hypothesize that there may be three channels driving the effects observed, each of which has different policy implications for better implementation.

H1 SEL activities yield higher learning through the change they cause in the tutor-student relationship. Creating a climate of trust in the class, improving the relationship between the tutor and the students, and potentially increasing rapport. This is important because SEL would benefit most if taught by teachers who also teach other classes instead of specialized SEL teachers.

H2 SEL activities yield higher learning through the change they cause on the tutors' skills. The tutors who learned SEL improved their emotional intelligence. By improving these skills, they became more aware of their students’ needs, and thus, they became better teachers. In this case, SEL interventions could be delivered directly to teachers through professional development and would not need to reduce instruction time devoted to other subjects.
H3 SEL activities yield higher learning through the change they cause in the students. The students under the SEL condition, improved their emotional intelligence, and made the class easier to teach. This allowed the tutor to enjoy the class more and thus invest more effort. This would imply that teaching kids socioemotional skills can benefit them in Math learning regardless of who teaches SEL and that there are possibly spillover effects in other disciplines. In this case, the opportunity cost is more likely to be outweighed by the benefits observed in other classes.

2023-10-23

2024-07-31

Our main outcome is the academic learning of students measured by the Math and Spanish score on an evaluation designed by us, which is an adaptive exam. Students will answer a version of this evaluation at baseline and at endline. The second main outcome is the student-tutor attachment measured using the student version of the teacher–student relationship inventory (S-TSRI) (Ang, Ong, & Li 2020).

Given that we are now focusing on the well-being of students and tutors, we will add an additional measure with respect to the 2022/2023 RCT, which is the SDQ 20 both for tutors and students (Nijenhuis, 2010). Finally, we’ll also analyze features we will extract from the recordings of the audio of tutoring sessions. These will give us an insight of how the interactions between the students and tutors changed due to the different interventions and see how these interactions relate to the success of the tutoring sessions. For instance, variables that we are considering are the tone in which the tutor speaks (encouraging, scolding, etc.), the frequency of each student’s interventions, the number of questions asked by students and tutors, the number of times that the tutor calls the students by name and students call each other by name, the length of their classes, among other measures that we are still working on developing.

A The first treatment will be assigning students to receive N SEL activities designed by pedagogist Cimenna Chao besides math.

B In this treatment tutors will be asked to teach Math and N History lessons to one group. Additionally the tutors of these groups will participate in N sessions designed to improve their emotional intelligence. The sessions will only be held in groups of tutors, and thus their students will never be exposed to the content, thus we can isolate the effect of the SEL activities on the tutors.

C In this treatment, each group of students will be assigned two tutors. One of the tutors will teach them Math and the N history activities. The other tutor will solely teach them the N SEL activities aforementioned. The reason for this treatment to exist is to isolate the effect SEL has on students.
D In this treatment, tutors will be asked to provide the students with Math and the N history lessons. This treatment was tested as-is on the past RCT and compared to A, where it yielded lower learning. We will compare A,B and C to D to test H1,H2 and H3.

E In this treatment, tutors will be asked to spend all instruction time teaching Math to their students with no additional activities, comparing treatments arms A,B,C will to E will allow us to benchmark the learning gains of SEL against a world where time devoted to SEL is used for instruction. Rather than the comparison with D where instruction time devoted to SEL does not substitute instruction time devoted to math. The comparison between D and E will allow us to identify the opportunity cost of SEL activities.

Due to time constraints some groups will receive only N=5 and most groups will receive N=8 either history or SEL classes. In the main specification we will control for this and we will do a subgroup analysis differentiating the groups that received partial treatment N=5 from the groups that received the full treatment.

Not available

We randomize using a computer.

We are randomizing students to treatment arms once they are assigned to a treatment arm, they are randomly grouped in groups of 5 students of the same grade. On the other side, we randomize tutors to treatments and then, within each treatment, randomly assign groups to tutors. We will cluster standard errors at the tutor level for the main specification. Each tutor teaches 1 to 4 groups of 5 students.

Yes

at least 1500 tutors depending on the ability of our partner organization to recruit participants.

15,000 students.

We will have 300 tutors per treatment arm.

Assuming an attrition rate of 25% on the students' side, that each tutor takes on average 2 groups of 5 students each, the MDE for the three ICC scenarios is: 0.045 for ICC of 0.05, 0.078 for ICC of 0.15 and 0.1 for ICC 0.25.