The effects of a large-scale social and emotional learning program on educational outcomes

The intervention consists of conveying the content of the well-established Lions Quest Skills for Growing (LQ SfG) program during a designated school lesson in randomly assigned treatment schools. As a social and emotional learning (SEL) program, LQ SfG aims at helping children to make good decisions, among others, in terms of forward-looking behaviors such self-discipline, impulse-control, and goal-setting, or in terms of social awareness and relationship skills such as working cooperatively, empathy, and perspective-taking.

The program has a longstanding history and follows a curriculum that is implemented by the children’s regular teachers in the classroom environment. It is characterized by very interactive and diverse teaching with example stories, open discussions, role plays, and group work.

The full schedule comprises six units that each consist of several lessons: 1) building a positive learning community, 2) personal development, 3) social development, 4) health and prevention, 5) leadership and service, and 6) a reflection section on what has been learned.

In cooperation with the Lions Clubs International Foundation, learning materials got translated and adopted to the local context (for instance, pictures of children in the materials were changed to depict Bangladeshi rather than U.S. American or Indian children). In the course of our program adoption, school teachers got trained as LQ teachers in three-day workshops by international, qualified LQ trainers and received a textbook with detailed instructions. On top, refreshment trainings were organized half-way through the program. Children got a student journal in which they could summarize the topics and do homework. Parents were invited to a single, mid-program meeting with the LQ teachers, implementation staff, and local education authorities in their children’s school.

The program ran over the course of one school year following an official 28-week schedule (that omitted unit 5) between February and October 2019. Lessons took place on a weekly basis.

Each lesson lasts for about 30 minutes and is divided into four parts. First, the teacher presents an every-day situation and identifies together with the students why this situation is problematic (“discovery phase”). Second, students are encouraged to share similar experiences and the class discusses reasons and solutions for the problem (“connecting phase”). When learning how to make good decisions, for example, children are taught to act according to the “Think, Predict, Choose Model.” Being confronted with a decision, they are trained to follow a traffic-light approach and step back, calm down, reflect on their options and the consequences, and carefully work out what to do. They also discuss how to keep up their motivation for tedious tasks by not following immediate impulses but reminding themselves of why this goal is important to them or where they have been successful before. Third, students reenact the presented situations in role plays or solve tasks in pairs or small groups and employ the solutions and strategies they have talked about (“practicing phase”). Finally, teachers assign a homework related to the week's topic (“applying phase”). Students are, for instance, asked to discover and solve similar situations in their daily life and document their progress in the student journal.

Please see https://www.lions-quest.org/explore-our-selcurriculum/elementary-school-social-and-emotional-learning/ for more information.

2019-02-01

2019-10-31

Measures of teachers’ teaching style, their beliefs, attitudes, and behaviors with respect to teaching

We will interview teachers with an online questionnaire (browser-based, programmed in SurveyCTO) comprising the following measures:

i) Follow-up elicitation of teaching style (preferred teaching practices, use of visual aids, rating transfer of knowledge vs. transfer of values), love of job (How much do you enjoy being a teacher?), and days of absence with questions we have used in a short survey directly following the intervention.

ii) Measures of teaching style: scales and items taken and adapted from the OECD Teaching and Learning International Survey (TALIS) and the Measures of Effective Teaching (MET) program (Tripod’s survey instrument 7Cs of Effective Teaching), as well as statements regarding extrinsic motivation (rewards vs. punishments) and warmth in teaching (similar to Alan and Mumcu, 2023). For further information, please see the programs’ websites https://www.oecd.org/education/talis/ and https://usprogram.gatesfoundation.org/news-and-insights/articles/measures-of-effective-teaching-project.

iii) Questions regarding teachers’ beliefs, attitudes, and behaviors with respect to teaching: perceived importance and responsibility of teachers for students’ achievements, time spent preparing for school lessons, and, following Alan and Mumcu (2023), competence-beliefs, growth-mindset (based on Dweck, 2006), and critical-thinking (based on Sosu, 2013) items.

iv) Measures of teaching approaches specifically with respect to official Lions-Quest (LQ, the intervention program) goals: statements to be rated to measure perceptions of the importance of official LQ goals as well as to measure perceived competence in promoting these goals (based on Talvio et al., 2015).

v) LQ-related questions: questions regarding the effect of the program on classroom environment, on teachers’ skills in interacting with children and explaining materials, and on their motivation.


References

Alan, S. and I. Mumcu (2023). Nurturing Childhood Curiosity to Enhance Learning: Evidence from a Randomized Pedagogical Intervention. Forthcoming in American Economic Review.

Dweck, C. S. (2006). Mindset: The New Psychology of Success. Random House.

Sosu, E. M. (2013). The Development and Psychometric Validation of a Critical Thinking Disposition Scale. Thinking Skills and Creativity, 9:107-119.

Talvio, M. , M. Berg, E. Ketonen, E. Komulainen, and K. Lonka (2015). Progress in Teachers’ Readiness to Promote Positive Youth Development among Students during the Lions Quest Teaching Workshop. Journal of Education and Training Studies, 3(6):1-13.

Measures of teachers’ personality in terms of the Big 5 taxonomy, locus of control, as well as time and social preferences (A.); if feasible, school exam results of children (B.); assessments of intervention effects on teachers’ teaching and reports of how schooling was handled during the Covid-19 pandemic by schools’ headmasters (C.)

A. Teacher level: The teachers’ questionnaire will further contain the following measures:

i) Big 5 personality scale: 16-item Big-5 inventory derived from John et al. (1991) and evaluated in Gerlitz and Schupp (2005) and Weinhardt and Schupp (2011).

ii) Locus of control personality scale: 10 different items adapted from Rotter (1966) and used, e.g., in the 2005-wave of the German Socio-Economic Panel (SOEP).

iii) Time and social preferences: items from the Global Preferences Survey (Falk et al., 2018) regarding time preferences and social preferences (altruism and reciprocity).

B. School level: If feasible, we will collect school exam results from our sample schools:

Exam results are for regular school exams in six subjects (math, Bangla, English, science, social studies, and religion). They are designed by school authorities (official education boards) and graded by teachers. We will try to gather results for multiple years following the intervention.

C. School level: We will interview schools’ headmasters in person (CAPI, programmed in SurveyCTO) with a questionnaire comprising the following questions (besides refining the matching of teachers to subjects and grades they taught during the intervention period):

i) For program schools, questions regarding program impacts on teachers’ teaching style and regarding the selection of program teachers.

ii) For all schools, questions to collect information on how schooling was handled during the Covid-19 pandemic, that is, tracking of school closures, remote learning opportunities, and school-specific inputs.


References

Falk, A., A. Becker, T. Dohmen, B. Enke, D. Huffman, and U. Sunde (2018). Global Evidence on Economic Preferences. The Quarterly Journal of Economics, 133(4):1645-1692.

Gerlitz, J.-Y. and J. Schupp (2005). Zur Erhebung der Big-Five-Basierten Persönlichkeitsmerkmale im SOEP. Research Notes, 4. DIW, Berlin.

John, O. P., E. M. Donahue, and R. L. Kentle (1991). The Big Five Inventory—versions 4a and 5. University of California, Berkeley, Institute of Personality and Social Research.

Rotter, J. B. (1966). Generalized Expectancies for Internal versus External Control of Reinforcement. Psychological Monographs: General and Applied, 80 (1):1-28.

Weinhardt, M. and J. Schupp (2011). Multi-Itemskalen im SOEP Jugendfragebogen. Data Documentation, 60. DIW, Berlin.

We have a single treatment and a single control group. The 135 elementary schools in our sample are divided into 69 treatment and 66 control group schools by stratified randomization. In cooperation with the Lions Clubs International Foundation, we implemented the social and emotional learning (SEL) program Lions Quest Skills for Growing (PreK-5) in grades 2 to 5 of the 69 treatment schools. The remaining 66 schools in the control group did not receive any treatment.

Not available

Randomization was done by a computer using the statistic program Stata. In order to gain a balanced treatment-control group setting, we stratified by subdistrict (our schools are located in 11 subdistricts differing in their schooling authorities), by distance of the school to its respective subdistrict capital (as a proxy for schooling quality) and village literacy rate. No re-randomization was done, but the first draw taken.

The treatment was randomized on school level.

Yes

Treatment was assigned to 69 out of a total of 135 schools (hence, 66 schools in control group). These schools are serving 150 villages that we randomly selected from 11 subdistricts belonging to four districts of Bangladesh.

The RCT is based on 3,000 households from the 150 villages that are served by the 135 elementary schools in our sample: sampled by selecting at least 20 children from grades 2 to 5 of each school in 2019 (larger samples from schools serving multiple sample villages, e.g., about 40 children per school if the school serves two villages). We have a child sample that covers more than 3,000 students attending grades 2 to 5 in 2019 plus, if they have siblings, information on one of them (if they have siblings, the additionally sampled sibling is chosen randomly). Further, in general, two teachers per school from treatment and control schools got selected: constitutes a sample of 277 teachers who have been interviewed in a short survey directly following the intervention at the beginning of 2020. In treatment schools, these are Lions-Quest (LQ, the intervention program) teachers. In control schools, two teachers got randomly selected. We plan to follow up with this sample of teachers. If feasible, we extend the sample to other teachers who have taught at the 135 elementary schools during the intervention period.

69 schools in treatment group, 66 schools in control group

Power calculations for the number of schools/students have been conducted for the original data collection described in AEARCTR-0003129. We now collect additional information for teachers in the schools that participated in the RCT.