|
Field
Trial Status
|
Before
in_development
|
After
on_going
|
|
Field
Abstract
|
Before
Evidence suggests that traumatic events, such as pandemics and wars, can impact children’s learning, socio-emotional development, and sense of protection (Quintana-Domeque and Ródenas-Serrano, 2017; Almond et al. 2018). Ukraine’s education system faces critical constraints in providing high-quality education to its students on their path toward recovery after disruptions to schooling and learning due to years of pandemic-related school closures. While children in many countries have gone back to school, the return to in-person education has been hindered by a lack of security, significant student and teacher displacement, and school damages posed by 6 months of the Russian invasion. Currently, learning losses in Ukraine are estimated to be over one year (Angrist et al, 2022), with learning outcomes falling below the lowest-performing countries in Europe which will have substantial impacts on human capital development in the country. To mitigate the impact of these traumatic events on children, the Ukrainian education system must find new strategies for supporting learning recovery and increasing learning equity while children are not able to return to in-person schooling.
High dosage small group instructional tutoring can be a powerful strategy to improve learning outcomes and cognitive and socioemotional skills, as it offers students a massive increase in personalized instruction, enabling teaching at the right level (Banerjee et al., 2015). To test the effectiveness of these programs in a conflict-affected setting, we study a tutoring program offering supplemental learning in math and Ukrainian language and psychosocial support. The target population of the tutoring program is Ukrainian students in grades 5 to 10 who are seeking supplemental support beyond the standard online schooling schedule. Initially, students are placed in groups of 3 and will receive 3 hours of tutoring per week for 6 weeks by paid-for tutors through the Edmodo online platform. The implementing partner is Teach for Ukraine.
The impact evaluation will include three waves, through which we will test for the effectiveness of varying certain attributes, including program length, group size, content allocation, group composition by ability, and tutors as role models. Students will be recruited in each wave. All waves will have one control and one treatment group. Waves 2 and 3 will have an additional group composed of students who participated in the control group in the previous wave who will receive the treatment. In this group, we will either test the effect of group composition or pilot an attribute to be tested in the following wave.
The effects of the tutoring program will be measured in math and Ukrainian language test scores and socioemotional skills. As secondary outcomes, we will measure the effects of the intervention on expectations, mental health, time use, attendance to the tutoring activities, and attitudes towards tutoring.
|
After
Evidence suggests that traumatic events, such as pandemics and wars, can impact children’s learning, socio-emotional development, and sense of protection (Quintana-Domeque and Ródenas-Serrano, 2017; Almond et al. 2018). Ukraine’s education system faces critical constraints in providing high-quality education to its students on their path toward recovery after disruptions to schooling and learning due to years of pandemic-related school closures. While children in many countries have gone back to school, the return to in-person education has been hindered by a lack of security, significant student and teacher displacement, and school damages posed by 6 months of the Russian invasion. Currently, learning losses in Ukraine are estimated to be over one year (Angrist et al, 2022), with learning outcomes falling below the lowest-performing countries in Europe which will have substantial impacts on human capital development in the country. To mitigate the impact of these traumatic events on children, the Ukrainian education system must find new strategies for supporting learning recovery and increasing learning equity while children are not able to return to in-person schooling.
High dosage small group instructional tutoring can be a powerful strategy to improve learning outcomes and cognitive and socioemotional skills, as it offers students a massive increase in personalized instruction, enabling teaching at the right level (Banerjee et al., 2015). To test the effectiveness of these programs in a conflict-affected setting, we study a tutoring program offering supplemental learning in math and Ukrainian language and psychosocial support. The target population of the tutoring program is Ukrainian students in grades 5 to 10 who are seeking supplemental support beyond the standard online schooling schedule. Initially, students are placed in groups of 3 and will receive 3 hours of tutoring per week for 6 weeks by paid-for tutors through an online platform. The implementing partner is Teach for Ukraine (https://teachforukraine.org/en/).
The impact evaluation will include three waves, through which we will test for the effectiveness of varying certain attributes, including program length, group size, content allocation, group composition by ability, and tutors as role models. Students will be recruited in each wave. All waves will have one control and one treatment group.
The effects of the tutoring program will be measured in math and Ukrainian language test scores, socioemotional skills, and mental health. As secondary outcomes, we will measure the effects of the intervention on expectations, time use, attendance to the tutoring activities, and attitudes towards tutoring.
|
|
Field
Trial End Date
|
Before
September 29, 2023
|
After
August 31, 2024
|
|
Field
Last Published
|
Before
December 16, 2022 04:03 PM
|
After
December 28, 2023 01:45 PM
|
|
Field
Intervention (Public)
|
Before
The intervention under evaluation consists of a tutoring program offering supplemental learning in math and Ukrainian language and psychosocial support. The learning component includes subject-specific academic content in accordance with Ukrainian educational programs. The socio-emotional part includes activities to support student’s social and emotional well-being. The total scheduled time for psychosocial support activities is 30 mins per week. The target population of the tutoring program is Ukrainian students in grades 5 to 10 who seek supplemental support beyond the standard online schooling schedule. Students are placed in groups of 3 and will receive 3 hours of tutoring per week for 6 weeks by paid-for tutors through the Edmodo online platform. The implementing partner is Teach for Ukraine.
|
After
The intervention under evaluation consists of a tutoring program offering supplemental learning in math and Ukrainian language and psychosocial support. The learning component includes subject-specific academic content in accordance with Ukrainian educational programs. The psychosocial support component includes activities to enhance student’s social and emotional well-being. The target population of the tutoring program is Ukrainian students in grades 5 to 10 who seek supplemental support beyond the standard online schooling schedule.
The structure of the intervention will vary in each wave. In wave 1 of the intervention, we implement a "Barebones Program" where the group size is three students, the number of hours per week is 3, and the number of subjects is 2 (math and Ukrainian language).
In wave 2, we keep the same structure of the program as in wave 1, but we implement a variation--that we call the "enhanced program"-- with three additional features. First, we conduct a short assessment during the enrolment stage to have a proxy for student knowledge prior to the beginning of the tutoring program and used this measure to rank students and sort them in groups based on ability levels. Second, we use the information in this short assessment to prepare and share a diagnostic report with tutors for each group assigned to them, containing aggregated information on how well each group performed relative to the average for all students. Third, we developed short curriculum-based formative assessments and shared with tutors during the second week of the program to be used as a guide for tutors to assess and learn how well students were doing on several learning outcomes.
In the third wave, we use the same structure of the program as in wave 2 but we work with a team of psychologists and specialists from the Harvard Program in Refugee Trauma (HPRT) to enhance the psychosocial support component. In the updated curriculum for this component, we include activities related to breathing exercises, mental check ins, parables, and other activities.
|
|
Field
Intervention End Date
|
Before
June 30, 2023
|
After
April 30, 2024
|
|
Field
Primary Outcomes (End Points)
|
Before
Math test score
Ukrainian language test score
Socioemotional skills (locus of control, GRIT/perseverance)
|
After
Math test score
Ukrainian language test score
Mental health
|
|
Field
Experimental Design (Public)
|
Before
The impact evaluation will be conducted in three waves of tutoring. In each wave, we will test for the effectiveness of varying certain attributes, including program length, group size, group composition by ability, tutor feedback, and tutor value-added. The exact attribute to be tested in each wave will depend on the results from the previous wave.
Students will be recruited in each wave. The tutoring program will be advertised on social network platforms and disseminated by the Ministry to schools around the country, influencers in the Education Sector, and TFU. Students interested in taking part in the tutoring program will register online, provide responses to a questionnaire, and provide their own consent as well as parental consent. Students who submit this information will become part of the study sample and, for all the 3 waves, will be randomly assigned to the treatment or the control group.
The randomization procedure for each wave will consist of stratified household-level randomization. The stratification variables for each wave will be student grade, preferred schedule (Mon-Tue-Wed or Wed-Thu-Fri), and if they were in or outside Ukraine. Then, students randomly allocated to a treatment status will be randomly assigned to each of the tutors. All treatment groups will receive a variation of the tutoring program in groups of 3 students.
All waves will have one control and one treatment group. For waves 2 and 3, we will add an additional treatment group that will consist of students in the control group in the previous wave. For example, in wave 2, we will work with three groups. We randomly assign enrolled students to treatment or control, and we will also treat students in the control group from wave 1. These students will be randomly assigned to a specific attribute, such as providing teachers feedback or estimating tutor value-added.
|
After
The impact evaluation will be conducted in three waves of tutoring. In each wave, we will test for the effectiveness of varying certain attributes and following an agile experimenting approach. The exact attribute to be varied and tested in each wave will depend on the results from the previous wave.
Students will be recruited in each wave. The tutoring program will be advertised on social network platforms and disseminated by the Ministry to schools around the country, influencers in the Education Sector, and TFU. Students interested in taking part in the tutoring program will register online, provide responses to a questionnaire, and provide their own consent as well as parental consent. Students who submit this information will become part of the study sample and, for all the 3 waves, will be randomly assigned to the treatment or the control group.
The randomization procedure for each wave will consist of a two-stage stratified household-level randomization. In each wave, at the first stage, all enrolled students are randomly assigned to treatment or control groups. The stratification variables are whether the parent has completed higher education and if they were living in Ukraine for wave 1 and the interaction between region (Central, Eastern, Southern, Western, Out of Ukraine) and if the parent has completed higher education in waves 2 and 3. Those in the treatment groups receive the tutoring activities, and the ones in the control groups enrolled in the online platform and were allowed to interact among each other but did not receive the tutoring sessions.
For the second stage, we assign students to different tutoring groups stratifying by grade and preferred schedule in wave 1. The stratification variables in waves 2 and 3 included grade and preferred schedule and they were also ranked by ability using a short knowledge test in math and Ukrainian language collected during the registration period. Students are then sorted into small groups based on their ranking position.
|
|
Field
Planned Number of Clusters
|
Before
8,000 households
|
After
At least 8,000 households
|
|
Field
Planned Number of Observations
|
Before
10,000 students
|
After
At least 9,600 students
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
Wave 1:
Treatment 1 --> 1,200
Control --> 2,000
Wave 2.
Treatment 2 --> 2,000
Treatment 3 (Controls from wave 2) --> 1,400
Control --> 2,000
Wave 3.
Treatment 4 --> 2,000
Treatment 5 (Controls from wave 2) --> 1,400
Control --> 2,000
Using pilot data, we are assuming attrition rates of 30% for the control group across waves.
|
After
Treatment --> 4,000 households; 4,800 students
Control --> --> 4,000 households; 4,800 students
|
|
Field
Power calculation: Minimum Detectable Effect Size for Main Outcomes
|
Before
Assuming attrition of 30%, take up of 52%, a power of 80%, and a type I error rate of 0.05, for a sample size of 3,000, we estimate an MDE of 0.13SD.
|
After
Assuming attrition of 30%, take up of 52%, a power of 80%, and a type I error rate of 0.05, for a sample size of 9,600, we estimate an MDE of 0.08SD.
|
|
Field
Intervention (Hidden)
|
Before
The intervention under evaluation consists of a tutoring program offering supplemental learning in math and Ukrainian language and psychosocial support. The learning component includes subject-specific academic content in accordance with Ukrainian educational programs. The socio-emotional part includes activities to support student’s social and emotional well-being. The total scheduled time for psychosocial support activities is 30 mins per week. The target population of the tutoring program is Ukrainian students in grades 5 to 10 who seek supplemental support beyond the standard online schooling schedule. Students are placed in groups of 3 and will receive 3 hours of tutoring per week for 6 weeks by paid-for tutors through the Edmodo online platform. The implementing partner is Teach for Ukraine.
|
After
|
|
Field
Did you obtain IRB approval for this study?
|
Before
No
|
After
Yes
|
|
Field
Secondary Outcomes (End Points)
|
Before
Personal and Academic Expectations
Mental health (stress and anxiety)
Time use
Attendance of the tutoring activities
Attitudes towards tutoring
|
After
Social emotional skills
Personal and Academic Expectations
Time use
Attendance of the tutoring activities
Attitudes towards tutoring
|
|
Field
Pi as first author
|
Before
No
|
After
Yes
|