After School Tutoring: A randomised controlled trial

The main objective of the tutoring program is to improve the learning outcomes of rural children. Global Development Research Initiative (GDRI), an NGO, implemented the intervention in collaboration with the Directorate of Primary Education in Bangladesh. The intervention offered free outside school hours tutoring for two years from 2013 to 2015. The intervention took place in public schools located in two south-western districts of Bangladesh, Khulna and Shatkhira.

In year 1, the tutoring program started in March 2014 among students in grade 3 and grade 4 and lasted for nine months until November 2014 when the students sat for the end-of-year exam. In year 2, the tutoring started in late February with the same cohort of students who progressed to grade 4 and grade 5, respectively. The program ended in November 2015.

We implemented a two-stage RCT design. The first stage involved assigning schools in these districts into treatment or control groups. In the second stage, students in each grade were randomly selected for free private tutoring. Thus, some students from the same grade did not receive free tuition from the program. Schools were randomly assigned to one of the three types of treatments- free small group tutoring to random groups of students(PT, treatment 1); free tutoring plus an additional weekly one-to-one session to random groups of students (HPT, treatment 2) ; free small group tutoring to most central students in each classroom (PT-central, treatment 3).

In PT treatment (treatment 1), students received tutoring for about 2 hours a day in a small group/batch, three days a week. The tutors taught mathematics and English, the two subjects that most Bangladeshi students find difficult. The tutors helped the students complete homework, prepare for exams, and understand the relevant basic concepts to develop more foundational skills.

In the HPT treatment (treatment 2), in addition to private tutoring as described above, the tutors also made home visits once a week to provide individual tutoring sessions for each student. The purpose of adding an individual session through home visit was to make sure that students were taught in accordance with their abilities and individual needs were better assessed and addressed. Tutors were also instructed to focus on the students' individual needs, so each student gets the opportunity to learn something that is difficult to cover in a group setting. During the home visit, tutors spent about an hour at students’ homes.

In PT-central treatment (treatment 3), treated students receive private tutoring which is same as treatment 1, but selected based on students' centrality, only central students in each classroom was treated. Centrality is calculated based on baseline social network survey in which each student was asked to nominate who are their best friends in his/her classroom, then students were ranked from high to low in terms of their centrality. Only students with high centrality in each classroom were selected to receive treatment.

2014-02-01

2015-11-01

The primary outcomes are represented by cognitive skills and non-cognitive skills of students. Cognitive skills are represented by English and Math test scores of students, collected at different points of time. The tests were conducted among all students, especially for evaluating the intervention, and hence they are different from school-administered final exams or other exams. The tests focused on actual learning to understand how much students really have learned.

Our second group of outcome variables consist of measures of the noncognitive skills of the students. To obtain these measures, we designed several questions to explore three main domains of noncognitive skills of each student: self-control, social skills, and motivation. School teachers (not the private tutors) provided an evaluation of a student in these dimensions. The questions collected to measure each skill are listed in the detail section.

The test items consist of both multiple- choice and short questions. The test is intended to assess problem-solving capacities in mathematics (e.g., geometric skills and complex worded problems), and English comprehension requiring students verbal, language, and reasoning skills. Local school teachers and educators were consulted to ensure that the tests are appropriate for the grade level. In addition, test scores were collected in the range of 0 to 100. In order to make it comparable across groups, we normalize these test scores of each student by subtracting the average test score of the control group in the respective year and dividing by the standard deviation.

In terms of non-cognitive skills, they are measured by three dimensions including self control, social skills and motivation to learn of a student. The questions used to measure each category are listed below.

Self control
(1) Is the student regular in class? 1=Yes, 0=No
(2) Does the student regularly do his homework? 1=Yes, 0=No
(3) Does the student regularly do his homework? 1=Yes, 0=No

Social skills
(1) How adaptive is the student in making friends? 1=Moderate, 2=Much, 3=Very much
(2) How adaptive is the student in sustaining relationships? 1=Moderate, 2=Much, 3=Very much
(3) How does the student behave with classmates? 1=Competitive, 2=Irreconcilable 3=Friendly

Motivation
(1) How much attentive is the student in class? 1=Moderate, 2=Much, 3=Very much
(2) How desirous of study is the student? 1=Moderate, 2=Much, 3=Very much
(3) How eager is the student in learning lessons? 1=Moderate, 2=Much, 3=Very much
(4) Involvement in taking challenging job 1=Moderate, 2=Much, 3=Very much
(5) How much does the student enjoy doing challenging jobs 1=Moderate, 2=Much, 3=Very much
(6) Does the student take challenging work? 1=Sometimes, 2=Often, 3=Very often

The standardization of non-cognitive skills are as follows. After answers are collected for each skill, we aggregate the corresponding questions into one variable by calculating the average across the responses (all responses are numerical), so all measures take values in [0, 1]. We then normalize the measures by subtracting the mean and then dividing by the standard deviation of the control group. We also construct an Index, which is the average of these three measures and it represents the overall non-cognitive ability of a student.

Apart from primary outcomes, we collected other outcomes in order to identify the possible channels to explain why private tutoring intervention matters, these outcomes are represented by different surveys conducted during and after the completion of the intervention.

Firstly, we collected student survey at the end of 2015 just before the completion of the intervention, the survey gathered information on the distribution of out-of-school time between studies, leisure, self-esteem, ambition, confidence, etc. The answers were obtained from survey responses of parents about their children, and students’ self-reporting their behaviors. The purpose of student survey is to measure how much effort students have spent on study in order to evaluate whether private tutoring intervention shaped students' behavior.

Next, we collected parent survey one year after the intervention finished in 2017. The key variables measure parent behavior and investment towards a child’s education. The purpose of parent survey is to understand whether the program increase or decrease parent effort devoted on their children's education. By examining parents’ time and resource allocation on their children, we are able to understand whether private tutoring increased or decreased parent effort and whether there are persistent effect on parents after intervention finished.

In addition, we collected teacher survey in August, 2015 which contains information on teaching methods and teachers’ level of effort in the class. These data were collected by random visits to the classrooms by the enumerators during the intervention and the teachers were not pre-informed about the visits. The objective of teacher survey is to assess whether tutoring program would change the behavior of class teachers. For example, they may be more engaged in class activities as the program could a threat to their position ; Or they could be less engaged in class activities given students already obtained help from private tutors.

Finally, we also collected cognitive skills of students one year after the intervention finished in order to assess whether private tutoring generated long-term effects that persist after the intervention finished. The cognitive skills are represented by PSC exam scores. PSC exams are externally administered public exam for the grade 5 students, called the primary school certificate (PSC) exam. A student needs to pass the PSC exam to move to secondary school, and the results for this PSC exam are taken seriously by teachers, parents, and students.

As summarised above, the secondary outcomes mainly come from three aspects of surveys including student survey, parent survey and teacher survey.

The student survey consists of questions that describe student effort devoted on study, represented by the amount of time they spent on out-of-school activities. Outcomes include how much time spent on personal study, household chores, watching TV, sports participation, and study with tutors. The amount of time are measured by hours spent in a day regarding above activities.

The parent survey focused on parents' behavior towards a child’s education including time spent helping them study, money spent on study materials and tutoring, as well as whether they monitor their child’s performance. Some variables are continuous, For example, the number of hours spent by parents to study with children, the amount of money invested on children education and tutoring. Some variables are categorical values such as 1 to 5 where 1 is not at all and 5 is very much. In order to make interpretation of results easy, we converted categorical values into 0 or 1 where 0 stands for no and 1 stands for yes.

The teacher survey records their effort devoted in class. We considered several aspects of teachers’ effort within the classroom, such as whether the teacher arrived on time to class, gave homework or other assignments, checked the homework, administered exams, used any teaching material (such as syllabus, lesson plans etc.) or tools (such as maps, charts, posters etc.) to deliver their lessons, etc. Some questions are recorded as a Yes or No, while some questions are recorded in different scales such as 1 to 5 where 1 is not at all and 5 is very much. Similar as parent survey, we converted categorical values into 0 or 1 where 0 stands for no and 1 stands for yes.

We use a two-stage randomization design where we first assign each school to one of the four experimental groups described above in the intervention section, and then assign students within treatment schools (PT, HPT, and PT-central) to receive treatment. In total, 254 primary (public) schools participated in our study randomly selected from two rural districts in Bangladesh, Satkhira and Khulna.

These 254 primary schools were randomly allocated into treatment and control groups, resulting in 100 schools allocated to the control group and the other 154 schools allocated to one of the treatment groups: 58 schools were randomly assigned to Treatment 1 (PT: private tutoring only ), 58 schools were randomly assigned to Treatment 2 (HPT: private tutoring and home visits), and 38 schools were randomly assigned to Treatment 3 (T3: centrality-based private tutoring). The school-level randomization was conducted based on school-level characteristics collected in the baseline survey such as facility, transportation, number of rooms, teacher characteristics and so on. Successful randomization is guaranteed given school-level characteristics are balanced across PT, HPT and PT-central groups.

After school-level randomization, we implemented student-level randomization. In details, within PT and HPT schools, a subgroup of students in each classroom were randomly chosen to receive the treatment. We used baseline test scores of students to randomize and guarantee that baseline test scores of students are balanced across groups and across treated and untreated students. In the 38

Treatment 3 (PT-central) schools, treated students were not chosen at random but selection was based on their eigenvector centrality. Eigenvector centrality is calculated according to network survey implemented in the baseline survey. In fact, in each school, only students with the highest centrality were treated and these students only received private tutoring, i.e., the same intervention as treated students in PT treatment (treatment 1). All students in each classroom were asked to nominate and rank ten of their closest friends among their classmates. To be more precise, we asked each student to pick their ten best friends and rank them from 1 (very close friends) to 10 (least close friends). Then centrality measures the degree of influence a student is among his classmates.

Randomization is done by a computer based on student-level and school-level characteristics collected in baseline surveys

Two levels of randomization, one is school-level randomization (randomize schools into treatment and control groups), the second one is student-level randomization (randomize treated and untreated students within treatment schools)

Yes

254 schools

14,216 students

58 schools PT treatment: private tutoring only, 58 schools HPT treatment: private tutoring and home visit, 38 schools PT-central treatment: private tutoring to central students, 100 control school