Experimental Design
In terms of method, the research will be based on a field experiment with three-level design - schools, classes, and students - with two levels of nesting - classes to the school, and students to the class - where the schools, level 3-unit, are randomly assigned to treatment or control groups. The study will be a school-level randomised field experiment formed by a control group and a treatment group of schools.
The interventions will occur in the primary and lower secondary state-run schools of the city of Rio de Janeiro that provide education for young people from the first to the ninth grade. The choice for the city was driven by the availability of the Court of Accountability of Rio de Janeiro and the Secretary of Education to partner to conduct out the experiment. The city of Rio has one of the largest education systems in South America with 1,543 schools and nurseries and 644,000 students from nurseries to special schools. The sample of sixty schools will be chosen randomly from 1,001 (one thousand and one) units responsible for 430,510 (four hundred and thirty thousand five hundred and ten) pupils from 1st to 9th year. Thereafter, the control and treatment groups will be formed randomly by 30 schools each.
The experimental sample will be chosen from a pool of three different types of education units: 233 schools responsible exclusively for children from the 1st to 5th year, 222 responsible exclusively for pupils from the 6th to 9th grade, and 546 responsible for children from 1st to 9th year. It is important to analyse the balance between the control and treatment groups considering a variable correlated with the results of interest, that is, students’ achievement. The idea is to use the Brazilian Basic Education Assessment System (Saeb) of 2019, the last national assessment available, as a representation of the students learning outcome. It is a standardised test applied every two years by the National Institute of Educational Studies and Research Anísio Teixeira (Inep), an autonomous federal government agency. The agency discloses the data by school, what is needed to prevent bias as follows. The goal is to ensure the balance between control and treatment groups regarding each of the three types of schools detailed previously in this paragraph.
Following the technique used by Fryer (2017), this study will use a matched-pair randomisation procedure. Firstly, the schools will be listed, for each type of school, in a descending order based on the score reached in the Saeb 2019 (pre-treatment score). Each type of school will appear on a separated ordered list thus it will have three lists. Every two schools in each of the three ordered lists will be matched forming 30 pairs. For instance, consider the list where appears only the schools responsible exclusively for children from the 1st to 9th year, it will be separated as follow based on the ordered list: school on the 1st place with the school on the 2nd, 3rd with 4th, etc. It will guarantee that the two schools in each pair will have grades from Saeb very close. For each pair, one school will be randomly assigned to the control group and the another to the treatment group. It will be made for each pair in the three ordered lists ensuring the balance between the control and treatment group. Therefore, the two groups of schools will have a very balanced score of student achievement.
As part of the methodology, the study will define the operational actions, i.e. the treatment, to be implemented in the schools using base management measures compiled from decades of research in management and education presented in Bloom et al. (2015).
The 23 management practices are the basis of the international management index also created by Bloom et al. (2015). Each of the twenty-three practices is assessed considering a grade from 1 to 5. For instance, consider practice 1. (a), Bloom et al. (2015) define that a school receives a score of 1 if "No clear or institutionalised instructional planning processes or protocols exist; little verification or follow up is done to ensure consistency across classrooms.", a score 3 if "School has defined instructional planning processes or protocols to support instructional strategies and materials and incorporate some flexibility to meet student’s needs; monitoring is only adequate.", and a score 5 if "School has implemented a clearly defined instructional planning process designed to align instructional strategies and materials with learning expectations and incorporate flexibility to meet student needs; these are followed up on through comprehensive monitoring or oversight.".
Grade 1 reflects the lack of the respective best high-level management practice, grade 3 represents that a part of the respective best high-level management practice exists, and grade 5 will be attributed to the school that has the respective best high-level management practice totally implemented. The grade average of the twenty-three best management practices will represent the management index of each school. Therefore, this study will focus on the implementation of the operational practices needed to increase the management index of each school in the treatment group of the experiment.
On the one hand, the study will use the international management index as presented previously based on the best management practices to show the management level of each school. The international management index will be measured through double-checked face-to-face interviews with the school principals. The interviews will be recorded to enable that a different evaluator from the inter-institutional team conducts a new assessment of the answers given by the school manager. Finally, the management index of each school will be the average of its scores in each of the twenty-three management measures.
On the other hand, the average of mathematics, science, and Portuguese language tests will represent the measure of pupil learning outcome. The tests will be applied in two classes of the same year per school. For schools responsible exclusively for children from the 1st to 5th year, the tests will be applied in classes of the 5th year. For schools responsible exclusively for children from 6th to 9th year or schools responsible for children from 1st to 9th year, the tests will be applied to classes of the 9th year. Therefore, the average result of the two classes of the same year will represent the average learning outcome of the students at the respective school. The selected classes will be assessed semi-annually during the experiment to allow comparisons through the time in each school.
There will be five rounds of matched assessment, management vs pupil outcomes, semi-annually. The first round of assessment will diagnose the situation of schools on the treatment and control groups, considering the international management index described previously and the average score of the students on the tests. The first round is supposed to happen in September 2021 and last on August 2023. The schedule was adapted due to the impact of coronavirus in Brazil.
Before the first round of matched assessment, it will be created a plan describing operational management practices needed to improve the management situation of the schools. In fact, the plan is almost complete. The goal of this plan is to guide the schools to improve their management index considering each of the 23 high-level management practices presented by Bloom et al. (2015).
After the diagnosis (first assessment round), the plan of implementation will be reviewed considering the management situation and the management index value of each school within the treatment group. Following the review, the intervention will start intending to last 1 year. The intervention means that the inter-institutional team will be at the schools helping the school managers to implement the plan of management actions.
The plan of implementation is only a baseline, that is, other operational management practices could be created during the intervention period always following the high-level management practices defined by Bloom et al. (2015). Some operational practices may not be implemented due to real obstacles encountered in the schools. Each school may have different levels of implementation due to the different situations in each school. As it is expected that changes occur during the execution, the life cycle approach for the project is the adaptive life cycle Institute, 2017). This research intends to measure the average impact of the operational practices implemented in the treatment group on pupil outcomes.
After the intervention period, it will start the monitoring period where the schools will be monitored regarding the maintenance or not of the operational management practices developed and implemented.
This procedure will make it possible to measure the average impact of the intervention during the year in which it took place and up to one year after the end of the implementation on the student’s achievement. It is relevant to highlight that this research seeks a real-world application. This means that if the operational management practices developed and implemented really impact the student’s achievement in a way that persuades and influences politicians and policymakers, it will be possible to scale the project of implementation of the best management practices for the entire system of schools in the city of Rio de Janeiro.