|
Field
Trial Title
|
Before
Management, institutional arrangements, and education: a three-level clustered randomised field experiment in Rio de Janeiro
|
After
Management, institutional arrangements, and education: a randomised field experiment in Rio de Janeiro
|
|
Field
Abstract
|
Before
What is the causal effect of management on schools? How can the independent public audit institutions as the courts of accountability act directly to generate better public policies?
One goal of the study is to analyse with a three-level randomised field experiment design if management can explain large differences in school performance. The focus will be on the school-level management practices of primary and lower secondary schools.
In order to find rigorous answers to scientific questions, researchers have used controlled randomised experiments. The method is used to infer causal relationship between variables from a study. The current research is intended to answer "What is the causal effect of management on education?". Therefore, it is needed to answer "How would students and schools who participated of the program have fared in the absence of the program?" and "How would those who were not exposed to the program have fared in the presence of the program?". Control groups, treatment groups and randomisation allow scientists to obtain an average impact of the program on a group of individuals by comparing them to a group who were not exposed to the program.
Using randomised trials, Esther Duflo and Abhijit Banerjee are reinventing development economics. The couple and their friend, Harvard economist Michael Kremer, awarded the 2019 Nobel Prize in economics as a consequence of their scientific and evidence-driven economics tackle the world poverty problem. The Nobel Prize committee said "Their experimental research methods now entirely dominate development economics".
According to the cited Nobel Prize winners, working with local partners has enabled conducting many randomised field evaluations in developing countries. Therefore, the experiment is intended to occur in Brazil, specifically in the city of Rio de Janeiro where it is possible to succeed with partnerships. Brazil is one of the largest economies in the world. It is a country with marked inequality, violence, and poverty. The city of Rio de Janeiro is the capital of the state of Rio de Janeiro and it is situated in the Southeast of the country. It has the second largest population in Brazil with almost 7 million people \parencite{site}.
According to The World Bank, 81\% of primary schools and 73\% of secondary schools around the world are state-run schools. Therefore, in order to generate wide results, the study will analyse schools operated, managed, and controlled by public authorities.
The second goal of the research is to explore how the government external control institutions, which have recognised qualified staff within the public administration around the world, can bring innovations to the state such as randomised field experiments in partnership with the policymakers.
Government external audit institutions have attracted growing interests of citizens, governments and non governmental institutions around the world because they play a very relevant role to avoid corruption and reducing waste \parencite{MeloMarcusAndré2009PaIC}.
The are two models of independent public external audit institutions in the world, the courts, and the audit offices. These organisations have worked historically assessing legal, financial, and accounting aspects of public spending. In the last decades, these institutions have worked also through performance audits to analyse economy, efficiency, and effectiveness of the public policies and organisations using well-established methods as founded on the International Standards of Supreme Audit Institutions 3000 (ISSAI 3000). More recently, there has been a global movement to broaden the scope and the theoretical basis of the public policies analysis made by the government audit institutions. That is, in addition to the analysis considering economy, efficiency, and effectiveness of the public policies already made in the performance audits, the idea is to evaluate also the impact and relevance of public spending using the scientific framework of social sciences.
The system of courts of auditors/account/accountability is encountered in countries such as France, Italy, Portugal, Germany, Netherlands, and Brazil. The audit offices are found in countries such as the United States of America, the United Kingdom, India, and Sweden.
Article written by Laffan claims that the European Court of Auditors had to strive to become a "living institution" and to find its place in the Union's order. According to the author, one of the most important factors for the constant evolution has been its search for cooperative relations with other EU institutions. Following the idea of cooperation and partnership, this research will explore how the government external audit institutions can acting in a new role, bring innovations and effectiveness to the public administration.
Brazil has thirty-three independent courts of accountability that act in the national, regional e local level and its members are life-tenure judges. Furthermore, there are specialised independent life-tenure prosecutors and autonomous auditors with strong stability. These strong guarantees aim at ensuring the independence of the Courts from the Government. This research is intended to explore, through a partnership with the Municipal Secretariat of Education of Rio de Janeiro, the use of the Court of Accountability of the city of Rio de Janeiro (TCMRJ), responsible to control four billion pounds in public policies each year, to conduct a randomised field experiment to test with scientific rigour the performance of selected public policies.
The Municipal Court of Accountability of Rio de Janeiro will conduct a three-level clustered randomised field experiment ex ante in partnership with the policy maker in order to verify a policy before the spending starts.
The research is intended to bring stronger evidence than previous studies on the question of whether management can impact student learning. Furthermore, it can lead the courts of accountability or audit offices to a new way of acting, bringing, in partnership with policy makers, innovation and better public policies for the society.
|
After
Although economists and other social scientists have worked hard trying to disentangle what factors can explain differences between performances across schools, there are few studies about the impact of management on education. For example, the discussion about management on productivity has not caught strong attention from economists (Bloom et al., 2012, p. 2). In the case of causal evidence coming from randomized trials, the gap is even larger. Despite a growing number of randomized experiments conducted by social researchers to address critical scientific questions on policies in the last decades, very few trials have been undertaken to explore the causal consequences of management on productivity (Duflo, 2007; Fryer, 2017). Thus, this research intends to reduce this knowledge gap by applying a rigorous scientific method, precisely a randomized field experiment, to understand our first research question, i.e., whether there is a causal relationship between school management and pupil learning. Understanding how management affects schools within and across countries is complex because of the lack of comparable measures to systematically assess management practices (Bloom et al., 2015). Therefore, as this research intends to use the same international management index created by Bloom et al. (2015), it will allow direct comparisons within Brazil and across different countries.
Although Brazil is one of the largest economies in the world, the country has marked levels of inequality and poverty. These two problems can create a unique pervasive dynamic in which individuals remain trapped in environments with scarce opportunities for upward social mobility (Banerjee & Duflo, 2007). High-quality public and free education are vital to breaking this vicious cycle, remaining the most reliable intergenerational way out of poverty for millions of families (UNICEF, 2020).
It is relevant to highlight that Brazil has more than 5,500 municipalities with the same standardized public educational system as Rio de Janeiro. The Brazilian educational system is bi-partite composed of private and public schools. The private system holds only 18.58% of the students from clearly better-off families, and the public system is responsible for the remaining 81,42% of pupils from mostly disadvantaged families. Even though the same Federal laws regulate the public system, they face diverse cultural and socioeconomic realities. Many schools are in income-deprived areas such as slums or favelas and deal with very complex social phenomena. This study's management practices have been developed to be flexible to adaptation and implementation in different realities. A key factor for adaptability is the inter-institutional team. It is thought to be dynamic and ready to adapt the baseline plan whenever necessary.
The City of Rio de Janeiro is the capital of the state of Rio de Janeiro and has the second-largest population among Brazilian cities, with almost 7 million people (IBGE, 2020). It has one of the largest education systems in South America, with 1,543 schools and nurseries and 644,000 students from nurseries to special schools. The Rio de Janeiro case is remarkable because many schools are located within slums and favelas dominated by drug dealers, where even the police cannot enter and where families are exposed to extreme marginality and violence daily. Exposure to extra-school vulnerabilities directly influences the learning conditions. In this context, management practices are even more crucial to ensure meaningful learning and increased growth prospects for pupils. The project will deal with 80 schools randomly selected from the Rio de Janeiro school population, mostly from poor city regions.
The idea is to improve education by implementing best management practices reported in the academic literature through the offer of in locus consultancy to selected schools. Considering the scarcity of resources in underdeveloped and developing countries, it is crucial to create feasible public policies. Aiming at a viable project, a partnership between government institutions in Rio de Janeiro was signed by the Court of Accountability of Rio de Janeiro (TCMRJ), Municipal Secretariat of Education (SMERJ), Dr. Tiago Cavalcanti (Faculty of Economics – University of Cambridge), Professor Shailaja Fennell (Centre of Development Studies – University of Cambridge), and Felipe Galvão Puccioni (Principal Investigator – TCMRJ). Based on the best practices shown in the academic literature, a flexible plan has been developed to deal with different schools, cities, and country realities. A randomized field experiment will be conducted to test a causal relationship between the best management practices that will be implemented and educational results. Eighty schools were randomly selected and assigned by pair-matching in two groups: treatment and control. For two years, the team, starting in 2022, will help school principals and managers to implement the best management practices.
The management practices of this study are adaptable for each school depending on the management level encountered on the diagnostic phase. The flexibility and customization of these practices are possible because the project is thought to be implemented by an interinstitutional team formed by marked professionals hired from the partners' institutions. The team will adapt the best management practices baseline plan to each school depending on the school, city, and country reality. Although the experiment is in Rio de Janeiro, it is supposed to be easily implemented in different social contexts globally with the appropriate adaptation to local realities.
This study could lead to a deeper understanding of state run-schools, which represent 81% of primary schools and 73% of secondary schools around the world, according to The World Bank (2020). Even though principals play a critical role and there is an increasing consensus on what they must do to be effective (Fryer & Dobbie, 2013; Fryer, 2014), this study will go further and analyze not only the school principals but also other managers within the schools who help the school heads to manage the educational units.
The second research question is: How can the government audit institutions, such as the courts of accountability, act directly to generate better public policies? These institutions have attracted the growing interest of citizens, governments, and non-governmental institutions worldwide because they play a relevant role in avoiding corruption and reducing public resources waste (Melo et al., 2009). More recently, there has been a movement to broaden the scope and the theoretical basis of the public policies analysis conducted by governmental audit institutions. In addition to analyzing the economy, efficiency, and effectiveness of public policies already made in performance audits, the idea is also to evaluate the utility and impact of public spending using the scientific framework of social sciences.
Brazil uses the court system and has thirty-three independent accountability courts that act at the national, regional, and local levels, and its members are life-tenure judges. In addition, there are specialized independent life-tenure prosecutors and autonomous auditors with strong stability against arbitrary dismissal. These strong guarantees aim at ensuring the independence of the Courts from the government. This study, in an innovative path, will check how the Municipal Court of Accountability of Rio de Janeiro (TCMRJ) - which is responsible for controlling thirty billion "reais" (four billion pounds) in public policies each year – will conduct a two-level clustered randomized field experiment ex-ante in partnership with policymakers to verify a policy before it is implemented. Therefore, this research aims also to explore how a government external control institution, specifically the Municipal Court of Accountability of Rio de Janeiro, can bring innovations, in partnership with policymakers, by conducting randomized field experiments.
In summary, the research is intended to bring more substantial evidence than previous studies on whether management can impact school performance. Furthermore, it can lead the courts of accountability or audit offices to a new way of acting, bringing, in partnership with policymakers, innovation and better public policies for the society.
The project is innovative in different ways. Firstly, the interinstitutional design is unique. It involves a partnership between the controller and the controlled government to develop, based on the academic literature, implement, with an inter-institutional team, and test, through a randomized field experiment, a set of the best management practices. Secondly, we use government institutions to develop, implement and test your own policies through a very rigorous scientific path. Assessing the causal relationship between management and educational results is innovative and challenging. Thirdly, the project is developing an innovative path of action for the government independent audit institutions by conducting randomized field experiments in partnership with policymakers to test public policies before those are implemented.
|
|
Field
Trial Start Date
|
Before
October 01, 2020
|
After
November 15, 2021
|
|
Field
Trial End Date
|
Before
April 26, 2024
|
After
June 24, 2024
|
|
Field
Last Published
|
Before
June 24, 2021 07:35 AM
|
After
November 12, 2021 10:20 AM
|
|
Field
Intervention (Public)
|
Before
The project of intervention will occur based on a partnership between the Municipal Secretariat of Education of Rio de Janeiro - SMERJ and the Court of Accountability of the City of Rio de Janeiro - TCMRJ. A collaboration term is needed because the project will be executed in the schools controlled by the Secretary of Education using human resources from the Court and from the Secretary. The document defines the responsibilities of each part and the implementation operational procedures of the experiment. Marked professionals from the Court and Secretariat will form the inter-institutional team that will deliver the treatment and placebo measures in the schools. A formal designation of the inter-institutional team, as defined in the collaboration term, will be done with a joint act signed by the President of the Court and the Secretary of Education.
It will be created, defined, and implemented operational practices based on the best management practices compiled by Bloom et al. (2015) from decades of research in management and education. The study cited absorbs management measures from previous research across other sectors such as manufacturing, retail, health care sectors, but with changes in order to adjust the framework to the school context. Therefore, sixteen management basic measures are applicable to different sectors as showed in Bloom et al. (2014), seven are specially designed to the management of schools (Bloom et al., 2015).
The research conducted by Bloom et al. (2015) shows the best management practices in a high-level view. Therefore, it is important to develop, based on the cited high-level definition of the best management practices, low-level operational practices to be implemented in schools aiming at a better school management situation. Considering this fact, operational practices - considering the reality of the Brazilian educational system and based on the high-level best management measures presented in Bloom et al. (2015) - will be developed in order to improve the management of the schools in Rio de Janeiro.
The implementation of the operational practices will be done by the inter-institutional team formed by outstanding professionals from the Court of Accountability and the Secretariat of Education. They will work autonomously focused on the improvement of the management index of each school assigned to the treatment group. It is relevant to highlight that only the inter-institutional team will have information about what are the schools assigned to treatment and control groups. This fact is included in the collaboration term.
|
After
The project of intervention will occur based on a partnership between the Municipal Secretariat of Education of Rio de Janeiro - SMERJ and the Court of Accountability of the City of Rio de Janeiro - TCMRJ. A collaboration term is needed because the project will be executed in the schools controlled by the Secretary of Education using human resources from the Court and from the Secretary. The document defines the responsibilities of each part and the implementation operational procedures of the experiment. Marked professionals from the Court and Secretariat will form the inter-institutional team that will deliver the treatment measures in the schools. A formal designation of the inter-institutional team, as defined in the collaboration term, will be done with a joint act signed by the President of the Court and the Secretary of Education.
It will be created, defined, and implemented operational practices based on the best management practices compiled by Bloom et al. (2015) from decades of research in management and education. The study cited absorbs management measures from previous research across other sectors such as manufacturing, retail, health care sectors, but with changes in order to adjust the framework to the school context. Therefore, sixteen management basic measures are applicable to different sectors as showed in Bloom et al. (2014), seven are specially designed to the management of schools (Bloom et al., 2015).
The research conducted by Bloom et al. (2015) shows the best management practices in a high-level view. Therefore, it is important to develop, based on the cited high-level definition of the best management practices, low-level operational practices to be implemented in schools aiming at a better school management situation. Considering this fact, operational practices - considering the reality of the Brazilian educational system and based on the high-level best management measures presented in Bloom et al. (2015) - will be developed in order to improve the management of the schools in Rio de Janeiro.
The implementation of the operational practices will be done by the inter-institutional team formed by outstanding professionals from the Court of Accountability and the Secretariat of Education. They will work autonomously focused on the improvement of the management index of each school assigned to the treatment group. It is relevant to highlight that only the inter-institutional team will have information about what are the schools assigned to treatment and control groups. The schools in the control group will not know they are in the study.
These facts are included in the collaboration term.
|
|
Field
Intervention Start Date
|
Before
October 04, 2021
|
After
January 24, 2022
|
|
Field
Intervention End Date
|
Before
September 30, 2022
|
After
July 24, 2023
|
|
Field
Primary Outcomes (End Points)
|
Before
Students' scores in math, science and Portuguese tests
|
After
Students' scores from a standardised assessment conducted bimonthly by the Municipal Secretariat of Education of Rio de Janeiro.
|
|
Field
Experimental Design (Public)
|
Before
In terms of method, the research will be based on a field experiment with three-level design - schools, classes, and students - with two levels of nesting - classes to the school, and students to the class - where the schools, level 3-unit, are randomly assigned to treatment or control groups. The study will be a school-level randomised field experiment formed by a control group and a treatment group of schools.
The interventions will occur in the primary and lower secondary state-run schools of the city of Rio de Janeiro that provide education for young people from the first to the ninth grade. The choice for the city was driven by the availability of the Court of Accountability of Rio de Janeiro and the Secretary of Education to partner to conduct out the experiment. The city of Rio has one of the largest education systems in South America with 1,543 schools and nurseries and 644,000 students from nurseries to special schools. The sample of sixty schools will be chosen randomly from 1,001 (one thousand and one) units responsible for 430,510 (four hundred and thirty thousand five hundred and ten) pupils from 1st to 9th year. Thereafter, the control and treatment groups will be formed randomly by 30 schools each.
The experimental sample will be chosen from a pool of three different types of education units: 233 schools responsible exclusively for children from the 1st to 5th year, 222 responsible exclusively for pupils from the 6th to 9th grade, and 546 responsible for children from 1st to 9th year. It is important to analyse the balance between the control and treatment groups considering a variable correlated with the results of interest, that is, students’ achievement. The idea is to use the Brazilian Basic Education Assessment System (Saeb) of 2019, the last national assessment available, as a representation of the students learning outcome. It is a standardised test applied every two years by the National Institute of Educational Studies and Research Anísio Teixeira (Inep), an autonomous federal government agency. The agency discloses the data by school, what is needed to prevent bias as follows. The goal is to ensure the balance between control and treatment groups regarding each of the three types of schools detailed previously in this paragraph.
Following the technique used by Fryer (2017), this study will use a matched-pair randomisation procedure. Firstly, the schools will be listed, for each type of school, in a descending order based on the score reached in the Saeb 2019 (pre-treatment score). Each type of school will appear on a separated ordered list thus it will have three lists. Every two schools in each of the three ordered lists will be matched forming 30 pairs. For instance, consider the list where appears only the schools responsible exclusively for children from the 1st to 9th year, it will be separated as follow based on the ordered list: school on the 1st place with the school on the 2nd, 3rd with 4th, etc. It will guarantee that the two schools in each pair will have grades from Saeb very close. For each pair, one school will be randomly assigned to the control group and the another to the treatment group. It will be made for each pair in the three ordered lists ensuring the balance between the control and treatment group. Therefore, the two groups of schools will have a very balanced score of student achievement.
As part of the methodology, the study will define the operational actions, i.e. the treatment, to be implemented in the schools using base management measures compiled from decades of research in management and education presented in Bloom et al. (2015).
The 23 management practices are the basis of the international management index also created by Bloom et al. (2015). Each of the twenty-three practices is assessed considering a grade from 1 to 5. For instance, consider practice 1. (a), Bloom et al. (2015) define that a school receives a score of 1 if "No clear or institutionalised instructional planning processes or protocols exist; little verification or follow up is done to ensure consistency across classrooms.", a score 3 if "School has defined instructional planning processes or protocols to support instructional strategies and materials and incorporate some flexibility to meet student’s needs; monitoring is only adequate.", and a score 5 if "School has implemented a clearly defined instructional planning process designed to align instructional strategies and materials with learning expectations and incorporate flexibility to meet student needs; these are followed up on through comprehensive monitoring or oversight.".
Grade 1 reflects the lack of the respective best high-level management practice, grade 3 represents that a part of the respective best high-level management practice exists, and grade 5 will be attributed to the school that has the respective best high-level management practice totally implemented. The grade average of the twenty-three best management practices will represent the management index of each school. Therefore, this study will focus on the implementation of the operational practices needed to increase the management index of each school in the treatment group of the experiment.
On the one hand, the study will use the international management index as presented previously based on the best management practices to show the management level of each school. The international management index will be measured through double-checked face-to-face interviews with the school principals. The interviews will be recorded to enable that a different evaluator from the inter-institutional team conducts a new assessment of the answers given by the school manager. Finally, the management index of each school will be the average of its scores in each of the twenty-three management measures.
On the other hand, the average of mathematics, science, and Portuguese language tests will represent the measure of pupil learning outcome. The tests will be applied in two classes of the same year per school. For schools responsible exclusively for children from the 1st to 5th year, the tests will be applied in classes of the 5th year. For schools responsible exclusively for children from 6th to 9th year or schools responsible for children from 1st to 9th year, the tests will be applied to classes of the 9th year. Therefore, the average result of the two classes of the same year will represent the average learning outcome of the students at the respective school. The selected classes will be assessed semi-annually during the experiment to allow comparisons through the time in each school.
There will be five rounds of matched assessment, management vs pupil outcomes, semi-annually. The first round of assessment will diagnose the situation of schools on the treatment and control groups, considering the international management index described previously and the average score of the students on the tests. The first round is supposed to happen in September 2021 and last on August 2023. The schedule was adapted due to the impact of coronavirus in Brazil.
Before the first round of matched assessment, it will be created a plan describing operational management practices needed to improve the management situation of the schools. In fact, the plan is almost complete. The goal of this plan is to guide the schools to improve their management index considering each of the 23 high-level management practices presented by Bloom et al. (2015).
After the diagnosis (first assessment round), the plan of implementation will be reviewed considering the management situation and the management index value of each school within the treatment group. Following the review, the intervention will start intending to last 1 year. The intervention means that the inter-institutional team will be at the schools helping the school managers to implement the plan of management actions.
The plan of implementation is only a baseline, that is, other operational management practices could be created during the intervention period always following the high-level management practices defined by Bloom et al. (2015). Some operational practices may not be implemented due to real obstacles encountered in the schools. Each school may have different levels of implementation due to the different situations in each school. As it is expected that changes occur during the execution, the life cycle approach for the project is the adaptive life cycle Institute, 2017). This research intends to measure the average impact of the operational practices implemented in the treatment group on pupil outcomes.
After the intervention period, it will start the monitoring period where the schools will be monitored regarding the maintenance or not of the operational management practices developed and implemented.
This procedure will make it possible to measure the average impact of the intervention during the year in which it took place and up to one year after the end of the implementation on the student’s achievement. It is relevant to highlight that this research seeks a real-world application. This means that if the operational management practices developed and implemented really impact the student’s achievement in a way that persuades and influences politicians and policymakers, it will be possible to scale the project of implementation of the best management practices for the entire system of schools in the city of Rio de Janeiro.
|
After
What is the causal effect of management on pupils’ achievement? How can independent public audit institutions such as the accountability courts act directly to generate better public policies? The treatment will be a set of specific management practices developed based on a growing literature about the strong association between these practices and productivity in schools, manufacturing, and hospitals which can be measured, taught in business schools, and recommended by consultants. These specific management practices will be implemented, measured, and tested through a randomized field experiment on a sample of state-run schools from Rio de Janeiro.
Evaluation design: The causal connection between management and student learning can be answered using a quantitative framework. The answer to the question is intended to be conclusive. Its focus is on answering “What,” “Where,” and “When” (Bray, 2008, p.299). To answer these queries, the research will be based on a field experiment with a two-level design - schools and students - with one level of nesting - students to schools - where the schools, level 2-unit, are randomly assigned to treatment and control groups. The random assignment to the groups solves the selection problem making it possible to attribute the differences between groups to the treatment applied. The study can be considered a school-level clustered randomized field experiment or, in other words, a two-level clustered randomized field experiment.
A sample of eighty schools will be chosen randomly from these 1,001 (one thousand and one) units responsible for pupils from 1st to 9th year. After that, the control and treatment groups will be formed randomly by 40 schools each. It is vital to analyze the balance between control and treatment groups considering a variable related to the interest results, that is, students’ achievement, but controlled by socioeconomic features. The idea is to use the Brazilian Basic Education Assessment System (Saeb) of 2019, the last national assessment available, to represent the students’ learning outcomes; the Brazilian Socioeconomic Level (NSE) of 2019, the last national assessment index of this type available. Saeb is a standardized test applied along with a questionnaire that is the basis of the NSE every two years. They are applied by the National Institute of Educational Studies and Research Anísio Teixeira (Inep), an autonomous federal government agency. NSE is an index formed by combining two elements: parents’ education and possession of goods and services. The agency discloses the data by school, what is needed to prevent bias as it is presented in the next paragraph. The goal is to ensure the balance between control and treatment groups regarding each of the three types of schools.
This study will use a matched-pair randomization procedure. Firstly, a linear regression of the Saeb of 2019 on the NSE of the 1,001 population of schools in Rio de Janeiro will generate the constant α and coefficient β1. For each school in the sample of eighty schools, using α and β1 from the first regression, it will be generated the expected Saeb by the equation SAEB_Expected = α+β1∗NSE2019. Therefore, Saeb2019−SaebExpected will show for each school a number that represents the distance from the tendency line given by Saeb_Expected controlled by NSE. Thus, the schools will be listed in descending order based on the number reached from the previous subtraction. Every two schools will be matched, forming 40 pairs: school in the 1st place with the school on the 2nd, 3rd with 4th, etc. One school will be randomly assigned to the control group and another to the treatment group for each pair. This will be performed for each pair, ensuring the balance between control and treatment groups. Therefore, the two groups of schools will have a balanced score of students’ achievements controlled for the socioeconomic level and randomization.
Twenty-three management practices are the basis of the international management index created by Bloom et al. (2015). Each of the twenty-three practices is assessed considering a scale from 1 to 5. For instance, consider practice 1. (a), Bloom et al. (2015) define that a school receives score 1 if “No clear or institutionalized instructional planning processes or protocols exist; little verification or follow-up is done to ensure consistency across classrooms,” a score 3 if “ school has defined instructional planning processes or protocols to support instructional strategies and materials and incorporate some flexibility to meet students’ needs; monitoring is only adequate,” and a score 5 if “ school has implemented a clearly defined instructional planning process designed to align instructional strategies and materials with learning expectations and incorporate flexibility to meet student needs; these are followed up on through comprehensive monitoring or oversight.” Grade 1 reflects the lack of the respective best high-level management practice; grade 3 represents that a part of the respective best high-level management practice exists; and grade 5 will be attributed to the school that has the respective best high-level management practice totally implemented. The grade average of the twenty-three best management practices will represent the management index of each school. Therefore, this study will focus on implementing the operational practices needed to increase the management index of each school in the treatment group of the experiment.
The schools in the control group will be isolated from the project to avoid the “contamination effect”; no one knows which schools are in the control group, only the interinstitutional team. The contamination effect can be avoided leaving the participants of an experiment not knowing which group they are in, treatment or control. It is relevant to highlight that only the inter-institutional team will have information about the schools assigned to the treatment and the control groups. This fact is included in the signed collaboration term. The team will support the school managers in the schools of the treatment group to increase the management level of each school.
On the one hand, the study will use the international management index as presented previously based on the best management practices to show the management level of each school. The international management index will be measured through double-checked face-to-face interviews with the school principals before and after the experiment. The interviews will be recorded to enable a different evaluator from the interinstitutional team to conduct a new assessment of the answers given by the school manager. Finally, the management index of each school will be the average of its scores in each of the twenty-three management measures. On the other hand, the average of mathematics, science, and reading tests will represent the measure of pupil learning outcomes. The educational system of Rio de Janeiro has a bi-monthly general assessment of the students’ learning already implemented and running.
There will be three rounds of matched assessment, management vs. pupil outcomes. The first round of evaluation will diagnose the situation of schools on the treatment and control groups, considering the international management index described previously and the average score of the students on the tests. The second round of matched assessment will happen just after the intervention period to allow measurement of the average impact of the operational practices implemented in the treatment group on pupil outcomes. After the intervention period, it will start the monitoring period where schools will be monitored regarding the maintenance or not of the operational management practices developed and implemented. The third round will occur just after the period of monitoring.
This procedure will make it possible to measure the average impact of the intervention just after the implementation and one year after the end of the implementation on the students’ achievement. It is relevant to highlight that this research seeks a real-world application. This means that if the operational management practices developed and implemented really impact the students’ achievement in a way that persuades and influences politicians and policymakers, it will be possible to scale the project up by implementing the best management practices for the entire system of schools in the city of Rio de Janeiro.
|
|
Field
Randomization Method
|
Before
In terms of method, the research will be based on a field experiment with three-level design - schools, classes, and students - with two levels of nesting - classes to the school, and students to the class - where the schools, level 3-unit, are randomly assigned to treatment or control groups. The study will be a school-level randomised field experiment formed by a control group and a treatment group of schools.
The interventions will occur in the primary and lower secondary state-run schools of the city of Rio de Janeiro that provide education for young people from the first to the ninth grade. The choice for the city was driven by the availability of the Court of Accountability of Rio de Janeiro and the Secretary of Education to partner to conduct out the experiment. The city of Rio has one of the largest education systems in South America with 1,543 schools and nurseries and 644,000 students from nurseries to special schools. The sample of sixty schools will be chosen randomly from 1,001 (one thousand and one) units responsible for 430,510 (four hundred and thirty thousand five hundred and ten) pupils from 1st to 9th year. Thereafter, the control and treatment groups will be formed randomly by 30 schools each.
The experimental sample will be chosen from a pool of three different types of education units: 233 schools responsible exclusively for children from the 1st to 5th year, 222 responsible exclusively for pupils from the 6th to 9th grade, and 546 responsible for children from 1st to 9th year. It is important to analyse the balance between the control and treatment groups considering a variable correlated with the results of interest, that is, students’ achievement. The idea is to use the Brazilian Basic Education Assessment System (Saeb) of 2019, the last national assessment available, as a representation of the students learning outcome. It is a standardised test applied every two years by the National Institute of Educational Studies and Research Anísio Teixeira (Inep), an autonomous federal government agency. The agency discloses the data by school, what is needed to prevent bias as follows. The goal is to ensure the balance between control and treatment groups regarding each of the three types of schools detailed previously in this paragraph.
Following the technique used by Fryer (2017), this study will use a matched-pair randomisation procedure. Firstly, the schools will be listed, for each type of school, in a descending order based on the score reached in the Saeb 2019 (pre-treatment score). Each type of school will appear on a separated ordered list thus it will have three lists. Every two schools in each of the three ordered lists will be matched forming 30 pairs. For instance, consider the list where appears only the schools responsible exclusively for children from the 1st to 9th year, it will be separated as follow based on the ordered list: school on the 1st place with the school on the 2nd, 3rd with 4th, etc. It will guarantee that the two schools in each pair will have grades from Saeb very close. For each pair, one school will be randomly assigned to the control group and the another to the treatment group. It will be made for each pair in the three ordered lists ensuring the balance between the control and treatment group. Therefore, the two groups of schools will have a very balanced score of student achievement.
|
After
In terms of method, the research will be based on a field experiment with two-level design - schools, and students - with one level of nesting - students to schools - where the schools, level 2-unit, are randomly assigned to treatment or control groups. The study will be a school-level randomised field experiment formed by a control group and a treatment group of schools.
The interventions will occur in the primary and lower secondary state-run schools of the city of Rio de Janeiro that provide education for young people from the first to the ninth grade. The choice for the city was driven by the availability of the Court of Accountability of Rio de Janeiro and the Secretary of Education to partner to conduct out the experiment. The city of Rio has one of the largest education systems in South America with 1,543 schools and nurseries and 644,000 students from nurseries to special schools. The sample of sixty schools will be chosen randomly from 1,001 (one thousand and one) units responsible for 430,510 (four hundred and thirty thousand five hundred and ten) pupils from 1st to 9th year. Thereafter, the control and treatment groups will be formed randomly by 30 schools each.
The experimental sample will be chosen from a pool of three different types of education units: 233 schools responsible exclusively for children from the 1st to 5th year, 222 responsible exclusively for pupils from the 6th to 9th grade, and 546 responsible for children from 1st to 9th year. The first step will be the random sampling of 80 schools from the population of schools from the city of Rio de Janeiro. It will be done with an institutional computer from the Court of Accountability using the program Stata. The procedure will be inspected by representatives from the Court of Accountability of Rio de Janeiro (TCMRJ), Municipal Secretariat of Education of Rio de Janeiro (SMERJ), Interinstitutional Team, and by Prosecutors.
Afterward, it is important to analyse the balance between the control and treatment groups considering a variable correlated with the results of interest, that is, students’ achievement. The idea is to use the Brazilian Basic Education Assessment System (Saeb) of 2019, the last national assessment available, as a representation of the students learning outcome. It is a standardised test applied every two years by the National Institute of Educational Studies and Research Anísio Teixeira (Inep), an autonomous federal government agency. The agency discloses the data by school, what is needed to prevent bias as follows. The goal is to ensure the balance between control and treatment groups regarding each of the three types of schools detailed previously in this paragraph.
Following the technique used by Fryer (2017), this study will use a matched-pair randomisation procedure. Firstly, the schools will be listed, for each type of school, in a descending order based on the score reached in the Saeb 2019 (pre-treatment score). Each type of school will appear on a separated ordered list thus it will have three lists. Every two schools in each of the three ordered lists will be matched forming 30 pairs. For instance, consider the list where appears only the schools responsible exclusively for children from the 1st to 9th year, it will be separated as follow based on the ordered list: school on the 1st place with the school on the 2nd, 3rd with 4th, etc. It will guarantee that the two schools in each pair will have grades from Saeb very close. For each pair, one school will be randomly assigned to the control group and the another to the treatment group. It will be made for each pair in the three ordered lists ensuring the balance between the control and treatment group. Therefore, the two groups of schools will have a very balanced score of student achievement.
|
|
Field
Planned Number of Clusters
|
Before
60 Schools
|
After
80 Schools
|
|
Field
Planned Number of Observations
|
Before
Two classes of each school will be assessed. Each class that will be assessed has an average of 35 students. Therefore, 4200 pupils will be observed.
|
After
Approximately, 32,000 students from 80 schools will be observed.
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
30 schools control and 30 schools treatment
|
After
40 schools control and 40 schools treatment
|
|
Field
Power calculation: Minimum Detectable Effect Size for Main Outcomes
|
Before
The experiment will be a three-level cluster randomised assignment experiment with students nested within classes and classes nested within schools. It will use a hierarchical linear model with randomisation and treatment occurring on the highest level, that is, level 3 (schools).
The research intends to implement only one treatment to schools in the treatment group. It is defined that the proportion of schools that will be randomly assigned to the treatment group will be P = 0.5. Therefore, the proportion of schools in the control group will be 1-P =0.5. The total sample size randomly chosen from the population will be K = 60 and the mean number of level 2 units (classrooms) per level 3 unit (school) will be J=2. According to data from the Municipal Secretariat of Education of Rio de Janeiro, the average size of a class in the 5th year and the 9th year is n = 35. Let's use plausible ICC from Konstantopoulos (2008), and edges and Hedberg (2007). These studies indicate that most of the school-level intraclass correlations, ICC_3, go from 0.1 to 0.2 and that classroom-level intraclass correlations, ICC_2, range from 0.067 to 0.134.
Being conservative and assuming a model without any covariate, if it is used the largest values presented in the previous paragraph for ICC_3 = 0.2 and ICC_2 = 0.134, the MDES will be 0.387. If it is used the smaller values for ICC, ICC_3 = 0.1 and ICC_2 = 0.067, the MDES will be 0.281. Both values are plausible values to the effect of management on pupil outcomes considering previous studies.
|
After
MDES between 0.195 and 0.273 considering school-level intraclass correlation from 0.1 to 0.2 and without covariates.
|
|
Field
Additional Keyword(s)
|
Before
Management, randomised field experiment, multilevel model, clusters
|
After
Management, randomised field experiment, multilevel model, clusters, causality, causal effect, randomised controlled trial, RCT, field RCT
|
|
Field
Building on Existing Work
|
Before
|
After
No
|