Back to History Current Version

The impact of an agile management policy on education. Evidence from a randomised field experiment in Brazil

Last registered on June 22, 2021

Pre-Trial

Trial Information

General Information

Title
Management, institutional arrangements, and education: a three-level clustered randomised field experiment in Rio de Janeiro
RCT ID
AEARCTR-0007669
Initial registration date
May 15, 2021

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
May 17, 2021, 10:31 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
June 22, 2021, 12:26 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Primary Investigator

Affiliation
Trinity College - University of Cambridge

Other Primary Investigator(s)

PI Affiliation
Trinity College - University of Cambridge

Additional Trial Information

Status
In development
Start date
2020-10-01
End date
2024-04-26
Secondary IDs
Abstract
What is the causal effect of management on schools? How can the independent public audit institutions as the courts of accountability act directly to generate better public policies?

One goal of the study is to analyse with a three-level randomised field experiment design if management can explain large differences in school performance. The focus will be on the school-level management practices of primary and lower secondary schools.

In order to find rigorous answers to scientific questions, researchers have used controlled randomised experiments. The method is used to infer causal relationship between variables from a study. The current research is intended to answer "What is the causal effect of management on education?". Therefore, it is needed to answer "How would students and schools who participated of the program have fared in the absence of the program?" and "How would those who were not exposed to the program have fared in the presence of the program?". Control groups, treatment groups and randomisation allow scientists to obtain an average impact of the program on a group of individuals by comparing them to a group who were not exposed to the program.

Using randomised trials, Esther Duflo and Abhijit Banerjee are reinventing development economics. The couple and their friend, Harvard economist Michael Kremer, awarded the 2019 Nobel Prize in economics as a consequence of their scientific and evidence-driven economics tackle the world poverty problem. The Nobel Prize committee said "Their experimental research methods now entirely dominate development economics".

According to the cited Nobel Prize winners, working with local partners has enabled conducting many randomised field evaluations in developing countries. Therefore, the experiment is intended to occur in Brazil, specifically in the city of Rio de Janeiro where it is possible to succeed with partnerships. Brazil is one of the largest economies in the world. It is a country with marked inequality, violence, and poverty. The city of Rio de Janeiro is the capital of the state of Rio de Janeiro and it is situated in the Southeast of the country. It has the second largest population in Brazil with almost 7 million people \parencite{site}.

According to The World Bank, 81\% of primary schools and 73\% of secondary schools around the world are state-run schools. Therefore, in order to generate wide results, the study will analyse schools operated, managed, and controlled by public authorities.

The second goal of the research is to explore how the government external control institutions, which have recognised qualified staff within the public administration around the world, can bring innovations to the state such as randomised field experiments in partnership with the policymakers.

Government external audit institutions have attracted growing interests of citizens, governments and non governmental institutions around the world because they play a very relevant role to avoid corruption and reducing waste \parencite{MeloMarcusAndré2009PaIC}.

The are two models of independent public external audit institutions in the world, the courts, and the audit offices. These organisations have worked historically assessing legal, financial, and accounting aspects of public spending. In the last decades, these institutions have worked also through performance audits to analyse economy, efficiency, and effectiveness of the public policies and organisations using well-established methods as founded on the International Standards of Supreme Audit Institutions 3000 (ISSAI 3000). More recently, there has been a global movement to broaden the scope and the theoretical basis of the public policies analysis made by the government audit institutions. That is, in addition to the analysis considering economy, efficiency, and effectiveness of the public policies already made in the performance audits, the idea is to evaluate also the impact and relevance of public spending using the scientific framework of social sciences.

The system of courts of auditors/account/accountability is encountered in countries such as France, Italy, Portugal, Germany, Netherlands, and Brazil. The audit offices are found in countries such as the United States of America, the United Kingdom, India, and Sweden.

Article written by Laffan claims that the European Court of Auditors had to strive to become a "living institution" and to find its place in the Union's order. According to the author, one of the most important factors for the constant evolution has been its search for cooperative relations with other EU institutions. Following the idea of cooperation and partnership, this research will explore how the government external audit institutions can acting in a new role, bring innovations and effectiveness to the public administration.

Brazil has thirty-three independent courts of accountability that act in the national, regional e local level and its members are life-tenure judges. Furthermore, there are specialised independent life-tenure prosecutors and autonomous auditors with strong stability. These strong guarantees aim at ensuring the independence of the Courts from the Government. This research is intended to explore, through a partnership with the Municipal Secretariat of Education of Rio de Janeiro, the use of the Court of Accountability of the city of Rio de Janeiro (TCMRJ), responsible to control four billion pounds in public policies each year, to conduct a randomised field experiment to test with scientific rigour the performance of selected public policies.

The Municipal Court of Accountability of Rio de Janeiro will conduct a three-level clustered randomised field experiment ex ante in partnership with the policy maker in order to verify a policy before the spending starts.

The research is intended to bring stronger evidence than previous studies on the question of whether management can impact student learning. Furthermore, it can lead the courts of accountability or audit offices to a new way of acting, bringing, in partnership with policy makers, innovation and better public policies for the society.
External Link(s)

Registration Citation

Citation
Galvão Puccioni, Felipe and Tiago Cavalcanti. 2021. "Management, institutional arrangements, and education: a three-level clustered randomised field experiment in Rio de Janeiro." AEA RCT Registry. June 22. https://doi.org/10.1257/rct.7669-1.2000000000000002
Experimental Details

Interventions

Intervention(s)
The project of intervention will occur based on a partnership between the Municipal Secretariat of Education of Rio de Janeiro - SMERJ and the Court of Accountability of the City of Rio de Janeiro - TCMRJ. A collaboration term is needed because the project will be executed in the schools controlled by the Secretary of Education using human resources from the Court and from the Secretary. The document defines the responsibilities of each part and the implementation operational procedures of the experiment. Marked professionals from the Court and Secretariat will form the inter-institutional team that will deliver the treatment and placebo measures in the schools. A formal designation of the inter-institutional team, as defined in the collaboration term, will be done with a joint act signed by the President of the Court and the Secretary of Education.

It will be created, defined, and implemented operational practices based on the best management practices compiled by Bloom et al. (2015) from decades of research in management and education. The study cited absorbs management measures from previous research across other sectors such as manufacturing, retail, health care sectors, but with changes in order to adjust the framework to the school context. Therefore, sixteen management basic measures are applicable to different sectors as showed in Bloom et al. (2014), seven are specially designed to the management of schools (Bloom et al., 2015).

The research conducted by Bloom et al. (2015) shows the best management practices in a high-level view. Therefore, it is important to develop, based on the cited high-level definition of the best management practices, low-level operational practices to be implemented in schools aiming at a better school management situation. Considering this fact, operational practices - considering the reality of the Brazilian educational system and based on the high-level best management measures presented in Bloom et al. (2015) - will be developed in order to improve the management of the schools in Rio de Janeiro.

The implementation of the operational practices will be done by the inter-institutional team formed by outstanding professionals from the Court of Accountability and the Secretariat of Education. They will work autonomously focused on the improvement of the management index of each school assigned to the treatment group. The team also will have to implement some practices, with a very reduced intensity if compared to the measures applied to the treatment group, in the schools assigned to the control group to avoid the "contamination effect". The contamination effect can be avoided leaving the participants of an experiment not knowing which group they are in, treatment or control. Therefore, placebo measures will be applied to the schools in the control group. It is relevant to highlight that only the inter-institutional team will have information about what are the schools assigned to treatment and control groups. This fact is included in the collaboration term.


Intervention Start Date
2021-10-04
Intervention End Date
2022-09-30

Primary Outcomes

Primary Outcomes (end points)
Students' scores in math, science and Portuguese tests
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Brazilian Basic Education Assessment System (Saeb) is a standardised test applied every two years by the National Institute of Educational Studies and Research Anísio Teixeira (Inep), an autonomous federal government agency.
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
In terms of method, the research will be based on a field experiment with three-level design - schools, classes, and students - with two levels of nesting - classes to the school, and students to the class - where the schools, level 3-unit, are randomly assigned to treatment or control groups. The study will be a school-level randomised field experiment formed by a control group and a treatment group of schools.

The interventions will occur in the primary and lower secondary state-run schools of the city of Rio de Janeiro that provide education for young people from the first to the ninth grade. The choice for the city was driven by the availability of the Court of Accountability of Rio de Janeiro and the Secretary of Education to partner to conduct out the experiment. The city of Rio has one of the largest education systems in South America with 1,543 schools and nurseries and 644,000 students from nurseries to special schools. The sample of sixty schools will be chosen randomly from 1,001 (one thousand and one) units responsible for 430,510 (four hundred and thirty thousand five hundred and ten) pupils from 1st to 9th year. Thereafter, the control and treatment groups will be formed randomly by 30 schools each.

The experimental sample will be chosen from a pool of three different types of education units: 233 schools responsible exclusively for children from the 1st to 5th year, 222 responsible exclusively for pupils from the 6th to 9th grade, and 546 responsible for children from 1st to 9th year. It is important to analyse the balance between the control and treatment groups considering a variable correlated with the results of interest, that is, students’ achievement. The idea is to use the Brazilian Basic Education Assessment System (Saeb) of 2019, the last national assessment available, as a representation of the students learning outcome. It is a standardised test applied every two years by the National Institute of Educational Studies and Research Anísio Teixeira (Inep), an autonomous federal government agency. The agency discloses the data by school, what is needed to prevent bias as follows. The goal is to ensure the balance between control and treatment groups regarding each of the three types of schools detailed previously in this paragraph.

Following the technique used by Fryer (2017), this study will use a matched-pair randomisation procedure. Firstly, the schools will be listed, for each type of school, in a descending order based on the score reached in the Saeb 2019 (pre-treatment score). Each type of school will appear on a separated ordered list thus it will have three lists. Every two schools in each of the three ordered lists will be matched forming 30 pairs. For instance, consider the list where appears only the schools responsible exclusively for children from the 1st to 9th year, it will be separated as follow based on the ordered list: school on the 1st place with the school on the 2nd, 3rd with 4th, etc. It will guarantee that the two schools in each pair will have grades from Saeb very close. For each pair, one school will be randomly assigned to the control group and the another to the treatment group. It will be made for each pair in the three ordered lists ensuring the balance between the control and treatment group. Therefore, the two groups of schools will have a very balanced score of student achievement.

As part of the methodology, the study will define the operational actions, i.e. the treatment, to be implemented in the schools using base management measures compiled from decades of research in management and education presented in Bloom et al. (2015).

The 23 management practices are the basis of the international management index also created by Bloom et al. (2015). Each of the twenty-three practices is assessed considering a grade from 1 to 5. For instance, consider practice 1. (a), Bloom et al. (2015) define that a school receives a score of 1 if "No clear or institutionalised instructional planning processes or protocols exist; little verification or follow up is done to ensure consistency across classrooms.", a score 3 if "School has defined instructional planning processes or protocols to support instructional strategies and materials and incorporate some flexibility to meet student’s needs; monitoring is only adequate.", and a score 5 if "School has implemented a clearly defined instructional planning process designed to align instructional strategies and materials with learning expectations and incorporate flexibility to meet student needs; these are followed up on through comprehensive monitoring or oversight.".

Grade 1 reflects the lack of the respective best high-level management practice, grade 3 represents that a part of the respective best high-level management practice exists, and grade 5 will be attributed to the school that has the respective best high-level management practice totally implemented. The grade average of the twenty-three best management practices will represent the management index of each school. Therefore, this study will focus on the implementation of the operational practices needed to increase the management index of each school in the treatment group of the experiment.

On the one hand, the study will use the international management index as presented previously based on the best management practices to show the management level of each school. The international management index will be measured through double-checked face-to-face interviews with the school principals. The interviews will be recorded to enable that a different evaluator from the inter-institutional team conducts a new assessment of the answers given by the school manager. Finally, the management index of each school will be the average of its scores in each of the twenty-three management measures.

On the other hand, the average of mathematics, science, and Portuguese language tests will represent the measure of pupil learning outcome. The tests will be applied in two classes of the same year per school. For schools responsible exclusively for children from the 1st to 5th year, the tests will be applied in classes of the 5th year. For schools responsible exclusively for children from 6th to 9th year or schools responsible for children from 1st to 9th year, the tests will be applied to classes of the 9th year. Therefore, the average result of the two classes of the same year will represent the average learning outcome of the students at the respective school. The selected classes will be assessed semi-annually during the experiment to allow comparisons through the time in each school.


There will be five rounds of matched assessment, management vs pupil outcomes, semi-annually. The first round of assessment will diagnose the situation of schools on the treatment and control groups, considering the international management index described previously and the average score of the students on the tests. The first round is supposed to happen in September 2021 and last on August 2023. The schedule was adapted due to the impact of coronavirus in Brazil.

Before the first round of matched assessment, it will be created a plan describing operational management practices needed to improve the management situation of the schools. In fact, the plan is almost complete. The goal of this plan is to guide the schools to improve their management index considering each of the 23 high-level management practices presented by Bloom et al. (2015).

After the diagnosis (first assessment round), the plan of implementation will be reviewed considering the management situation and the management index value of each school within the treatment group. Following the review, the intervention will start intending to last 1 year. The intervention means that the inter-institutional team will be at the schools helping the school managers to implement the plan of management actions.

The plan of implementation is only a baseline, that is, other operational management practices could be created during the intervention period always following the high-level management practices defined by Bloom et al. (2015). Some operational practices may not be implemented due to real obstacles encountered in the schools. Each school may have different levels of implementation due to the different situations in each school. As it is expected that changes occur during the execution, the life cycle approach for the project is the adaptive life cycle Institute, 2017). This research intends to measure the average impact of the operational practices implemented in the treatment group on pupil outcomes.

After the intervention period, it will start the monitoring period where the schools will be monitored regarding the maintenance or not of the operational management practices developed and implemented.

This procedure will make it possible to measure the average impact of the intervention during the year in which it took place and up to one year after the end of the implementation on the student’s achievement. It is relevant to highlight that this research seeks a real-world application. This means that if the operational management practices developed and implemented really impact the student’s achievement in a way that persuades and influences politicians and policymakers, it will be possible to scale the project of implementation of the best management practices for the entire system of schools in the city of Rio de Janeiro.
Experimental Design Details
Not available
Randomization Method
In terms of method, the research will be based on a field experiment with three-level design - schools, classes, and students - with two levels of nesting - classes to the school, and students to the class - where the schools, level 3-unit, are randomly assigned to treatment or control groups. The study will be a school-level randomised field experiment formed by a control group and a treatment group of schools.

The interventions will occur in the primary and lower secondary state-run schools of the city of Rio de Janeiro that provide education for young people from the first to the ninth grade. The choice for the city was driven by the availability of the Court of Accountability of Rio de Janeiro and the Secretary of Education to partner to conduct out the experiment. The city of Rio has one of the largest education systems in South America with 1,543 schools and nurseries and 644,000 students from nurseries to special schools. The sample of sixty schools will be chosen randomly from 1,001 (one thousand and one) units responsible for 430,510 (four hundred and thirty thousand five hundred and ten) pupils from 1st to 9th year. Thereafter, the control and treatment groups will be formed randomly by 30 schools each.

The experimental sample will be chosen from a pool of three different types of education units: 233 schools responsible exclusively for children from the 1st to 5th year, 222 responsible exclusively for pupils from the 6th to 9th grade, and 546 responsible for children from 1st to 9th year. It is important to analyse the balance between the control and treatment groups considering a variable correlated with the results of interest, that is, students’ achievement. The idea is to use the Brazilian Basic Education Assessment System (Saeb) of 2019, the last national assessment available, as a representation of the students learning outcome. It is a standardised test applied every two years by the National Institute of Educational Studies and Research Anísio Teixeira (Inep), an autonomous federal government agency. The agency discloses the data by school, what is needed to prevent bias as follows. The goal is to ensure the balance between control and treatment groups regarding each of the three types of schools detailed previously in this paragraph.

Following the technique used by Fryer (2017), this study will use a matched-pair randomisation procedure. Firstly, the schools will be listed, for each type of school, in a descending order based on the score reached in the Saeb 2019 (pre-treatment score). Each type of school will appear on a separated ordered list thus it will have three lists. Every two schools in each of the three ordered lists will be matched forming 30 pairs. For instance, consider the list where appears only the schools responsible exclusively for children from the 1st to 9th year, it will be separated as follow based on the ordered list: school on the 1st place with the school on the 2nd, 3rd with 4th, etc. It will guarantee that the two schools in each pair will have grades from Saeb very close. For each pair, one school will be randomly assigned to the control group and the another to the treatment group. It will be made for each pair in the three ordered lists ensuring the balance between the control and treatment group. Therefore, the two groups of schools will have a very balanced score of student achievement.
Randomization Unit
School
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
60 Schools
Sample size: planned number of observations
Two classes of each school will be assessed. Each class that will be assessed has an average of 35 students. Therefore, 4200 pupils will be observed.
Sample size (or number of clusters) by treatment arms
30 schools control and 30 schools treatment
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
The experiment will be a three-level cluster randomised assignment experiment with students nested within classes and classes nested within schools. It will use a hierarchical linear model with randomisation and treatment occurring on the highest level, that is, level 3 (schools). The research intends to implement only one treatment to schools in the treatment group. It is defined that the proportion of schools that will be randomly assigned to the treatment group will be P = 0.5. Therefore, the proportion of schools in the control group will be 1-P =0.5. The total sample size randomly chosen from the population will be K = 60 and the mean number of level 2 units (classrooms) per level 3 unit (school) will be J=2. According to data from the Municipal Secretariat of Education of Rio de Janeiro, the average size of a class in the 5th year and the 9th year is n = 35. Let's use plausible ICC from Konstantopoulos (2008), and edges and Hedberg (2007). These studies indicate that most of the school-level intraclass correlations, ICC_3, go from 0.1 to 0.2 and that classroom-level intraclass correlations, ICC_2, range from 0.067 to 0.134. Being conservative and assuming a model without any covariate, if it is used the largest values presented in the previous paragraph for ICC_3 = 0.2 and ICC_2 = 0.134, the MDES will be 0.387. If it is used the smaller values for ICC, ICC_3 = 0.1 and ICC_2 = 0.067, the MDES will be 0.281. Both values are plausible values to the effect of management on pupil outcomes considering previous studies.
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number