Management, institutional arrangements, and education: a randomised field experiment in Rio de Janeiro

Last registered on November 12, 2021

Pre-Trial

Trial Information

General Information

Title
Management, institutional arrangements, and education: a randomised field experiment in Rio de Janeiro
RCT ID
AEARCTR-0007669
Initial registration date
May 15, 2021
Last updated
November 12, 2021, 10:20 AM EST

Locations

Primary Investigator

Affiliation
Trinity College - University of Cambridge

Other Primary Investigator(s)

PI Affiliation
Trinity College - University of Cambridge

Additional Trial Information

Status
In development
Start date
2021-11-15
End date
2024-06-24
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Although economists and other social scientists have worked hard trying to disentangle what factors can explain differences between performances across schools, there are few studies about the impact of management on education. For example, the discussion about management on productivity has not caught strong attention from economists (Bloom et al., 2012, p. 2). In the case of causal evidence coming from randomized trials, the gap is even larger. Despite a growing number of randomized experiments conducted by social researchers to address critical scientific questions on policies in the last decades, very few trials have been undertaken to explore the causal consequences of management on productivity (Duflo, 2007; Fryer, 2017). Thus, this research intends to reduce this knowledge gap by applying a rigorous scientific method, precisely a randomized field experiment, to understand our first research question, i.e., whether there is a causal relationship between school management and pupil learning. Understanding how management affects schools within and across countries is complex because of the lack of comparable measures to systematically assess management practices (Bloom et al., 2015). Therefore, as this research intends to use the same international management index created by Bloom et al. (2015), it will allow direct comparisons within Brazil and across different countries.

Although Brazil is one of the largest economies in the world, the country has marked levels of inequality and poverty. These two problems can create a unique pervasive dynamic in which individuals remain trapped in environments with scarce opportunities for upward social mobility (Banerjee & Duflo, 2007). High-quality public and free education are vital to breaking this vicious cycle, remaining the most reliable intergenerational way out of poverty for millions of families (UNICEF, 2020).

It is relevant to highlight that Brazil has more than 5,500 municipalities with the same standardized public educational system as Rio de Janeiro. The Brazilian educational system is bi-partite composed of private and public schools. The private system holds only 18.58% of the students from clearly better-off families, and the public system is responsible for the remaining 81,42% of pupils from mostly disadvantaged families. Even though the same Federal laws regulate the public system, they face diverse cultural and socioeconomic realities. Many schools are in income-deprived areas such as slums or favelas and deal with very complex social phenomena. This study's management practices have been developed to be flexible to adaptation and implementation in different realities. A key factor for adaptability is the inter-institutional team. It is thought to be dynamic and ready to adapt the baseline plan whenever necessary.

The City of Rio de Janeiro is the capital of the state of Rio de Janeiro and has the second-largest population among Brazilian cities, with almost 7 million people (IBGE, 2020). It has one of the largest education systems in South America, with 1,543 schools and nurseries and 644,000 students from nurseries to special schools. The Rio de Janeiro case is remarkable because many schools are located within slums and favelas dominated by drug dealers, where even the police cannot enter and where families are exposed to extreme marginality and violence daily. Exposure to extra-school vulnerabilities directly influences the learning conditions. In this context, management practices are even more crucial to ensure meaningful learning and increased growth prospects for pupils. The project will deal with 80 schools randomly selected from the Rio de Janeiro school population, mostly from poor city regions.
The idea is to improve education by implementing best management practices reported in the academic literature through the offer of in locus consultancy to selected schools. Considering the scarcity of resources in underdeveloped and developing countries, it is crucial to create feasible public policies. Aiming at a viable project, a partnership between government institutions in Rio de Janeiro was signed by the Court of Accountability of Rio de Janeiro (TCMRJ), Municipal Secretariat of Education (SMERJ), Dr. Tiago Cavalcanti (Faculty of Economics – University of Cambridge), Professor Shailaja Fennell (Centre of Development Studies – University of Cambridge), and Felipe Galvão Puccioni (Principal Investigator – TCMRJ). Based on the best practices shown in the academic literature, a flexible plan has been developed to deal with different schools, cities, and country realities. A randomized field experiment will be conducted to test a causal relationship between the best management practices that will be implemented and educational results. Eighty schools were randomly selected and assigned by pair-matching in two groups: treatment and control. For two years, the team, starting in 2022, will help school principals and managers to implement the best management practices.

The management practices of this study are adaptable for each school depending on the management level encountered on the diagnostic phase. The flexibility and customization of these practices are possible because the project is thought to be implemented by an interinstitutional team formed by marked professionals hired from the partners' institutions. The team will adapt the best management practices baseline plan to each school depending on the school, city, and country reality. Although the experiment is in Rio de Janeiro, it is supposed to be easily implemented in different social contexts globally with the appropriate adaptation to local realities.

This study could lead to a deeper understanding of state run-schools, which represent 81% of primary schools and 73% of secondary schools around the world, according to The World Bank (2020). Even though principals play a critical role and there is an increasing consensus on what they must do to be effective (Fryer & Dobbie, 2013; Fryer, 2014), this study will go further and analyze not only the school principals but also other managers within the schools who help the school heads to manage the educational units.

The second research question is: How can the government audit institutions, such as the courts of accountability, act directly to generate better public policies? These institutions have attracted the growing interest of citizens, governments, and non-governmental institutions worldwide because they play a relevant role in avoiding corruption and reducing public resources waste (Melo et al., 2009). More recently, there has been a movement to broaden the scope and the theoretical basis of the public policies analysis conducted by governmental audit institutions. In addition to analyzing the economy, efficiency, and effectiveness of public policies already made in performance audits, the idea is also to evaluate the utility and impact of public spending using the scientific framework of social sciences.

Brazil uses the court system and has thirty-three independent accountability courts that act at the national, regional, and local levels, and its members are life-tenure judges. In addition, there are specialized independent life-tenure prosecutors and autonomous auditors with strong stability against arbitrary dismissal. These strong guarantees aim at ensuring the independence of the Courts from the government. This study, in an innovative path, will check how the Municipal Court of Accountability of Rio de Janeiro (TCMRJ) - which is responsible for controlling thirty billion "reais" (four billion pounds) in public policies each year – will conduct a two-level clustered randomized field experiment ex-ante in partnership with policymakers to verify a policy before it is implemented. Therefore, this research aims also to explore how a government external control institution, specifically the Municipal Court of Accountability of Rio de Janeiro, can bring innovations, in partnership with policymakers, by conducting randomized field experiments.

In summary, the research is intended to bring more substantial evidence than previous studies on whether management can impact school performance. Furthermore, it can lead the courts of accountability or audit offices to a new way of acting, bringing, in partnership with policymakers, innovation and better public policies for the society.

The project is innovative in different ways. Firstly, the interinstitutional design is unique. It involves a partnership between the controller and the controlled government to develop, based on the academic literature, implement, with an inter-institutional team, and test, through a randomized field experiment, a set of the best management practices. Secondly, we use government institutions to develop, implement and test your own policies through a very rigorous scientific path. Assessing the causal relationship between management and educational results is innovative and challenging. Thirdly, the project is developing an innovative path of action for the government independent audit institutions by conducting randomized field experiments in partnership with policymakers to test public policies before those are implemented.
External Link(s)

Registration Citation

Citation
Galvão Puccioni, Felipe and Tiago Cavalcanti. 2021. "Management, institutional arrangements, and education: a randomised field experiment in Rio de Janeiro." AEA RCT Registry. November 12. https://doi.org/10.1257/rct.7669-1.4000000000000004
Experimental Details

Interventions

Intervention(s)
The project of intervention will occur based on a partnership between the Municipal Secretariat of Education of Rio de Janeiro - SMERJ and the Court of Accountability of the City of Rio de Janeiro - TCMRJ. A collaboration term is needed because the project will be executed in the schools controlled by the Secretary of Education using human resources from the Court and from the Secretary. The document defines the responsibilities of each part and the implementation operational procedures of the experiment. Marked professionals from the Court and Secretariat will form the inter-institutional team that will deliver the treatment measures in the schools. A formal designation of the inter-institutional team, as defined in the collaboration term, will be done with a joint act signed by the President of the Court and the Secretary of Education.

It will be created, defined, and implemented operational practices based on the best management practices compiled by Bloom et al. (2015) from decades of research in management and education. The study cited absorbs management measures from previous research across other sectors such as manufacturing, retail, health care sectors, but with changes in order to adjust the framework to the school context. Therefore, sixteen management basic measures are applicable to different sectors as showed in Bloom et al. (2014), seven are specially designed to the management of schools (Bloom et al., 2015).

The research conducted by Bloom et al. (2015) shows the best management practices in a high-level view. Therefore, it is important to develop, based on the cited high-level definition of the best management practices, low-level operational practices to be implemented in schools aiming at a better school management situation. Considering this fact, operational practices - considering the reality of the Brazilian educational system and based on the high-level best management measures presented in Bloom et al. (2015) - will be developed in order to improve the management of the schools in Rio de Janeiro.

The implementation of the operational practices will be done by the inter-institutional team formed by outstanding professionals from the Court of Accountability and the Secretariat of Education. They will work autonomously focused on the improvement of the management index of each school assigned to the treatment group. It is relevant to highlight that only the inter-institutional team will have information about what are the schools assigned to treatment and control groups. The schools in the control group will not know they are in the study.
These facts are included in the collaboration term.

Intervention Start Date
2022-01-24
Intervention End Date
2023-07-24

Primary Outcomes

Primary Outcomes (end points)
Students' scores from a standardised assessment conducted bimonthly by the Municipal Secretariat of Education of Rio de Janeiro.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Brazilian Basic Education Assessment System (Saeb) is a standardised test applied every two years by the National Institute of Educational Studies and Research Anísio Teixeira (Inep), an autonomous federal government agency.
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
What is the causal effect of management on pupils’ achievement? How can independent public audit institutions such as the accountability courts act directly to generate better public policies? The treatment will be a set of specific management practices developed based on a growing literature about the strong association between these practices and productivity in schools, manufacturing, and hospitals which can be measured, taught in business schools, and recommended by consultants. These specific management practices will be implemented, measured, and tested through a randomized field experiment on a sample of state-run schools from Rio de Janeiro.

Evaluation design: The causal connection between management and student learning can be answered using a quantitative framework. The answer to the question is intended to be conclusive. Its focus is on answering “What,” “Where,” and “When” (Bray, 2008, p.299). To answer these queries, the research will be based on a field experiment with a two-level design - schools and students - with one level of nesting - students to schools - where the schools, level 2-unit, are randomly assigned to treatment and control groups. The random assignment to the groups solves the selection problem making it possible to attribute the differences between groups to the treatment applied. The study can be considered a school-level clustered randomized field experiment or, in other words, a two-level clustered randomized field experiment.

A sample of eighty schools will be chosen randomly from these 1,001 (one thousand and one) units responsible for pupils from 1st to 9th year. After that, the control and treatment groups will be formed randomly by 40 schools each. It is vital to analyze the balance between control and treatment groups considering a variable related to the interest results, that is, students’ achievement, but controlled by socioeconomic features. The idea is to use the Brazilian Basic Education Assessment System (Saeb) of 2019, the last national assessment available, to represent the students’ learning outcomes; the Brazilian Socioeconomic Level (NSE) of 2019, the last national assessment index of this type available. Saeb is a standardized test applied along with a questionnaire that is the basis of the NSE every two years. They are applied by the National Institute of Educational Studies and Research Anísio Teixeira (Inep), an autonomous federal government agency. NSE is an index formed by combining two elements: parents’ education and possession of goods and services. The agency discloses the data by school, what is needed to prevent bias as it is presented in the next paragraph. The goal is to ensure the balance between control and treatment groups regarding each of the three types of schools.

This study will use a matched-pair randomization procedure. Firstly, a linear regression of the Saeb of 2019 on the NSE of the 1,001 population of schools in Rio de Janeiro will generate the constant α and coefficient β1. For each school in the sample of eighty schools, using α and β1 from the first regression, it will be generated the expected Saeb by the equation SAEB_Expected = α+β1∗NSE2019. Therefore, Saeb2019−SaebExpected will show for each school a number that represents the distance from the tendency line given by Saeb_Expected controlled by NSE. Thus, the schools will be listed in descending order based on the number reached from the previous subtraction. Every two schools will be matched, forming 40 pairs: school in the 1st place with the school on the 2nd, 3rd with 4th, etc. One school will be randomly assigned to the control group and another to the treatment group for each pair. This will be performed for each pair, ensuring the balance between control and treatment groups. Therefore, the two groups of schools will have a balanced score of students’ achievements controlled for the socioeconomic level and randomization.

Twenty-three management practices are the basis of the international management index created by Bloom et al. (2015). Each of the twenty-three practices is assessed considering a scale from 1 to 5. For instance, consider practice 1. (a), Bloom et al. (2015) define that a school receives score 1 if “No clear or institutionalized instructional planning processes or protocols exist; little verification or follow-up is done to ensure consistency across classrooms,” a score 3 if “ school has defined instructional planning processes or protocols to support instructional strategies and materials and incorporate some flexibility to meet students’ needs; monitoring is only adequate,” and a score 5 if “ school has implemented a clearly defined instructional planning process designed to align instructional strategies and materials with learning expectations and incorporate flexibility to meet student needs; these are followed up on through comprehensive monitoring or oversight.” Grade 1 reflects the lack of the respective best high-level management practice; grade 3 represents that a part of the respective best high-level management practice exists; and grade 5 will be attributed to the school that has the respective best high-level management practice totally implemented. The grade average of the twenty-three best management practices will represent the management index of each school. Therefore, this study will focus on implementing the operational practices needed to increase the management index of each school in the treatment group of the experiment.

The schools in the control group will be isolated from the project to avoid the “contamination effect”; no one knows which schools are in the control group, only the interinstitutional team. The contamination effect can be avoided leaving the participants of an experiment not knowing which group they are in, treatment or control. It is relevant to highlight that only the inter-institutional team will have information about the schools assigned to the treatment and the control groups. This fact is included in the signed collaboration term. The team will support the school managers in the schools of the treatment group to increase the management level of each school.

On the one hand, the study will use the international management index as presented previously based on the best management practices to show the management level of each school. The international management index will be measured through double-checked face-to-face interviews with the school principals before and after the experiment. The interviews will be recorded to enable a different evaluator from the interinstitutional team to conduct a new assessment of the answers given by the school manager. Finally, the management index of each school will be the average of its scores in each of the twenty-three management measures. On the other hand, the average of mathematics, science, and reading tests will represent the measure of pupil learning outcomes. The educational system of Rio de Janeiro has a bi-monthly general assessment of the students’ learning already implemented and running.

There will be three rounds of matched assessment, management vs. pupil outcomes. The first round of evaluation will diagnose the situation of schools on the treatment and control groups, considering the international management index described previously and the average score of the students on the tests. The second round of matched assessment will happen just after the intervention period to allow measurement of the average impact of the operational practices implemented in the treatment group on pupil outcomes. After the intervention period, it will start the monitoring period where schools will be monitored regarding the maintenance or not of the operational management practices developed and implemented. The third round will occur just after the period of monitoring.

This procedure will make it possible to measure the average impact of the intervention just after the implementation and one year after the end of the implementation on the students’ achievement. It is relevant to highlight that this research seeks a real-world application. This means that if the operational management practices developed and implemented really impact the students’ achievement in a way that persuades and influences politicians and policymakers, it will be possible to scale the project up by implementing the best management practices for the entire system of schools in the city of Rio de Janeiro.
Experimental Design Details
Not available
Randomization Method
In terms of method, the research will be based on a field experiment with two-level design - schools, and students - with one level of nesting - students to schools - where the schools, level 2-unit, are randomly assigned to treatment or control groups. The study will be a school-level randomised field experiment formed by a control group and a treatment group of schools.

The interventions will occur in the primary and lower secondary state-run schools of the city of Rio de Janeiro that provide education for young people from the first to the ninth grade. The choice for the city was driven by the availability of the Court of Accountability of Rio de Janeiro and the Secretary of Education to partner to conduct out the experiment. The city of Rio has one of the largest education systems in South America with 1,543 schools and nurseries and 644,000 students from nurseries to special schools. The sample of sixty schools will be chosen randomly from 1,001 (one thousand and one) units responsible for 430,510 (four hundred and thirty thousand five hundred and ten) pupils from 1st to 9th year. Thereafter, the control and treatment groups will be formed randomly by 30 schools each.

The experimental sample will be chosen from a pool of three different types of education units: 233 schools responsible exclusively for children from the 1st to 5th year, 222 responsible exclusively for pupils from the 6th to 9th grade, and 546 responsible for children from 1st to 9th year. The first step will be the random sampling of 80 schools from the population of schools from the city of Rio de Janeiro. It will be done with an institutional computer from the Court of Accountability using the program Stata. The procedure will be inspected by representatives from the Court of Accountability of Rio de Janeiro (TCMRJ), Municipal Secretariat of Education of Rio de Janeiro (SMERJ), Interinstitutional Team, and by Prosecutors.

Afterward, it is important to analyse the balance between the control and treatment groups considering a variable correlated with the results of interest, that is, students’ achievement. The idea is to use the Brazilian Basic Education Assessment System (Saeb) of 2019, the last national assessment available, as a representation of the students learning outcome. It is a standardised test applied every two years by the National Institute of Educational Studies and Research Anísio Teixeira (Inep), an autonomous federal government agency. The agency discloses the data by school, what is needed to prevent bias as follows. The goal is to ensure the balance between control and treatment groups regarding each of the three types of schools detailed previously in this paragraph.

Following the technique used by Fryer (2017), this study will use a matched-pair randomisation procedure. Firstly, the schools will be listed, for each type of school, in a descending order based on the score reached in the Saeb 2019 (pre-treatment score). Each type of school will appear on a separated ordered list thus it will have three lists. Every two schools in each of the three ordered lists will be matched forming 30 pairs. For instance, consider the list where appears only the schools responsible exclusively for children from the 1st to 9th year, it will be separated as follow based on the ordered list: school on the 1st place with the school on the 2nd, 3rd with 4th, etc. It will guarantee that the two schools in each pair will have grades from Saeb very close. For each pair, one school will be randomly assigned to the control group and the another to the treatment group. It will be made for each pair in the three ordered lists ensuring the balance between the control and treatment group. Therefore, the two groups of schools will have a very balanced score of student achievement.
Randomization Unit
School
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
80 Schools
Sample size: planned number of observations
Approximately, 32,000 students from 80 schools will be observed.
Sample size (or number of clusters) by treatment arms
40 schools control and 40 schools treatment
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
MDES between 0.195 and 0.273 considering school-level intraclass correlation from 0.1 to 0.2 and without covariates.
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number