|
Field
Experimental Design (Public)
|
Before
What is the causal effect of management on pupils’ achievement? How can independent public audit institutions such as the accountability courts act directly to generate better public policies? The treatment will be a set of specific management practices developed based on a growing literature about the strong association between these practices and productivity in schools, manufacturing, and hospitals which can be measured, taught in business schools, and recommended by consultants. These specific management practices will be implemented, measured, and tested through a randomized field experiment on a sample of state-run schools from Rio de Janeiro.
Evaluation design: The causal connection between management and student learning can be answered using a quantitative framework. The answer to the question is intended to be conclusive. Its focus is on answering “What,” “Where,” and “When” (Bray, 2008, p.299). To answer these queries, the research will be based on a field experiment with a two-level design - schools and students - with one level of nesting - students to schools - where the schools, level 2-unit, are randomly assigned to treatment and control groups. The random assignment to the groups solves the selection problem making it possible to attribute the differences between groups to the treatment applied. The study can be considered a school-level clustered randomized field experiment or, in other words, a two-level clustered randomized field experiment.
A sample of eighty schools will be chosen randomly from these 1,001 (one thousand and one) units responsible for pupils from 1st to 9th year. After that, the control and treatment groups will be formed randomly by 40 schools each. It is vital to analyze the balance between control and treatment groups considering a variable related to the interest results, that is, students’ achievement, but controlled by socioeconomic features. The idea is to use the Brazilian Basic Education Assessment System (Saeb) of 2019, the last national assessment available, to represent the students’ learning outcomes; the Brazilian Socioeconomic Level (NSE) of 2019, the last national assessment index of this type available. Saeb is a standardized test applied along with a questionnaire that is the basis of the NSE every two years. They are applied by the National Institute of Educational Studies and Research Anísio Teixeira (Inep), an autonomous federal government agency. NSE is an index formed by combining two elements: parents’ education and possession of goods and services. The agency discloses the data by school, what is needed to prevent bias as it is presented in the next paragraph. The goal is to ensure the balance between control and treatment groups regarding each of the three types of schools.
This study will use a matched-pair randomization procedure. Firstly, a linear regression of the Saeb of 2019 on the NSE of the 1,001 population of schools in Rio de Janeiro will generate the constant α and coefficient β1. For each school in the sample of eighty schools, using α and β1 from the first regression, it will be generated the expected Saeb by the equation SAEB_Expected = α+β1∗NSE2019. Therefore, Saeb2019−SaebExpected will show for each school a number that represents the distance from the tendency line given by Saeb_Expected controlled by NSE. Thus, the schools will be listed in descending order based on the number reached from the previous subtraction. Every two schools will be matched, forming 40 pairs: school in the 1st place with the school on the 2nd, 3rd with 4th, etc. One school will be randomly assigned to the control group and another to the treatment group for each pair. This will be performed for each pair, ensuring the balance between control and treatment groups. Therefore, the two groups of schools will have a balanced score of students’ achievements controlled for the socioeconomic level and randomization.
Twenty-three management practices are the basis of the international management index created by Bloom et al. (2015). Each of the twenty-three practices is assessed considering a scale from 1 to 5. For instance, consider practice 1. (a), Bloom et al. (2015) define that a school receives score 1 if “No clear or institutionalized instructional planning processes or protocols exist; little verification or follow-up is done to ensure consistency across classrooms,” a score 3 if “ school has defined instructional planning processes or protocols to support instructional strategies and materials and incorporate some flexibility to meet students’ needs; monitoring is only adequate,” and a score 5 if “ school has implemented a clearly defined instructional planning process designed to align instructional strategies and materials with learning expectations and incorporate flexibility to meet student needs; these are followed up on through comprehensive monitoring or oversight.” Grade 1 reflects the lack of the respective best high-level management practice; grade 3 represents that a part of the respective best high-level management practice exists; and grade 5 will be attributed to the school that has the respective best high-level management practice totally implemented. The grade average of the twenty-three best management practices will represent the management index of each school. Therefore, this study will focus on implementing the operational practices needed to increase the management index of each school in the treatment group of the experiment.
The schools in the control group will be isolated from the project to avoid the “contamination effect”; no one knows which schools are in the control group, only the interinstitutional team. The contamination effect can be avoided leaving the participants of an experiment not knowing which group they are in, treatment or control. It is relevant to highlight that only the inter-institutional team will have information about the schools assigned to the treatment and the control groups. This fact is included in the signed collaboration term. The team will support the school managers in the schools of the treatment group to increase the management level of each school.
On the one hand, the study will use the international management index as presented previously based on the best management practices to show the management level of each school. The international management index will be measured through double-checked face-to-face interviews with the school principals before and after the experiment. The interviews will be recorded to enable a different evaluator from the interinstitutional team to conduct a new assessment of the answers given by the school manager. Finally, the management index of each school will be the average of its scores in each of the twenty-three management measures. On the other hand, the average of mathematics, science, and reading tests will represent the measure of pupil learning outcomes. The educational system of Rio de Janeiro has a bi-monthly general assessment of the students’ learning already implemented and running.
There will be three rounds of matched assessment, management vs. pupil outcomes. The first round of evaluation will diagnose the situation of schools on the treatment and control groups, considering the international management index described previously and the average score of the students on the tests. The second round of matched assessment will happen just after the intervention period to allow measurement of the average impact of the operational practices implemented in the treatment group on pupil outcomes. After the intervention period, it will start the monitoring period where schools will be monitored regarding the maintenance or not of the operational management practices developed and implemented. The third round will occur just after the period of monitoring.
This procedure will make it possible to measure the average impact of the intervention just after the implementation and one year after the end of the implementation on the students’ achievement. It is relevant to highlight that this research seeks a real-world application. This means that if the operational management practices developed and implemented really impact the students’ achievement in a way that persuades and influences politicians and policymakers, it will be possible to scale the project up by implementing the best management practices for the entire system of schools in the city of Rio de Janeiro.
|
After
What is the causal effect of management on pupils’ achievement? How can independent public audit institutions such as the accountability courts act directly to generate better public policies? The treatment will be a set of specific management practices developed based on a growing literature about the strong association between these practices and productivity in schools, manufacturing, and hospitals. These practices can be measured, taught in business schools, and recommended by consultants. These specific management practices will be implemented, measured, and tested through a randomized field experiment on a sample of state-run schools from Rio de Janeiro.
Evaluation design: A quantitative framework can answer the causal connection between management and student learning. The answer to the question is intended to be conclusive. Its focus is on answering “What,” “Where,” and “When” (Bray, 2008, p.299). To answer these queries, the research will be based on a field experiment with a two-level design - schools and students - with one level of nesting - students to schools - where the schools, level 2-unit, are randomly assigned to treatment and control groups. The random assignment to the groups solves the selection problem making it possible to attribute the differences between groups to the treatment applied. The study can be considered a school-level clustered randomized field experiment or, in other words, a two-level clustered randomized field experiment.
A sample of eighty schools will be chosen randomly from these 1,001 (one thousand and one) units responsible for pupils from 1st to 9th year. After that, the control and treatment groups will be formed randomly by 40 schools each. It is vital to analyze the balance between control and treatment groups considering a variable related to the interest results: students’ achievement. The idea is to use the Brazilian Basic Education Assessment System (Saeb) of 2019, the last national assessment available, to represent the students’ learning outcomes. Saeb is a standardized test applied every two years. It is applied by the National Institute of Educational Studies and Research Anísio Teixeira (Inep), an autonomous federal government agency. The agency discloses the data by school, what is needed to prevent bias as presented in the next paragraph. The goal is to ensure the balance between control and treatment groups regarding each of the three types of schools.
This study will use a matched-pair randomization procedure. Firstly, the schools will be divided into two groups, schools mainly focused on 1st to 5th year (1st segment), and schools mainly focused on 6th to 9th year (2nd segment). It is important to separate the schools into two groups considering their years' focus to ensure that both groups will end up, after the randomisation, with the same number of the two different types of school. Thus, the schools will be ordered descending in each type of school group, forming two lists. Every two schools in each list will be matched, forming pairs: school in the 1st place with the school on the 2nd, 3rd with 4th, etc., for each list. Within each pair, one school will be randomly assigned to the control group and another to the treatment group. Therefore, the two groups of schools should have a balanced score of students’ achievements (variable of interest). Pre-tests will be conducted to analyse balance in observable variables.
Twenty-three management practices are the basis of the international management index created by Bloom et al. (2015). Each of the twenty-three practices is assessed considering a scale from 1 to 5. For instance, consider practice 1. (a), Bloom et al. (2015) define that a school receives a score of 1 if “No clear or institutionalized instructional planning processes or protocols exist; little verification or follow-up is done to ensure consistency across classrooms,” a score 3 if “ school has defined instructional planning processes or protocols to support instructional strategies and materials and incorporate some flexibility to meet students’ needs; monitoring is only adequate,” and a score 5 if “ school has implemented a clearly defined instructional planning process designed to align instructional strategies and materials with learning expectations and incorporate flexibility to meet student needs; these are followed up on through comprehensive monitoring or oversight.” Grade 1 reflects the lack of the respective best high-level management practice; grade 3 represents that a part of the respective best high-level management practice exists; and grade 5 will be attributed to the school with the respective best high-level management practice totally implemented. The grade average of the twenty-three best management practices will represent the management index of each school. Therefore, this study will focus on implementing the operational practices needed to increase the management index of each school in the treatment group of the experiment.
The schools in the control group will be isolated from the project to avoid the “contamination effect”; no one knows which schools are in the control group, only the interinstitutional team. The contamination effect can be avoided by leaving the experiment participants not knowing which group they are in, treatment or control. It is relevant to highlight that only the inter-institutional team will have information about the schools assigned to the treatment and the control groups. This fact is included in the signed collaboration term. The team will support the school managers in the schools of the treatment group to increase the management level of each school.
On the one hand, the study will use the international management index presented previously based on the best management practices to show the management level of each school. The international management index will be measured through double-checked face-to-face interviews with the school principals before and after the experiment. The interviews will be recorded to enable a different evaluator from the interinstitutional team to conduct a new assessment of the answers given by the school manager. Finally, the management index of each school will be the average of its scores in each of the twenty-three management measures. On the other hand, the average of mathematics, science, and reading tests will represent the measure of pupil learning outcomes. The educational system of Rio de Janeiro has a bi-monthly general assessment of the students’ learning already implemented and running.
There will be three rounds of matched assessment, management vs pupil outcomes. The first round of evaluation will diagnose the situation of schools on the treatment and control groups, considering the international management index described previously and the average score of the students on the tests. The second round of matched assessment will happen just after the intervention period to allow measurement of the average impact of the operational practices implemented in the treatment group on pupil outcomes. After the intervention period, it will start the monitoring period where schools will be monitored regarding the maintenance or not of the operational management practices developed and implemented. The third round will occur just after the period of monitoring.
This procedure will make it possible to measure the average impact of the intervention just after the implementation and one year after the end of the implementation on the students’ achievement. It is relevant to highlight that this research seeks a real-world application. This means that if the operational management practices developed and implemented really impact the students’ achievement in a way that persuades and influences politicians and policymakers, it would be possible to scale the project up by implementing the best management practices for the entire system of schools in the city of Rio de Janeiro.
|