Experimental Design
SETTING
The experiment focuses on teachers in 39 schools coming from 15 different regions of Romania. The teachers used in this experiment are those that teach the 9 academically relevant subjects for which students have to undertake exams: mathematics, native language, English language, physics, biology, chemistry, computer science, history and geography. All the schools in this experiment have voluntarily purchased the rights to use an on-line education platform which allows teachers to interact more efficiently with students and parents. While the schools using this environment might not be fully representative of the average school (they have better performing students on average), there will be no selection into treatment since the schools are not aware of being part of a field experiment.
The usage of the platform comes at a small monthly cost which is covered by the parents of students. The main features on the platform are the ability to post grades and attendance on-line, and communicate in real-time through messages sent between all parties involved.
Access to this data allows me to monitor the performance of all students and teachers in the school for an entire academic year. In particular, I can observe the (i) grades and (ii) attendance of all the students in the school, which will be used as the main performance measures for teachers.
Schools will be randomly assigned to either treatment and control, and they will be stratified on (1) the baseline performance of the students at the school level, (2) the average teacher value added at the school level and (3) the size of the school. The baseline performance of the school is given by an average of the first grade that all the students in the school receive in the beginning of the year. The size of the school is given by the number of teachers employed at the unit.
Schools which are assigned to the treatment group will receive "public recognition". More precisely, a sub-sample of the "best performing teachers" will be publicly praised through a message posted on-line through the education platform. The first round of recognition will be unannounced, and it will come as a surprise. The first round of intervention will announce that public praise will be given in the future, throughout the rest of the year, to ensure that all teachers have the same expectations. Subsequent rounds of recognition will be given at regular time-intervals until the end of the academic year.
QUALIFYING FOR RECOGNITION
Deciding which teachers are top-performers can be tricky since schools differ in quality and there is no official standard requirement from teachers. Since the recognition is publicly given, it is important that it is perceived as deserved by all observers. In other words, one wants to avoid situations where too few teachers qualify for recognition, or too many.
Top performers will be determined on the basis of student grade improvement, referred henceforth as Teacher Value Added (TVA). Namely, for each teacher in the school the change in student performance will be recorded over a period of 3 and a half months. By the end of December, most students will have at least two grades for the 9 subjects of interest.
Those teachers who are in the top 25% highest TVA within their subject will qualify for recognition and will be publicly praised. In this experiment, praise will be publicly given to teachers, such that everyone in the school, including students and their parents, will be able to observe the top performers. Due to this design feature, randomization will be done at the school level. As such, in the sample of 39 schools, a random half will be assigned to treatment and a random half will be assigned to control.
RANDOMIZATION
Since schools differs somewhat in the sample, stratification on some characteristics is important. The three main strata to be considered are (1) student quality, (2) teacher quality and (3) school size.
Firstly, because there is a positive selection of the best students into higher quality schools, the initial grade students receive in the beginning of the year is a good proxi for this selection effect. To account for differences in the underlying quality of the students at each school, the randomization will be stratified by the baseline performance of the school. Baseline performance is an average of all the initial grades of the students across the 9 academic subjects of interest.
Secondly, since schools can hire teachers with different quality levels on average, the distribution of teacher value added across schools is another important factor. The average teacher value added at the school level will be the second characteristic on which the sample will be stratified.
Furthermore, I will also stratify the sample on the size of the school. Post randomization, I will check whether the treatment is balanced across regions, degree of urbanization and on whether the school is public or private.
INTERVENTION
In the schools which qualify for treatment, all teachers, students and parents will be able to see on the school's electronic messaging board a message praising the top performers in the school. For the exact phrasing on the message see the attached document titled "Experiment design and analysis plan"