Public Recognition and Long Run Employee Performance

The intervention is the provision of public praise to the best performing teachers in a random sample of schools. In particular, teachers will qualify for recognition if they are among the top 25% performers in terms of teacher value-added, since the beginning of the school year. Recognition is given publicly through the means of an on-line message which is viewed by all school staff, students and parents.
The first round of public recognition is unannounced and comes as a surprise. Subsequent rounds of recognition are announced and the intervention is repeated two more times until the end of the school year.

2018-01-22

2018-04-30

There are three outcomes:
1. The change in teacher value added for all teachers before and after each of the 3 rounds of recognition.
2. The change in student attendance for all teachers before and after each of the 3 rounds of recognition.
3. Finals student grades on a high-stake exam in the end of the school year.

Teacher value added is given by the gap between the first grade and a weighted average of all the subsequent grades of a student, where the final grade is given a weight of 50% and for all other intermediate grades, the remaining weight is equally distributed. The first grade is considered to be the baseline performance of a student, in the beginning of an academic year. Although teachers can have the same classes for a number of years in a row, the material each year is significantly different and thus reflects a new baseline start for each student. The final grade before the Christmas break is given a higher weight for two reasons: (i) it is the least noisy measurement of teacher value added since it allows for the longest period of time to pass since the baseline performance grade is recorded and (ii) final tests before the Christmas break tend to be more general and include a larger share of the material studied up to that point, avoiding situations where students can perform poorly on one particular chapter, despite having a higher performance overall.

For each teacher, the value added will be a weighted average of all the individual student level added values since the beginning of the academic year. Those teachers who are in the top 25% highest TVA within their subject will qualify for recognition and will be publicly praised.

SETTING

The experiment focuses on teachers in 39 schools coming from 15 different regions of Romania. The teachers used in this experiment are those that teach the 9 academically relevant subjects for which students have to undertake exams: mathematics, native language, English language, physics, biology, chemistry, computer science, history and geography. All the schools in this experiment have voluntarily purchased the rights to use an on-line education platform which allows teachers to interact more efficiently with students and parents. While the schools using this environment might not be fully representative of the average school (they have better performing students on average), there will be no selection into treatment since the schools are not aware of being part of a field experiment.

The usage of the platform comes at a small monthly cost which is covered by the parents of students. The main features on the platform are the ability to post grades and attendance on-line, and communicate in real-time through messages sent between all parties involved.

Access to this data allows me to monitor the performance of all students and teachers in the school for an entire academic year. In particular, I can observe the (i) grades and (ii) attendance of all the students in the school, which will be used as the main performance measures for teachers.

Schools will be randomly assigned to either treatment and control, and they will be stratified on (1) the baseline performance of the students at the school level, (2) the average teacher value added at the school level and (3) the size of the school. The baseline performance of the school is given by an average of the first grade that all the students in the school receive in the beginning of the year. The size of the school is given by the number of teachers employed at the unit.

Schools which are assigned to the treatment group will receive "public recognition". More precisely, a sub-sample of the "best performing teachers" will be publicly praised through a message posted on-line through the education platform. The first round of recognition will be unannounced, and it will come as a surprise. The first round of intervention will announce that public praise will be given in the future, throughout the rest of the year, to ensure that all teachers have the same expectations. Subsequent rounds of recognition will be given at regular time-intervals until the end of the academic year.

QUALIFYING FOR RECOGNITION

Deciding which teachers are top-performers can be tricky since schools differ in quality and there is no official standard requirement from teachers. Since the recognition is publicly given, it is important that it is perceived as deserved by all observers. In other words, one wants to avoid situations where too few teachers qualify for recognition, or too many.

Top performers will be determined on the basis of student grade improvement, referred henceforth as Teacher Value Added (TVA). Namely, for each teacher in the school the change in student performance will be recorded over a period of 3 and a half months. By the end of December, most students will have at least two grades for the 9 subjects of interest.

Those teachers who are in the top 25% highest TVA within their subject will qualify for recognition and will be publicly praised. In this experiment, praise will be publicly given to teachers, such that everyone in the school, including students and their parents, will be able to observe the top performers. Due to this design feature, randomization will be done at the school level. As such, in the sample of 39 schools, a random half will be assigned to treatment and a random half will be assigned to control.

RANDOMIZATION

Since schools differs somewhat in the sample, stratification on some characteristics is important. The three main strata to be considered are (1) student quality, (2) teacher quality and (3) school size.

Firstly, because there is a positive selection of the best students into higher quality schools, the initial grade students receive in the beginning of the year is a good proxi for this selection effect. To account for differences in the underlying quality of the students at each school, the randomization will be stratified by the baseline performance of the school. Baseline performance is an average of all the initial grades of the students across the 9 academic subjects of interest.

Secondly, since schools can hire teachers with different quality levels on average, the distribution of teacher value added across schools is another important factor. The average teacher value added at the school level will be the second characteristic on which the sample will be stratified.

Furthermore, I will also stratify the sample on the size of the school. Post randomization, I will check whether the treatment is balanced across regions, degree of urbanization and on whether the school is public or private.

INTERVENTION

In the schools which qualify for treatment, all teachers, students and parents will be able to see on the school's electronic messaging board a message praising the top performers in the school. For the exact phrasing on the message see the attached document titled "Experiment design and analysis plan"

Randomization done using STATA

Randomization is done at the school levels, stratified according to:
-baseline performance of the students
-baseline performance of teachers
-school size

Yes

39 schools and/or 850 teachers.

850 teachers and 20.000 students.

There will be 21 schools in the treatment group and 18 schools in the control group.

The minimum detectable effect size for our main outcomes are in the attached document "Experiment design and analysis plan".