Experimental Design Details
Recruitment of schools
We will contact every Hungarian primary school with an invitation letter sent to the schools’ central e-mail address that is publicly available. The invitation letter will be addressed to the school principal. Principals will be asked to forward the e-mail to the Hungarian and mathematics teachers in the school.
The first 200 Hungarian and 200 mathematics teachers who are willing to participate can take part in the online experiment. The data collection, however, will be closed at 30 September 2021 even if a lower number of teachers participate until then. The number of teachers who are allowed to participate in the experiment will be limited to five mathematics and five Hungarian teachers per school. Teachers will receive 10.000 HUF (equivalent to 34.4 USD) for participation. Thus, our target population consists of teachers who teach Hungarian grammar and literature or mathematics in the 5th-8th grades of Hungarian primary schools.
The experimental procedure
We have constructed mathematics and Hungarian tests from exercises used in prior admission tests to secondary education. The mathematics test consists of six exercises. The Hungarian grammar and literature test consists of five exercises, including a 10-12 sentence-long essay on a gender-neutral topic (advantages and disadvantages of written tests and oral presentations in school).
We have constructed six-six different solutions of the mathematics and Hungarian test, respectively. We pre-tested these solutions among teachers not included in the final sample. The pre-test verified sufficient variation in teachers’ evaluations, which shows that teachers assessed the same test version differently, which leaves space for teachers’ subjective evaluations.
We created four name sets. Two name sets contained 2 Roma (1 male, 1 female) and 4 non-Roma (2 male, 2 female) names. The other two name sets contained 4 Roma (2 male, 2 female) and 2 non-Roma (1 male, 1 female) names.
Roma family names were selected based on a study that investigated which family names are most often perceived as Roma by the Hungarian population (Váradi, 2012). Roma first names were selected based on the frequency of first names of Roma students participating in two recent data collections among primary school students in ethnically mixed primary schools in Hungary. Non-Roma family names were selected from the most frequent family names in Hungary, but less frequent names that are typically perceived as non-Roma family names (e.g., ending with a “y”) were also used. Non-Roma first names were selected from the ten most frequent female and ten most frequent male names given in 2006.
We created test packages for mathematics and Hungarian tests separately. From the six test versions and four name sets, eight different packages were created because eight is the lowest number of combinations to ensure that every test is presented with an assigned Roma and non-Roma male and female name by an equal frequency. Each package contained all the six tests and one of the name sets. Within one package, one test is always assigned to one particular name. In the eight packages, every test is assigned two times to a Roma male name, two times to a non-Roma male name, two times to a Roma female name, and two times to a non-Roma female name.
We will assign the packages randomly to the teachers. Within the packages, teachers will see the tests in a random order.
Teachers’ tasks in the experiment
Teachers have four tasks in the experiment:
1. They evaluate mathematics or Hungarian grammar and literature tests on a 30-point scale first. Students’ names on the tests will be fictive and randomly assigned. Teachers are not provided with a solution because we want to avoid influencing teachers’ grading practices.
2. Teachers recommend a grade based on the evaluation of the test on the 5-grade scale used in the Hungarian educational system (1 = fail, 2 = pass, 3 = satisfactory, 4 = good, 5 = excellent).
3. They recommend a secondary school track to the students whose tests they evaluated. (1 – vocational school, 2 – secondary vocational school, 3 – secondary grammar school). Besides the points and grades assigned by the teacher in the given subject, we will provide an additional information: fictive points the student received for their test in the other subject.
4. Teachers are asked to fill out a questionnaire focusing on background information, seating arrangements in the class, and tracking in school.
Experimental manipulation
Our experimental manipulation consists of a random manipulation of students’ names on the tests. Since this is a light-touch manipulation, our design makes sure to communicate students’ name many times to teachers:
1. Students’ name stands on the sheet before the first exercise with students’ handwriting (e.g., “Kolompár Ramóna”).
2. Students’ name appears before and after each exercise (e.g., “You now see the first exercise of Kolompár Ramóna’s test.” / “Ön most Kolompár Ramóna dolgozatának első feladatát látja.” “How many points do you give to the first exercise of Kolompár Ramóna’s test from a maximum of 5 points?” / “Hány pontra értékeli Kolompár Ramóna dolgozatának 1. feladatát a maximális 5 pontból?”).
3. After teachers evaluated all exercises, all students’ names appear again in one table. Teachers have to indicate the grade corresponding to each test in this table.
4. After teachers graded the tests, all names appear again in a new table, in which teachers have to recommend a secondary school track.