Experimental Design
For the recruitment experiment:
We implement four treatments to study the effect of the suggested research topic on participation. Headmasters’ either receive an invitation to participate in a survey (control treatment) or receive an invitation to participate in one of three collaborative projects. For the collaborative projects, we vary the suggested research topic and whether schools can receive a financial reward for participation. All collaborative projects are presented in the same way and with equal length. The treatment variation consists of the first and last paragraph of the e-mail, announcing our plan to conduct an experiment about the respective research topic within schools. The fourth paragraph of the e-mail informs about monetary incentives, if applicable, i.e., two schools can win a 700 Euro budget in the case of participating.
Control Treatment: In the Control Treatment, we ask schools to participate in an online survey. There, we ask about the headmasters’ point of view regarding the collaboration between academia and schools, i.e., how insights gained in academic research can be integrated into the everyday life of schools. Importantly, answering the survey does not involve participation in any experimental study and it requires a minimum of the headmasters’ time – approximately five minutes. Due to the low stakes of the survey and the time frame, we interpret the responsiveness in the survey as the schools’ baseline responsiveness in dealing with inquiries of academic researchers.
E-Learning Treatment: In the E-Learning Treatment, we suggest participating in a study on the use of electronic devices in education. The presented research question is to find out which types of electronic testing formats can be implemented in schools and how they perform compared to traditional pen and paper exams.
Parental-Involvement Treatment: In the Parental-Involvement Treatment, we ask for participation in a study aiming at analyzing the effect of getting parents involved in their children’s education. This treatment is motivated by recent academic research using
electronic devices (e.g., text-messaging) to reduce information frictions between parents and children by informing, for example, about the students’ behavior in class and their academic performance (see, e.g., Bergman and Chan, 2019; Kraft and Rogers, 2015). These studies
show that active participation of parents in their children’s education can lead to favorable educational and behavioral outcomes.
Integration-Migrant-Children Treatment: In the Integration-Migrant-Children Treatment, we ask schools to participate in a study to analyze how students with a migration background and language difficulties can best be integrated into classroom education. This topic was inspired by the increasing migration to Germany in 2015/16, which was covered widely in the media. It still constitutes a major challenge for schools to rapidly integrate non-German-speaking children into the school environment.
For the experiment on overlap and precision:
Division of the Sample in Treatment and Control Group
The population of schools considered consists of 3,305 schools. From this pool, we randomly draw 12 sub-samples. To investigate how strongly balance decreases with an increasing number of treatment arms, we also draw sub-samples consisting of different and increasing numbers of schools, so that we can assign between one and six treatment groups with equal numbers of schools. For the 12 sub-samples we draw disjunct groups of equal sizes that are comparable to the ones randomly drawn. We assess balance in covariates with the omnibus test of equivalence between groups introduced by Hansen and Bowers (2008).
In this way, we will obtain 24 sub-samples consisting of 12 pairs of pair-wise comparable sub-samples. Of each pair, we randomly allocate one sub-sample to the minMSE approach (i.e., the treatment group ‘balance’), and the other sub-sample to a comparison method (i.e., the control group). For 10 pairs, pure randomization is used as comparison method, and for one pair each, re-randomization based on t-statistics and pair-wise matching will be the comparison methods, respectively.
Treatment Assignment for Remaining Schools
After having allocated the schools in 12x2 sub-samples (matching/minMSE sub-samples, re-randomization/minMSE sub-samples,
and ten randomization/minMSE sub-samples) to experimental groups, around one third of the sample will still not be assigned an experimental group (this is intentionally; otherwise, chances to obtain comparable sub-groups in the 12x2 draws decreases). Taking into account the treatment assignments already made by then, using the minMSE method, we allocate those remaining schools to the control and the treatment groups, with the restriction of having the group sizes as equal as possible and the goal of achieving overall balance across treatments in the whole sample. The resulting assignment to experimental groups will be checked with respect to balance with the omnibus test by Hansen and Bowers (2008).