Sample size: planned number of observations
RCT within the tech firm:
We start the RCT at the end of October 2022. The company confirmed that we can run the RCT until the end of October 2023; we aim to extent the RCT at least until the end of April 2024.
We are not able to predict the exact number of observations in the RCT. Because of that, we summarize here the process how the data will be collected and our expected number of observations.
Each year, the tech company is posting on average between 80 and 120 of job ads for engineering, business administration, IT and science (e.g., physics, biology) positions in the plant that cooperates with us. All of these job ads will be part of the RCT.
For each job ad, the company receives approximately between 2 and 60 applications. In our RCT, one observation is one application. Standard errors will be clustered at the job ad level.
We receive detailed data from the company about each application (e.g. gender, employment history, educational background) and data from the hiring process (e.g. classification of the quality of the applicant from the firm’s recruiters, which applicant is invited to an interview).
We seek to recruit survey participants whose educational background matches with the requirements listed in the job ad. In the laboratory pools, we select participants based on their ﬁeld of study. All subjects whose ﬁeld of study ﬁts to the advertised position are invited to participate in the survey. Yet, we do not know how many suitable candidates are available in the laboratory pools, as we do not know which jobs will be advertised by the company during next year (and thus not the requirements).
For 15-30 job ads (if possible) we will run the survey in the lab (the exact number depends on the total number of vacancies and the size of the subject pools / response rates in the labs). As there is uncertainty with regard to the required educational background needed for the positions and the positions are partly highly specialized, we do not know yet, whether we will be able to ﬁnd sufficiently many suitable potential survey participants for the lab. Our aim is to collect for each job ad in total at least 45 observations (at least 15 survey participants for each treatment group), but we aim at 90 observations for each job ad (30 survey participants for each treatment group). Note that it is possible that this number will vary because of varying response rates in the different labs and because of the uncertainty with respect to the ﬁt of the subjects’ background (job experience, education) to the job ads.
We made some adjustments compared to our previous version of the preregistration (see the former version from November 18, 2022). Our initial plan was to run surveys for 20 job ads in cooperation with the market research agency Bilendi. However, as we started to pilot our surveys with them, we experienced a very low response quality. Simple quality checks (like reading the job ad for at least 20 seconds) led to a dropout of roughly 75% of the sample. Because of the low answer quality as well as potential selection bias resulting from it, we decided to stop working with Bilendi before analyzing the data; we conduct our whole survey via various econ labs across German speaking countries. Moreover, for statistical reasons, we decided to reduce the targeted number of job ads used in the survey for the sake of an increased number of participants in each survey wave. As the subject pool in the labs are mainly students with no or limited job experience, we are also planning to analyze in our RCT heterogeneous effects with respect to job experience.