There is a long-standing interest in the determinants of education and occupation choice. At the same time, there is the question how targeted interventions can help improve decision-making in this context. We provide experimental evidence for the role of professional career counselors and online self-assessment tools by evaluating a German-wide policy for high school students.
The policy is part of an initiative by the German federal employment agency implemented in 2019 to improve individuals’ career choice throughout their life. At the high school level, the initiative focuses on the decision-making with respect to post-secondary education and occupation. In addition, there is the interest to equalize educational opportunities for students and improve knowledge about the whole spectrum of career opportunities. To achieve these goals, the federal employment agency is revising and expanding their services offered to high school students.
For our analysis, we focus on the two central parts of the policy and consider them in two separate treatments: direct interaction of professionally trained career counselors with students (treatment 1), and a self-assessment online tool that analyzes the fit of different fields of study and occupations (treatment 2).
Specifically, treatment 1 considers the expansion of face-to-face contact with students. Prior to the implementation of the policy, counselors already offered one-on-one talks with students, although primarily at the vicinities of the employment agency, and gave presentations informing about post-secondary education at schools. With the policy, first, there is a focus on offering one-on-one meetings at the school premises to allow easy access and closer interactions and second, counselors offer more group activities such as presentations to high schools. For treatment 1, students in treated schools get access to the intensified counselling, whereas those at control schools have access only to the pre-policy program. We therefore expect the treatment to lead to differences in the contact between students and counselors as a first stage.
In contrast, treatment 2 evaluates the effects of a self-assessment online tool developed by the employment agency. In this tool, students fill out skill tests and express their agreement to a broad set of statements. Based on the answers, the tool provides a ranking and additional information for suitable fields of study and vocational trainings. The employment agency considers the tool an effective way to help high school students in their decision-making and intends to encourage more students to complete the according tests. In the treatment, we provide information about the tests and incentives to complete them. As a result, we expect a difference in the probability of completion by treatment status and analyze the effects.
To implement our treatment 1, we cooperate with employment agencies (Agenturen für Arbeit) in 42 districts in eight federal states (Bundesländer). We contacted all high schools (n=940) in these employment agency districts and invited them to participate in a large-scale experiment on the effects of career counselors on education and occupation choice. We randomly allocated the participating schools in a treatment and control group using pairwise randomization stratified by local employment agency districts and date of declaring participation. To facilitate the implementation of the treatment for the employment agencies we randomized at three points in time. After each of the three randomizations, we communicated the results to the employment agency in the districts where the schools are located to allow them to prepare the treatment. Participating schools did not receive any information about their treatment status and treatment started only after the baseline data collection took place.
For the implementation of treatment 2 we build on the recruitment process of schools for treatment 1. We randomize students in both treatment and control group of treatment 1 into a treatment and control group for treatment 2. After this second randomization, there are four groups (no treatment, only treatment 1, only treatment 2, treatment 1 and treatment 2).
217 schools (Treatment 1: 109 TG and 108 CG) agreed to be part of the baseline sample and to take part in our trial. Within these schools, we focus on the high school students of the graduating classes of 2020 and 2021 to assess the effects of our treatments. We invited the students to fill out surveys in context of our study without mentioning the exact treatments of the experiment. The participation was voluntary and in seven of the eight federal states, the consent of the respective parents was required.
The baseline survey (Q_0) took place as a paper assisted personal interview (PAPI) during class time. In the participating schools, a professional data collection team (Infas/IEA) conducted Q_0 between September and November 2019. Overall, we expect around 9,000 students to participate in Q_0.
We collect data through four additional questionnaires. These questionnaires will take place outside the school context as computer assisted web or telephone interview (CAWI/CATI). We plan to interview the participating students in February to March 2020 (Q_1), October to December 2020 (Q_2), February to March 2021 (Q_3) and October to December 2021 (Q_4).
The questionnaires include information about occupational expectations, believes, personality measures, school performance, take-up of our treatments and further information to capture the primary and secondary outcomes described below.
We also collect data through surveys from the parents and school principals of participating students. For the parents, we plan two questionnaires in CAWI/CATI format of which the first will take place directly after Q_0. For the school principals, we collected data through a paper-based questionnaire at the same time of Q_0.
Furthermore, we enrich our data collection with administrative data of students and parents who gave consent to data linkage. The administrative data include information about students’ meetings with the career counselors of the employment agencies and labor market outcomes such as employment, wages, and occupation.
In addition to the data sources described above, we plan to evaluate treatment 2 by collecting data about the utilization of the online self-assessment tool.
To assess the success of the policy we select outcome variables that are informative about the educational and occupational decisions of high school students.
As primary outcome, we consider the choice of post-secondary education after high school completion. In the case of Germany, most high school graduates either start a college degree in a particular field of study or choose vocational training for a specific occupation. Depending on their decision, students then select either their field of study or occupation, respectively. They can also decide to defer further formal education and instead start temporary internships, voluntary work, etc. We therefore analyze the impact of the treatment along this dimension as well.
These outcome variables provide a short-term assessment of the policy and an analysis close to implementation of the policy (Q_0–Q_2). For medium-run effects, we continue surveying students with a focus on the persistence in education and occupation choice (Q_3-Q_4). On the one hand, students may drop out of university or vocational training all together or they may switch between fields of study or occupation.
The average effect allows for a general assessment of the policy, however, it may mask important heterogeneity. We consider socioeconomic status and gender as two important subgroup categories that can speak to the policy goal of educational equality. We further provide an analysis by students’ post-secondary education intention at baseline to examine the malleability of intentions and the associated direction of adjustments.
To further pin down the importance of different channels, we use secondary outcome variables. The survey questionnaire permits the investigation of various explanations including knowledge about post-secondary education, beliefs about future outcomes, and secondary education outcomes such as grades and school satisfaction.
The information given in the registration reflect the current state of the trial. At this point, we collected the baseline data and implemented treatment 1 in coordination with the employment agencies. The external data collection company currently processes the data and subsequently will provide the results of the baseline survey.