Evaluation of a vocational school initiative to increase graduation

The intervention is called OS I BRANCHEN (Us in the Industry) and represents a new approach to the 20 week’s basic course at Danish vocational schools, called Grundforløb 2. Henceforth referred to with the common abbreviation GF2.

Background
The GF2 takes place at the vocational school as preparation for the main course. The main course consists of several apprenticeship and school modules. The intention of GF2 is to teach the students a range of vocational skills prior to their first apprenticeship.

The ROCKWOOL Foundation Interventions Unit has interviewed many GF2 students during the development of the intervention. Many students have chosen a vocational education because they want to be active in a workplace and use their skills to create value and make a difference for others. Quite a few have had bad experience with prior schooling and are eager to get out in the industry. Some students have limited experience with the industry and therefore need exposure to the industry to discover the possibilities and the everyday life in a company through practical experiences. Most students experience that the individual responsibility for their education and finding an apprenticeship is a heavy burden. The intervention is developed as a response to these student experiences. The intervention does not grant student apprenticeships, but elements of the intervention is meant to provide students with tools and experiences, which they can utilize in the process of getting an apprenticeship agreement with a company. Mostly, students carry the responsibility for getting an apprenticeship agreement with a workplace. However, a recent tri-party agreement on the vocational school area has shifted some of the responsibility of getting apprenticeship agreement away from the students towards the schools. The schools must then assist the students not able to get an apprenticeship agreement themselves. The tri-party agreement has not been fully implemented yet at the point of writing and therefore the effect of the responsibility shift is unknown, and there is uncertainty about the practical implementation of the tri-party agreement.

The intervention is being implemented and trialed at three vocational schools in Denmark. Zealand Business College (ZBC), NEXT Uddannelse København (NEXT) and Center for Erhvervsrettede Uddannelser Lolland-Falster (CELF). The intervention is being rolled out to the following nine educations or areas of education: Carpentry, Electrician, Bricklaying, Data & Communication, Car Mechanic, Plumbing, Social and Health Care (Social and health care assistants and social and health care helpers), Foods (Baker & confectioner, gastronome, waiter, nutrition assistant), and Business (Retail, trade, event coordinator).

The schools were recruited for the trial based on three parameters: They had a range of the educations which the intervention was designed for, the enrollment into these educations were large enough to utilize the network dynamics of the intervention, both in-between students and between the students and the companies, and lastly that graduation rates on the GF2 left room for improvement.

The intervention
The intervention rests on two principles:
- “The first day in school is the first day in the industry”
- “We succeed as a team”
The activities in the intervention are designed to give the students real-life experiences and a professional network. In all activities the students will try out the industry in a professional collaboration in a real world context. The intervention is delivered by existing personnel at the participating schools. The personnel participate in an on-boarding course prior to starting the intervention.

2022-06-05

2024-06-30

Short-term: graduation of the GF2 program at 8 months after enrollment

The intervention is at the GF2 program that lasts 20 weeks, excluding vacations. In a previous analysis of the timing of GF2-graduation (see Groes et al 2021), we have found that measuring graduation 8 months after starting the GF2 program is a time where we capture most of the graduates. At the eight months mark, we therefore measure student graduation from the GF2 program.

The secondary outcomes originate from two data sources: students’ answers to survey questions and data from the administrative record. Our first secondary outcome is the students’ mental well-being, which we measure by survey questions. The second and third set of secondary outcomes are students’ achievements and input factors, measured form the administrative records.

The secondary outcome for students’ mental health is captured in a student survey. Rather than solely relying on the short-term effect to appear through graduation rate, we believe the intervention potentially can impact the mental well-being of students as students should feel more supported and motivated with their studies and experience that they belong and are able to add value to their new profession.
Mental well-being will be measured based upon seven questions from the Warwick-Edinburg Mental Well-being Scale (Clarke et al 2011, Koushede et al 2018, Powell et al 2013). Five of the questions are taking from the Short Warwick-Edinburg Mental Wellbeing Scale while the last two have been chosen because we believe that they measure what the intervention is trying to achieve. We will construct one mental wellbeing score by taking the sum of the seven questions.
The other secondary outcomes are explorative and originates from the administrative records. First, they capture the students’ potential of a better education match, the students’ achievement, measured as the students’ grades, and the students’ probability of signing an apprenticeship. Second, they capture input factors measured as student and teacher attendance, which may change because of the intervention.
The intervention is meant to give students a better chance to graduate from the GF2 program. However, graduation from the GF2 program is not the final outcome for the vocational students’ education because after the GF2 program comes the vocational main program of around 3 years after which the student receives a certification for the labor market. We therefore include a secondary outcome measure 8 months after initial enrollment to see if the students are enrolled in any education or have graduated from the GF2 program. This measure at 8 months after initial enrollment will capture both the students who graduated the GF2 program and it will also capture students who have started a different education.
The intervention is meant to give students a better idea from the beginning of the program about whether the chosen education is the right one. This means that one consequence of the intervention could be that some students will drop out earlier when they realize that the specific education is not a good match. Some of these students who drop out because of a bad match may have improved skills due to the intervention that can help them choose another education than the one they drop out from (such as the ability to network, a better understanding of the industry, and understanding drop-out as learning rather than failure). These students should be better prepared to start a different education and stay with it. By measuring students’ graduation and any other educational enrollment 8 months after initial enrollment, we capture both the restarters and the students who graduated from the main program.
The students receive grades in some of the subjects that are not practical (e.g. math). The practical exam is a pass/fail. If the intervention increases the probability of graduating from the GF2, we could expect the grade distribution to improve. The average grade of the graduates will depend on the composition of the graduates and it is therefore not clear if the average grade, conditional on passing the exam, will increase or decrease. However, as a fraction of all the enrolled students, we should expect the grades to improve. To measure whether the intervention has an impact on the high end of the grade distribution, we will measure the effect of the intervention on a variable that takes value one if the student achieves a high grade and takes value zero if the student passes but achieves a non-high grade, the student fails, or if the student drops out, which capture all the enrolled students.
We hope to include a measure of the probability of obtaining an apprenticeship at a firm and the date at which the agreement with the firm is made. If the intervention is successful at connecting the students more to the industry and that the network groups helps, we expect that the fraction of students finding an apprenticeship with a firm increases just as the fraction of students having an apprenticeship at the school should decrease . We will use the False Discovery Rate (Benjamin and Hochberg 1995, Andersen 2008) to obtain q-values for the hypotheses tests corresponding to the following two outcomes: probability of a high grade and probability of obtaining an apprenticeship.
Finally, we expect the students’ attendance to increase if the students can see an added benefit to coming to class from the intervention. The teachers’ attendance could also be affected if students’ higher motivation leads teachers to be more motivated and less likely to suffer from stress or other sicknesses as a result of the intervention. We will use the False Discovery Rate (Benjamin and Hochberg 1995, Andersen 2008) to obtain q-values for the hypotheses tests corresponding to the following two outcomes: students’ attendance and teachers’ attendance.

Outcomes that will be used to explain the mechanisms at play
Through the student survey, which the student answers at week 3 and 13 of the GF2, we also ask questions covering topics such as: (1) self-efficacy, (1) feeling of “succeeding as a team”, (3) sense of belonging to the profession, (4) perception of the curriculum, (5) feeling of having value, (6) Number of contacts with workplaces. These six concepts are tightly related to the underlying theory of change of the intervention.
(1) Self-efficacy will be measured through an index which includes the answers to 2 questions which the research team has formulated. (2) Feeling of “succeeding as a team” will be measured using an index which includes the answers to 3 questions. (3) Sense of belonging is measured through an index including 2 questions uncovering if the student is uncertain about their belonging in the profession. (4) perception of curriculum is measured through 1 question uncovering if the student finds the acquired skills and experiences in GF2 to have value later on. (5) Feeling of having value is related to if the student feels like they can contribute to the workplace and vocation of their choice. (6) number of contacts with workplaces is measured as one question asking about this. We will use the False Discovery Rate (Benjamin and Hochberg 1995, Andersen 2008) to obtain q-values for the hypotheses tests corresponding to the following six outcomes: self-efficacy index, succeeding as a team index, sense of belonging on the profession index, perception of curriculum, perception of having value, and number of contacts with workplaces.

Identification
The identification of the treatment effect of the intervention on the outcomes will come from the stratified random staggered roll out of the intervention. See below for further details on the randomization.

Estimation
The econometrics literature that exploits the staggered roll out of a policy has expanded very significantly in the last four years (see for instance Borusyak et al 2022, Chaisemartin and D’HaultfŒuille 2018 and 2022, Freyaldenhoven et al 2021, Gardner 2021, Goodman-Bacon 2021, Roth et al 2022, Sun and Abraham 2020, Roth et al 2022). However, most of the literature has focused on non-random staggered rollout, in which identification of the treatment effects relies on the common trends assumption.

Recently, both Athey and Imbens (2022) and Roth and Sant’Anna (2022) have focused on the random staggered rollouts. Athey and Imbens (2022) assumes that the treatment effect is homogeneous with respect to the adoption date (invariance to history). Both papers assume a non-clustered design, in which the adoption date of a unit i is randomized, and the outcome for such unit i is observed during the entire time period. However, in our set up, we observe the outcomes of repeated cross sections (the students of a particular education-location in each semester, who will be different from semester to semester). Some of the ideas proposed by Athey and Imbens (2022) seem relatively straightforward to adapt to the repeated cross-section case. However, in order to use the methods proposed by Roth and Sant’Anna (2022), we would need to average the data at education-location-semester level, but this would lead to loss of power unless the intracluster correlation was one. Because the literature on staggered roll outs is evolving very quickly, we prefer not to select an estimation method at the moment of writing the pre-analysis plan.

Some secondary outcome variables are measured using a student survey. We anticipate the response rate to be different depending on whether the intervention is being implemented or not at the time of the survey. To deal with possible selection bias, we will use a Heckman Selection model in which the randomized reminders to answer the survey will be used for the exclusion restriction.

Not available

The randomization, or the order in which the education-locations can begin implementing the intervention, was done in-person at the ROCKWOOL Foundations premises.
The randomization was stratified. Education locations were grouped into stratas according to the type of education that they provide (i.e. carpentry) as well as logistical constraints. The stratification allows to minimize the chance that all locations of a given education, say all locations which teaches Carpentry, started the intervention at the same time. When the number of education-locations was too low for a given education, we grouped several different educations in the same strata.
Each strata had pre-defined sequences which determined the roll out scheme over a five semester period. The pre-defined sequences took into account logistical constraints such as not starting too many education-locations at the beginning of the trial. The education-locations where then randomly picked and applied to a sequence, which determined when the education-location location was going to implement the intervention.
The randomization was done by randomly picking downward-facing cards, which identified the individual education-locations. The order of the draw determined the roll out scheme for the stepped wedge trial.
The directors of education at the partnering schools participated in the draft to increase local support for the roll-out scheme among the implementation teams. ROCKWOOL personnel also attended the draft. The randomization was recorded and saved for documentation.

The unit of randomization is an education-location, e.g. Carpentry in the city of Slagelse. Within some areas of education, different educations are taught together in GF2: All food educations are taught together, the same are all business educations and also the two social and health care educations. To avoid spillovers, we grouped these educations together that take lessons together during GF2.
One school can have several education-locations for each education. In some cases, the same vocational teachers work at multiple locations, and here we grouped the education-locations together into one cluster to avoid spillovers.

Yes

At the time of writing the pre-analysis plan, we have recruited 3 schools which provide 33 education-locations (clusters). We are planning to recruit an additional school into the study which will provide additional education-locations.

The number students participating in the trial, will depend on future enrolment into vocational schools and specifically the education-locations participating in the trial. Based on intake from historical administrative data, we estimate around 7600 students, which wil increase when we recruit an additional school.

Because of the staggered rollout design, all clusters end up receiving the intervention at some point.

Using block bootstrap on the administrative data, we estimate a power of 0.70 for an effect size of 5 percentage points, and 0.83 for an effect size of 6 percentage points in two-tail tests at 5% significance. We expect the power level to improve as we are planning to recruit another school into the study.

Pre-analysis plan for the evaluation of OS I BRANCHEN