Labour market information and study choice: One-size-fits-all?

In the Netherlands, secondary schools have the obligation to provide career guidance to their pupils. Often, they will do this by contracting a private company that specialized in career guidance. In collaboration with a career counseling platform called Qompas, one of the largest in the Netherlands, we propose a field experiment to answer the following research questions: What format of labour market information has the strongest effect on study choice? Does displaying labour market information using a pictograph (bar chart or icon arrays), a (neutral or colourful) risk ladder or a table have a different effect on study choice depending on the students’ numeracy, literacy, graph literacy and economic skills? Is there heterogeneity by gender, socio-economic status, time and risk preferences and migration background?

Our target population are 3rd year students at the general and academic high school tracks in the Netherlands, who use Qompas’ resources to help them choose their profile in secondary education as part of their school curriculum. We are building on the experiment conducted by de Koning (2022) in cooperation with Qompas; who showed that information has a positive effect on belief’s accuracy in the short-term, and occupational preferences. Nonetheless, there are several differences between that study and ours, as we would like to take into consideration students’ individual characteristics and to analyse their response to different formats of information.

Students take several tests that are already included in the platform regarding their competences, interests and high school profile preferences. Based on pupils’ answers, Qompas presents the occupation sectors that fit best each student, including a list of example occupations. We will ask the pupils to make a shortlist of 10 favourite occupations and rank them in terms of their preferences (ranking A). The average gross hourly wage corresponding with the shortlisted occupations will be displayed for the students in the treatment groups.
Each treatment group would see the corresponding information in a different format: table, (colourful or neutral) risk ladder or bar chart or icon arrays. Students that belong to the control group will not have access to wage information. Right after receiving the information, treatment group students shall rank the shortlisted occupations again (ranking B), which helps test the short run effect of the information intervention on preferences. We would compare the average wage corresponding to the preferred occupations among the pupils that received information in different formats and those that did not, given gender, socioeconomic background, migration background, numeracy, literacy, economic and graph literacy, time and risk preferences. On two later points in time during the experiment, after students have gone through other assignments, we ask all students again to rank a (reduced) list of occupations based on their preferences (rankings C and D) in order to study the mid run effect of the information treatment.

Reference
de Koning, B. K. (2022). Empirical studies on information, beliefs, and choices in education and work [Doctoral Thesis, Maastricht University]. Maastricht.

2023-09-17

2024-06-30

- Ranking of preferred occupations.

- Short term average wage differences of most preferred occupation(s).

- Mid term changes in wage of most preferred occupation(s).

- Heterogeneity analyses. Interaction terms between the treatment dummies and the following personal characteristics will be included: literacy, numeracy, graph literacy, economic skills, gender, migration background, socioeconomic status (proxied by a quality of life indicator at the postcode level), time and risk preferences.

- Ranking of preferred occupations. Pupils have a shortlist of 10 occupations that are a good match to them and are asked to rank them in terms of their preferences (Ranking A). Immediately after, pupils that are in treatment schools receive labour market information about their shortlisted occupations in the corresponding format. Then, they are asked to re-rank them (Ranking B). Moreover, all pupils will be asked to re-rank their top 10 preferred occupations a two later points during the school year (Rankings C and D). We thus want to test whether students in the treatment groups change their rankings after receiving the information and whether there are differences depending on the format of presenting the information.

- Short term average wage differences of most preferred occupation(s): we would like to estimate whether the average hourly wage of the treatment group immediately after receiving information (Ranking B) is statistically different from the average hourly wage of the control group in their first ranking (Ranking A). The average treatment effect (ATE) for each of the treatment groups would thus be estimated by running a linear regression of the average hourly wage of the top (1/2/3) preferred occupation(s) of individual i on five dummy variables that represent being allocated to each of the treatment groups.

- Mid term changes in wage of most preferred occupation(s). We wish to estimate whether the difference between the average wage change in the treatment group (post – pre treatment) and the average wage change in the control group is statistically significant. First, we will compare the average hourly wage of the top (1,2,3) occupations reported in two later moments in time (Rankings C and D) with the first ranking (Ranking A) within the control and treatment groups. The average treatment effect (ATE) for each of the treatment groups would thus be estimated by running a linear regression of the change in average hourly wage of the top (1/2/3) preferred occupation(s) of individual i on five dummy variables that represent being allocated to each of the treatment groups.

- Heterogeneity analyses. Interaction terms between the treatment dummies and the following personal characteristics will be included: literacy, numeracy, graph literacy, economic skills, gender, migration background, socioeconomic status (proxied by a quality of life indicator at the postcode level), time and risk preferences.

- Self reported high school profile choice.

- Actual high school profile choices.

- Enrolment in higher education programs.

- Self reported high school profile choice: Third year high school students in the Netherlands must choose their profile for the remaining of their secondary school (two or three years, depending on their track). There are four options: science and technology; science and health; economics and society; culture and society. Qompas provides guidance for pupils to choose the profile that is more suitable to them, a choice that is reported in the platform. Given the randomization of the schools into control and treatment groups, we would expect that, in absence of the intervention, the self reported distribution of pupils among profiles would be the same between control and treatment groups. Therefore, we aim at testing whether the share of pupils in each profile is significantly different between the groups. First, we would run the test in absolute terms, and then conditional on students actually receiving information corresponding to occupations from more than one profile. If all of the shortlisted occupations of a student are associated with the same profile, we do not expect any change in their profile choice as a consequence of the treatment. However, if the shortlist comprises occupations from multiple profiles, it is possible that the treatment influences the profile choice. Nonetheless, given that the profile choice in the Qompas platform is self-reported and filling it in is not compulsory, there may be potential drawbacks, such as reduced sample size.

- Actual high school profile choices: By matching our experimental data with administrative data for the full population of students, we wish to compare whether the actual distribution of pupils among high school profiles is different between the control and treatment groups.

- Enrolment in higher education programs: We are planning to study whether pupils who received information in different formats make different higher education choices regarding labour market outcomes than students who did not get access to information. We will thus merge our experimental data with administrative data in order to have access to enrolment in higher education. We wish to compare whether students in the treatment groups pursue higher education programs with higher expected wages than students in the control group. We will use the gross wage 1.5 years after graduation obtained in School Leavers Surveys.

Students in the third year of the academic and the general track of high school in the Netherlands must choose a profile for the remaining of their secondary education. A large number of schools hire the services of career counselling companies, such as Qompas. Therefore, as part of their career guidance activities in their school curriculum, students already answer multiple tests regarding skills and preferences within the Qompas platform. These tests are presented in the following order: 1) “Profile choice test”, 2) “Competency test” and 3) “Interest questionnaire”. At a later stage, pupils are asked to fill in a Provisional profile choice and finally, their Definite profile choice. Within the platform, students receive information of what profiles and occupations would be a good match for them based on their responses to the test. The aim of our experiment would be to estimate the effect of providing labour market information on the gross hourly wage for these shortlisted occupations in different formats on students’ preferences for occupations and their study profile choice.

Prior to the beginning of the experiment, certain preparations have been done. First of all, having a direct match between the occupations that students will select and the labour market information is essential. Therefore, we substituted the occupational groups that are listed under the 11 categories of occupational sectors in the “Interest test” in Qompas with the ROA-BRC 2014 list of occupations, which is the adaptation to the Dutch context of the International Standard Classification of Occupations (ISCO). A graph literacy test has been incorporated into the platform, based on Okan et al. (2019). Furthermore, we have also included a shorter adaptation of the economic competences test from Oberrauch et al. (2023). Only schools whose headmaster has given consent will participate in the experiment. Moreover, students have the option to opt out of the experiment if they so wish.

Our intervention would take place after the “Interest questionnaire”. When pupils fill in the so-called “Interest questionnaire”, Qompas provides an overview on how the following 11 occupational sectors match them in a 5-point scale ranging from Very much (represented by ++) to Not at all (displayed as - -). A list of the occupations that are comprised within each sector is shown when clicking in the arrow on the right side of the category. All students will be asked to first select their preferred 10 occupations out of the top 3 of broad categories of occupations (beroepensectoren). Then we give them the option to choose additional existing occupations and exchange them for any of the already selected ones, so they keep a total of 10. Once they have their shortlist of 10 occupations, we would ask them to rank them (Ranking A). We have decided to have a shortlist of 10 occupations instead of 5 to reduce the risk of having too little variation between occupations (e.g. if a student chooses 5 occupations that are very similar in terms of wage, the intervention would have no effect).

Immediately after, students allocated to each treatment group will receive the average hourly wage corresponding to their shortlisted occupations in the format of table (T1), neutral risk ladder (T2), colourful risk ladder (T3), bar chart (T4) or icon arrays (T5), respectively. The average hourly wage will refer to the most up to date information available (currently 2021). Students in the control group schools will not receive information. Students in the treatment groups will be asked to rank these occupations according to their preferences immediately after seeing the information (Ranking B), in order to measure the short term effect of the intervention. Furthermore, all students (control and treatment groups) will be asked to rank their preferred occupations again when they fill in their provisional (Ranking C) and definite profile choice (Ranking D), which are completed, on average, 15 and 31 days after the interest questionnaire, respectively.

References:
Oberrauch, L., Kaiser, T., & Seeber, G. (2023). Measuring economic competence of youth with a short scale. Journal of Economic Psychology, 97, 102633.
Okan, Y., Janssen, E., Galesic, M., & Waters, E. A. (2019). Using the Short Graph Literacy Scale to Predict Precursors of Health Behavior Change. Medical Decision Making, 39(3), 183-195. https://doi.org/10.1177/0272989x19829728

Not available

We do a two-step randomisation: first we randomise at the school level whether schools get access to wage information or not. Then, within the schools that receive information, we randomise at the classroom level in which format the information is displayed (treatments 1 - 5).

Randomisation method for the school level randomisation:
Originally we received a dataset with 255 schools that were already in the system (i.e., schools that use the Qompas methods). We use the cvcrand command from STATA, which performs a covariate constrained randomisation of clusters in a two-arm parallel cluster randomised trial:

cvcrand aantalleerlingen20212022 aantal_leerlingen pc4 cm em ng nt qualitylife qompas_intensity size sizeqompas qualitylifecat qompas_int_cat classrooms, ntotal_cluster(255) ntrt_cluster(192) categorical (size sizeqompas qualitylifecat qompas_int_cat classrooms) stratify(size sizeqompas qualitylifecat qompas_int_cat classrooms) seed(12345)

* varlist are the baseline cluster-level variables to constrain on:
- aantalleerlingen20212022: registered students per school in the platform
- aantal_leerlingen: total number of students in the school
- pc4: postcode (4 digit version) of the school
- cm: number of students in the culture and society profile
- em: number of students in the economics and society profile
- ng: number of students in the nature and health profile
- nt: number of students in the nature and technology profile
- qualitylife: quality of life indicator at the 4 digit postcode (https://data.overheid.nl/en/dataset/leefbaarometer-meting-20201)
- qompas_intensity: qompas intensity usage, determined by the share of tests completed at the school level
- size: categorical variable that divides schools in quartiles based on the number of registered students
- sizeqompas: categorical variable that divides schools in quartiles based on the number of students registered in the platform
- qualitylifecat: categorical variable that divides schools in quartiles based on the quality of life indicator
- qompas_int_cat: categorical variable that divides schools in quartiles based on the intensity of usage of the platform
- classrooms: categorical variable that divides schools in quartiles based on the number of classrooms per school

* ntotal_cluster are the total amount of clusters, in other words, 255 schools
* ntrt_cluster refers to the amount of clusters that should be allocated to the treatment group, in this case, 192 schools (roughly 75%)
* categorical specifies which variables are categorical (size, sizeqompas, qualitylifecat, qompas_int_cat classrooms)
* stratify: we specify which categorical variables on which to stratify: size, sizeqompas, qualitylifecat, qompas_int_cat, classrooms. Each of them has 5 categories: 4 quartiles and missing values. They represent the number of students the school has, the number of students in Qompas platform, the quality of life indicator and the share of students that complete all tests in the platform
* seed: we set the randomisation seed to 12345, to make sure every time that the randomisation is the same

We check the summary statistics for the variables taken into consideration for the randomisation, by treatment or not. We also conducted a variance ratio test (sdtest) to check whether the variances of the two subgroups are equal or not, in order to take it into account in the t test below

bys _allocation: summarize qompas_intensity, detail
sdtest qompas_intensity, by(_allocation) // we fail to reject the null hypothesis that the variances are equal
histogram qompas_intensity, by (_allocation)

bys _allocation: summarize qualitylife, detail
sdtest qualitylife, by(_allocation) // we fail to reject the null hypothesis that the variances are equal
histogram qualitylife, by (_allocation)

bys _allocation: summarize aantal_leerlingen, detail
sdtest aantal_leerlingen, by(_allocation) // we fail to reject the null hypothesis that the variances are equal
histogram aantal_leerlingen, by (_allocation)

bys _allocation: summarize aantalklassen, detail
sdtest aantalklassen, by(_allocation) // we fail to reject the null hypothesis that the variances are equal
histogram aantalklassen, by (_allocation)

bys _allocation: summarize aantalleerlingen20212022, detail
sdtest aantalleerlingen20212022, by(_allocation) // we reject the null hypothesis that the variances are equal: adjust for this in the t test
histogram aantalleerlingen20212022, by(_allocation)

sdtest em, by(_allocation) // we reject the null hypothesis that the variances are equal: adjust for this in the t test
sdtest cm, by(_allocation) // we reject the null hypothesis that the variances are equal: adjust for this in the t test
sdtest nt, by(_allocation) // we fail to reject the null hypothesis that the variances are equal
sdtest ng, by(_allocation) // we fail to reject the null hypothesis that the variances are equal

bys _allocation: egen school_classrooms = total(aantalklassen)
tab school_classrooms // check the total number of classrooms by treatment group: 367 classrooms in control group, 1102 in treatment group
bys _allocation: summarize cm, detail //
bys _allocation: summarize em, detail //
bys _allocation: summarize ng, detail //
bys _allocation: summarize nt, detail //


We then run some balancing tests:
tabstat qualitylife, by(_allocation) statistics(mean sd p25 p50 p75)
ttest qualitylife, by (_allocation) // difference is not significant
ttest qompas_intensity, by(_allocation) // difference is not significant
ttest aantal_leerlingen, by(_allocation) // difference is not significant
ttest aantalleerlingen20212022, by(_allocation) unequal // difference is not significant
ttest aantalklassen, by (_allocation) // difference is not significant

ttest em, by(_allocation) unequal // difference is not significant
ttest cm, by(_allocation) unequal // difference is not significant
ttest nt, by(_allocation) // difference is not significant
ttest ng, by(_allocation) // difference is not significant

Later in time, we received a list of new schools that have just hired Qompas' services for the first time and for which there is no historical data. Therefore, these schools get allocated using the command randtreat from STATA:
randtreat, generate(_allocation) setseed(12345) unequal(1/4 3/4) misfits(global)


Randomisation at the classroom level regarding the format in which information is displayed:
It is done within Qompas' platform using the roll of a dice.

We pursue a two-stage randomization. First, 25% of the schools will be randomly allocated into a control group that will not receive information, and the remaining 75% of the schools will receive information. Then, on the second stage, classrooms within the treatment schools are randomly assigned into the following five treatment groups: table, neutral risk ladder, colourful risk ladder, bar chart and icon arrays.

Yes

Approximately 300 schools, with an average of 5.8 classes per school. Therefore, we expect approximately 1740 classes.

Between 25,000 and 30,000 pupils

We pursue a two-stage randomization. First, 25% of the schools will be randomly allocated into a control group that will not receive information, and the remaining 75% of the schools will receive information. Then, on the second stage, classrooms within the treatment schools are randomly assigned into the following five treatment groups:

- Table: 1/3 of classrooms within an _allocation=1 school
- Risk ladder no colour: 1/6 of classrooms within an _allocation=1 school
- Risk ladder colour: 1/6 of classrooms within an _allocation=1 school
- Bar chart: 1/6 of classrooms within an _allocation=1 school
- Coin icon arrays: 1/6 of classrooms within an _allocation=1 school

We received a list of schools that were already registered in Qompas' system, which were randomised as follows:
- 192 schools in the extended treatment group that receives information
- 63 schools in the control group that does not receive information
There is an average of 5.8 classes per schools. Therefore, there are 368 classes in the control group and 1102 to be allocated into the different treatment groups. Then, we planned the following allocation:
- 368 classes in the table treatment
- 184 classes in the neutral ladder
- 184 classes in the colourful ladder
- 184 classes in the bar chart
- 184 classes in the icon arrays

Later in time, 49 new schools joined the platform, for which we have no information. We expect 12 schools (average of 70 classes) to join the control group and 37 to join the treatment group. Thus, on average we expect approximately 214.6 classes to be allocated among the 5 treatment groups:
- 71 for the table
- 35 for the neutral ladder
- 36 for the colourful ladder
- 35 for the bar chart
- 36 for the icon arrays

All in all, we thus expect the following number of classes per group:
- 438 for the control group
- 439 for the table
- 219 for the neutral ladder
- 220 for the colourful ladder
- 219 for the bar chart
- 220 for the icon arrays

We use the STATA command power to calculate the power in a test comparing two independent means in a cluster randomised design. We focus on the wage changes of the preferred occupation post-pre treatment. Based on de Koning (2022), who also conducts an information experiment in the Qompas' platform, we assume an intraclass correlation coefficient of 0.003. We assume 438 classes in the control group and 219 in the smallest treatment group. We assume 20 students per class and a standard deviation of 1. We assume that the wage change of the preferred occupation will be 0 for the treatment group. Therefore, to obtain a statistical power of 80% and a significance level of 5%, the minimum detectable effect size is 0.0533. power twomeans 0, k1(438) k2(219) m1(20) m2(20) rho(0.003) power(0.8) Study parameters: - alpha = 0.0500 - power = 0.8000 - m1 = 0.0000 - sd = 1.0000 Cluster design: - K1 = 438 - K2 = 219 - M1 = 20 - M2 = 20 - N1 = 8,760 - N2 = 4,380 - rho = 0.0030 Estimated effect size and experimental-group mean: - delta = 0.0533 - m2 = 0.0533 Reference de Koning, B. K. (2022). Empirical studies on information, beliefs, and choices in education and work [Doctoral Thesis, Maastricht University]. Maastricht.