|
Field
Experimental Design (Public)
|
Before
In this experiment, we aim to investigate the role of labor market information in education decisions. Dutch preparatory vocational education students select five occupations that fit their preferences and aptitudes according to a test. They are then asked to rank these occupations on the basis of how much they would like to pursue that occupation and state their beliefs about the labor market prospects of the occupations. Subsequently, we provide a number of treatment groups with information about the actual labor market prospects of the different occupations and observe how this changes the students' attitudes towards the occupations, their beliefs about the labor market prospects of the occupations and their study profile choice. Moreover, we estimate whether the use of information is influenced by who provides it and whether more information influences students' sensitivity to the information.
The proposed experiment takes place within the online career counseling platform of Qompas. The platform provides preparatory vocational education students with computer assignments and tests to do in class. One of the tests is the so-called occupation test. In this test, students answer a number of questions about themselves and potential occupations. Based on the answers given, Qompas calculates a score for each of the 353 occupations in their system. This score represents how well the occupation fits the student's preferences and aptitudes. After the test, the students are shown their scores. They are subsequently asked what study profile they intend to choose. In the Netherlands, students choose a profile after the first two years in preparatory vocational education. This profile dictates the courses students take and therefore what study programs they will be eligible for. Once this question has been answered, the students select the five occupations they are most interested in pursuing out of the twenty occupations they scored highest on. After picking the occupations, the students are shown information about the work content of these occupations. Subsequently, they are asked to rank the five occupations on the basis of how much they would like to pursue that occupation. When making the ranking, the study profiles associated with the different occupations are shown. The experiment starts after the first ranking.
After the ranking, the students are asked about their beliefs about the labor market prospects of the different occupations. They are asked to estimate both the opportunities of finding a job in six years and the current hourly wage of intermediate vocational education graduates for the selected occupations. The opportunities of finding a job can be either very good, good, reasonable, bad or very bad. The options for the hourly wage range between 10 and 26 Euros, with 1 Euro intervals. These beliefs are elicited so that we know in what direction the provided information influences the beliefs of the student.
Once the beliefs are elicited, students are shown a number of screens, depending on the treatment group they belong to. On the school level, students are randomized into five groups: one control group and four treatment groups.
The control group is shown no information about the labor market prospects of different occupations and is therefore shown no sender either. Treatment 1 is given information about just the job opportunities and shown a human information sender. Treatment 2 gets information on just job opportunities as well, but is not shown a human information sender. Instead, students are told the research center for education and the labor market provides them with the information. Treatment 3 and 4 are shown the same senders as treatments 1 and 2, respectively. However, in treatments 3 and 4, students are provided with information about both the opportunities of finding a job and hourly wages.
The information about job opportunities is taken from the research center for education and the labor market’s biannual labor market forecasts. These forecasts predict the job opportunities for 113 different occupational groups in six years. The Qompas occupations are divided over these occupational groups. The information about the gross hourly wage is taken from administrative data. It is the median hourly wage for individuals who graduated from an intermediate vocational education program and work in the occupational group associated with the Qompas occupation.
The human ‘information senders’ are employees of the research center for education and the labor market. Senders are divided into four groups: inexperienced male researchers, experienced male researchers, inexperienced female researchers and experienced female researchers. A researcher that does not have a Ph.D. (yet) is considered inexperienced, whereas a researcher with a Ph.D. is considered experienced. For each sender, the name and experience are shown. Gender is not explicitly mentioned, but the names of all senders are indicative of their gender and the word ‘researcher’ is different for men and women in Dutch. No pictures of the sender are shown either, so as to avoid bias caused by appearance unrelated to status or gender.
To ensure that all groups have some time to think about their ranking based on being asked about their beliefs and potentially the information presented to them, all groups are shown a video about work in general after the first ranking (control group) or information provision (treatment groups). The video does not mention the relation between education and job opportunities or wages.
After the video, the students are asked to rank the five selected occupations a second time. They are shown their first ranking and asked whether they want to change anything. To estimate how strongly beliefs change, beliefs are elicited again after the second ranking. Eliciting beliefs a second time allows us to see how beliefs are updated and is especially important to analyze the effect of information overload.
It is possible that students select occupations with very bad labor market prospects only. In this case, providing information is not very useful. To counteract this, students who selected only occupations that have very bad, bad or reasonable job opportunities are shown a number of alternative occupations. Based on a sample of historical data, approximately 20% of students is expected to select only such occupations. The occupations suggested are those with the best labor market prospects out of the twenty occupations they selected, ranked first on job opportunities and then on hourly wages. The difference between the control group and treatment groups lies in the information provided. The control group gets no information about why these occupations are suggested. Treatments 1 and 2 only get information about the job opportunities and treatments 3 and 4 get information about both the job opportunities and the hourly wages. After the alternative occupations are suggested, the students are asked whether they want to change their ranking one last time. In this ranking, they are allowed to include the alternative occupations. If students were not shown alternative occupations, they are not asked to rank the occupations a third time.
The ranking is subsequently included in their student file. The students are supposed to periodically review this file.
Before the end of the school year, the students have to choose a study profile. Unfortunately, it is not possible to match individual data from the experiment to administrative data. We therefore observe the choice of profile through a survey in the Qompas system. Historically, the response rate of this survey is low. Therefore, we also include a question about the intended profile choice at the end of the experiment.
|
After
The proposed experiment takes place within the online career counseling platform for preparatory vocational education (vmbo in Dutch) of Qompas (www.qompas.nl). Qompas is a private company that, among other things, provides career guidance services to secondary and post-secondary education students in the Netherlands. The platform provides preparatory vocational education students with computer assignments and tests to do in class. The assignments and tests help the students get to know what they like, what they're good at and what occupations might fit them. All the information generated throughout the various steps of the Qompas assignments are stored in an individual `student file'. The students are supposed to periodically review this file. One of the tests is the so-called `occupation test'. During this test, students answer a number of questions about themselves and potential occupations. Based on the answers given, Qompas calculates a score for each of the 353 occupations in their system. This score represents how well the occupation fits the student's preferences and abilities. Our experiment starts right after this test.
After the test, students are first asked what study profile they intend to choose. We refer to this as the prior intended profile choice.
After this, the students are shown the twenty occupations that fit them best according to the test. They are subsequently asked to select the five occupations they are most interested in pursuing out of these twenty occupations. After selecting the five occupations, the students are shown information about what the work in the different occupations entails. After the students are provided with this information, they are asked to rank the five occupations on the basis of how much they want to pursue that occupation. We refer to this as the prior ranking of occupations. When making the ranking, the study profiles associated with the different occupations are shown.
After the ranking, the students are asked about their beliefs about the labor market prospects of the different occupations. They are asked to estimate both the opportunities of finding a job in six years and the current hourly wage of intermediate vocational education graduates for the selected occupations. The opportunities of finding a job can be either very bad, bad, reasonable, good or very good. The options for the hourly wage range between €10,- and €26,- with €1,- intervals. We refer to these beliefs as prior beliefs.
Once the beliefs are elicited, students are shown a number of screens, depending on the treatment group they belong to. Randomization of students into control and treatment groups is done at the school level. There is one control group and there are four treatment groups. For the exact randomization strategy, see the corresponding section. The control group is shown no information about the labor market prospects of different occupations and is therefore shown no sender either. The treatment groups are provided with information about either just the job opportunities of their selected occupations or both the job opportunities and hourly wage levels of their selected occupations. This information is provided by Maastricht University's Research Center for Education and the Labor Market (www.roa.nl). As part of one of its research programs, this research center has developed labor market forecasts (http://roa.sbe.maastrichtuniversity.nl/?portfolio=poa-project-onderwijs-arbeidsmarkt-2; For more information about the forecasts, please contact d.fouarge[at]maastrichtuniversity.nl). These forecasts predict the job opportunities for 113 different occupational groups for the coming six years. The Qompas occupations are matched to these occupational groups. The information about the gross hourly wage is taken from administrative data. It is the mean hourly wage for individuals who graduated from an intermediate vocational education program and work in the occupational group associated with the Qompas occupation. The reason for not taking the seemingly more relevant median hourly wage is that preparatory vocational education students probably know the definition of the mean, but may not know the definition of the median.
Treatment 1 and 2 are given information about just the job opportunities. Treatments 3 and 4 get information on both the job opportunities and the hourly wage levels for different occupations.
In treatments 1 and 3, the information is presented by a reseacher from the Research Center for Education and the Labor Market. These `information senders' are divided into four groups: low-status males, high-status males, low-status females and high-status females. In this context, status is defined by the seniority of the information sender. A researcher that does not have a Ph.D. (yet) is considered low-status, whereas a researcher with a Ph.D. is considered high-status. To ensure understanding, the designation presented to students is either 'beginning researcher' or 'experienced researcher' (In Dutch: `beginnend onderzoek(st)er' and `ervaren onderzoek(st)er'). The reason for not presenting the different statuses as `junior' and `senior', respectively, is that we are again worried about a lack of understanding. `Beginning' and `experienced' are more commonly used in the scenario described above in Dutch than in English. For each sender, the name and status are shown. Gender is not explicitly mentioned, but the names of all senders are indicative of their gender and the Dutch word for `researcher' is different for men and women. No pictures of the sender are shown, so as to avoid bias caused by appearance unrelated to status or gender.
In treatments 2 and 4, no human information sender is specified. Instead, students are told the Research Center for Education and the Labor Market provides them with the information.
To decide on the way in which to display the information, we held a short survey among students of a high school that does not work with Qompas, but is otherwise very similar. Approximately 72% (44 out of 61) of students indicated that the current way of displaying information was the most clear to them.
We are primarily interested in the effect of the information about job opportunities. Therefore, we did not include a treatment group that only gets information about hourly wages, so as to preserve power. The main reason for this is that the information about job opportunities is a forecast of the situation in six years, which is more relevant for the students than the current hourly wage levels.
Next, after the first ranking (control group) or information provision (treatment groups), all groups are shown a video (https://www.youtube.com/watch?v=YJ78VDQrO3c) about work in general. We do this to allow both treated and non-treated students some additional reflection time. The video does not mention any particular occupations or the importance of job opportunities and wages.
After the video, the students are asked to make a second ranking. They are shown their first ranking and asked whether they want to change anything. We refer to this as the posterior ranking at second elicitation. After this second ranking, beliefs are again elicited. We refer to these beliefs as the posterior beliefs.
It is possible that students select occupations with very bad labor market prospects only. Based on a sample of historical data, approximately 20% of students is expected to select only such occupations. In this case, providing information is not very useful. To deal with this, students who selected only occupations that have very bad, bad or reasonable job opportunities are shown a number of alternative occupations. The occupations suggested are those with the best labor market prospects out of the twenty occupations they selected, ranked first on job opportunities and then on hourly wages. The difference between the control group and treatment groups lies in the information provided. The control group gets no information about why these occupations are suggested. The treatment groups are told these occupations have better prospects. Treatments 1 and 2 then get information about just the job opportunities and treatments 3 and 4 get information about both the job opportunities and the hourly wages.
After the alternative occupations are suggested, the students can decide to learn more about these occupations or not. We refer to this as the decision to learn more about the alternative occupations or not. If the students decide to learn more about these occupations, they are thereafter asked whether they want to change their ranking one last time. In this ranking, the students are allowed to include the alternative occupations. Initially, the new occupations are placed at the bottom of the ranking in a random order. We refer to this ranking as the posterior ranking at third elicitation. The final top five occupations are subsequently included in the student file.
A number of months after the experiment, the second-year students have to choose a study profile. Students are supposed to indicate what study profile they chose in the `definitive profile choice survey'. We refer to this as the actual profile choice. We also include a question about the intended profile choice at the end of the experiment. In the analyses, we refer to this as the posterior intended profile choice. The fourth-year students have to choose a study program a number of months after the experiment. This choice is captured in one of the other Qompas assignments. Naturally, the study program choices of the current third- and second-year students are captured a year and two years later, respectively.
|
|
Field
Randomization Method
|
Before
Stratified randomization done on a computer.
|
After
We employ a stratified randomization procedure at the school level. The reason for randomizing at the school level is twofold. First, it reduces the chance of there being spillover effects between the different treatments. Second, we expected schools to be less willing to participate if some of their students were to be provided with information, whereas others weren't. As stated, 286 schools participate in the experiment. Last year, the occupation test was completed by approximately 32,750 students. A third of the schools is assigned to the control group. The remaining two thirds are divided equally over the treatment groups.
Schools are stratified on the basis of three characteristics: the available study profile choices in the school, the number of students that completed the occupation test last year, and the quality of life indicator of neighborhoods the students come from. For the available profile choices, we rely on data from Qompas. This data includes information on what study profiles schools do not provide. Unfortunately, no data can either mean that the school provides all profiles or that it did not register the possibilities in the Qompas system. The number of students that completed the occupation test last year is also registered by Qompas. However, data is again not available for all schools. If no data is available in this case, however, the number is predicted using the number of newly registered students in the Qompas system and the total number of students in the school itself (Data from Dutch education executive agency; https://duo.nl/open_onderwijsdata/databestanden/vo/leerlingen/leerlingen-vo-2.jsp; Retrieved: 22-06-2018). If data on one of the two is not available, the number is predicted using just the available measure. For the quality of life in neighborhoods students come from, we rely on the quality of life indicator developed by the Ministry of the Interior and Kingdom Relations (https://data.overheid.nl/data/dataset/leefbaarometer-2-0---meting-2016; Retrieved: 22-06-2018). All neighborhoods in the Netherlands have a score, ranging from 1 (very low quality of life) to 9 (very high quality of life). For every school, we calculate the weighted average quality of life indicator score of the neighborhoods the school's student body comes from. If no data on the residential location of students is available, we predict the average quality of life indicator score using the score of the school's neighborhood.
We use a block design to randomize. Because the profile choice is one of our outcome variables and largely determines the variety of occupations the students are likely to be interested in, we first seek balance on this dimension. We divide the schools into three groups: predetermined choice (only one profile type available), limited choice (between one and three profile types available) and unknown. In many cases, schools in the unknown category will offer all profile types. Within these groups, we subsequently rank schools based on the number of students that completed the occupation test last year. Groups are split in three based on this dimension. Because schools vary a lot in size, this improves the balance in terms of sample size. Lastly, within each of the now nine groups, schools are ranked on the basis of the weighted average quality of life indicator score. These groups are then further split in two. Increased balance on this dimension is important as we intend to estimate heterogeneous effects based on the indicator. In the end, we are left with eighteen strata.
To assign schools to a treatment group, we use randtreat, a user-written STATA command (Alvaro Carril, 2015. "RANDTREAT: Stata module to randomly assign treatments uneven treatments and deal with misfits," Statistical Software Components S458106, Boston College Department of Economics, revised 13 Apr 2017.) Within each stratum, schools are randomly assigned to the different treatment types according to the specified division. As not every stratum contains a perfect multitude of six schools, this will result in some schools not being assigned. We deal with these unassigned schools by recreating strata as mentioned above, omitting the division in two based on the weighted average quality of life indicator score. Within each of the now nine strata, schools are again randomly assigned. For unassigned schools arising from this procedure, we repeat the procedure once more, now stratifying only based on the freedom of profile choice. The last few remaining unassigned schools are sorted based on the freedom of profile choice. They are then assigned based on a randomly ordered list of the control and treatment groups.
As stated, within the sender treatment (treatment groups 1 and 3), the actual sender shown is randomized at the individual level. This means that each student part of a school allocated to treatment 1 or treatment 3 is shown a random sender.
|