Back to History

Fields Changed

Registration

Field Before After
Trial Status in_development on_going
Abstract This document describes the research design and analysis strategy of our field experiment, designed to boost training and employment of unemployed. We start with a detailed outline of the intervention, which takes place in early 2021 during a Covid-19 induced partial lockdown in the region of Lower Austria (Niederösterreich) in Austria. Unemployed receive an email newsletter, which for some contains a training voucher and additional information on job vacancies. We designed multiple different treatment arms to separate out direct effects of raising awareness, supporting reciprocity, and strengthening autonomy. We provide a detailed discussion of our sample selection, variables used and the handling of the data to make the analysis as transparent and replicable as possible. We report the outcomes of our stratified randomization. Further, we state our hypotheses and outcomes of interest motivated by the active labour market policy evaluation literature. Finally, we conclude by specifying our statistical approach to inference. This document describes the research design and analysis strategy of our field experiment, designed to boost training and employment of unemployed. We start with a detailed outline of the two parts of the intervention, which take place in early 2021 during a Covid-19 induced partial lockdown in the region of Lower Austria (Niederösterreich) in Austria. Unemployed receive an email newsletter, which for some contains a training voucher and additional information. In the first intervention, we designed multiple different treatment arms to separate out direct effects of raising awareness, supporting reciprocity, and strengthening perceived autonomy. In the second intervention, we send out variations of the training voucher email, which are informed by a pre-intervention survey. We provide a detailed discussion of our sample selection, variables used and the handling of the data to make the analysis as transparent and replicable as possible. We report the outcomes of our stratified randomization. Further, we state our hypotheses and outcomes of interest motivated by the active labour market policy evaluation literature. Finally, we conclude by specifying our statistical approach to inference.
Last Published March 01, 2021 05:42 AM March 10, 2021 11:22 AM
Intervention (Public) In February and March 2021, we launch a field experiment designed in cooperation and implemented by the Public Employment Service (PES) of Lower Austria (Arbeitsmarktservice Niederösterreich (AMS NÖ)). The aim is to increase training and employment among the unemployed by increasing participation in and completion of training programs. The intervention consists of an email newsletter that invites unemployed, who have been registered as unemployed for 3 to 12 months, to voluntarily contact the PES to arrange a consultation on training programs. The newsletters will be sent in three waves on February 9th, February 16th, and March 9th. Individuals are assigned to the waves depending on their duration of unemployment. Three different treatment arms vary the type of information provided and the (perceived) autonomy that the unemployed have in choosing a training program. The different treatment conditions are as follows: 1. Group: control 2. Group: treatment with newsletter 3. Group: treatment with newsletter, and voucher 4. Group: treatment with newsletter, voucher, and information prime Group 1 functions as the control group and is not contacted at all. Group 2 receives a newsletter that includes an invitation to a consultation to discuss potential training programs with the PES' job counselor and provides information about existing financial incentives to start a training program. In addition, groups 3 and 4 receive a voucher (Figure 2) worth € 15.000,-, which can be redeemed to take part in training programs provided by the PES. Alternatively, the voucher can be redeemed in consultation with the PES for any outside training for up to € 3.000,-. The groups receiving the voucher further obtain a list of typical training programs as part of the newsletter. This should motivate the unemployed in these two groups to already think about their preferred training program before the consultation at the PES. Finally, job counselors are instructed to take serious the voucher received by unemployed. The treatment is designed to increase self-initiative for the unemployed and raise awareness for the financial value of such training programs, thus inducing reciprocity. Finally, group 4 receives in addition to the voucher an information treatment consisting of a list of occupations with the highest number of job vacancies. This information treatment is intended to counteract a frequently mentioned concern related to asymmetric information in the use of training vouchers: unemployed allegedly do not have enough information to make an informed choice about their optimal training program (Strittmatter, 2016). It will additionally increase (perceived) autonomy as it encourages even more to think about potential course choices before the consultation at the PES. In general, all groups (including the control group) have access to the same training programs, both provided by the PES as well as outside training. The intervention, thus, consists of the variation in the type of information provided. Additionally, it varies the actual and perceived autonomy that the unemployed have in choosing their courses. In February and March 2021, we launch a field experiment designed in cooperation and implemented by the Public Employment Service (PES) of Lower Austria (Arbeitsmarktservice Niederösterreich (AMS NÖ)). The aim is to increase training and employment among the unemployed by increasing participation in and completion of training programs. Unemployed receive an email newsletter, which for some contains a training voucher and additional information. In the first intervention, we designed multiple different treatment arms to separate out direct effects of raising awareness, supporting reciprocity, and strengthening perceived autonomy. In the second intervention, we send out variations of the training voucher email, which are informed by a pre-intervention survey. First Intervention: The newsletters for the first intervention are sent out in three waves, to which the unemployed are assigned based on their unemployment duration (6-9 months, 9-12 months, and 2-4 months). Here, only those with a valid email-address can be contacted as the newsletters are sent solely via email. Three different treatment arms vary the type of information provided and the perceived autonomy that the unemployed have in choosing a training program. The different treatment conditions are as follows: 1. Group: control 2. Group: treatment with newsletter 3. Group: treatment with newsletter, and voucher 4. Group: treatment with newsletter, voucher, and information prime Group 1 functions as the control group and is not contacted at all. Group 2 receives a newsletter that includes an invitation to a consultation to discuss potential training programs with the PES' job counselor and provides information about existing financial incentives to start a training program. In addition, groups 3 and 4 receive a voucher (Figure 2) worth € 15.000,-, which can be redeemed to take part in training programs provided by the PES. Alternatively, the voucher can be redeemed in consultation with the PES for any outside training for up to € 3.000,-. The groups receiving the voucher further obtain a list of typical training programs as part of the newsletter. This should motivate the unemployed in these two groups to already think about their preferred training program before the consultation at the PES. Finally, job counselors are instructed to take serious the voucher received by unemployed. The treatment is designed to increase self-initiative for the unemployed and raise awareness for the financial value of such training programs, thus inducing reciprocity. Finally, group 4 receives in addition to the voucher an information treatment consisting of a list of occupations with the highest number of job vacancies. This information treatment is intended to counteract a frequently mentioned concern related to asymmetric information in the use of training vouchers: unemployed allegedly do not have enough information to make an informed choice about their optimal training program (Strittmatter, 2016). It will additionally increase (perceived) autonomy as it encourages even more to think about potential course choices before the consultation at the PES. In general, all groups (including the control group) have access to the same training programs, both provided by the PES as well as outside training. The intervention, thus, consists of the variation in the type of information provided. Additionally, it varies the perceived autonomy that the unemployed have in choosing their courses. Importantly, the control group refers to the status quo without intervention, meaning that they are not made worse off by our intervention. Second intervention: Intervention 2 includes all unemployed, who have been registered unemployed for at least 12 months, with or without email-address. Those without an email-address receive the newsletters from the second intervention by post. Additionally, those without an email-address, who would have been included in the first intervention, because of their unemployment duration, also receive the second intervention treatments. To target the second intervention in the best way possible, we ran a pre-intervention survey, which is sent to all who were unemployed longer than a year and had a valid email address. The aim of the survey is to get a better understanding of why unemployed do not participate in training, to then address these reasons and possible solutions more directly in the second newsletter treatment. The second intervention consists again of sending different variations of a newsletter. To maximize the positive effects on participants, we use the most successful newsletter variation from the first intervention, group 3, as a baseline (see Figure 5). On top, we include additional information as treatment variation. Each treatment consists of a few lines that help tackling a specific concern of unemployed to not participate in training. The pre-intervention survey informs the treatment selection and design. The findings point to three reasons that seem most prevalent in preventing or motivating unemployed to take up PES offered training opportunities, which are backed up by anecdotal evidence by PES experts. The treatment variation is: 1. Group: Baseline including newsletter with voucher. The baseline group receives the newsletter with the voucher from the first intervention (from group 3), as this seemed to be the most effective in this short time frame. 2. Group: Baseline plus information on financial support: Financial constraints prevent unemployed from participating in longer term training offered by the PES. Therefore, we strengthen the information on financial support during course participation, making clear that the financial assistance provided by the PES amounts to at least 1000€ a month. 3. Group: Baseline plus information on benefits of training: Unemployed aspire to quickly find a job, and ideally a better one after having completed a training course. Therefore, we provide additional information on the the benefits of training programs. These include average re-employent rates and lower likelihood to become unemployed again after course completion. 4. Group: Baseline plus information on individualistic support: In the survey, unemployed were sceptical that the PES would offer well-matching training programs. Unemployed seek more tailor-made consultations or course offers. Therefore, we emphasize that consultations will be adjusted to fit best the individual needs and skills of the unemployed. Treatment assignment in both interventions is random and stratified on unemployment duration, gender, age, region, education, and sending procedure (mail or post in the 2nd intervention).
Intervention End Date March 10, 2021 March 24, 2021
Experimental Design (Public) The experimental design is based on a stratified randomized assignment to four different treatment and control goups. The randomization is conducted for each of three waves separately, thus practically stratifying by the three categories of unemployment duration dividing the three waves. In addition, we use the other stratification variables as specified in table 1 to construct strata, i.e. blocks. The treatment assignment is in a next step conducted randomly within these strata. First analyses of short-term treatment effects are intended to be carried out with outcome data provided by the PES in several rounds in 2021. Longer-term effects are intended to be estimated with data provided by the PES each year until 2026. This will allow us to estimate long-term effects up to at least 5 years after the intervention. Please refer to the pre-analysis plan for further details. The experimental design is based on a stratified randomized assignment to four different treatment and control groups. In the first intervention, the randomization is conducted for each sending date separately, thus practically stratifying by the three categories of unemployment duration dividing the three waves. In addition, we use the other stratification variables as specified in table 1 to construct strata, i.e. blocks. The treatment assignment is in a next step conducted randomly within these strata. For intervention 2, the randomization is conducted separately for those without email and shorter than 12 months unemployment duration, and for the long-term unemployed with and without email. In the second case, we stratified additionally by how the treatment was sent, i.e. via mail or post. In the first case, the sample size was too small to be able to stratify along unemployment duration within this sample. However, in the whole sample, it is then practically stratified by being unemployed for more or less than a year. In addition, we use the other stratification variables as specified in table 1 to construct strata, i.e. blocks. The treatment assignment is in a next step conducted randomly within these strata. First analyses of short-term treatment effects are intended to be carried out with outcome data provided by the PES in several rounds in 2021. Longer-term effects are intended to be estimated with data provided by the PES each year until 2026. This will allow us to estimate long-term effects up to at least 5 years after the intervention. Please refer to the pre-analysis plan for further details.
Planned Number of Observations 12,000 unemployed 21,000 unemployed
Sample size (or number of clusters) by treatment arms 25% of unemployed in the sample per each of 4 treatment and control arms Intervention 1: 11,000, where 25% of unemployed are allocated to each of the 4 treatment and control arms Intervention 2: 14,000, where 25% of unemployed are allocated to each of the 4 treatment and control arms
Did you obtain IRB approval for this study? No Yes
Secondary Outcomes (End Points) Newsletter read + Clicks; Training take-up; Type of training Newsletter read + Clicks Responses via phone or mail Contacts with PES case workers Training take-up Type of training
Secondary Outcomes (Explanation) The following secondary outcomes are not per se desired outcomes, but can be seen as mechanisms leading to the primary outcomes described above. In this context we will look at whether or not the email was read, newsletter clicks, and course take-up. We will differentiate by the type of course to check whether the intervention also changes training choices. The following secondary outcomes are not per se desired outcomes, but can be seen as mechanisms leading to the primary outcomes described above. In this context we will look at whether or not the email was read, newsletter clicks, who responded to the PES via phone or mail, contacts with the PES caseworker, and course take-up. We will differentiate by the type of course to check whether the intervention also changes training choices.
Back to top

Analysis Plans

Field Before After
Document
PAP_training_vouchers_Austria.pdf
MD5: 04bc8113a364646551b294265ceec32b
SHA1: c0a3425cd9c2aabe657dd607e0c424df10814e4d
Back to top

Documents

Field Before After
Description This document shows the pre-intervention survey in english.
Back to top

Irbs

Field Before After
IRB Name Departmental Research Ethics Committee, Department of Social Policiy and Intervention, University of Oxford
IRB Approval Date March 04, 2021
IRB Approval Number C1A_20_005
Back to top
Field Before After
IRB Name Competence Center for Experimental Research, Vienna University of Economics and Business
IRB Approval Date March 06, 2021
IRB Approval Number WU-HSRP-2021-002
Back to top

Fields Removed

Analysis Plans

Field Value
Document
PAP_training_vouchers_Austria.pdf
MD5: 74e0bf988ded2531cb3312fea54ca0c0
SHA1: b3c80a741afb96e1569c0acde77d2d86e76df2c5
Back to top