|
Field
Trial Title
|
Before
Can automatic feedback improve the motivation and labor market outcomes of Swedish job seekers?
|
After
Stimulating controlled or autonomous motivation of job seekers. What works best?
|
|
Field
Abstract
|
Before
Job search is a difficult process often associated with failed job applications and lost social connections, making it difficult for unemployed workers to stay motivated. Then, what is the best way for the public employment service (PES) to enhance motivation? Research in psychology – Self-Determination Theory (Ryan and Deci, 2000) differentiates between different sources of motivation. Controlled motivation occurs when people search for jobs, because they feel pressured to do so. Autonomous occurs when people search for jobs because they find it interesting, or if they search for jobs because they find it meaningful and/or personally relevant.
In this project, we set up a large scale randomized controlled trial (RCT) to study the impact of an intervention that triggers controlled motivation and another that induces autonomous motivation of unemployed job seekers on the following set of outcomes: (i) self-determined motivation as measured by a psychological scale (Da Motta Veiga and Gabriel, 2016); (ii) job search intensity (quantity) as measured both via surveys and register data (based on monthly activity reports submitted by the job seekers), (iii) job search quality as measured by the extent of engagement in metacognitive activities; (iv) the outcomes of job search: job finding and job quality as measured by objective indicators (such as the wage, and the contractual and effective job duration) and subjective indicators of job satisfaction. We also aim at measuring the extent to which initial motivation at the start of unemployment moderates the impact of the intervention on these outcomes. Objective job search outcomes will be measured by register data, while electronic surveys will measure the job search intensity and the subjective outcomes.
The intervention consists for each condition (controlled and autonomous motivation) of a series of six electronic messages that are sent to unemployed job seekers in Sweden during the first half year of unemployment each time just prior to the deadline at which these unemployed have to send a monthly activity report to the PES. These messages aim at enhancing the motivation to search for jobs either by reinforcing external pressure in the controlled condition or by triggering internal motivation in the autonomous condition. A control group receives no messages. The population involved in this RCT will consist of a 2/3 random sample of all Swedes who start a spell of insured unemployment between January 20, and December 19, 2020. Depending on how high this population is relative to that in preceding years the size of this population ranges between 82,500 to 105,000 individuals. These individuals are in turn randomly assigned with 25% probability to each of the two aforementioned conditions and with a 50% probability to the control group.
References:
Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55, 68-78.
Da Motta Veiga, Serge P. & Grabriel, Allison S. (2016). The Role of Self-Determined Motivation in Job Search: A Dynamic Approach. Journal of Applied Psychology, 101(3), 350-361.
|
After
Finding a suitable job is a central objective of most people as it is one of the key drivers of their well-being. However, job search is a difficult process often associated with failed job applications and lost social connections, making it difficult for unemployed workers to stay
motivated. Hence, a key public policy question is how the public employment service (PES) can best strengthen and keep up the motivation of job seekers. Research in economics and psychology seems to result in conflicting advice. In psychology – Self-Determination Theory (SDT) (Deci and Ryan, 1985; 2000; 2012) differentiates between different sources of motivation. Controlled motivation occurs when people search for jobs, because they feel pressured to do so. Autonomous motivation occurs when people search for jobs because they find it interesting, or if they search for jobs because they find it meaningful and/or personally relevant. According to SDT autonomous motivation yields better results than controlled motivation in that it predicts that it leads to more effective job search effort and, hence, in a higher likelihood of finding a job. By contrast, in economics standard job search theory (JST) (Ehrenberg and Oaxaca, 1976; Mortensen, 1977) predicts that controlling job seekers stimulates job search and job finding more strongly. This research aims at empirically testing the validity of these conflicting theories.
To this aim, we set up a large scale randomized controlled trial (RCT) to study the effects of an intervention dispensed to unemployed job seekers aimed at triggering controlled and autonomous motivation. We will examine whether the intervention affects labor market outcomes and job search behavior as predicted by the SDT, or that they are consistent with standard JST instead.
The intervention consists of a series of six electronic messages that are sent for each condition that aims to be triggered (controlled or autonomous motivation) to unemployed job seekers in Sweden during the first half year of unemployment. A control group receives no messages. The initial sampling plan involved the drawing of a 2/3 random sample of all Swedes who start a spell of insured unemployment between January 20, and December 19, 2020. These individuals are in turn randomly assigned with 25% probability to each of the two aforementioned conditions and with a 50% probability to the control group. Due to the COVID-19 crisis the intervention temporarily halted and the intervention was shifted such that a new start date was set to May 20, 2020.
References:
Deci, E. L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in human behavior. New York: Plenum Publishing Co.
Deci, E. L., & Ryan, R. M. (2000). Self-determination theory: A macrotheory of human motivation, development, and health. Canadian Psychology, 49, 182-185.
Deci, E. L., & Ryan, R. M. (2012). Motivation, personality, and development within embedded social contexts: An overview of self-determination theory. In R. M. Ryan (Ed.), Oxford handbook of human motivation (pp. 85-107). Oxford, UK: Oxford University Press.
Ehrenberg, R. & Oaxaca, R. (1976). Unemployment insurance, duration of unemployment and subsequent wage gain. American Economic Review 5, 754–766.
Mortensen, D.T. (1977). Unemployment insurance and job search decisions. Industrial and Labor Relations Review 30, 505–517.
|
|
Field
Last Published
|
Before
March 02, 2020 03:50 PM
|
After
April 20, 2021 08:09 AM
|
|
Field
Intervention Start Date
|
Before
February 26, 2020
|
After
June 30, 2020
|
|
Field
Intervention End Date
|
Before
June 30, 2021
|
After
June 30, 2022
|
|
Field
Primary Outcomes (End Points)
|
Before
(i) Self-determined motivation based on surveys sent to the participants and the control group about 3.5 months after entry in unemployment;
(ii) Job search intensity (quantity): number of hours typically spent on job search activities in the past weeks (survey sent 3.5 months after entry in unemployment) and the number of reported job applications and other job search activities (from register data);
(iii) Job search quality measured by the extent of engagement in metacognitive activities (surveys about 3.5 months and 9 months after entry in unemployment);
(iv) Outcomes of job search: job finding and job quality as measured by objective indicators (such as the wage, unemployment duration, employment status and the contractual and effective job duration) and by subjective indicators of job satisfaction (based on exit surveys sent one month after exit from unemployment).
|
After
(i) Job search effort as measured by the average number of job applications per month 2 to 7 months after entry in unemployment;
(ii) Job finding as measured by an indicator equal to one if unemployment is left within 7 months of entry in unemployment, and zero otherwise.
|
|
Field
Primary Outcomes (Explanation)
|
Before
The measure of self-determined motivation is based on the psychological scale in Da Motta Veiga and Gabriel (2016), while the measure of job search quality measured by the extent of engagement in metacognitive activities is based on a measure developed by Turban, Stevens and Lee (2009). For both measures some items are slightly modified to take into account the specific local context and the requirements of the PES.
References:
Da Motta Veiga, Serge P. & Grabriel, Allison S. (2016). The Role of Self-Determined Motivation in Job Search: A Dynamic Approach. Journal of Applied Psychology, 101(3), 350-361.
Turban, D. B., Stevens, C. K., & Lee, F. K. (2009). Effects of conscientiousness and extraversion on new labor market entrants’ job search: The mediating role of metacognitive activities and positive emotions. Personnel Psychology, 62, 553–573.
|
After
(i) Number of job applications: average number of job applications per month between months 2 to 7 since entry in unemployment, using data from the activity reports. It includes formal job applications, unsolicited/spontaneous applications and job applications following vacancy referrals and proposals. Sample consists of those who submitted a report. Data are used from the activity reports; periods in which no activity reports are handed in are ignored in calculating the average. The same holds for all variables based on the activity reports.
(ii) Job finding: Defined as leaving unemployment within 7 months. Measured using administrative data from the Swedish PES. Unemployment includes full-time unemployment and participation in an active labor market program. Defined as in Cheung et al. (2019).
Reference:
Cheung M, J Egebark, A Forslund, L Laun, M Rödin & J Vikström (2019), “Does job search assistance reduce unemployment? Experimental evidence on displacement effects and mechanisms”, IFAU Working Paper 2019:25.
|
|
Field
Experimental Design (Public)
|
Before
The population involved in this RCT will consist of a 2/3 random sample of all Swedes who start a spell of insured unemployment between January 20, and December 19, 2020.
There are two treatment group conditions (controlled or autonomous motivation) and one control group, all individuals who entered unemployment and registered at the Public Employment Service (PES) at some point between January 20, and December 19, 2020. The size of the control group is double as large as that of the treatment groups. The controlled treatment group condition receives a series of six electronic messages (of about 100-120 words) during the first half year of unemployment (so conditional on still being unemployed). The content of these messages aims at reinforcing external pressure. The autonomous treatment group condition also receives these six messages according to the same timing, but their content aims rather at triggering autonomy, competence and relatedness, the three inborn needs of people according to the Self-Determination Theory. The control group receives no messages.
|
After
Because of the Corona crisis the start date and end date of the intervention was shifted (see the Pre-analysis plan for more details).
The population involved in this RCT will consist of a 2/3 random sample of all Swedes who start a spell of insured unemployment between May 20, 2020 and December 31, 2021 (or earlier in case the PES decides to stop the intervention earlier).
There are two treatment group conditions (controlled or autonomous motivation) and one control group, all individuals who entered unemployment and registered at the Public Employment Service (PES) at some point between May 20, 2020 and December 31, 2021 (or earlier in case the PES decides to stop the intervention prematurely). The size of the control group is double as large as that of the treatment groups. The controlled treatment group condition receives a series of six electronic messages (of about 100-120 words) during the first half year of unemployment (so conditional on still being unemployed). The content of these messages aims at reinforcing external pressure. The autonomous treatment group condition also receives these six messages according to the same timing, but their content aims rather at triggering autonomy, competence and relatedness, the three inborn needs of people according to the Self-Determination Theory. The control group receives no messages.
|
|
Field
Planned Number of Clusters
|
Before
2/3 of all individuals who entered unemployment in Sweden and registered at the Public Employment Service between January 20, and December 19, 2020 will be sampled. This is expected to comprise between 82,500 and 105,000 individuals.
|
After
Planned number of clusters is equal to the planned number of individuals/observations (see below).
|
|
Field
Planned Number of Observations
|
Before
2/3 of all individuals who entered unemployment in Sweden and registered at the Public Employment Service between January 20, and December 19, 2020 will be sampled. This is expected to comprise between 82,500 and 105,000 individuals.
|
After
2/3 of all individuals who entered unemployment in Sweden and registered at the Public Employment Service between May 20, 2020 and December 31, 2021 will be sampled, unless the PES decides to stop the intervention earlier. In case the intervention takes place throughout the planned interventions period the sample size is estimated to comprise 275,000 individuals (calculation based on extrapolations).
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
The treatment groups are each of size ¼ of the total sample size, i.e. in expectation between 20,625 to 26,250 individuals. The control group is expected to be ½ of the total sample size, i.e. 41,250 to 52,500 individuals.
|
After
The treatment groups are each of size ¼ of the total sample size, i.e. in expectation each 75,000 individuals. The control group is expected to be ½ of the total sample size, i.e. 125,000 individuals.
|
|
Field
Power calculation: Minimum Detectable Effect Size for Main Outcomes
|
Before
For the power calculations we consider two sets of outcome variables. The first set of outcome variables uses the administrative files and can, hence, be based on the full sample. The second set of outcome variables are based on 3 electronic surveys sent by e-mail: (1) an initial survey at the start of the unemployment spell, (2) an intermediate survey after about 3.5 months of unemployment, and (3) an exit survey one month after exit from unemployment. We plan to send initial surveys to about 50% of the retained treated sample and to about 25% of the control sample, i.e. to between 10,300 and 13,125 individuals in each treatment arm (between 30,900 and 39,375 individuals in total). By contrast, intermediate surveys are planned to be sent to the full treatment population and 50% of the control population, i.e. to between 20,600 and 26,250 individuals in each treatment arm (between 61,800 and 78,750 individuals in total). Nevertheless, these numbers should be regarded as targets. Due to budgetary and administrative restrictions we might have to reduce these targets. Based on a pilot survey we set the first survey response rate equal to 30%. Conditional on response in a prior survey, the subsequent response rate is assumed to be 70%. These response rates are taken into account when conducting the power analysis for the outcomes that are based on the survey outcomes. For outcomes based on the administrative data only, these response rates are set to 100%
Since we have no clear a priori about the means and standard deviations of the outcome variables, we conduct the power analyses for two cases: (1) for a standardized outcome variable, i.e. a variable of which the mean is normalized to zero and the standard deviation to one, and for (2) a discrete dichotomous variable with two possible outcomes and with mean equal to 0.5. We set in all power analyses the significance level to 5% and the power to 80%.
We consider for each outcome three different effects: treatment 1 (T1) versus control (C), T2 versus C and T1 versus T2. Because the control group is double as large as each of the treated groups, the power analysis for the outcome variables based on administrative files differs according to the considered treatment effect: the minimum detectable effect size (MDE) will be larger for the contrast between T1 and T2 than for the other two contrasts. By contrast, since in the surveys we set the sample size to be equal for each treatment arm, the MDE is the same for all three treatment effects.
1. Set of outcome variables based on admin files
(i) For the standardized outcome (mean equal to zero and standard deviation equal to one) the MDE ranges, depending on aforementioned sample size ranges, between 0.0212 and 0.0239 of a standard deviation when we consider the contrasts between T1 (or T2) and C. For the contrast between T1 and T2 the MDE ranges between 0.0245 and 0.0276.
(ii) For the binary discrete outcome with mean equal to 0.5, the MDE for the contrasts between T1 (or T2) and C ranges between 0.0106 and 0.0119. For the contrast between T1 and T2 it ranges between 0.0130 and 0.0146.
2. Set of outcome variables based on the intermediate and the outcome surveys. For outcomes based on these surveys, we consider that the response rate of 30% reduces the effective sample size by treatment arm to between 6,180 and 7,950. Hence the MDE of a standardized outcome ranges between 0.0444 and 0.0504, while for a binary discrete outcome it ranges between 0.0222 and 0.0252.
3. Set of outcome variables based on an interaction between variables in the initial survey on the one hand, and, on the other hand, outcomes in a) the administrative data, b) the intermediate survey and the outcome survey. This aims, amongst other, at measuring the moderating effect of initial motivation on the other outcomes.
(i) the administrative data, the effective survey size must only be reduced by the non-response rate (70%) in the initial survey. The effective sample size by treatment arm ranges therefore between 3,090 and 3,938. The MDE of a standardized interacted outcome therefore ranges between 0.0631 and 0.0713.
(ii) the intermediate survey and the outcome survey, the effective survey size must be reduced by the non-response rate in the initial survey (70%) as well as by the non-response response rate in the intermediate survey, conditional on response in the initial survey (30%). The effective sample size by treatment arm therefore ranges between 2,163 and 2,757. The MDE of a standardized interacted outcome therefore ranges between 0.0755 and 0.0852.
|
After
Based on the above estimations the minimum detectable effect sizes (MDE's) of the primary outcomes are for a significance level of 5% and a power at 80%:
(i) for the average number of job applications between months 2 and 7 since entry in unemployment (mean = 4.2 and standard deviation = 6.0) equal to 0.0776 (or 1.8% of the mean) for the comparison of the treatment to the control condition, while it is equal to 0.0868 (or 2.1% of the mean) for the comparison of the two treatment conditions to each other;
(ii) for leaving unemployment within 7 months (mean = 0.63 and standard deviation = 0.48) equal to 0.0062 (or 1.0% of the mean) for the comparison of the treatment to the control condition, while it is equal to 0.0069 (or 1.1% of the mean) for the for the comparison of the two treatment conditions to each other.
The mean and standard deviation of these outcomes are estimates on the basis of historical data. More information can be found in the pre-analysis plan.
|
|
Field
Did you obtain IRB approval for this study?
|
Before
No
|
After
Yes
|
|
Field
Secondary Outcomes (End Points)
|
Before
An interaction between, on the one hand, the aforementioned measure of self-determined motivation as measured in an initial survey within the first month of registration in unemployment, and, on the other hand, outcomes in a) the administrative data, b) the intermediate survey (about 3.5 months after entry in unemployment) and the outcome survey (one month after exit from unemployment). This aims, amongst other, at measuring the moderating effect of initial motivation on the other outcomes.
|
After
(i) Job search motivation:
o Controlled job search motivation: Mean of six items from the intermediate survey;
o Controlled job search motivation: Mean of four items from the intermediate survey;
(ii) Job search effort:
o Fraction of times clicked on url's in messages;
o Activity reporting: fraction of times an individual hands in an activity report (1-3, 4-6, 1-6 months after entry in unemployment);
o Number of job applications: average number of job applications per month over the number of months considered (2-4, 5-7);
o Survey-reported job search effort: number of hours per week spent on job search activities, using data from the intermediate survey;
(iii) Job search quality:
o Spontaneous applications: Average number of spontaneous applications per month (months 2-4, 5-7, 2-7);
o Survey-reported job search quality: Mean of five items from the intermediate survey;
(iv) Job search outcomes:
o Job interviews: average number of job interviews per month considering the following months after entry in unemployment: 2-4, 5-7, 2-7;
(v) Job finding:
o Job finding: leaving unemployment within 4, and 12 months;
o Days unemployed during the first year after entry to unemployment;
(vi) Job quality:
o Employment duration: If the first period of non-unemployment exceeds 3, 6 or 12 months or not;
o Wage first job: Full-time equivalent monthly wage rate from Statistics Sweden;
o First job lasts more than 3/6/12 months;
o Job satisfaction: One item from the exit survey;
o Perceived fit: One two-item scale and three separate items from the exit survey;
o Stay intention: One item from the exit survey;
o Composite perceived job quality score: mean of job satisfaction, perceived fit and stay intention.
|
|
Field
Secondary Outcomes (Explanation)
|
Before
|
After
For details how secondary outcomes are constructed see the Pre-analysis plan.
|