|
Field
Trial Title
|
Before
Effects of Structured Support from Leader on Employee Engagement: Experimental Evidence from Grocery Stores
|
After
Effects of Structured Support from Leader on Employee Satisfaction, Intention to Quit and Store Performance: Experimental Evidence from Grocery Stores
|
|
Field
Trial Start Date
|
Before
February 05, 2023
|
After
August 28, 2023
|
|
Field
Trial End Date
|
Before
December 31, 2024
|
After
December 31, 2025
|
|
Field
Last Published
|
Before
February 21, 2023 10:34 AM
|
After
September 08, 2023 03:02 PM
|
|
Field
Intervention Start Date
|
Before
February 20, 2023
|
After
September 11, 2023
|
|
Field
Intervention End Date
|
Before
December 31, 2023
|
After
October 29, 2023
|
|
Field
Primary Outcomes (End Points)
|
Before
primary outcomes:
Employee level outcomes:
Short term:
intention to quit, work engagement, satisfaction
Long term:
employee turnover
Store-level outcomes:
Index on store performance both Short-term (average March-April) and Long-term (Average May – September)
|
After
Update after pilot but prior to implementation: removed engagement as pilot showed little variation in this variable, updated dates
Primary outcomes :
Employee level outcomes:
Short term:
intention to quit, work satisfaction
Long term:
employee turnover
Store-level outcomes:
Index on store performance both Short-term (October-November) and Long-term (December-April)
|
|
Field
Primary Outcomes (Explanation)
|
Before
primary outcomes:
Employee level outcomes (from the three targeted employees):
-Short term:
Intention to quit: measured using Michaels and Spector’s (1982) 3-Item Turnover Intentions Scales which uses a 6-point Likert scale. We will build an index using equal weighting.
Work engagement: measured using the Norwegian version of the 9-item version of the Utrecht Work Engagement Scale (Nerstad, Richardsen, & Martinussen, 2010) aggregated with equal weights.
Satisfaction: measured using the 18-item basic need satisfaction scale by Van Broek et al. (2010) aggregated with equal weights.
-Long term:
employee turnover: binary variable indicating whether the employee is still working in the store 6 months after the intervention
Store level outcomes:
store performance: index consisting of the following administrative measures provided by the cooperation partner: percent satisfied customers per month, contribution margin per month, and shrinkage per month relative to revenue. We are using a principal component analysis to generate the index.
|
After
Primary outcomes:
Employee level outcomes (from the two targeted employees):
-Short term:
Intention to quit: measured using Michaels and Spector’s (1982) 3-Item Turnover Intentions Scales which uses a 6-point Likert scale. We will build an index using equal weighting.
Satisfaction: measured using the 18-item basic need satisfaction scale by Van Broek et al. (2010) aggregated with equal weights.
-Long term:
employee turnover: binary variable indicating whether the employee is still working in the store 6 months after the intervention
Store level outcomes:
store performance: index consisting of the following administrative measures provided by the cooperation partner: percent satisfied customers per month, contribution margin per month, and shrinkage per month relative to revenue. We are using a principal component analysis to generate the index.
|
|
Field
Experimental Design (Public)
|
Before
We conduct a field experiment in a large grocery store chain in Norway to examine the effects of structured support from leaders on employees’ engagement and, subsequently, core KPIs of the stores. In the experiment, leaders will work with a leadership training app for 7-weeks. In addition, the leaders are encouraged to target three employees they work with closely while using the app. The content and features of the app differ by treatment arms.
|
After
Update after pilot but prior to implementation: removed engagement as an outcome variable and reduced the number of selected employees to two instead of three
We conduct a field experiment in a large grocery store chain in Norway to examine the effects of structured support from leaders on employees’ satisfaction and, subsequently, core KPIs of the stores. In the experiment, leaders will work with a leadership training app for 7-weeks. In addition, the leaders are encouraged to target two employees they work with closely while using the app. The content and features of the app differ by treatment arms.
|
|
Field
Randomization Method
|
Before
Stratified randomization blocked on the contributions margin per month, Intention to quit at store level, and the share of satisfied customers, all median split based on baseline data. In addition, we have generated a block including only two stores in which the targeted employees have a 0% fulltime employment position. The fact that the position is set to 0% is likely an error in the data, but we expect that these employees might be different from those with a fixed positive employment rate. We randomized using the stratarand command in STATA on the resulting 9 strata. In the extended trial, we will also block on corporation (4-5 different corporations) and might not need to have a block for the 0% fulltime employment position.
|
After
Stratified randomization blocked on the contributions margin per month, Intention to quit at store level, and the share of satisfied customers, all median split based on baseline data. In addition, we have generated a block including only two stores in which the targeted employees have a 0% fulltime employment position. The fact that the position is set to 0% is likely an error in the data, but we expect that these employees might be different from those with a fixed positive employment rate. We randomized using the stratarand command in STATA on the resulting 9 strata. In the extended trial, we will also block on corporation (4-5 different corporations) and might not need to have a block for the 0% fulltime employment position. In the extended trial, we will also block on corporation (4-5 different corporations) and might not need to have a block for the 0% fulltime employment position.
Update on extended trial randomization after pilot but prior to implementation:
In the extended trial, we use stratified randomization blocked on corporation, Intention to quit at store level and for large corporations the contributions margin per month based on baseline data. For small corporations (less than 10 stores) we stratify using a median split on intention to quit. For medium corporations (more than 10 but less than 20 stores) we use intention to quit terciles, for large corporations (more than 20 stores) we use intention to quit terciles and median split on the contributions margin to generate strata. We randomized using the stratarand command in STATA on the resulting 31 strata.
|
|
Field
Planned Number of Clusters
|
Before
First trial: 42.
Extension:If the trial goes well, we hope to expand the number of participating stores to 150. If we do not have to make major changes after the first trial, we will stack the data of both trials.
|
After
First trial: 42.
Extension:If the trial goes well, we hope to expand the number of participating stores to 150. If we do not have to make major changes after the first trial, we will stack the data of both trials.
Update Extension: We were able to recruit 169 stores. Due to poor take-up in the first trial, we do not intend to stack the data.
|
|
Field
Planned Number of Observations
|
Before
First trial: 42 stores.
For each store, we observe at least the manager, the assistant manager, and 2 two additional employees.
For individual-level employee outcomes, this gives us a sample of 3*42=126.
For store-level outcomes, we have 42 observations.
Extension: 150 stores.
For each store, we observe at least the manager, the assistant manager, and 2 two additional employees.
For individual-level employee outcomes, this gives us a sample of 3*150=450.
For store-level outcomes, we have 150 observations.
|
After
First trial: 42 stores.
For each store, we observe at least the manager, the assistant manager, and 2 two additional employees.
For individual-level employee outcomes, this gives us a sample of 3*42=126.
For store-level outcomes, we have 42 observations.
Extension: 150 stores.
For each store, we observe at least the manager, the assistant manager, and 2 two additional employees.
For individual-level employee outcomes, this gives us a sample of 3*150=450.
For store-level outcomes, we have 150 observations.
Update Extension:169 stores.
For each store, we observe at least the manager, the assistant manager, and one additional employee.
For individual-level employee outcomes, this gives us a sample of 2*169=338.
For store-level outcomes, we have 169 observations.
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
Observations will be split equally between treatment and control. For the first trial, this means 21 stores in treatment and 21 stores in control, or 63 employees in treatment and 63 employees in control. For the extension, this means 75 stores in treatment and 75 stores in control, or 225 employees in treatment and 225 employees in control.
|
After
Observations will be split equally between treatment and control. For the first trial, this means 21 stores in treatment and 21 stores in control, or 63 employees in treatment and 63 employees in control. For the extension, this means 75 stores in treatment and 75 stores in control, or 225 employees in treatment and 225 employees in control.
Update extension: We plan to have 85 stores in treatment and 84 stores in control, or 170 employees in treatment and 168 in control.
|
|
Field
Power calculation: Minimum Detectable Effect Size for Main Outcomes
|
Before
We calculate the MDES using PowerUP! Software Dong, N. and Maynard, R. A. (2013).
First trial: Individual level outcomes: We have 9 blocks and, on average, 4.67 clusters per block with 3 observations in each cluster. Assuming constant treatment effects, an ICC of 0.01, an alpha of 0.05, and a power of 80%, we achieve a MDES of 0.58 standard deviations. This MDES can be further reduced by taking into account the explanatory power of the blocking variable and the inclusion of control variables, especially baseline measures for the outcome variable. If we, for example, assume that the blocking variable can explain 20% of the variance and an R^2 of 0.8, the MDES reduces to 0.27 standard deviations.
Store level outcomes: We have 9 blocks with, on average, 4.67 stores per block. Assuming constant treatment effects, an alpha of 0.05, and a power of 80%, we achieve a MDES of 0.99. This MDES can be reduced by taking into account the explanatory power of the blocking variable and the inclusion of control variables, especially baseline measures for the outcome variable. If we, for example, assume the proportion of variance in Level 1 outcome explained by Block and Level 1 covariates is 0.8, the MDES reduces to 0.44 standard deviations.
Extension: We are assuming that we have additional 4 corporations which are of similar size as the one participating in the first trial.
Individual level outcomes: We have 32 blocks and, on average, 4.67 clusters per block with 3 observations in each cluster. Assuming constant treatment effects, an ICC of 0.01, an alpha of 0.05, and a power of 80%, we achieve a MDES of 0.29. standard deviations. This MDES can be further reduced by taking into account the explanatory power of the blocking variable and the inclusion of control variables, especially baseline measures for the outcome variable. If we, for example, assume that the blocking variable can explain 20% of the variance and an R^2 of 0.8, the MDES reduces to 0.14 standard deviations.
Store level outcomes: We have 32 blocks with, on average, 4.67 stores per block. Assuming constant treatment effects, an alpha of 0.05, and a power of 80%, we achieve a MDES of 0.53. This MDES can be further reduced by taking into account the explanatory power of the blocking variable and the inclusion of control variables, especially baseline measures for the outcome variable. If we, for example, assume that the blocking variable can explain 20% of the variance and an R^2 of 0.8, the MDES reduces to 0.23 standard deviations.
|
After
We calculate the MDES using PowerUP! Software Dong, N. and Maynard, R. A. (2013).
First trial: Individual level outcomes: We have 9 blocks and, on average, 4.67 clusters per block with 3 observations in each cluster. Assuming constant treatment effects, an ICC of 0.01, an alpha of 0.05, and a power of 80%, we achieve a MDES of 0.58 standard deviations. This MDES can be further reduced by taking into account the explanatory power of the blocking variable and the inclusion of control variables, especially baseline measures for the outcome variable. If we, for example, assume that the blocking variable can explain 20% of the variance and an R^2 of 0.8, the MDES reduces to 0.27 standard deviations.
Store level outcomes: We have 9 blocks with, on average, 4.67 stores per block. Assuming constant treatment effects, an alpha of 0.05, and a power of 80%, we achieve a MDES of 0.99. This MDES can be reduced by taking into account the explanatory power of the blocking variable and the inclusion of control variables, especially baseline measures for the outcome variable. If we, for example, assume the proportion of variance in Level 1 outcome explained by Block and Level 1 covariates is 0.8, the MDES reduces to 0.44 standard deviations.
Extension: We are assuming that we have additional 4 corporations which are of similar size as the one participating in the first trial.
Individual level outcomes: We have 32 blocks and, on average, 4.67 clusters per block with 3 observations in each cluster. Assuming constant treatment effects, an ICC of 0.01, an alpha of 0.05, and a power of 80%, we achieve a MDES of 0.29. standard deviations. This MDES can be further reduced by taking into account the explanatory power of the blocking variable and the inclusion of control variables, especially baseline measures for the outcome variable. If we, for example, assume that the blocking variable can explain 20% of the variance and an R^2 of 0.8, the MDES reduces to 0.14 standard deviations.
Store level outcomes: We have 32 blocks with, on average, 4.67 stores per block. Assuming constant treatment effects, an alpha of 0.05, and a power of 80%, we achieve a MDES of 0.53. This MDES can be further reduced by taking into account the explanatory power of the blocking variable and the inclusion of control variables, especially baseline measures for the outcome variable. If we, for example, assume that the blocking variable can explain 20% of the variance and an R^2 of 0.8, the MDES reduces to 0.23 standard deviations.
Update Extension:
Individual level outcomes: We have 31 blocks and, on average, 5.45 clusters per block with 2 observations in each cluster. Assuming constant treatment effects, an ICC of 0.01, an alpha of 0.05, and a power of 80%, we achieve a MDES of 0.32. standard deviations. This MDES can be further reduced by taking into account the explanatory power of the blocking variable and the inclusion of control variables, especially baseline measures for the outcome variable. If we, for example, assume that the blocking variable can explain 20% of the variance and an R^2 of 0.8, the MDES reduces to 0.15 standard deviations.
Store level outcomes: We have 31 blocks with, on average, 5.45 stores per block. Assuming constant treatment effects, an alpha of 0.05, and a power of 80%, we achieve a MDES of 0.45. This MDES can be further reduced by taking into account the explanatory power of the blocking variable and the inclusion of control variables, especially baseline measures for the outcome variable. If we, for example, assume that the blocking variable can explain 20% of the variance and an R^2 of 0.8, the MDES reduces to 0.20 standard deviations.
|
|
Field
Intervention (Hidden)
|
Before
All participating stores receive access to an online application providing several leadership-training topics. The app lists three employees that leaders should work with while using the app. The treatment version of the app includes two additional features to facilitate structured support from leaders: First, it includes a tool to encourage the leaders to set up weekly development meetings with the three employees. This tool helps leaders to organize meeting notes, set goals, and prompts the leader to schedule follow-up meetings. Second, to strengthen leaders’ beliefs in development opportunities, it includes a module designed to help leaders develop a growth mindset for their own abilities as a leader and the growth potential of their employees.
|
After
All participating stores receive access to an online application providing several leadership-training topics. The app lists two employees that leaders should work with while using the app. The treatment version of the app includes two additional features to facilitate structured support from leaders: First, it includes a tool to encourage the leaders to set up weekly development meetings with the two employees. This tool helps leaders to organize meeting notes, set goals, and prompts the leader to schedule follow-up meetings. Second, to strengthen leaders’ beliefs in development opportunities, it includes a module designed to help leaders develop a growth mindset for their own abilities as a leader and the growth potential of their employees.
|
|
Field
Secondary Outcomes (Explanation)
|
Before
leader’s need support (emlpoyee reported): 12-item survey measure developed by Tafvelin and Stenling (2019).
frequency of leader employee meetings (employee reported):
Question: How frequently do you have conversations with your leader
(1)almost every day (2) 2-3 times per week (3) once a week (4) 2-3 times a month (5) once a month or less
leader mindset: measured using a 13 item survey measure developed by Kangas et al (2022)
|
After
leader’s need support (employee reported): 12-item survey measure developed by Tafvelin and Stenling (2019).
frequency of leader employee meetings (employee reported):
Question: How frequently do you have a work-related conversations with your leader
(1)almost every day (2) 2-3 times per week (3) once a week (4) 2-3 times a month (5) once a month or less
We intend to validate this by checking consistency with frequency of meetings reported by the leader for the 2 chosen employees.
leader mindset: measured using a 13 item survey measure developed by Kangas et al (2022)
|