Effects of Upward Feedback on Supportive Leadership, Job Satisfaction and Engagement: An Experimental Field Investigation

The setting:
We develop an upward feedback technology in which employees give feedback to their nearest leader. The employees answer six questions about their engagement, work satisfaction, and their nearest leader’s supportive behavior using a web application. The employees are encouraged by their leaders to use the feedback technology during the working day. Leaders in both the treatment and control group will see the answers their employees gave on work engagement and satisfaction. Employees provide feedback every third week over 12 weeks. The leaders receive the average score they have achieved on the questions, but will not be able to identify individual employees’ answers. The information is provided in the form of bar charts which show the development of the average score over the weeks. The charts also provide information about how many employees have answered each question. The feedback the leaders receive is private and will not be shared with anyone else in the company.

The intervention:
Leaders in the treatment group will also receive feedback on how the employees perceive their leadership, based on four questions about the leader’s supportive behavior.

This project consists of two subprojects:
Subproject 1 will use the data from the first trial, which will run in March 2023 to investigate the effect of upward feedback on supportive leadership behaviors.

Subproject 2 will use the data of the first trial, and if possible additional trials to investigate the effect of feedback on employee satisfaction, engagement, intention to quit and quitting, and KPIs of the hospital.

2023-03-09

2023-05-22

Subproject 1:
Supportive Leadership (survey measure, perceived by employees)
Supportive Leadership (feedback technology)

Subproject 2:
Intention to quit (survey measure)
Work Engagement (survey measure)
Job Satisfaction (survey measure)
Quitting (provided by cooperation partner)

Subproject 1:
Supportive Leadership (survey measure): Employees’ rating of their nearest leader’s supportive behavior is measured using the Need Support 12-item scale (Tafvelin & Stenling, 2018). The scale consists of three dimensions: autonomy, competence, and relatedness. Participants respond on a 5-point Likert scale, ranging from 1 (never/almost never) to 5 (always).

Supportive Leadership (feedback technology): Employees’ rating of their nearest leader’s supportive behavior is measured using four items from the feedback technology. We will combine the items into one index for supportive leadership using equal weights. Participants respond on a 5-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree).

1. During the last three weeks, my immediate leader has given me constructive feedback on my work.
2. During the last three weeks, my immediate leader has recognized what I do well.
3. During the last three weeks, my immediate leader has been friendly and accommodating.
4. During the last three weeks, my immediate leader has been working towards a working culture where we see and support each other.

Subproject 2:
Work Engagement (survey measure): Employees’ self-reported engagement is measured using the UWES-9 Utrecht Work Engagement 9-item scale (Schaufeli et al., 2003). The measure consists of three dimensions capturing vigor, dedication, and absorption. Participants respond on a 7-point Likert scale, ranging from 0 (never) to 6 (every day).

Job Satisfaction (survey measure): Employees’ self-reported satisfaction is measured using the (BNS-18) Basic Need Satisfaction 18-item scale (Van den Broeck et al., 2010). The measure consists of three dimensions: autonomy, competence, and relatedness. Participants respond on a 5-point Likert scale, ranging from 1 (totally disagree) to 5 (totally agree).

Intention to quit (survey measure): Employees’ self-reported intention to quit is measured using Michaels and Spector’s (1982) 3-item Turnover Intention Scale. Participants respond on a 6-point Likert scale, ranging from 1 (strongly disagree) to 6 (strongly agree).

Quitting: Indicator whether an employee has quit between the baseline and the endline survey.

Subproject 1:
Dimension of Supportive leadership (Feedback technology):
feedback, recognition, friendliness, and support.

Subproject 2:
Mechansims:
Supportive Leadership (survey measure, perceived by employees)
Supportive Leadership (feedback technology)

Department level Outcomes:
Sick leave (Provided by cooperating partner)

Only for departments with patient care:
- Index for service quality

Subproject 1:
Dimension of Supportive leadership (Feedback technology):
In the feedback technology, the supportive leadership questions are divided into four different dimensions: feedback, recognition, friendliness, and support.

Feedback is measured using a single item:
1. During the last three weeks, my immediate leader has given me constructive feedback on my work.

Recognition is measured using a single item:
2. During the last three weeks, my immediate leader has recognized what I do well.

Friendliness is measured using a single item:
3. During the last three weeks, my immediate leader has been friendly and accommodating.

Support is measured using a single item:
4. During the last three weeks, my immediate leader has been working towards a working culture where we see and support each other.

Subproject 2:
Individual level:
Supportive Leadership (survey measure): Employees’ rating of their nearest leader’s supportive behavior is measured using the Need Support 12-item scale (Tafvelin & Stenling, 2018). The scale consists of three dimensions: autonomy, competence, and relatedness. Participants respond on a 5-point Likert scale, ranging from 1 (never/almost never) to 5 (always).

Supportive Leadership (feedback technology): Employees’ rating of their nearest leader’s supportive behavior is measured using four items from the feedback technology. We will combine the items into one index for supportive leadership using equal weights. Participants respond on a 5-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree).


1. During the last three weeks, my immediate leader has given me constructive feedback on my work.
2. During the last three weeks, my immediate leader has recognized what I do well.
3. During the last three weeks, my immediate leader has been friendly and accommodating.
4. During the last three weeks, my immediate leader has been working towards a working culture where we see and support each other.

Department level:

Sick leave (cooperating partner data): Average number of days employees are on sick leave per month.

Only for departments with patient care:
Index for service quality: equal weight average of standardized measures of waiting time and re-admission rate per month and department. Both outcomes will be provided by the cooperation partner)

We conduct a field experiment in a hospital setting to investigate the effects of upward feedback on supportive leadership, work engagement, job satisfaction, intention to quit and quitting, and as a second step core key KPIs. In the experiment, leaders in both the treatment and the control group will receive feedback from their employees every three weeks over 12 weeks. Depending on the treatment the feedback covers different topics.

We conduct a field experiment in a hospital setting to investigate the effects of upward feedback on supportive leadership, work engagement, job satisfaction, intention to quit and overtime, and core KPIs. Participation in the research project is voluntary and departments could opt into using the feedback technology.

Using block randomization, the leaders, who have volunteered to use the feedback technology, are randomized into treatment and control. We block on the division the department belongs to. For the implementation in March, two departments do not share the division with any other department, we put those in one block. Lastly, we split the block of the emergency care divisions in two blocks based on whether the departments are located in the countryside or the urban area. This leads to a total of 7 blocks for the implementation in March. Both treatment and control will receive feedback on employee engagement and satisfaction, but only the treatment group will receive feedback on supportive leadership.

When estimating treatment impacts, we use several model specifications investigating robustness to adding sets of control variables. All models control for blocks. We will also include additional controls, which are selected based on model fit tested without the inclusion of the treatment variable.

We investigate differential treatment effects across leader’s sex, leader’s tenure (median split), implementation readiness (measured using an adapted version of the 9-items of Bleses et al., 2022, median split) and baseline supportive leadership behaviors as measured in the feedback technology (median split).

We will check the data for unrealistic answers and set these values to missing if, for example, a participant provides a year of birth later than 2005.

Hypotheses
a) Subproject 1:
Research question 1 – Can an upward feedback intervention change the behavior of leaders to be more supportive?

H1 Employees in the treatment group report higher supportive leadership behavior from their leader than employees in the control group in the feedback technology.
H2 Employees in the treatment group report higher supportive leadership behavior from their leader than employees in the control group in the survey.


Secondary hypotheses (investigating the individual items):

H3 Employees in the treatment group report higher scores on feedback than the control group.
H4 Employees in the treatment group report higher scores on recognition than the control group.
H5 Employees in the treatment group report higher scores on friendliness than the control group.
H6 Employees in the treatment group report higher scores on support than the control group.


b) Subproject 2:
Research question 1 – Can an upward feedback intervention improve employee outcomes?

H1 Employees in the treatment group report higher work engagement than employees in the control group.
H2 Employees in the treatment group report higher job satisfaction than employees in the control group.
H3 Employees in the treatment group report lower intentions to quit than employees in the control group.
H 4 Employees in the treatment group are less likely to quit.



Secondary Hypotheses:

H1 Departments in the treatment group score higher on the performance index than departments in the control group.
H2 Employees in the treatment group report less overtime than employees in the control group.

Stratified randomization based on divisions using the STRATARAND command in STATA by John Ternovski

Leader

Yes

Implementation March 2023:
Nr. Of cluster: 23 departments
If possible, we would like to extend the project to more departments (subproject 2). We would like to stack the data in the future, but we do not have enough information at the moment to know what the sample size would be.

Implementation March 2023: Department level outcomes: 23 observations Individual level outcomes: (employees without leaders) The average department size is 27 employees, leading to a potential sample size of 622 observation. We expect a response rate of 50% leading to a sample of 311 employees.

Implementation March 2023:
Clusters will be as equally split across treatments as possible: 11 in the treatment and 12 in the control.

Implementation March 2023: We use PowerUp! software to conduct our power calculation. For individual-level data, we are able to get an MDES of 0.513 standard deviations with alpha = 0.05 and a power of 80% assuming the 23 leaders (cluster), 13 employees per department, and an intraclass correlation of 0.05. For department-level data, we are able to get an MDES of 1,323 with alpha = 0.05 and a power of 80%, assuming 23 independent observations at the department-level. We will include baseline control variables which should help us improve power. For example, if we assume the control variables can explain 80% of the variance in the outcome variables, the MDES is reduced to 0.37 at the individual-level, and 0.59 at the department-level.