Experimental Design Details
The SM team manage a dynamic pipeline of organisations that have expressed interest in principle; have completed a questionnaire detailing company structure, their motivations for inclusion and shared an organogram of the organisation. Discussions between the organisation and SM team clarify whether the organisation is appropriate for the research [having a management/report structure that allows clustering by department/branch/unit] and the organisation is willing to adhere to the terms of the evaluation.
At the point where a Participation Agreement has been signed and the structure of clustering agreed, the organisation sets a start date 6 weeks from the point of this agreement. During this 6-week period the team use an organisation’s organogram as the basis for randomisation. Randomisation is carried out separately for each organisation as they reach the point of committing fully to the training and evaluation; with the organogram and meetings providing the information needed for stratification and creation of distinct units/clusters to be randomised to treatment and control.
Clustering and Stratification: one of the key challenges for this study is the creation of a theoretically and empirically justified approach to our criteria for the type/level of randomisation within each organisation; that can be applied systematically across organisations. There is substantial variation within and between organisations in the size of departments, branches, manufacturing locations and other units used to create clusters; and this is accompanied by variation in how these units are structured between organisations.
This study tests for impacts amongst employee reports, arising from a treatment administered to their line manager(s). Therefore, a key criterion for the creation of clusters is that they reflect line-management responsibilities. Initial discussions with organisations clarify the extent to which they have a structure that can be split into clusters of staff with clear line-management responsibility (i.e. not overlapping); and the extent to which these are separated geographically and/or operationally. This allows the team to create clusters of employees who interact on a personal and operational basis; and have some distance from other clusters – so that if a manager achieves improved outcomes, they will be reflected in the responses of their reports, but will not spill over to other units, teams and managers.
This is a relatively straightforward theoretical justification. It is in the practical implementation that challenges arise. Here we set out some of the key issues that have arisen from this approach to randomisation and decisions made, beginning with a challenge that helps clarify our approach to levels.
Geographically separate outlets [whether retail, hospitality, contract services etc] are ideal as they present a low potential for contamination across treatment and control clusters. However, an issue we have encountered with this type of unit is how to allocate area/regional managers with cross-cutting responsibilities.
Consider the example of Organisation X that has 8 geographically separate retail branches and a head office with 4 separate departments - (i) Accountancy/Finance; (ii) HR/Marketing; (iiii) IT/sales and (iv) Logistics. If Organisation X confirms these 12 branches/departments have the required operational/geographic separation, we would ideally randomise at this level – the line managers trained in units (i) to (iv) and the eight branches will have an equivalent level of seniority, over employee reports .
However, in Organisation X, we may have two area managers each with responsibility for 4 of the 8 branches. Some organisations have an understandable desire to include area managers; and in some limited cases, suggest they need to be trained to support the branch managers. An approach that randomised at area-manager level in Organisation X would reduce the number of clusters to 6. There would be two units - each incorporating an area manager, four branches and their managers - and four head office units, incorporating first line managers. As with the issue of cluster size variability, this complication in our statistical approach to levels is driven by a combination of practical and theoretical considerations.
In most instances, organisations are content to train at the level of branch manager [using the example of Org X] and have area managers untrained in the first wave of wait-listed treatment. In discussion with organisations, this is our preferred option. Whilst there may be concerns that a treatment impact is less likely in branches where line managers, but not the area manager, has been trained, this approach does not risk contamination across clusters, from area managers with cross-cutting responsibilities, and can more clearly be argued to be randomising at the same level as back-office units within organisations. However, in a small number of instances, organisational structures are such that we are forced to randomise and cluster at the level of area manager – for instance, in organisations that have branch manager vacancies being filled by area managers; or where there is a real potential for the organisation to drop out because of this issue.
This randomisation at area manager level is our least preferred option and in a small number of organisations where this is an issue, we prefer to work towards a compromise – with randomisation at the level of branch manager, but all area managers also trained. Depending on the number of organisations that are eventually randomised using this option, the data collected will be used to consider the implications of these decisions – that is, potential for contamination across treatment and control when area managers are trained; and potential for dissolution of treatment impact, where they are not [see Implementation and process evaluation]. In all these contexts, the data collected allow us to clearly differentiate the approaches taken and the approach to randomisation provides a within-organisation treatment and control comparison.
Clearly, the extent to which these different options are pursued is dictated by the team’s interaction with organisations – so the extent to which we do or do not have area manager engagement is endogenous to the organisation. However, as branches are randomised independently of whether area managers are or are not trained, we will have randomly allocated instances of the deviations from our preferred approach. This will allow us to shed some light on a debate within the literature, where managers are seen to lack confidence in tackling conflict, due to a lack of senior management support . The power calculations in this protocol are not focused on testing this issue, rather it is an issue of implementation and process evaluation.
It is hopefully clear why one would wish to consider separately [via stratification] back-office and front-of-house/shop-floor units. The SM intervention is designed to equip managers with conflict resolution skills and our approach to stratification recognises that the nature of conflict, and the wider workplace context, differ on factory or retail ‘shop floors’, when compared to head or back-office operations. For instance, there are often greater opportunities for home-working and other flexibilities in head/back-office operations, whilst employees on the ‘shop-floor’ are more often dealing directly with customers and/or daily production targets. These differences impact both the nature of management challenges and the potential for treatment impacts.
It is worth noting that variability in cluster size (both in terms of number of employee reports and managers) and additional variability in function (above and beyond any shop floor/back-office distinction) have been considered as drivers for additional stratification. For instance, we have considered stratification according to unit size, but across organisations there is substantial variability in what might constitute a ‘large’ or ‘small’ cluster; and the same applies to operational variability above and beyond the stratification chosen.
Randomisation: Having identified which clusters to allocate to the two strata, a different seed is used for each organisation and stratum [recorded], obtained from a site generating random numbers for this purpose . Units in stratum one [operational clusters] are randomly ordered; units in stratum 2 randomly ordered [head office clusters] and then the first 50% of the randomised list of units for each stratum are allocated to treatment.
Ultimately our approach to randomisation is driven by the realities of engagement with organisations working in dynamic competitive environments. A managed pipeline provides batches of organisations that are ready to start the trial at a specific point in time – for many, missing this window of opportunity removes them from the study. As each organisation reaches a point of being ready to begin mainstage, the SM team work at pace to randomise within that organisation and begin the trial. This does mean that in some cases managers are aware of whether they are in the initial treatment group (all managers are aware they will be treated at some point), at the time Pulse Survey responses are secured from their employee reports. All Pulse Survey responses, randomisations and manager correspondences are however time stamped, so we can identify instances where the baseline from employee reports is not secured prior to the randomisation outcome being revealed to their managers.