x

Please fill out this short user survey of only 3 questions in order to help us improve the site. We appreciate your feedback!
Scalable Cyber Interventions Accelerating Productivity Practice for SMEs' programme (Cyber Well) explores whether the use of an online game can enhance the productivity of Dorset SMEs through encouraging the adoption of better data management and cyber security practices.
Last registered on October 30, 2020

Pre-Trial

Trial Information
General Information
Title
Scalable Cyber Interventions Accelerating Productivity Practice for SMEs' programme (Cyber Well) explores whether the use of an online game can enhance the productivity of Dorset SMEs through encouraging the adoption of better data management and cyber security practices.
RCT ID
AEARCTR-0006595
Initial registration date
October 21, 2020
Last updated
October 30, 2020 7:09 AM EDT
Location(s)
Primary Investigator
Affiliation
BCP Council
Other Primary Investigator(s)
PI Affiliation
Limetools
Additional Trial Information
Status
On going
Start date
2020-04-06
End date
2020-11-30
Secondary IDs
N/A
Abstract
Scalable Cyber Interventions Accelerating Productivity Practice for SMEs' programme (Cyber Well) is a Randomised Control Trial to test whether the use of online gamification can enhance the productivity of Dorset SMEs through encouraging the adoption of better data management and cyber security practices. Cyber Well is an integrated training and self-delivered programme building on four Proof of Concept studies in Dorset. The studies showed evidence that SMEs' growth and productivity can suffer directly from poor digital data management and careless cyber security behaviour. Furthermore, evidence shows that SMEs are unlikely to win new business contracts that increasingly require national security and data management accreditation. Conversely, those that have good practices through a combination of new contracts, compliance certification and organisational behaviour change increase productivity.

The primary outcomes being evaluated are whether the deployment of a cyber game with nudge learning increase the cyber and data knowledge resilience in SMEs, and whether this style of teaching encourages attitudinal changes and increases productivity in SMEs related to cyber behaviours and certification in a more effective way than typical cyber ‘push-learning’ training.

The trial design is a parallel group, two-arm superiority trial with 1:1 allocation ratio of Control: Intervention (aim: 150 SMEs in each). The project team ran three batches of delivery. The first batch started in March 2020 and the final batch in August 2020. Participant eligability requirements included that they were: South England based; an SME; and had enough de Minimis allocation.

The main comparison will be between the Intervention and the Control groups. This between groups comparison will take place following each quantitative data collection phase, i.e. at baseline, at initial follow-up and at two-month follow-up. Due to the use of block randomisation the team will also be able to compare performance across different sized businesses and across different job roles (IT or non-IT role) within each arm. In addition, within group comparisons across the three time points of baseline, initial follow-up and two-month follow-up and for each arm will be conducted. This may be done as a factorial analysis, depending on if the data meets the requirements for such a test (see below).

The analysis to be done will be determined by the normality and distribution of the data. In the requirements for ANOVA are met then the survey items with continuous outcomes such as most of the survey attitudes questions will be analysed using a 2 x 3 mixed ANOVA (intervention vs control/ baseline to initial follow-up to two—month follow-up). Post-hoc tests main effects and interactions will be conducted as appropriate.If the requirements for ANOVA are not met, then the Friedman test will be used to determine changes on continuous responses within each arm over the three quantitative data collection time points. Mann-Whitney will be used to make comparisons between each arm at each data collection point, with Bonferroni corrections applied as appropriate. Chi-square analysis will be used for any outcomes with binary choices, such as some of the true/ false items within the baseline and follow-up surveys. An appropriate variation on a chi-square such as McNemar’s test will be used to establish the statistical significance in any changes from baseline to follow-up.

Descriptive analysis will be used to report outcomes from any response items that use ranking, such as the attitude item on perception of barriers to implementing good cyber security practice. Due to COVID restrictions, the numbers of participants has been much lower than anticipated. As such, the project team decided to interview 20 of the participants who completed the training. This will provide insight into the training and rich evidence to substantiate the quantitative data. The team are currently in the final phase of data collection and will begin writing the final report in the next month.
External Link(s)
Registration Citation
Citation
Pleva, Megan and Matthew Robson. 2020. "Scalable Cyber Interventions Accelerating Productivity Practice for SMEs' programme (Cyber Well) explores whether the use of an online game can enhance the productivity of Dorset SMEs through encouraging the adoption of better data management and cyber security practices.." AEA RCT Registry. October 30. https://doi.org/10.1257/rct.6595-1.1.
Experimental Details
Interventions
Intervention(s)
The Intervention has been designed to maximise participation (and learning) compared to more standard ways of learning. Drama, storylines, character immersion and nudges will all be utilised.

The Intervention comprises of:
• Baseline Questionnaire
An online tool/game split into 4 sections
• Module 1: the cyber game (including general cyber practices) which will take approximately 50 minutes to complete
• 15 Minute Nudge following Module 1
• Module 2: the cyber essentials game will take approximately 40 mins to complete (including GDPR and cyber information related to the cyber essentials process).
• 25-minute Nudge following Module 2
• Post Trial Baseline Questionnaire
• Final 2 Month Follow up Baseline Questionnaire

The overall course will take approximately two hours and ten minutes in total, over a two-month period with a week-long gap between each module and nudge. The weeklong gap between each interaction with the participant is to ensure that in case participants do not have time or forget to complete that week’s task, there is time in the ‘mop up’ week to account for this.
Intervention Start Date
2020-04-06
Intervention End Date
2020-11-30
Primary Outcomes
Primary Outcomes (end points)
We are measuring 4 primary outcome measures; (1) retention, (2) perceived risk of cyber security, (3) perceived importance of cybersecurity, and (4) awareness of practises already in place.
Primary Outcomes (explanation)
1. RETENTION
Comparing who: the control group and the intervention group
Outcome to Change: Using baseline knowledge surveys the team will seek to understand if people retained information differently between the control and the intervention group.

2. IMPORTANCE OF CYBER SECURITY
Comparing who: the control group and the intervention group.
Outcome to Change: Using Qualtrics interaction data to understand how often the user engaged with the nudge learning (every week, bi-weekly etc), how interested in extra material they are post Intervention (tested in the 2 month follow up), how they felt the nudge learning impacted their experience (tested in the 2 month follow up and qualitative interviews), how they feel this has impacted their working day with regards to cyber (tested in the two month follow up), If any current practices have changed within the Intervention and control group (tested in the baseline and two month follow up).

3. PERCEIVED RISK
Comparing who: the control group and the intervention group and general insights from both groups.
Outcome to Change: To understand how the SME’s perceive cyber risk and if that has changed based on the intervention and control. To understand what the barriers are that prevent SME productive cyber security behaviours in the workplace. To understand the importance that SME’s place on perceived risk of cyber security and whether they understand the effect this can have on organisational productivity.
Qualitative interviews and the two-month evaluation follow up will help the project team gain a richer understanding of how the SMEs feel that understanding more about cyber and certification has impacted their current or future productivity and what has physically changed.
The interviewer will use open-ended questions during the interviews. Participants will be interviewed from both groups and analysed using thematic analysis.

4. AWARENESS OF PRACTICES IN PLACE
Comparing who: the control group and the intervention group
Outcome to Change: Using baseline data and nudge data to understand whether the intervention encouraged awareness changes and inspired SME’s to take responsibility for cyber threat
Secondary Outcomes
Secondary Outcomes (end points)
Secondary Outcomes (explanation)
Experimental Design
Experimental Design
The trial design will be a parallel group, two-arm superiority trial with 1:1 allocation ratio of Control: Intervention. The project team are aiming to run two batches of delivery. The first batch will start running at the beginning of March 2020. The second batch (recruitment dependant) will aim to be delivered one month after that.

Once recruited onto the programme, the eligibility of SMEs to participate in the programme will be ascertained by checking that they are: Dorset based; an SME; and have enough de Minimis allocation. If the SME does have Cyber Essentials, they will not be removed from the project, but they will be randomised evenly across the two groups.

Once the sample is finalised baseline data (first questionnaire/test) will be taken. Following the baseline data collection process, the randomisation process will be delivered. This will be ‘blocked randomisation’ and will ensure that the number of participants in the trail arms are as equal as possible. For example, stratification by organisation size (micro, small medium) followed by randomisation within strata will be used to ensure that the two study arms are comparable.
Experimental Design Details
Not available
Randomization Method
Randomisation will be blocked using a computer.
Randomization Unit
The strata will be in terms of:
size of business
IT or non-IT role within company
If there are any companies with cyber essentials, they will also be block randomised
Once potential SMEs have been identified they will be assessed to determine if they meet the eligibility criteria, as outlined in a previous section. Each SME will be classed as micro, small or medium, as per the standard definitions. The job role (IT or non-IT) of the contact person at the SME will be also be recorded. This will be used to stratify the overall sample. From within each strata SMEs will then be randomised into either the control or intervention arm.
Was the treatment clustered?
No
Experiment Characteristics
Sample size: planned number of clusters
N/A
Sample size: planned number of observations
N/A
Sample size (or number of clusters) by treatment arms
The planned treatment arms are 150 per arm. Due to Covid and extenuating circumstance, the participant group are using mixed methods approach and interviewing 20 participants from the cohort to generate rich and insightful data.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB
INSTITUTIONAL REVIEW BOARDS (IRBs)
IRB Name
IRB Approval Date
IRB Approval Number