RCT of Adoption of Digital Accounting and Payment Technology (ADAPT)

Details of the interventions for each group with sufficient detail to allow replication.
200 SMEs in Cheshire will be recruited for the Trial.
Treatment group
Through the Trial, 100 of these SMEs, selected at random, will receive two interventions:
• Access to an online portal, where they can read a static written ‘good practice guide’ on the use of Digitally Automated Accounting and Payment Technologies. Access to the guide will be provided following completion of the EOI form, approval of eligibility for the trial and completion of the baseline survey on the online portal, allowing for immediate access on point of recruitment for all participants (i.e. on a rolling basis over a short window of 4-6 weeks) and prior to the webinar events.
Attendance at an interactive online ‘Good Practice’ webinar event delivered by specialists at Barclays Bank. Once assigned to the treatment group, SMEs will be asked to confirm which event date, from a range of available options, they will attend. This event will provide an opportunity to learn from technology experts from the Bank about good practice in using Barclays’ digital payment technologies. Additionally, the webinar will provide experiential, peer-to-peer learning opportunities to share/compare experiences and learn from each other in a small group environment. The event will include an interactive presentation by Barclays, with Q&A and a facilitated networking discussion. Ten events will be delivered, with 10 SMEs in each event, over a five-day period. The same individual at Barclays will deliver each event to ensure consistency.
Control group
The other 100 SMEs randomly allocated to the Control Group will only get access to the static written ‘good practice guide’ on the use of Digitally Automated Accounting and Payment Technology via the ADAPT portal.

2021-03-01

2021-04-30

Several outcome measures have been designed to test the intervention’s effect on awareness, understanding and adoption of digital payment technologies.
Primary outcomes
CO1: SMEs have an increased level of awareness and understanding of digital payment technologies and the associated benefits/risks
To measure the effects of the intervention on awareness and understanding of digital payment technologies, in collaboration with Circle Leadership and Barclays, we have developed a short knowledge test which will be included into the baseline and post-intervention surveys. The test will include questions to assess businesses’ awareness of existing technologies and their potential benefits, as well practical aspects of using them.
Outcome measure: test result (continuous measure)
Analysis metric: change in the test score from baseline to immediate post-intervention survey
Method of aggregation: mean per study group (treatment vs control group)
Time point of interest:
• Before receiving support (baseline survey)
• Immediate post-intervention (survey)
• 6 month post-intervention (survey)
Note: awareness and understanding will be assessed immediately post-intervention and 6-months post-intervention to understand whether this outcome has been sustained.
Survey questions: see the baseline survey attached
CO2: SMEs make progress towards adopting a new to firm digital Accounting and Payment Technology solution.
To measure the effects of the intervention on adoption of digital payment technologies, in collaboration with Circle Leadership and Barclays, we have identified a set of necessary steps/actions needed to progress with adoption of a new digital Accounting and Payment Technology. The information on the number of those actions taken or anticipated by businesses will be collected post-intervention , while the baseline survey will include questions aimed at determining the pre-intervention level of adoption.
Digital Accounting and Payment Technology adoption actions taken
Action steps
• Browsed/considered different digital Accounting and Payment Technology options (undertaken needs analysis and identified potential solutions)
• Invested in external technology services/expertise
• Initiated contact with technology providers/vendors
• Participated in a demonstration of an online Accounting and Payment Technology platform
• Met with/held discussions with potential technology providers/vendors
• Participated in training and/or invested in staff training
• Trialled or piloted the use of new technology
• Decision made to continue use of new technology following pilot/trial
• Business case developed for investment in technology
• Investment made to purchase technology, optimising or expanding adoption across firm
• New technology deployed and integrated into company
• Other (please specify) Response options
• Taken already
• Not taken yet but may take in future – follow-on question to estimate the probability action will be taken within the next 12 months
• Not taken yet and unlikely to take in future
• Not sure
• Not applicable


Outcome measure: steps/actions taken/anticipated since the intervention
Analysis metric: number of steps/actions taken/anticipated to adopt a new digital Accounting and Payment Technology.
Method of aggregation: mean per study group (treatment vs control group)
Time point of interest:
• Pre-intervention level of adoption recorded at the baseline (survey)
• Immediate post-intervention (survey)
• 6 months post-intervention (survey)
Survey questions: see the baseline survey tool attached

We will undertake descriptive and qualitative analysis to answer complementary research questions. This will highlight the elements of the intervention and characteristics of businesses which are associated with positive outcomes.
There are two main reasons we are not proposing formal statistical analysis for complementary research questions: a) the nature of the outcomes; b) available sample size.
The reasons why the best practice event works for a particular business are inherently heterogeneous and in our opinion are better suited for qualitative analysis.
The most successful recruitment methods as well as the types of business benefitting from the intervention the most could be identified through formal statistical analysis of subgroups. However, it is likely the pandemic has narrowed the pool of eligible businesses by increasing usage of digital Accounting and Payment Technologies across the economy and weakened the expected effect of the intervention by forcing the best-practice events to take place online. Therefore, we prefer to use the available sample to analyse the core outcomes and lower the probability of a false negative (rejecting a true null hypothesis that the best practice events do not provide any additional benefit compared to the best practice guides) as much as possible. The sample size calculations are discussed in more detail in section 8.
The secondary objectives are:
1. What recruitment methods are most effective in engaging SMEs to participate in a Digitally Automated Accounting and Payment Technology focused business support programme?

Analysis metric: scores for CO1and CO2 by type of recruitment method (plus qualitative evidence)
Method of aggregation: mean scores for CO1 and CO2 per study group, descriptive analysis

Time point of interest:
• baseline: recruitment method recorded (survey)
• 2-3 months post-intervention (consultations)
• 6 months post-intervention (survey)
• 7 months post-intervention (case studies)

2. Are the good practice webinar events more effective among certain types of beneficiary SME? If so, what are the characteristics of those businesses (i.e. size/sector/location/exporting etc.)?

Analysis metric: scores for CO1and CO2 for different groups of businesses
Method of aggregation: mean scores for CO1 and CO2 per study group, descriptive analysis

Time point of interest:
• baseline: business characteristics recorded (survey)
• 6 months post-intervention (survey)


3. For businesses that have progressed through different stages of the adoption process, how/why is a peer-to-peer good practice webinar event effective in influencing behaviours?

Analysis metric: testing the assumptions in the Theory of Change (described in section 4)
Method of aggregation: qualitative evidence

Time point of interest:
• 2-3 months post-intervention (consultations)
• 6 months post-intervention (survey)
• 7 months post-intervention (case studies)

Sample size permitting (i.e. depending on the observed levels of attrition and whether the programme is oversubscribed) we will perform formal tests for secondary research objectives i.e. analyse CO1 and CO2 for groups of businesses and recruitment methods. However, we state in advance that these calculations will be secondary, and we should not dismiss the results of the analysis of primary outcomes as happening by chance if they appear statistically significant when the results for the sub-group analysis do not.

The core intervention of this RCT will be the delivery of a series of virtual ‘Good Practice’ events to be delivered online by Barclays Bank. These events will be hosted by technology experts from the Bank who will share their experience and knowledge of the use of digital payment technologies, and provide opportunities for peer to peer learning.
This proposed intervention is based on an approach tested in a Proof of Concept under Round 1 of the Business Basics Programme. The ADAPT Proof of Concept tested whether it was possible to harness the expertise of larger corporates in the borough, to share their best practice knowledge, to encourage SMEs in Cheshire to adopt productivity boosting digital automation practices and technology. The proof of concept focused on digital automation in manufacturing and marketing businesses.
The evaluation of the Proof of Concept found that the intervention did have a significant effect on SMEs at the early stages of adoption (based on the BEIS Adoption Model). For example, 100% of participants reported that the ADAPT best practice visit had persuaded them of the benefits of adopting digital automation technology.
The concept was however, only tested on a relatively small treatment group and was not compared against any alternative form of treatment. Additionally, due to the time constraints of the Proof of Concept, it was not possible to gather evidence of how the positive changes identified on the ‘immediate outcomes’ of the intervention may lead to changes on ‘intermediate impacts’ such as a decision to invest in/adopt the digital technology and implement the practices within the business.
This RCT will test the intervention with a larger population sample and identify the added average treatment effect to that of a ‘control’ treatment of exchanging good practice, such as the provision of good practice written ‘Information Guides’.
The format of the event has been adapted from an in-person intervention, delivered at the Barclays Global Technology Centre in Knutsford, Cheshire, to virtual, online events, due to restrictions associated with COVID-19.
During the Trial, 100 SMEs from Cheshire will be randomly selected to attend one of the virtual ‘Good Practice’ events delivered online, and receive a ‘Good Practice Guide’ on the use of digital banking and payment technologies to resolve issues with late payments. A further 100 SMEs will be randomly selected to receive only the ‘Good Practice Guide’.
The Trial will last 13 months, starting on 1st January 2021 and ending by 31st January 2022. This will allow for the collection of longitudinal data from both the treatment and control groups on the 'intermediate outcomes/impacts' at appropriate intervals throughout the Trial.
To estimate the effect of the intervention, we will compare the outcomes between the treatment and control groups taking into account the baseline levels of awareness, understanding and adoption of digital payment technologies to assess the net impact of the good practice events. Formal statistical tests (including t-test and regression adjustment, discussed in more detail in section 7) will be used to determine whether the observed differences between the treatment and control groups are statistically significant.

Description of randomisation methods used to generate the allocation sequence
- pure
- stratified/blocked (please specify strata) - recommended
- paired
- cluster (please specify)

To reduce the standard errors in our analysis of the core outcomes we will use stratified randomisation.

The final decision on the number of strata and the list of stratification variables will be made after the recruitment has been completed (to check how homogeneous the group of recruited businesses is and ensure there are enough businesses in each strata). However, currently we are considering stratification according to the following four characteristics which are likely to explain the outcome variables and the treatment effect:
• Exporting vs non-exporting businesses
• B2B vs B2C businesses
• Baseline results of the ‘knowledge test’
• Baseline levels of technology adoption

Depending on the observed differences between participating businesses we may replace some of the characteristics mentioned above with business size and sector.

Firm level

Yes

200 firms

200 firms

100 in treatment group. 100 in control group.

Description of the statistical methods to be used to compare the groups on the primary and secondary outcome measures: statistical test (e.g. t-test, chi2-test, linear regression with covariates etc.) treatment of standard errors For both CO1 and CO2 we will first analyse the raw differences in outcomes between the treatment and comparison groups. For the CO1 this will be done by performing a one-tailed t-test for the differences in the mean increases in the scores. For the CO2 this will involve a one-tailed t-test for the differences in the mean number of actions/steps taken/anticipated since the intervention. Then, to improve the precision of the estimated effects, we will fit a regression based ANCOVA model : Y_i1=α+β_1 T_i+γX_io+〖δY〗_io+ε_(i,1) , Where Y_i1 – is the value of the post-intervention outcome variable Y_i0 – is the pre-intervention value of the outcome variable (for CO2 this is the level of adoption measured at baseline) T_i – is the treatment indicator equal to 1 for the treatment group and 0 for the control group X_i0 – is a set of control variables which will include all stratification variables and size and location of businesses (if not used for stratification) α – is a constant ε_(i,1) – is an error term. In case attrition or imperfect compliance cause the treatment and control groups to be of different size we will include the interaction term between the covariates and treatment indicator since this may result in greater precision gains (Lin, 2013 ) According to our calculations, depending on the levels of attrition, we will be able to detect an effect which is between 40% and 80% of the standard deviation. Generally, effects of such magnitude are considered large and if they are indeed present can be considered of great policy significance.