Making Accountants Digital Enablers (MADE)

The project will focus on the provision of an intervention for accountants to better equip them to provide knowledge, skills support and confidence to their SME customers so that SMEs feel better able to harness the benefits of digital technology across several specific applications.

We will use a randomised controlled trial (RCT) to compare different intensities of advice provided by accountants to SMEs (as well as no advice at all: to the control group). Using mixed methods we will evaluate statistical measures of changes in productivity, as well as conducting a realist evaluation of how the transfer of knowledge to SMEs is diffused to drive digital adoption.

Sage will recruit N=390 accountants (not just those that use Sage products) who will in turn will identify 5 of their closely matched SME customers to take part in the trial, totalling N=1950 SME participants.  Of the 5 SME participants identified by each of the accountants. This represents a large robust sample, and as discussed in detail later, gives excellent statistical power with respect to identifying even subtle changes in SME productivity across intervention groups.

2020-01-01

2020-02-29

The two primary productivity measures include: (1) number of invoices (2) revenue per invoice.

Once we know the nature of the historic data it may be sensible to look at a small number of secondary supporting / complementary productivity statistics, drawn from but far from exhausting the following list:

Measurement Rank 1

» Survival
» Revenue per Employee
» Revenue per No. Invoices
» Revenue per Total Assets
» Revenue per ICT Assets (ICT Assets is an estimate)
» Revenue per R&D Spend (R&D Spend is an estimate)
» Change in Creditor Days (Trade Creditors in the last quarter * 365 /Revenue)
» Change in Debtor Days (Trade Debtors in the last quarter * 365 /Revenue)

Measurement Rank 2

» Growth in Average Order Value (Revenue / No. Invoices)
» Growth in No. Customers
» No. Invoices per Employee
» No. Invoices per Total Assets
» No. Invoices per ICT Assets (ICT Assets is an estimate)
» No. Invoices per R&D Spend (R&D Spend is an estimate)

N/A

The primary outcome is productivity of SMEs following an intervention of knowledge transfer via an accountant. The focus of the evaluation is whether training accountants on digital technologies as a route to disseminate information on, and support SMEs in, digital adoption leads to increased productivity.

The primary outcome is productivity of SMEs following an intervention of knowledge transfer via an accountant. The focus of the evaluation is whether training accountants on digital technologies as a route to disseminate information on, and support SMEs in, digital adoption leads to increased productivity.
We will assess this by examining historic productivity data from eight quarters prior to the trials’ educational interventions to provide a robust set of baseline data. We will measure change in productivity quarterly for two quarters following the interventions in each trial arm to assess the digital adoption effect of knowledge transfer at different intensities. By using Kirkpatrick’s model of training evaluation we will also establish the trajectory of change where productivity has not yet been affected.
The study has been designed to answer two research questions:
1: Does the facilitation of digital adoption in UK SMEs by UK accountants increase productivity?
2: How does the facilitation of digital adoption in UK SMEs by UK accountants increase productivity?
Research Question 1 (RQ1) will use statistical hypothesis testing to test whether a relationship(s) does exist whilst RQ2 will examine the causal mechanisms underpinning such relationships using a realist evaluation design.
Trial design (RQ1)
From the Trial Research Design and its subsequent operationalization in the Question 3 Support Document this research question has been operationalized into three specific hypothesis tests, as displayed in Figure 2.
These tests will involve a two-way ANOVA analysis of Initial “Digital Savviness” (Blocks) versus SME intervention (Treatments) i.e. control, Foundation, transformation.
There are four levels of SME treatments contemplated by the arms of the trials and five levels of SME ‘digital savviness’ and one value for μ (the total of 1,950 SMEs). Each of these three hypotheses will be tested at a 5% level of significance.
The study is designed so that the four arm design can be aggregated (or collapsed) to a two arm design following the pilot exercise.
Process evaluation (RQ2)
Assuming that evidence of interactions are found within the three tested hypotheses, it is then important to understand the causal mechanism(s) underpinning these interaction, through a Realist Evaluation.
This will assess what works, for whom, in what respects, to what extent, in what circumstances, and how? This is a theory-driven analysis to identify the generative mechanisms that explain how digital adoption was produced and the influence of different SMEs contexts. An initial programme theory will specify embeddedness as an assumed mechanistic driver. Kirkpatrick’s model of training evaluation will be applied to demonstrate progression from knowledge to learning transfer to explain rates of diffusion.
Interviews and research diaries from accountants and SMEs from the active arms will be used to collect evidence. This will be supported by Realist And Meta-narrative Evidence Syntheses: Evolving Standards (RAMESES).
This evaluation will also seek unintended consequences of the intervention(s), including spill-over benefits to SME customers, other stakeholders (using stakeholder engagement interviews), and the broader UK economy: we will detect through surveys what digital adoption occurs beyond that of the six specific technologies contained in foundation and transformation. E.g. there may be an increase in expedited digital tax submission. We may also expect to capture a policy benefit to BEIS in the form of increased GDP for the SME component of UK Plc.

Northumbria will use the SME Matrix to assess the five SME customers by key characteristics from each accountant. Critically each accountant and their consultation of SMEs will be assigned to only one arm. The reason for this is ethical, relates to business ethics recognising that accountants would find problematic to offer different levels of advice across SMEs.
o The digital savviness match of the accountant and the SME
o The spread of SME digital savviness.
o The SME profile based upon the SME Matrix characteristics.
This will generate a sample of n =1950 SME customers to take part in the trial.
The consequence of assigning as accountant and their associate five SMEs to one arm requires us to find similar consultations to be placed on the other two arms, in order to balance out the study.
The Randomized Block Design (RBD) will contain three arms (two treatment groups and a control group). The two treatment groups represent two intensities of training that the accountants receive and disseminate to the SMEs: foundation and transformation (with transformational having additional enhanced features over foundation). Accountants will be allocated then two arms using a process of randomisation, which ensures that the number of accountants and associated SMEs is equal on each arm and through statistical testing that the characteristics discussed above are balanced.

The unit of data collection is the accountant and five associated SME customers. the unit of data analysis is the SME, since the study relates fundamentally to changes in SME business productivity. Nevertheless, other supplementary business information will be collected using both quantitative and qualitative means. The treatment was clustered at the level of the accountant.

Yes

390 Accountants

390 Accountants each with 5 SMEs = 390 x 5 = 1950 We intend to collect eight quarters of historic productivity data from each SME, plus three quarters of data 'during' the period of the study, plus one further quarter 'post' study = 12 quarters of productivity data per SME. as a consequence we will be calculating 23,400 measurements for each identified productivity change statistic.

130 Accountants per treatment arm x 5 = 650 SMEs

Please see the attached for a range of estimates for the minimum detectable effect size under different assumptions and ways of treating the multiple comparisons problem. these etsimates