Hand-holding and the power of free: can a low-cost tailored behavioural intervention carry SMEs over the adoption hurdle?

Last registered on January 05, 2022

Pre-Trial

Trial Information

General Information

Title
Hand-holding and the power of free: can a low-cost tailored behavioural intervention carry SMEs over the adoption hurdle?
RCT ID
AEARCTR-0006891
Initial registration date
December 13, 2020

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
December 14, 2020, 10:30 AM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
January 05, 2022, 6:57 AM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Primary Investigator

Affiliation
ALM Analytics & Consultancy Limited

Other Primary Investigator(s)

Additional Trial Information

Status
On going
Start date
2020-07-01
End date
2022-02-28
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
This is an investigation of barriers to the adoption of Artificial Intelligence (AI) technologies in the UK by small and medium-sized employers (abbreviated hereafter as SMEs). SMEs are defined as having under 250 employees. The research project was conceived prior to the Covid-19 pandemic, but will be implemented within the resumption of economic activity following the initial lockdown measures introduced in March 2020, and subsequent interventions. It is a test case to understand what encouragement mainstream SMEs need to adopt AI within their accounts payable, invoice management and book-keeping functions of their businesses, whilst also dealing with the uncertainties caused by both Covid-19 and the end of the EU transition period in December 2020.
External Link(s)

Registration Citation

Citation
Moody, Anthony. 2022. "Hand-holding and the power of free: can a low-cost tailored behavioural intervention carry SMEs over the adoption hurdle?." AEA RCT Registry. January 05. https://doi.org/10.1257/rct.6891
Sponsors & Partners

Sponsors

Partner

Experimental Details

Interventions

Intervention(s)
Intervention Start Date
2020-12-01
Intervention End Date
2021-11-30

Primary Outcomes

Primary Outcomes (end points)
Primary RCT experiment: Testing adoption of AI account payable technology

Outcome measure: The primary outcome measure is, “does the behavioural support cause more use of the technology?”, within the observed free introductory period (ending 30th September 2021), and hence more SMEs to adopt the technology. We define ‘adoption’ as having successfully used the Evolution Invoice software twice or more by the end of September 2021, with uses at least 30 days apart.

A successful use is uploaded a file to be decoded by the AI technology and then it returns a file with the extracted data back to the business.

At the simplest level we will compare the number of successful uses by businesses in each of the two trial arms. The expectation is that the behavioural support will cause more usage overall and more usage in more businesses.

We overlay that basic measure of volume of technology usage with making an inference about technology adoption. We conclude the technology cannot be said to be adopted if the usage does not span at least 30 days, since adoption requires evidence of recurrent usage. If the observed usage is all contained within a 30 day window then it will not be considered as having been adopted, no matter how much the technology was used. In other words, if an SME has not used the technology at all by 1st September then it cannot satisfy the adoption definition because usage won’t span the required duration before the end of the trial period.

In contrast, even if a business only uses the software twice, provided the gap in usage is at least 30 days then that counts as adoption, because the business will have shown a sustained commitment to trying to use the technology even if the volume of usage is very low.

This definition of adoption will generate a binary variable for our primary outcome measure (0 = not adopted; 1 = adopted), which will be the independent variable to which our predictive logistic model will be fitted, containing the treatment/control variable and any necessary dependent variables. We will also present the model using a standard OLS form, rather, than just a probit, to aid interpretation.
Primary Outcomes (explanation)
Primary RCT experiment: Testing adoption of AI account payable technology

Outcome measure: The primary outcome measure is, “does the behavioural support cause more use of the technology?”, within the observed free introductory period (ending 30th September 2021), and hence more SMEs to adopt the technology. We define ‘adoption’ as having successfully used the Evolution Invoice software twice or more by the end of September 2021, with uses at least 30 days apart.

A successful use is uploaded a file to be decoded by the AI technology and then it returns a file with the extracted data back to the business.

At the simplest level we will compare the number of successful uses by businesses in each of the two trial arms. The expectation is that the behavioural support will cause more usage overall and more usage in more businesses.

We overlay that basic measure of volume of technology usage with making an inference about technology adoption. We conclude the technology cannot be said to be adopted if the usage does not span at least 30 days, since adoption requires evidence of recurrent usage. If the observed usage is all contained within a 30 day window then it will not be considered as having been adopted, no matter how much the technology was used. In other words, if an SME has not used the technology at all by 1st September then it cannot satisfy the adoption definition because usage won’t span the required duration before the end of the trial period.

In contrast, even if a business only uses the software twice, provided the gap in usage is at least 30 days then that counts as adoption, because the business will have shown a sustained commitment to trying to use the technology even if the volume of usage is very low.

This definition of adoption will generate a binary variable for our primary outcome measure (0 = not adopted; 1 = adopted), which will be the independent variable to which our predictive logistic model will be fitted, containing the treatment/control variable and any necessary dependent variables. We will also present the model using a standard OLS form, rather, than just a probit, to aid interpretation.

Secondary Outcomes

Secondary Outcomes (end points)
Additional outcome measure: We will conduct a range of exploratory secondary analyses of this data to understand any differences beyond this simple adoption measure. These include understanding speed of adoption, post-adoption usage and time/labour saved.

The more complicated the invoice read by the AI software the greater the labour saving benefit. We can proxy this by extracting from the usage data the number of field entries or the number of characters populated into the data files returned to the businesses. This might be, say, 5 seconds per field entry or 0.3 seconds per character, which we can then multiply up using estimates of labour cost to quantify the notional value extracted by the adopting businesses over the course of the trial.

Using our understanding from work with one of Evolution AI’s corporate clients as a starting point, we plan to estimate this formula for deriving length of time to key an invoice by using some of the sample after the primary RCT concludes. Specifically, we will invite some of the SMEs that have not used the Evolution Invoice software by 1st September 2021 (and hence cannot become adopters) to work with us to construct a linear formula for the amount of time it takes to manually key an invoicing item using information that we will gather from a limited number of SMEs that will recruit to our expanding consultative group.

Secondary RCT experiment: Testing payment requirements on the transition to continued adoption

Outcomes measure: For those businesses that are considered to adopt the AI technology we will measure continued usage after the free trial has ended.

Provision of bank details for billing (for the 50% discount group) or selecting the continued free usage offer (for the further free offer group) during October 2021 is considered to constitute intention to use Evolution Invoice after the free period ends on 1st December 2021continued usage. Although this is a proxy for actual continued usage, it is the best that can be achieved within the timescale of the trial

Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The trial design is a parallel group, two-arm, superiority trial with a 50:50 allocation ratio.
Experimental Design Details
Randomization Method
Randomisation assigned against a sequence of computer-generated random numbers.
Randomization Unit
Firm
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
992 small to medium-sized firms
Sample size: planned number of observations
992
Sample size (or number of clusters) by treatment arms
n/a
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Our starting assumptions are a baseline rate of adoption of the “Evolution Invoice” AP technology (the 'outcome') in the control of 25%. We assume a 10 percentage point improvement in adoption rate (to 35%) in the treatment group. On-going adoption rate will be monitoring from the usage data behind the AI technology. From these two assumptions the required sample is calculated. The “relative risk” is 1.4 (i.e. 0.35/0.25), which equates to an odds ratio of 1.615. At a conventional α of 0.05 and β of 0.8, the required sample size in this scenario is 661. We aim to recruit over 1000 SMEs. With 1145 businesses in the sample, the trial would be powered to detect a relative risk of 1.3 (i.e. 32.5% adoption compared to 25% baseline). Similarly, if the baseline rate of adoption was much lower (say, 10%), then an achieved sample of 1145 businesses would still have sufficient power to detect a relative risk of 1.4.
Supporting Documents and Materials

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number
Analysis Plan

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials