Hand-holding and the power of free: can a low-cost tailored behavioural intervention carry SMEs over the adoption hurdle?

2020-12-01

2021-11-30

Primary RCT experiment: Testing adoption of AI account payable technology

Outcome measure: The primary outcome measure is, “does the behavioural support cause more use of the technology?”, within the observed free introductory period (ending 30th September 2021), and hence more SMEs to adopt the technology. We define ‘adoption’ as having successfully used the Evolution Invoice software twice or more by the end of September 2021, with uses at least 30 days apart.

A successful use is uploaded a file to be decoded by the AI technology and then it returns a file with the extracted data back to the business.

At the simplest level we will compare the number of successful uses by businesses in each of the two trial arms. The expectation is that the behavioural support will cause more usage overall and more usage in more businesses.

We overlay that basic measure of volume of technology usage with making an inference about technology adoption. We conclude the technology cannot be said to be adopted if the usage does not span at least 30 days, since adoption requires evidence of recurrent usage. If the observed usage is all contained within a 30 day window then it will not be considered as having been adopted, no matter how much the technology was used. In other words, if an SME has not used the technology at all by 1st September then it cannot satisfy the adoption definition because usage won’t span the required duration before the end of the trial period.

In contrast, even if a business only uses the software twice, provided the gap in usage is at least 30 days then that counts as adoption, because the business will have shown a sustained commitment to trying to use the technology even if the volume of usage is very low.

This definition of adoption will generate a binary variable for our primary outcome measure (0 = not adopted; 1 = adopted), which will be the independent variable to which our predictive logistic model will be fitted, containing the treatment/control variable and any necessary dependent variables. We will also present the model using a standard OLS form, rather, than just a probit, to aid interpretation.

Primary RCT experiment: Testing adoption of AI account payable technology

Outcome measure: The primary outcome measure is, “does the behavioural support cause more use of the technology?”, within the observed free introductory period (ending 30th September 2021), and hence more SMEs to adopt the technology. We define ‘adoption’ as having successfully used the Evolution Invoice software twice or more by the end of September 2021, with uses at least 30 days apart.

A successful use is uploaded a file to be decoded by the AI technology and then it returns a file with the extracted data back to the business.

At the simplest level we will compare the number of successful uses by businesses in each of the two trial arms. The expectation is that the behavioural support will cause more usage overall and more usage in more businesses.

We overlay that basic measure of volume of technology usage with making an inference about technology adoption. We conclude the technology cannot be said to be adopted if the usage does not span at least 30 days, since adoption requires evidence of recurrent usage. If the observed usage is all contained within a 30 day window then it will not be considered as having been adopted, no matter how much the technology was used. In other words, if an SME has not used the technology at all by 1st September then it cannot satisfy the adoption definition because usage won’t span the required duration before the end of the trial period.

In contrast, even if a business only uses the software twice, provided the gap in usage is at least 30 days then that counts as adoption, because the business will have shown a sustained commitment to trying to use the technology even if the volume of usage is very low.

This definition of adoption will generate a binary variable for our primary outcome measure (0 = not adopted; 1 = adopted), which will be the independent variable to which our predictive logistic model will be fitted, containing the treatment/control variable and any necessary dependent variables. We will also present the model using a standard OLS form, rather, than just a probit, to aid interpretation.

Additional outcome measure: We will conduct a range of exploratory secondary analyses of this data to understand any differences beyond this simple adoption measure. These include understanding speed of adoption, post-adoption usage and time/labour saved.

The more complicated the invoice read by the AI software the greater the labour saving benefit. We can proxy this by extracting from the usage data the number of field entries or the number of characters populated into the data files returned to the businesses. This might be, say, 5 seconds per field entry or 0.3 seconds per character, which we can then multiply up using estimates of labour cost to quantify the notional value extracted by the adopting businesses over the course of the trial.

Using our understanding from work with one of Evolution AI’s corporate clients as a starting point, we plan to estimate this formula for deriving length of time to key an invoice by using some of the sample after the primary RCT concludes. Specifically, we will invite some of the SMEs that have not used the Evolution Invoice software by 1st September 2021 (and hence cannot become adopters) to work with us to construct a linear formula for the amount of time it takes to manually key an invoicing item using information that we will gather from a limited number of SMEs that will recruit to our expanding consultative group.

Secondary RCT experiment: Testing payment requirements on the transition to continued adoption

Outcomes measure: For those businesses that are considered to adopt the AI technology we will measure continued usage after the free trial has ended.

Provision of bank details for billing (for the 50% discount group) or selecting the continued free usage offer (for the further free offer group) during October 2021 is considered to constitute intention to use Evolution Invoice after the free period ends on 1st December 2021continued usage. Although this is a proxy for actual continued usage, it is the best that can be achieved within the timescale of the trial

The trial design is a parallel group, two-arm, superiority trial with a 50:50 allocation ratio.

Randomisation assigned against a sequence of computer-generated random numbers.

Firm

No

992 small to medium-sized firms

992

n/a

Our starting assumptions are a baseline rate of adoption of the “Evolution Invoice” AP technology (the 'outcome') in the control of 25%. We assume a 10 percentage point improvement in adoption rate (to 35%) in the treatment group. On-going adoption rate will be monitoring from the usage data behind the AI technology. From these two assumptions the required sample is calculated. The “relative risk” is 1.4 (i.e. 0.35/0.25), which equates to an odds ratio of 1.615. At a conventional α of 0.05 and β of 0.8, the required sample size in this scenario is 661. We aim to recruit over 1000 SMEs. With 1145 businesses in the sample, the trial would be powered to detect a relative risk of 1.3 (i.e. 32.5% adoption compared to 25% baseline). Similarly, if the baseline rate of adoption was much lower (say, 10%), then an achieved sample of 1145 businesses would still have sufficient power to detect a relative risk of 1.4.