200SMEchallenge: Design-driven Open Innovation Challenge for 200 SMEs

Pursuing optimal design and user experience of digital products is key for companies to stay competitive in the market. User-centered design techniques inspired by design thinking, such as the “Design Sprint”, have the potential of improving the quality of digital products design. Yet, many SMEs are not aware of the added value of these techniques and are not equipped to adopt them. The 200SMEchallenge project aims at building evidence on the effectiveness of a two-day Design Sprint initiative coming in the format of a “UX Challenge” (User eXperience Challenge) on SMEs’ awareness and readiness to adopt user-centered design. More precisely, the project addresses the following evaluation question: for SMEs operating in the digital industry sector or other SMEs that develop products bearing digital interfaces, does participating in the UX Challenge, rather than not participating, enhance knowledge, awareness and intention to adopt the innovative approaches in the design of digital products? The evaluation features a randomized controlled trial that will be conducted in Winter-Spring 2021 and will involve 200 SMEs based in seven EU regions. The impact of the UX Challenge will be assessed on SMEs’ Digital Design Readiness and Awareness, measured with a set of survey-collected indices on knowledge, attitudes and planned actions related to digital design approaches. The project is funded by the European Commission (Grant Agreement number 824212 — 200SMEchallenge — H2020-INNOSUP-2018-2020).

External Link(s)

Registration Citation

Citation

Azzolini, Davide and Nicola Doppio. 2020. "200SMEchallenge: Design-driven Open Innovation Challenge for 200 SMEs." AEA RCT Registry. October 08. https://doi.org/10.1257/rct.6246-1.0

Sponsors & Partners

Interventions

Intervention(s)

The UX Challenge is scheduled in a two-day format and involve the following actors:
a) Companies: 8 SMEs operating in the digital industry sector or in other sectors but developing products bearing digital interfaces;
b) Solvers: are university students (including Ph.D. students) and young professionals (recently graduated students, e.g. up to 18 months from graduation) mainly with a background in UX design, interaction design and human-computer interaction (computer scientists, designers sociologists, psychologists, economists). Solvers are organized in teams, and each team is mentored by at least one senior mentor (a UX design professional);
c) Mentors: UX design professionals that will support teams in the execution of the UX Challenge. They may be either freelancers or affiliated with a design firm, or even a software company. The buy-in for them to take part in the Challenge is to get in touch with potential clients (SMEs) and potential new talents to hire;
d) Testers: are potential users or customers of the products selected in the Challenge. And at least 4 users have to be selected per product / team (plus one reserve). This way, if 8 companies are selected to the treatment group and will take part in the challenge, each partner will have to select some 40 citizens applying to 8 different profiles.

Day one is normally a Thursday and day 2 a Friday, but this can change depending on local conditions. There are slots that involve both the solvers and the companies (or the larger public), from those that involve the solvers only. One very important slot is the final event (a.k.a. “plenary”, during which all teams briefly pitch the results not only to companies, but to a larger public too).

The UX challenge is organized in five consecutive phases.
• Phase 1: Scoping the Challenge.
* Phase 2: Ideating the solutions.
* Phase 3: prototyping.
* Phase 4: Test.
* Phase 5: Tune and deliver.

NB: Due to the ongoing Covid-19 pandemic, the above description of the UX Challenge is subject to change.

Intervention Start Date

2021-01-17

Intervention End Date

2021-05-31

Primary Outcomes

Primary Outcomes (end points)

Digital Design Readiness and Awareness (DDRA): knowledge, attitudes and behaviors related to digital design approaches in SMEs' activity

Primary Outcomes (explanation)

DDRA will be measured through three specific outcomes:
● Knowledge of user-centered design & design sprint
● Attitudes towards user-centered design
● Intentions to adopt user-centered design

The three outcomes will be measured through batteries of items and will be analyzed independently and no single composite index will be computed.

Secondary Outcomes

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

The trial follows a parallel 2 arms design:
● Treated companies: receive access to the UX Challenge and post-experiment webinar;
● Control companies: have no access to the UX Challenge but can access the post-experiment webinar.

Experimental Design Details

Randomization Method

Randomization will be done in office by a computer using the statistical software STATA.

Randomization Unit

The unit of randomization is the single selected SME. If a company applies with more than one product, the company will enter the randomization procedure only with the product that is regarded as the most suitable.

Was the treatment clustered?

Experiment Characteristics

Sample size: planned number of clusters

Sample size: planned number of observations

200

Sample size (or number of clusters) by treatment arms

Allocation ratio: .28 (T= 56; C= 144). A larger control group is planned to increase the experiment’s statistical power.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Sample size is fixed, as it is constrained by the budget available for the intervention. The size of the control group is planned as being larger than the treatment group, deviating from an equal distribution of treated and controls, to gain higher statistical precision. Since estimates of the variance of our outcome variables are not available, power analysis is calculated both in terms of minimum detectable effect size (in case of continuous variables) and dichotomous variables (assuming the worse scenario of a .50 distribution of the outcome). Estimated minimum detectable effect size is: .39 SD Estimated minimum detectable effect (dichotomous variable): .20 percentage points The assumptions made to calculate the minimum detectable effects of the experiment are: a) statistical significance level (p-value) = .05; b) statistical power: 80%; c) proportion of randomization units assigned to treatment: 28%. A number of tweaks in the RCT design are put in place to improve the experiment’s statistical power: 1) using a larger control group in order to increase sample size; 2) employ a stratified randomization based on companies' location and turnover or sector; 3) statistical analysis of the treatment effects based on regression-adjusted estimation of the impact including covariates and pre-treatment outcomes.

Supporting Documents and Materials

IRB