Information Provision and Website Performance: Industrial firms in Spain

At the heart of our research is the exploration of the effects of information gaps on technology adoption and overall business performance. To this end, we inform a comprehensive set of textile firms in Spain. The research relied on a database provided by BTOMAIL, a company specialized in compiling and supplying email address lists segmented by product category and geographical area. BTOMAIL ensures the validity and updating of its data, complying with the privacy standards established by the GDPR. Then, we randomly select a group of these firms to provide them with more detailed feedback on their website improvements. The purpose of such an experiment is to estimate how the detail of feedback about online efficacy metrics influence the broader adoption of digital technologies. To conduct the study, we employ the following treatment arms:


Treatment Arm 1 (Benchmarked Information Provision): Firms in this group receive direct information regarding their website's performance. This information offers insights from various metrics, including website speed, search engine efficiency, and user engagement.


Treatment Arm 2 (Benchmarked Information Provision + Link to more detailed information): Firms in this treatment arm receive not only the benchmarked information provided in Treatment Arm 1 but also a supplementary link that allows them to access to more detailed information. In particular, this link directs them to a platform offering in-depth analyses of their website's performance and recommendations for improving efficiency and optimising the reach of their website.


Treatment Arm 3 (Benchmarked Information Provision + Link with more detailed information + Bimonthly Feedback): Firms in this treatment arm receive the services of Treatment Arm 2 and are also informed that their website performance will be monitored bimonthly. Under this treatment, firms are presented with a comprehensive report every other month, providing further insights into the long-term efficiency and reach of their websites.

We have opted not to include a control group devoid of any information. This decision is motivated by the RCT conducted by Adem et al. (2023) in the UK, which demonstrated that providing benchmark information to UK firms did not significantly affect subsequent firms’ decisions compared to the non-informed group at the conventional levels.

The information provided is compiled using a sophisticated toolkit that benchmarks individual firm data against industry standards. This toolkit draws insights from technology usage data sourced from built with, website speed information from batch speed, and search engine performance metrics from Moz.

The principal objective of this research is to estimate the effect of providing feedback on the web performance of Spanish textile firms on subsequent decisions to improve it.

Although preliminary findings within the distilling sector in the UK indicated that the provision of benchmark information had negligible effects, it is interesting to examine whether the textile sector in Spain— with its distinct regional characteristics, industry-specific nuances, and exposure to various treatment arms—responds similarly or exhibits different patterns of reaction.

2024-10-08

2025-03-31

Primary Outcomes (end points)

Our main objective is to assess whether offering information about the company's website performance incentivizes adopting new web-based technology.

Secondary Outcomes (end points)
Additional outcomes involve evaluating whether the company's financial performance saw enhancements.
Secondary Outcomes (explanation)

Heterogeneity analysis

We will conduct a heterogeneity analysis to examine whether the treatment effects vary based on the initial performance of the firms. Specifically, we will analyze whether firms performing above the median on the average of the nine baseline indicators respond differently to the treatments compared to firms performing below the median. Firms will be categorized based on their initial performance in the nine indicators (First-Contentful-Paint, First-Meaningful-Paint, Time to Interactive, Ranking Keywords, Speed-Index, Domain Authority, Page Views, Below Average Count, and Total Website Performance Score). Firms will be divided into two groups: those above the median and those below the median. The heterogeneity analysis will investigate if the treatment affects more to firms performing on average in the nine indicators above the median and firms performing below the median.

Experimental design: This study adopts a three-arm block randomized design. Within the block, firms are randomly assigned to one of the following:


Treatment Arm 1 (Benchmarked Information Provision)

Treatment Arm 2 (Benchmarked Information Provision + Link to more detailed information)

Treatment Arm 3 (Benchmarked Information Provision + Link to more detailed information+ + Bimonthly Feedback):

Methodology: Randomised Control Trial (RCT)
Sampling:. Random sampling will be conducted among the Spanish textile firms with information contained in the BTOMAIL Database for this study
Sample Size: 3,601 firms from the textile sector.
Interventions:
Treatment: All firms will get insights into their website's performance, including speed and search efficiency metrics. We will randomly choose two groups of firms to provide them with more detailed information about the firm's relative performance benchmarked against industry standards. A final group of firms is randomly chosen and provided with bimonthly feedback.


Duration: From 10th September 2024 to 24th September 2024.
Mode:

The dissemination of the insights will occur via electronic mail. We will initially introduce our collaborative Research Group to the firms, emphasizing our affiliations at Universidad Rey Juan Carlos, the University of Nottingham, and the University of Liverpool. A second e-mail regarding their website’s performance will be sent one week after this introductory email.

Experimental Arms:
Treatment Arm 1 (40% of firms, 1,440 firms): Firms in this group receive direct information regarding their website’s performance. This information offers insights from various metrics, including website speed, search engine efficiency, and user engagement.

Treatment Arm 2 (30% firms, 1,080 firms): Firms in this treatment arm receive not only the information provided in Treatment Arm 1 but also a supplementary link. This link directs them to a platform offering in-depth analyses of their website’s performance and recommendations for optimizing their site’s reach and efficiency.
.

Treatment Arm 3 (30% of firms, 1,080 firms): Firms in this treatment arm receive the services of Treatment Arm 2 and are also informed that their website performance will be monitored bimonthly. They will receive a comprehensive report every two months, providing further insights into the long-term efficiency and reach of their websites.


Outcome Metrics:
Primary Outcomes: The study will estimate the causal effects of the intervention on the firm’s website performance post-intervention paying special attention to whether the the firm adopts new web-based technologies as a result of the intervention.
Secondary Outcomes:
Experimental Design Details:
Randomization Method: A simple random assignment method will be used to select participating textile firms for the intervention. This will be achieved using tools such as Stata.
Randomization Unit: Random assignment occurs at the firm level.
Clustered Treatment: No

Not available

Randomization Method: A simple random assignment method will be used to select participating textile firms for the intervention. This will be achieved using tools such as Stata.

Randomization Unit: Random assignment occurs at the firm level.

No

Sample size: 3,601 firms from the Spanish textile industry sector.

Planned number of observations: Each participating firm will be observed for changes post-intervention.

Sample size by treatment arms:
Of the 3,601 firms in the sample, 1,800 perform above the median baseline, while the remaining 1,800 perform below the median. For Treatment Arm 1, there are 720 firms treated and 720 control firms in both above and below median performance categories. Treatment Arms 2 and 3 each have 540 treated and 540 control firms in the above and below median performance categories. In summary, Treatment Arm 1 will include 1,440 firms, Treatment Arm 2 will consist of 1,080 firms, and Treatment Arm 3 will also consist of 1,080 firms. Approximately 40% of the firms will be assigned to Treatment Arm 1, while 30% will be assigned to Treatment Arm 2, and the remaining 30% to Treatment Arm 3.

Minimum detectable effect size: Parameters will be delineated, with references from the current textile sector's achievements and other analogous studies. Using Adem et al. (2023) Summary statistics for a similar RCT in the UK distilling industry, we compute the following Minimum Detectable Effect and Implied Change Table 1. Minimum Detectable Effect and Implied Change of the Randomized Control Trial Treatment arm 1 Each treatment arms 2 & 3 Significance level Power Mean SD Minimum Detectable Effect Implied Change First-Contentful-Paint 720 540 5% 80% 1.408 0.744 0.1187 8.43 First-Meaningful-Paint 720 540 5% 80% 1.554 0.833 0.1330 8.56 First-Meaningful-Paint 720 540 5% 80% 2.490 1.485 0.2370 9.52 Time to interactive 720 540 5% 80% 3.189 2.168 0.3460 10.85 Ranking Keywords 720 540 5% 80% 24.910 143.485 22.9015 91.94 Speed-Index 720 540 5% 80% 3.268 2.127 0.3395 10.39 Domain Authority 720 540 5% 80% 17.114 14.036 2.2403 13.09 Page Views 720 540 5% 80% 4368.6 46671.2 7449.1408 170.52 Below Average Count 720 540 5% 80% 2.365 1.529 0.2440 10.32 Sensitivity analysis We evaluate the sensitivity of the results to different effect sizes and standard deviations using a representative variable, time to interactive, which takes the median value of the web measures included in Table 1. This representative variable has 1441 observations for the treatment arm 1 and 1080 observations for the treatment arms 2 and 3. Its mean value and standard deviation are 3.189 and 2.127, respectively. Thus, the effect size corresponding to testing the effect of treatment arm 1 compared to treatment arm 2 or 3 with 80% of power is 0.2399. Table 2 shows the statistical power of the mean difference test (treatment arm 1 vs treatment arm 2 or 3) associated with different effect sizes. It can be seen that small reductions in the effect size generate big losses in power. For example, an effect size of 0.2 is only associated with a power of 65%. Table 2. Statistical power associated with different mean differences alpha power N N1 N2 Effect size m1 m2 SD 0.05 0.052 2521 1441 1080 0.01 3.189 3.199 2.127 0.05 0.056 2521 1441 1080 0.02 3.189 3.209 2.127 0.05 0.064 2521 1441 1080 0.03 3.189 3.219 2.127 0.05 0.075 2521 1441 1080 0.04 3.189 3.229 2.127 0.05 0.090 2521 1441 1080 0.05 3.189 3.239 2.127 0.05 0.108 2521 1441 1080 0.06 3.189 3.249 2.127 0.05 0.129 2521 1441 1080 0.07 3.189 3.259 2.127 0.05 0.154 2521 1441 1080 0.08 3.189 3.269 2.127 0.05 0.183 2521 1441 1080 0.09 3.189 3.279 2.127 0.05 0.215 2521 1441 1080 0.1 3.189 3.289 2.127 0.05 0.250 2521 1441 1080 0.11 3.189 3.299 2.127 0.05 0.289 2521 1441 1080 0.12 3.189 3.309 2.127 0.05 0.330 2521 1441 1080 0.13 3.189 3.319 2.127 0.05 0.373 2521 1441 1080 0.14 3.189 3.329 2.127 0.05 0.418 2521 1441 1080 0.15 3.189 3.339 2.127 0.05 0.464 2521 1441 1080 0.16 3.189 3.349 2.127 0.05 0.510 2521 1441 1080 0.17 3.189 3.359 2.127 0.05 0.556 2521 1441 1080 0.18 3.189 3.369 2.127 0.05 0.602 2521 1441 1080 0.19 3.189 3.379 2.127 0.05 0.646 2521 1441 1080 0.2 3.189 3.389 2.127 0.05 0.689 2521 1441 1080 0.21 3.189 3.399 2.127 0.05 0.729 2521 1441 1080 0.22 3.189 3.409 2.127 0.05 0.766 2521 1441 1080 0.23 3.189 3.419 2.127 0.05 0.800 2521 1441 1080 0.24 3.189 3.429 2.127 0.05 0.831 2521 1441 1080 0.25 3.189 3.439 2.127 0.05 0.859 2521 1441 1080 0.26 3.189 3.449 2.127 0.05 0.884 2521 1441 1080 0.27 3.189 3.459 2.127 0.05 0.905 2521 1441 1080 0.28 3.189 3.469 2.127 0.05 0.923 2521 1441 1080 0.29 3.189 3.479 2.127 0.05 0.939 2521 1441 1080 0.3 3.189 3.489 2.127 0.05 0.952 2521 1441 1080 0.31 3.189 3.499 2.127 Table 3 shows the effect of changing the standard deviation of the representative sample on the power of the test. The effect of marginal changes in SD is also asymmetric in the neighbourhood of 80% power, as reducing the SD generates only small increases in power as we approach the upper bound of 100%. Table 3. Statistical power associated with different standard deviations alpha power N N1 N2 Effect size m1 m2 SD 0.05 0.9793 2,521 1,441 1,080 0.2416 3.189 3.431 1.5 0.05 0.9633 2,521 1,441 1,080 0.2416 3.189 3.431 1.6 0.05 0.9418 2,521 1,441 1,080 0.2416 3.189 3.431 1.7 0.05 0.9152 2,521 1,441 1,080 0.2416 3.189 3.431 1.8 0.05 0.8846 2,521 1,441 1,080 0.2416 3.189 3.431 1.9 0.05 0.8509 2,521 1,441 1,080 0.2416 3.189 3.431 2 0.05 0.8153 2,521 1,441 1,080 0.2416 3.189 3.431 2.1 0.05 0.7786 2,521 1,441 1,080 0.2416 3.189 3.431 2.2 0.05 0.7418 2,521 1,441 1,080 0.2416 3.189 3.431 2.3 0.05 0.7055 2,521 1,441 1,080 0.2416 3.189 3.431 2.4 0.05 0.6701 2,521 1,441 1,080 0.2416 3.189 3.431 2.5 0.05 0.6361 2,521 1,441 1,080 0.2416 3.189 3.431 2.6 0.05 0.6035 2,521 1,441 1,080 0.2416 3.189 3.431 2.7 0.05 0.5727 2,521 1,441 1,080 0.2416 3.189 3.431 2.8 0.05 0.5435 2,521 1,441 1,080 0.2416 3.189 3.431 2.9 0.05 0.5161 2,521 1,441 1,080 0.2416 3.189 3.431 3 0.05 0.4904 2,521 1,441 1,080 0.2416 3.189 3.431 3.1 0.05 0.4663 2,521 1,441 1,080 0.2416 3.189 3.431 3.2 0.05 0.4438 2,521 1,441 1,080 0.2416 3.189 3.431 3.3 0.05 0.4228 2,521 1,441 1,080 0.2416 3.189 3.431 3.4 0.05 0.4031 2,521 1,441 1,080 0.2416 3.189 3.431 3.5 Reference Anwar Adem, Richard Kneller, Cher Li (2023) Information constraints and technology efficiency: Field experiments benchmarking firms website performance. CESifo Working Paper 10457. https://www.cesifo.org/en/publications/2023/working-paper/information-constraints-and-technology-efficiency-field-experiments