Cultural institutions: The Effect of Physical and Digital Layouts on Visitor Experience (Part 2)

Last registered on June 26, 2022
View Trial History

Pre-Trial

Trial Information

General Information

Title
Cultural institutions: The Effect of Physical and Digital Layouts on Visitor Experience (Part 2)
RCT ID
AEARCTR-0009585
Initial registration date
June 22, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
June 26, 2022, 5:29 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country
Netherlands
Region
Amsterdam

Primary Investigator

Name
Abhishek Deshmane
Affiliation
IESE Business School
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Victor Martínez de Albéniz
PI Affiliation
IESE Business School
Contact Investigator
PI Name
Ali Aouad
PI Affiliation
London Business School
Contact Investigator

Additional Trial Information

Status
In development
Start date
2022-06-20
End date
2022-07-03
Keywords
Behavior, Education, Other
Additional Keywords
Museums, cultural operations, path analysis
JEL code(s)
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
A fundamental operational decision faced by museums and other major cultural institutions is the design of a display arrangement for their collection of items (artworks, objects, media contents), which in turn affects the visitors' interactions with the offered experiences. In this project, we collaborate with the Van Gogh Museum in Amsterdam to carry out experiments that help us causally identify potential links between layout-level decisions made by the experience providers and the visitor engagement, satisfaction levels, and learnings. The multimedia guides provided by the museum help us record visitors' movements and activities across time and space through granular event logs. Hence, the visitors navigate through both the physical layout, i.e. the arrangement of exhibits on display, and the digital layout, i.e. the tour of "Highlight" artworks recommended by the multimedia guide. We intend to analyse the interplay between these two types of content provision on the way visitors interact with the exhibits.

This study is the second part of our experimentation programme, where we test an optimized version of the Highlights tour displayed by the multimedia guide. Using a data-driven model and a simulation-based optimization algorithm, we identify a modified composition of the Highlights tour to increase the expected length of visitors' paths. Our intervention consists in removing three artworks from the Highlights tour and adding three other ones in replacement. We study the differences in the sequential choices made by the visitors across the treatment and control groups. The corresponding effect on the visitor engagement is measured, on a first level, through a Difference-in-Differences methodology; on a second level, we use a path-based Multinomial Logit model developed in previous research (paper in writing with same authors using historical observational data).
External Link(s)
Pre-registration for the preceding RCT study (part 1)

Registration Citation

Citation
Aouad, Ali, Abhishek Deshmane and Victor Martínez de Albéniz. 2022. "Cultural institutions: The Effect of Physical and Digital Layouts on Visitor Experience (Part 2)." AEA RCT Registry. June 26. https://doi.org/10.1257/rct.9585-1.0
Sponsors & Partners

Partner

Name
Van Gogh Museum
Type
none
URL
https://www.vangoghmuseum.nl/en
Experimental Details

Interventions

Intervention(s)
Intervention Start Date
2022-06-22
Intervention End Date
2022-07-03

Primary Outcomes

Primary Outcomes (end points)
We intend to track three different types of outcome variables. They are:

1. Visitor movement and clickstream activity: We track the path taken by the visitors across the museum during their visit using granular MMT logs. Based on this, we are able to build the following metrics:
a) Number of stops viewed/audio clips listened to (referred to as a "hit" hereinafter),
b) Number of hits per artwork,
c) Proportion of the audio clips listened to/abandonment,
c) Total visit time,
d) Time per artwork, and
e) Number of artworks skipped.

Note that from these individual outcomes, we can build our aggregate primary outcome measuring the hit rates of artworks on a day, which will be the focus of our initial DiD analysis.

2. Visitor satisfaction: For this, we set up an online survey that contains questions related to visit satisfaction. As a part of this project, we instated a system in the museum to nudge visitors to go online and fill out these surveys after finishing their visit. Through this survey, we are able to track:
a) Visitor overall satisfaction,
b) Quality of experience with respect to the multimedia guide, and
c) Willingness of the visitor to recommend the museum to their friends/colleagues/family.

3. Visitor learning: Finally, pedagogical goals are central to the operations of the museum. Our online survey also allows us to administer a quiz which the visitors can fill out towards the end of the visit. Four multiple choice questions regarding the artworks on display are asked to the visitors with the grades and correct answers made available after the conclusion of the survey and quiz.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The Van Gogh Museum provides multimedia guides (MMT) to its visitors, a standard offering across many museums in the world. The objective of MMTs is to enable visitors to engage with the artworks on display in an informed and interactive way. As a result of this, the visitors not only navigate the physical space in the museum, but also they also browse through the digital layout of the multimedia guide. Both types of content provision affect how visitors progress within the museum and their encounters with artworks. The MMT provides the recommendation of a 'Highlights tour' that covers the masterpieces of the museum, as well as a longer, more exploratory 'Leisure tour', which includes a number of additional artworks on display. In this study, we intend to understand the effect of the selection of artworks recommended by the MMT on visitor engagement.

Our previous research in collaboration with the Van Gogh Museum yielded a stochastic choice model to describe visitors' sequential exploration of the museum collection based on the MMT clickstream data. In this study, we leverage this model to construct an "optimized" Highlights tour based on counterfactual simulations. Our algorithm determines a subset of three artworks on the Highlights tour that are replaced by three other ones with the goal of maximising the expected number of artworks viewed by visitors, subject to various practical constraints imposed by the museum. We track the effect of this change is on visitor activity, measured by usage data of the MMT guide, and on visitor satisfaction levels, tracked through post-visit online surveys.
Experimental Design Details
Randomization Method
Due to the practical operational constraints faced by the museum and the presence of interference between simultaneous visitors, it is not possible to randomize the visitors' assignment to the different arms (i.e., compositions of Highlights tour) using a standard A/B test. Hence, we adopt a switchback design over two weeks, comprising one treatment and one control exposure, as explained above.

Each day is assigned to a condition: control (baseline MMT composition of the Highlights tour) and treatment according to a pre-determined schedule which remains unknown to the incoming visitors. Additionally, we construct a control group for our DiD analysis using the sample of visitors that exclusively utilise the Leisure tour, thereby remaining unaffected by our interventions on the Highlights tour.

Our switchback schedule comprises three weekly changes to the MMT guide, chosen to balance the day-of-week seasonality effects across the two arms. Note that since nearly all visitors make one-time visits throughout our study, there is no risk of temporal interference.
Randomization Unit
The outcomes are measured at the visitor session-level. Our analysis will be conducted with time-based clustered standard errors (two hours buckets and daily). We refer to the data analysis plan of Part 1 for a description of the specifications we will use in our analysis.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
There is no random assignment of the treatment conditions. Instead, we have constructed a switchback schedule over 14 days.
Sample size: planned number of observations
While the planned number of observations is contingent on the number of visitors that show up in the museum, based on the number of visitors recorded within a time period of two weeks of the study, we expect the following numbers: Visitor movement (audio guide-based): 1200 per day * 14 days = 16800 Visitor satisfaction (survey-based): 50 per day * 14 days = 700
Sample size (or number of clusters) by treatment arms
If C denotes the days with control condition, T are the days with the treatment condition, we expect sample size Num(.) to be:
Num(C) = Num(T) = 8400
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials

Documents

Document Name
IRB Protocol
Document Type
irb_protocol
Document Description
This document contains the project description, scope, and protocols submitted for the ethical review of research projects by the IRB of IESE Business School.
File
IRB Protocol

MD5: 108b03cd923bb6957257729946d7e6fd

SHA1: a9061ed72016c9f2d6d53716b512e45a98b79c2c

Uploaded At: June 13, 2022

IRB

Institutional Review Boards (IRBs)

IRB Name
Protocols submitted to IRB
IRB Approval Date
2022-03-31
IRB Approval Number
IESE.2022.09
Analysis Plan

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials