How to cold call firms? An application of multi-armed bandit optimization in corporate web surveys

Last registered on July 26, 2022

Pre-Trial

Trial Information

General Information

Title
How to cold call firms? An application of multi-armed bandit optimization in corporate web surveys
RCT ID
AEARCTR-0009801
Initial registration date
July 25, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
July 26, 2022, 1:50 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
University of Mannheim

Other Primary Investigator(s)

PI Affiliation
University of Mannheim
PI Affiliation
University of Mannheim

Additional Trial Information

Status
In development
Start date
2022-08-01
End date
2024-03-15
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Previous experiments show that (even slight) differences in the design of invitation letters result in significant differences of starting rates between the respective groups. Typically, these studies rely on static experimental groups and inference that allows to determine the best performing invitation letter only ex-post. Trading off exploration and exploitation, we apply a reinforcement learning method in an adaptive experiment, as proposed by Scott (2010). Using multi-armed bandit optimization, specifically Thompson Sampling, we may compare a substantially larger set of alternatives. We show how Thomson Sampling can be implemented in the context of survey research conducted by the German Business Panel (GBP) and how accurate inference can be drawn. We aim to identify the distinct elements of a corporate web survey invitation letter that maximize survey starting rates.
External Link(s)

Registration Citation

Citation
Gaul, Johannes, Davud Rostam-Afschar and Thomas Simon. 2022. "How to cold call firms? An application of multi-armed bandit optimization in corporate web surveys." AEA RCT Registry. July 26. https://doi.org/10.1257/rct.9801-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
We conduct an adaptive experiment using Thompson Sampling instead of an experimental setup with static group composition. The main advantage of using a reinforcement learning algorithm is that insights about treatment effects are continuously exploited while the experiment is conducted, optimizing output (survey starting rates) already during the exploitation phase of the (sequential) experiment. In addition, such research design allows us to compare a larger number of treatments than in previously conducted experiments that consider a maximum of 16 different invitation letters (Kaplowitz et al., 2011).

Our research design varies five key elements of a survey invitation letter. Specifically, we consider (1) the personalization of invitation letters, (2) emphasizing the authority and status of the sender, (3) survey link placement, (4) stressing compliance with data protection, and (5) using a distinct request style by offering potential survey respondents the opportunity to share their opinion or pleading for help. We implement each of these factors in two different ways, resulting in a full-factorial experiment with a total of 32 (2^5) treatments.
Intervention (Hidden)
We analyze optimal survey invitation letter design in the context of the GBP. The GBP is part of the Collaborative Research Center TRR 266 “Accounting for Transparency” and continuously surveys all German firms listed in the GBP contact database, for which email contact information is available and which have not withdrawn from participation in previous survey rounds. Each survey round typically lasts half a year. During this time, the GBP transmits approximately 1,000,000 survey invitations to German firms. Since survey invitations are sent out continuously each workday during each survey wave, roughly 8,000 survey requests are transmitted per day. This steady stream of sent out invitations (instead of invitations sent out all at once) allows us to sequentially exploit potential treatment effects of different invitation letters.

We aim to conduct our experiment in the fifth survey round of the GBP, which is scheduled to start on August 01, 2022. For our adaptive experimental design, we implement a Thompson Sampling algorithm that represents a state-of-the-art application of multi-armed bandit optimi-zation using a Bayesian decision rule. This reinforcement learning algorithm considers realized treatment effects and dynamically determines treatment group sizes to maximize our main outcome variable, survey starting rates.

We implement a burn-in-phase of four weeks, during which group sizes are equal (and static) among the 32 different invitation letters. Such a burn-in-phase mitigates the concern of sampling error during early phases of the experiment (Kaibel and Biemann, 2021). Starting on August 29, 2022, and lasting for at least the remaining 20 weeks of the GBP survey wave, we activate the Thompson algorithm, calculating group weights through reinforcement learning as detailed in the research proposal. The algorithm is run not daily but in batches once a week, evaluating treatment effects that have been realized thus far and calculating group sizes for the next week of survey invitations to be sent.
Intervention Start Date
2022-08-29
Intervention End Date
2023-01-20

Primary Outcomes

Primary Outcomes (end points)
survey starting rate
Primary Outcomes (explanation)
survey starting rate is used for determining dynamic group sizes

Secondary Outcomes

Secondary Outcomes (end points)
- survey linking agreement,
- panel retention rate,
- response behavior to sensitive survey questions
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
Randomization is conducted via random allocation of subjects into 32 different treatment groups. Yet, group sizes might differ during the course of the experiment. Within the first four weeks of the survey period, allocation to all groups happens with the same frequency.

Following this burn-in-phase, each week the bandit algorithm evaluates realized treatment effects and dynamically determines group sizes. In doing so, the bandit algorithm prioritizes relatively strong treatments versus relatively weak ones, while still testing the latter. Weights for allocation (size of the treatment groups) are calculated according to the formula stated in the section “Sampling design” of the research proposal. In sum, subjects are allocated to treatment groups at random, however the probability of receiving a specific treatment is updated weekly and is equivalent to the weights calculated by the Thompson Sampling algorithm.
Experimental Design Details
Randomization Method
Randomization done by a computer with equal probability for each of 32 invitation letters in the burn-in-phase.

To determine treatment group sizes after the burn-in-phase of the experiment, we use Thompson Sampling. Each week, the algorithm updates prior beliefs about success probabilities using the hitherto realized sample and calculates sampling weights for each treatment group that will be applied for the next week of invitations that will be sent.

The algorithm
i) uses independent prior beliefs over the success probability for each treatment,
ii) updates the distribution using Bayes' rule (considering the number of successes and trials),
iii) determines the treatment with the highest success probability, and updates the expected success distribution.

See research proposal for more detailed information.
Randomization Unit
Firm
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
No clusters
Sample size: planned number of observations
Approximately 1,250,000 invitation letters will be sent, of which we expect approx. 1,000,000 to be received, 300,000 to be opened, 30,000 to start the survey, 13,500 to finish the survey, and 8,100 to be willing to receive a follow-up survey
Sample size (or number of clusters) by treatment arms
During burn-in-phase: approximately 6,250 per invitation letter of which a starting rate can be observed for 1,500.

Sample sizes are adaptively determined after start of Thompson sampling (see section "Intervention").
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials