Show or Tell? Improving Agent Decision Making in a Tanzanian Mobile Money Field Experiment

Last registered on June 16, 2018

Pre-Trial

Trial Information

General Information

Title
Show or Tell? Improving Agent Decision Making in a Tanzanian Mobile Money Field Experiment
RCT ID
AEARCTR-0003080
Initial registration date
June 12, 2018

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
June 16, 2018, 12:50 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Primary Investigator

Affiliation
American University

Other Primary Investigator(s)

PI Affiliation
Harvard Business School
PI Affiliation
Harvard Business School
PI Affiliation
Pennsylvania State University

Additional Trial Information

Status
Completed
Start date
2016-01-01
End date
2016-10-17
Secondary IDs
Abstract
When workers make operational decisions, the firm's global knowledge and the worker's domain-specific knowledge complement each other. Oftentimes workers have the final decision-making power. Two key decisions a firm makes when designing systems to support these workers are: 1) what guidance to deliver, and 2) what kind of training (if any) to provide. We examine these choices in the context of mobile money platforms---systems that allow users in developing economies to deposit, transfer, and withdraw money using their mobile phones. Mobile money has grown quickly, but high stockout rates of currency persist due to sub-optimal inventory decisions made by contracted employees (called agents). In partnership with a Tanzanian mobile money operator, we perform a randomized controlled trial with 4,771 agents over eight weeks to examine how differing types of guidance and training impact the agents' inventory management. We find agents who are trained in person and receive an explicit, personalized, daily text message recommendation of how much electronic currency to stock are less likely to stock out. These agents are more likely to alter their electronic currency balance on a day (rebalance). In contrast, agents trained in person but who receive summary statistics of transaction volumes or agents who are notified about the program and not offered in-person training do not experience changes in stockouts or rebalances. We observe no evidence of learning or fatigue. Agent-level heterogeneity in the treatment effects shows that the agents who handle substantially more customer deposits than withdrawals benefit most from the intervention.
External Link(s)

Registration Citation

Citation
Acimovic, Jason et al. 2018. "Show or Tell? Improving Agent Decision Making in a Tanzanian Mobile Money Field Experiment." AEA RCT Registry. June 16. https://doi.org/10.1257/rct.3080-1.0
Former Citation
Acimovic, Jason et al. 2018. "Show or Tell? Improving Agent Decision Making in a Tanzanian Mobile Money Field Experiment." AEA RCT Registry. June 16. https://www.socialscienceregistry.org/trials/3080/history/30802
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Intervention Start Date
2016-08-23
Intervention End Date
2016-10-17

Primary Outcomes

Primary Outcomes (end points)
Float stockout: we define a float stockout event when the agent's float balance falls below the median size of a single CI transaction. This is a dummy variable for whether or not a float stockout event occurred on a day.
Rebalance: this is a dummy variable for whether the agent rebalanced at a bank or with a float runner on a day.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
In order to test which combination(s) of training and guidance leads to improvements for the agents, we ran a field experiment in partnership with our mobile money partner which has nearly 40,000 agents throughout Tanzania. In collaboration with our mobile money partner, we identified a subset of agents to include in the treatable population based on characteristics of the agents' transaction history during the pre-treatment period. From the population of over 16,000 agents in Dar es Salaam, we identified a treatable population of 4,771 agents by applying a set of filters before randomly assigning them to the treatment and control groups described below. In general, these filters ensure that agents had a minimum threshold of activity and experience, and also that agents' recommendations were consistent throughout the day so that they would not be harmed if they rebalanced at 1PM to the recommended levels intended for 8AM. See the paper for details on the exact filters applied.

The experiment proceeds as follows. First, 1,200 agents are randomly assigned to be treated in one of six ways with the remaining 3,571 in the control group. The six treatments come from three types of guidance provided via daily text message(s) delivered to the agent and two types of training provided. Guidance can be (1) information (Info), (2) recommendation (Rec), or (3) both (Both) information and recommendation. The training provided to the agents could be either (1) in-person (Train) or (2) a voice recording notification (Notify). Second, treated agents were invited to training and trained (if they chose to attend), or sent a voice recording notification depending on their treatment. Third, Phase One (August 23, 2016 to September 18, 2016) proceeds with treated agents receiving daily text messages at the end of the previous working day according to their guidance treatment. Finally, Phase Two (September 19, 2016 - October 17, 2016) proceeds with all agents receiving both information and recommendation text messages. We refer to the entire pre-treatment period as Phase Zero throughout the paper.

We comment here briefly on our notation. The control group is denoted Control. Each of the six treatment groups is denoted by two terms separated by a hyphen: {Train,Notify}-{Rec,Info,Both}. The first term denotes the type of training: Train if they were invited to in-person training or Notify if they were sent a voice notification text message. The second term denotes the type of guidance they received in Phase One: Rec if they received only an explicit recommendation in Phase One, Info if the received only the summary statistics of the 50th and 90th percentiles in Phase One, and Both if they received both a recommendation and information in Phase One. All treated agents received Both in Phase Two. For example, Train-Rec refers to agents who were invited to training that occurred on August 20th. These agents started receiving explicit recommendations on the evening of August 22nd (to be followed starting August 23rd). On the evening of September 18th, these agents began receiving both recommendation and information texts (as did every treated agent starting that evening).

Of the treated population, 750 were randomly assigned to the in-person training treatment with the remaining 450 receiving a voice recording notification. For the in-person training treatment, agents were invited by our mobile money partner to one of ten locations in Dar es Salaam. Our mobile money partner's trainers, all native Swahili speakers using presentation materials developed by the researchers (and translated into Swahili), spoke about the motivation of the DSS, ideas behind the recommendations, and how uncertainty plays a role in profits. The trainers were trained by the researchers in the days leading up to the training and two of the researchers visited the training locations on the day of training to make sure training proceeded as expected. Agents in the notification treatment were invited to call a pre-recorded message which explains that the program can help them, but does not go into as much depth as the in-person training. Text messages, the training presentation materials, and the questionnaire were all written by the researchers, translated from English into Swahili by a professional translation company in Dar es Salaam, and verified/approved by our mobile money partner both before and after translation.

Changing the guidance treatment in Phase Two was intended to be used as a test for whether there is a sequencing effect, i.e., are agents better off receiving first the information messages and then both or first receiving recommendation messages and then both or should they immediately receive both messages? This was based on a hypothesis that more guidance (information and recommendation) would be better than providing only one text message.
Experimental Design Details
Randomization Method
Randomization done via computer.
Randomization Unit
The randomization unit is an individual.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
4,771 agents
Sample size: planned number of observations
About 1.2 million agent-days
Sample size (or number of clusters) by treatment arms
3,571 control agents, 250 Train-Rec, 250 Train-Info, 250 Train-Both, 150 Notify-Rec, 150 Notify-Info, 150 Notify-Both
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
Pennsylvania State University IRB
IRB Approval Date
2016-04-21
IRB Approval Number
STUDY00004836

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials