Experimental Design
In order to test which combination(s) of training and guidance leads to improvements for the agents, we ran a field experiment in partnership with our mobile money partner which has nearly 40,000 agents throughout Tanzania. In collaboration with our mobile money partner, we identified a subset of agents to include in the treatable population based on characteristics of the agents' transaction history during the pre-treatment period. From the population of over 16,000 agents in Dar es Salaam, we identified a treatable population of 4,771 agents by applying a set of filters before randomly assigning them to the treatment and control groups described below. In general, these filters ensure that agents had a minimum threshold of activity and experience, and also that agents' recommendations were consistent throughout the day so that they would not be harmed if they rebalanced at 1PM to the recommended levels intended for 8AM. See the paper for details on the exact filters applied.
The experiment proceeds as follows. First, 1,200 agents are randomly assigned to be treated in one of six ways with the remaining 3,571 in the control group. The six treatments come from three types of guidance provided via daily text message(s) delivered to the agent and two types of training provided. Guidance can be (1) information (Info), (2) recommendation (Rec), or (3) both (Both) information and recommendation. The training provided to the agents could be either (1) in-person (Train) or (2) a voice recording notification (Notify). Second, treated agents were invited to training and trained (if they chose to attend), or sent a voice recording notification depending on their treatment. Third, Phase One (August 23, 2016 to September 18, 2016) proceeds with treated agents receiving daily text messages at the end of the previous working day according to their guidance treatment. Finally, Phase Two (September 19, 2016 - October 17, 2016) proceeds with all agents receiving both information and recommendation text messages. We refer to the entire pre-treatment period as Phase Zero throughout the paper.
We comment here briefly on our notation. The control group is denoted Control. Each of the six treatment groups is denoted by two terms separated by a hyphen: {Train,Notify}-{Rec,Info,Both}. The first term denotes the type of training: Train if they were invited to in-person training or Notify if they were sent a voice notification text message. The second term denotes the type of guidance they received in Phase One: Rec if they received only an explicit recommendation in Phase One, Info if the received only the summary statistics of the 50th and 90th percentiles in Phase One, and Both if they received both a recommendation and information in Phase One. All treated agents received Both in Phase Two. For example, Train-Rec refers to agents who were invited to training that occurred on August 20th. These agents started receiving explicit recommendations on the evening of August 22nd (to be followed starting August 23rd). On the evening of September 18th, these agents began receiving both recommendation and information texts (as did every treated agent starting that evening).
Of the treated population, 750 were randomly assigned to the in-person training treatment with the remaining 450 receiving a voice recording notification. For the in-person training treatment, agents were invited by our mobile money partner to one of ten locations in Dar es Salaam. Our mobile money partner's trainers, all native Swahili speakers using presentation materials developed by the researchers (and translated into Swahili), spoke about the motivation of the DSS, ideas behind the recommendations, and how uncertainty plays a role in profits. The trainers were trained by the researchers in the days leading up to the training and two of the researchers visited the training locations on the day of training to make sure training proceeded as expected. Agents in the notification treatment were invited to call a pre-recorded message which explains that the program can help them, but does not go into as much depth as the in-person training. Text messages, the training presentation materials, and the questionnaire were all written by the researchers, translated from English into Swahili by a professional translation company in Dar es Salaam, and verified/approved by our mobile money partner both before and after translation.
Changing the guidance treatment in Phase Two was intended to be used as a test for whether there is a sequencing effect, i.e., are agents better off receiving first the information messages and then both or first receiving recommendation messages and then both or should they immediately receive both messages? This was based on a hypothesis that more guidance (information and recommendation) would be better than providing only one text message.