Back to History Current Version

Adaptive Treatment Assignment for Policy Choice: Phone Enrollment for an Agricultural Extension Service

Last registered on June 24, 2019
View Trial History

Pre-Trial

Trial Information

General Information

Title
Adaptive Treatment Assignment for Policy Choice: Phone Enrollment for an Agricultural Extension Service
RCT ID
AEARCTR-0004263
Initial registration date
June 02, 2019

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
June 04, 2019, 11:28 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
June 24, 2019, 5:00 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country
India
Region
Odisha

Primary Investigator

Name
Anja Sautmann
Affiliation
World Bank
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Maximilian Kasy
PI Affiliation
Harvard University
Contact Investigator

Additional Trial Information

Status
In development
Start date
2019-06-03
End date
2019-08-31
Keywords
Agriculture, Other
Additional Keywords
adaptive treatment assignment, adaptive sampling, optimal policy choice
JEL code(s)
C93, C11, O22, O13
Secondary IDs
Abstract
The purpose of this experiment is to test the implementation of an adaptive experimental procedure designed by the authors in a real-world context.
The goal of many experiments is to inform the choice between different policies. However, standard experimental designs are geared toward point estimation and hypothesis testing. We consider the problem of treatment assignment in an experiment with several cross-sectional waves where the goal is to choose among a set of possible policies (treatments) for large-scale implementation. We show that optimal experimental designs learn from earlier waves by assigning more experimental units to the better-performing treatments in later waves. We propose a computationally tractable approximation of the optimal design, based on a modification of Thompson sampling.
We collaborate with a non-profit organization that supports smallholder farmers in developing countries by providing personalized agricultural advice through mobile phones in Odisha, India. We apply the proposed adaptive trial design to help the non-profit choose efficiently among several alternative call procedures (treatments) to enroll farmers into their service. Since the adaptive design is based on Bayesian optimization, it allocates more subjects to the most successful treatment arms in consecutive study waves, which improves learning and at the same time applies the more successful treatment variants earlier to more subjects.

External Link(s)

Registration Citation

Citation
Kasy, Maximilian and Anja Sautmann. 2019. "Adaptive Treatment Assignment for Policy Choice: Phone Enrollment for an Agricultural Extension Service." AEA RCT Registry. June 24. https://doi.org/10.1257/rct.4263-2.0
Former Citation
Kasy, Maximilian and Anja Sautmann. 2019. "Adaptive Treatment Assignment for Policy Choice: Phone Enrollment for an Agricultural Extension Service." AEA RCT Registry. June 24. https://www.socialscienceregistry.org/trials/4263/history/48639
Sponsors & Partners

Partner

Name
Precision Agriculture for Development
Type
ngo
URL
https://precisionag.org/

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Precision Agriculture for Development (PAD) is testing interactive voice response (IVR) phone calls with the purpose to enroll as many rice farmers as possible into Odisha’s farmer information service. The main content of the call consists in a set of enrollment questions and does not vary across treatments.
The experiment consists in varying how these phone calls are initiated. Possible treatment variants considered involved text messages prior to the call to announce the call time, starting the call with a jingle, calling at different times of the day, etc.
The final choice of treatments selected by PAD for testing was:
- sending no text message to announce the call, sending a text message 1 hour before, and sending a text message 24 hours before;
- calling farmers at 10 am, and calling farmers at 7pm.
Text messaging and call times were cross-randomized for a total of 6 treatment arms.
Each study wave takes two days (one day to send text messages, one day to carry out calls).
Intervention Start Date
2019-06-03
Intervention End Date
2019-06-30

Primary Outcomes

Primary Outcomes (end points)
Successful call completion: binary variable describing whether the call recipient answered five IVR questions asked during the call (answers to these questions enable processing).
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The study is conducted in Odisha, India, with farmers who are potential clients of Precision Agriculture for Development, an NGO that attempts to improve the lives of farmers by providing tailored farming advice.
Precision Agriculture for Development (PAD) is testing a variety of different interactive voice response (IVR) treatments. The purpose of these treatments is to enroll as many rice farmers as possible into Odisha’s farmer information service. PAD designed these treatments based on pilots and experience from other enrollment campaigns. The treatments are fully designed and implemented by PAD and their final choice was to test six distinct treatment arms (see above). The adaptive experimental procedure the authors designed helps PAD to choose the treatment that works best for their setting.
The experiment is conduced in waves of 600. The only information PAD has are a very large set of phone numbers provided by the government. An employee of PAD processes these numbers by querying if they are valid, and if they are on a do-not-disturb list. Only valid numbers not on the list are then used. PAD is setting aside 10'000 phone numbers that are up for calling in the target period for this experiment.
Experimental waves are carried out consecutively, where each wave takes two days to administer (text messages sent up to 24 hours ahead of time, two call times in morning and evening on the call day). In the first wave, 100 phone numbers are randomly selected from the set of 10'000 numbers and each is assigned randomly to one of the treatment arms. The outcomes of each experimental wave are submitted to an app programmed by the authors. The app uses treatment success rates in each treatment arm to update a flat prior on possible average success rates, and applies a modified Thompson sampling procedure to choose the number of phone numbers to be assigned to each treatment arm in the next wave. A new wave of 600 phone numbers are then randomly selected and assigned to treatment arms. Modified Thompson sampling is adapted from standard Thompson sampling for waves of size 1 by (a) reducing sampling variation by taking advantage of batch sampling (of 600 units per wave here), and (b) re-distributing experimental units from the best performing option to close competitors to improve learning. The experiment continues until 10'000 phone numbers are used.
Experimental Design Details
Randomization Method
Computer-based randomization.
Randomization Unit
Individual phone number.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
600 (or less) per wave up to a total of 10'000 numbers.
Sample size: planned number of observations
600 (or less) per wave for a total of 10'000 numbers.
Sample size (or number of clusters) by treatment arms
100 per wave. Number of waves is chosen as part of the experimental design during the trial. Final sample size approved by IRB may be up to 10'000.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
N/A - samples size is pre-determined and treatments are evaluated based on expected regret.
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
Massachusetts Institute of Technology, Committee on the Use of Humans as Experimental Subjects (COUHES)
IRB Approval Date
2019-04-09
IRB Approval Number
1903763252
Analysis Plan

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials