|
Field
Trial Status
|
Before
in_development
|
After
completed
|
|
Field
Abstract
|
Before
The purpose of this experiment is to test the implementation of an adaptive experimental procedure designed by the authors in a real-world context.
The goal of many experiments is to inform the choice between different policies. However, standard experimental designs are geared toward point estimation and hypothesis testing. We consider the problem of treatment assignment in an experiment with several cross-sectional waves where the goal is to choose among a set of possible policies (treatments) for large-scale implementation. We show that optimal experimental designs learn from earlier waves by assigning more experimental units to the better-performing treatments in later waves. We propose a computationally tractable approximation of the optimal design, based on a modification of Thompson sampling.
We collaborate with a non-profit organization that supports smallholder farmers in developing countries by providing personalized agricultural advice through mobile phones in Odisha, India. We apply the proposed adaptive trial design to help the non-profit choose efficiently among several alternative call procedures (treatments) to enroll farmers into their service. Since the adaptive design is based on Bayesian optimization, it allocates more subjects to the most successful treatment arms in consecutive study waves, which improves learning and at the same time applies the more successful treatment variants earlier to more subjects.
|
After
The purpose of this experiment is to test the implementation of an adaptive experimental procedure designed by the authors in a real-world context.
The goal of many experiments is to inform the choice between different policies. However, standard experimental designs are geared toward point estimation and hypothesis testing. We consider the problem of treatment assignment in an experiment with several cross-sectional waves where the goal is to choose among a set of possible policies (treatments) for large-scale implementation. We show that optimal experimental designs learn from earlier waves by assigning more experimental units to the better-performing treatments in later waves. We propose a computationally tractable approximation of the optimal design, based on a modification of Thompson sampling.
We collaborate with a non-profit organization that supports smallholder farmers in developing countries by providing personalized agricultural advice through mobile phones in India. We apply the proposed adaptive trial design to help the non-profit choose efficiently among several alternative call procedures (treatments) to enroll farmers into their service. Since the adaptive design is based on Bayesian optimization, it allocates more subjects to the most successful treatment arms in consecutive study waves, which improves learning and at the same time applies the more successful treatment variants earlier to more subjects.
|
|
Field
Last Published
|
Before
June 24, 2019 05:00 PM
|
After
May 06, 2021 04:23 PM
|
|
Field
Study Withdrawn
|
Before
|
After
No
|
|
Field
Intervention Completion Date
|
Before
|
After
July 06, 2019
|
|
Field
Data Collection Complete
|
Before
|
After
Yes
|
|
Field
Final Sample Size: Number of Clusters (Unit of Randomization)
|
Before
|
After
10000
|
|
Field
Was attrition correlated with treatment status?
|
Before
|
After
No
|
|
Field
Final Sample Size: Total Number of Observations
|
Before
|
After
10000
|
|
Field
Final Sample Size (or Number of Clusters) by Treatment Arms
|
Before
|
After
903 (10 am, no SMS), 3931 (10 am, SMS 1hr ahead), 2234 (10 am, 24 hrs ahead), 366 (6:30 pm, no SMS), 1081 (6:30pm, SMS 1 hr ahead), 1485 (6:30pm, SMS 24hrs ahead)
|
|
Field
Public Data URL
|
Before
|
After
https://www.econometricsociety.org/content/supplement-adaptive-treatment-assignment-experiments-policy-choice
|
|
Field
Is there a restricted access data set available on request?
|
Before
|
After
No
|
|
Field
Program Files
|
Before
|
After
Yes
|
|
Field
Program Files URL
|
Before
|
After
https://www.econometricsociety.org/content/supplement-adaptive-treatment-assignment-experiments-policy-choice
|
|
Field
Data Collection Completion Date
|
Before
|
After
July 06, 2019
|
|
Field
Is data available for public use?
|
Before
|
After
Yes
|
|
Field
Intervention (Public)
|
Before
Precision Agriculture for Development (PAD) is testing interactive voice response (IVR) phone calls with the purpose to enroll as many rice farmers as possible into Odisha’s farmer information service. The main content of the call consists in a set of enrollment questions and does not vary across treatments.
The experiment consists in varying how these phone calls are initiated. Possible treatment variants considered involved text messages prior to the call to announce the call time, starting the call with a jingle, calling at different times of the day, etc.
The final choice of treatments selected by PAD for testing was:
- sending no text message to announce the call, sending a text message 1 hour before, and sending a text message 24 hours before;
- calling farmers at 10 am, and calling farmers at 7pm.
Text messaging and call times were cross-randomized for a total of 6 treatment arms.
Each study wave takes two days (one day to send text messages, one day to carry out calls).
|
After
Precision Agriculture for Development (PAD) is testing interactive voice response (IVR) phone calls with the purpose to enroll as many rice farmers as possible into a farmer information service. The main content of the call consists in a set of enrollment questions and does not vary across treatments.
The experiment consists in varying how these phone calls are initiated. Possible treatment variants considered involved text messages prior to the call to announce the call time, starting the call with a jingle, calling at different times of the day, etc.
The final choice of treatments selected by PAD for testing was:
- sending no text message to announce the call, sending a text message 1 hour before, and sending a text message 24 hours before;
- calling farmers at 10 am, and calling farmers at 7pm.
Text messaging and call times were cross-randomized for a total of 6 treatment arms.
Each study wave takes two days (one day to send text messages, one day to carry out calls).
|
|
Field
Experimental Design (Public)
|
Before
The study is conducted in Odisha, India, with farmers who are potential clients of Precision Agriculture for Development, an NGO that attempts to improve the lives of farmers by providing tailored farming advice.
Precision Agriculture for Development (PAD) is testing a variety of different interactive voice response (IVR) treatments. The purpose of these treatments is to enroll as many rice farmers as possible into Odisha’s farmer information service. PAD designed these treatments based on pilots and experience from other enrollment campaigns. The treatments are fully designed and implemented by PAD and their final choice was to test six distinct treatment arms (see above). The adaptive experimental procedure the authors designed helps PAD to choose the treatment that works best for their setting.
The experiment is conduced in waves of 600. The only information PAD has are a very large set of phone numbers provided by the government. An employee of PAD processes these numbers by querying if they are valid, and if they are on a do-not-disturb list. Only valid numbers not on the list are then used. PAD is setting aside 10'000 phone numbers that are up for calling in the target period for this experiment.
Experimental waves are carried out consecutively, where each wave takes two days to administer (text messages sent up to 24 hours ahead of time, two call times in morning and evening on the call day). In the first wave, 100 phone numbers are randomly selected from the set of 10'000 numbers and each is assigned randomly to one of the treatment arms. The outcomes of each experimental wave are submitted to an app programmed by the authors. The app uses treatment success rates in each treatment arm to update a flat prior on possible average success rates, and applies a modified Thompson sampling procedure to choose the number of phone numbers to be assigned to each treatment arm in the next wave. A new wave of 600 phone numbers are then randomly selected and assigned to treatment arms. Modified Thompson sampling is adapted from standard Thompson sampling for waves of size 1 by (a) reducing sampling variation by taking advantage of batch sampling (of 600 units per wave here), and (b) re-distributing experimental units from the best performing option to close competitors to improve learning. The experiment continues until 10'000 phone numbers are used.
|
After
The study is conducted in India, with farmers who are potential clients of Precision Agriculture for Development, an NGO that attempts to improve the lives of farmers by providing tailored farming advice.
Precision Agriculture for Development (PAD) is testing a variety of different interactive voice response (IVR) treatments. The purpose of these treatments is to enroll as many rice farmers as possible into the farmer information service. PAD designed these treatments based on pilots and experience from other enrollment campaigns. The treatments are fully designed and implemented by PAD and their final choice was to test six distinct treatment arms (see above). The adaptive experimental procedure the authors designed helps PAD to choose the treatment that works best for their setting.
The experiment is conduced in waves of 600. The only information PAD has are a very large set of phone numbers provided by an outside party. An employee of PAD processes these numbers by querying if they are valid, and if they are on a do-not-disturb list. Only valid numbers not on the list are then used. PAD is setting aside 10'000 phone numbers that are up for calling in the target period for this experiment.
Experimental waves are carried out consecutively, where each wave takes two days to administer (text messages sent up to 24 hours ahead of time, two call times in morning and evening on the call day). In the first wave, 100 phone numbers are randomly selected from the set of 10'000 numbers and each is assigned randomly to one of the treatment arms. The outcomes of each experimental wave are submitted to an app programmed by the authors. The app uses treatment success rates in each treatment arm to update a flat prior on possible average success rates, and applies the Exploration Sampling algorithm to choose the number of phone numbers to be assigned to each treatment arm in the next wave. A new wave of 600 phone numbers are then randomly selected and assigned to treatment arms. Exploration Sampling is adapted from standard Thompson sampling for waves of size 1 by (a) reducing sampling variation by taking advantage of batch sampling (of 600 units per wave here), and (b) re-distributing experimental units from the best performing option to close competitors to improve learning. The experiment continues until 10'000 phone numbers are used.
|
|
Field
Planned Number of Observations
|
Before
600 (or less) per wave for a total of 10'000 numbers.
|
After
600 (or less) per wave for a total of 10'000 numbers.
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
100 per wave. Number of waves is chosen as part of the experimental design during the trial. Final sample size approved by IRB may be up to 10'000.
|
After
100 per wave in expectation. Number of units in each arm is chosen as part of the experimental design during the trial. Final sample size approved by IRB may be up to 10'000.
|
|
Field
Keyword(s)
|
Before
Agriculture, Other
|
After
Agriculture, Other
|
|
Field
Public locations
|
Before
No
|
After
Yes
|