|
Field
Abstract
|
Before
Information and search frictions in developing markets may induce agents to make distorted pricing and advertising choices, which in turn may generate further negative externalities that reinforce such frictions. While direct effects of information interventions to alleviate some of these frictions are extensively studied (e.g. Aker 2010; Aker and Mbiti 2010; Jensen 2007), potential spillovers and their mechanisms remain relatively under-explored. We wish to provide empirical insights into the ways in which new, individual price information in a developing economy alters market conditions, or ways in which the effects of new information are altered by such market conditions. To this end, we will conduct a private price information intervention to sellers on a listing platform for used vehicles in Pakistan, PakWheels.com, from June to August 2021. We will generate variation in the timing of treatment by market sub-sections via a two-step cluster-randomization design, in order to separate direct treatment effects from spillovers. In our primary analysis, we will measure direct and spillover effects on a) changes to the listing price, b) occurrence of transaction, c) transaction price, d) an advertising index, and e) page views as a proxy for buyer interest. In our secondary analysis, we will identify ways in which the treatment effects interact with, or in turn affect, market efficiency and structure.
|
After
In developing markets facing information and search frictions, how do agents access price information, update their beliefs about market conditions, and make pricing and other strategic choices? Also in such environments, what are the extent to which individual agents' access to information and their choices generate spillovers to other market participants? We explore these questions via a randomized control trial on a major online listing platform for used vehicles in Pakistan, where, along with other developing economies, increasing shares of transactions are shifting to online and mobile platforms. In our intervention we will provide estimates of transaction prices to sellers on a listing platform for used vehicles in Pakistan from February to April, 2022. We vary treatment saturation at the market-segment level by a two-stage randomization design so as to capture both direct and spillover effects. We measure the effect of providing private price information to sellers on their choices and outcomes, and capture spillover effects on competing sellers. In our primary analysis, we will detect direct and spillover effects on a) changes to the listing price, b) occurrence of transaction, c) transaction price, d) usage of advertising tools, and e) index of buyer attention. In our secondary analysis, we will identify ways in which the intervention interacts with, or in turn affects, market efficiency and structure.
|
|
Field
Trial End Date
|
Before
September 20, 2021
|
After
May 31, 2022
|
|
Field
Last Published
|
Before
June 14, 2021 11:35 AM
|
After
February 17, 2022 09:15 PM
|
|
Field
Intervention Start Date
|
Before
June 14, 2021
|
After
February 21, 2022
|
|
Field
Intervention End Date
|
Before
August 09, 2021
|
After
April 17, 2022
|
|
Field
Experimental Design (Public)
|
Before
We have designed a blocked, two-step randomization procedure to identify both direct treatment and spillover effects of the price-information intervention. This intervention will be carried out by the firm over 9 weeks.
The platform receives approximately 100,000 valid listings per month. Our sample is every seller that creates a post on the platform during the intervention period, except if we do not have sufficient data to provide a Price Calculator estimate for a given make-models/make-model-model-year. We define these exclusion criteria to be models that have self-reported transaction prices (which we use to provide the Machine-Learning price forecast) from at fewer than 10 listings per month, and to make-model-years with fewer than 3.33 reported transaction prices per month (which we use to provide information on the bounds). These criteria restrict our sample to approximately 88% of total posts, consisting of 73 distinct make-models.
Our two-step randomization process is as follows. In step 1, we will block-randomize the “cluster” of vehicles, defined as the make-model (e.g. Toyota Corolla), into 9 treatment groups, corresponding to the week in which treatment is given to some in the cluster. In step 2, we randomize posts into treatment based on the last digit of the user ID on PakWheels. In order to ensure that treatment and control groups are comparable over the primary outcome variables, we test for balance using listings data from a pre-treatment period with the same sample inclusion criteria as the experiment. We iterate the randomization procedure over 500 seeds, and randomly choose one seeds among the subgroup of assignments that failed to reject statistically significant differences in all primary outcome variables, adjusted for the false discover rate.
In step 1, we block the make-model clusters over standardized cluster-level means of primary outcome variables. Blocking is done with R’s blockTools package (Moore, 2012), which uses the optimal-greedy algorithm over the Mahalambois distance. We weight the five main outcome variables (log of price difference, occurrence of transaction, transaction price, advertising use, and demand index) twice as heavily as the cluster size. Our choice of weights is admittedly arbitrary, but we rationalize this as our objective is to balance over main outcome variables. Using these groupings, we will identify the first-stage assignment, i.e. the week in which some listings in the cluster is assigned treatment. In “Treat” clusters, we will randomly select 50% of new listings to receive the Price Calculator estimates, while none of the new listings in the “Control” clusters will receive them. On the 9th week, all new listings will receive the Price Calculator.
In step 2, listing-level randomization is done based on the last digit of their user-ID on PakWheels. The choice of 5 digits is fixed across cluster and time, so as to limit the extent of potential interference and for logistical simplicity. The second-stage treatment digits are chosen based on a random number generator.
The intervention is designed in a way that limits the extent of treatment non-compliance as much as possible; for those randomly assigned treatment, the Price Calculator estimate is given automatically on the user interface during the post-making process. One exception is
that the sellers that use PakWheels’ mobile app can only receive Price Calculator estimates once they update their mobile app after the start of the intervention period. On the other hand, there is no need for an update if the user accesses PakWheels.com via the web (including internet browsers on mobile phones), so anyone assigned to treatment would receive it automatically. This may generate selection into treatment conditional on assignment, based on a) users’ preference for PakWheels’ app and b) their propensity to update the app. To this end, we will plan on identifying both intend-to-treat as well as treatment-on-treated effects. We will use the assignment variable as an instrument for treatment.
|
After
We have designed a two-step randomization procedure to identify both direct treatment and spillover effects of the price-information intervention. This intervention will be carried out by the firm over 8 weeks.
The platform receives approximately up to 100,000 valid listings per month. Our sample is every seller that creates a post on the platform during the intervention period, except if we do not have sufficient data to provide a Price Calculator estimate for a given make-models/make-model-model-year. We define these exclusion criteria to based on sufficient availability of self-reported transaction prices. These criteria restrict our sample to approximately 90% of total posts, consisting of approximately 70 distinct make-models.
Our two-step randomization process is as follows. In step 1, we will block-randomize the “cluster” of vehicles, defined as the make-model (e.g. Toyota Corolla), into 3 treatment groups: control, medium saturation (50%) and high saturation (90%). In step 2, we randomize posts into treatment based on the last digit of the user ID on PakWheels. In order to ensure that treatment and control groups are comparable over the primary outcome variables, we test for balance using listings data from a pre-treatment period with the same sample inclusion criteria as the experiment. We iterate the randomization procedure over 500 seeds, and randomly choose one seeds among the subgroup of assignments that failed to reject statistically significant differences in all primary outcome variables, adjusted for the false discover rate.
In step 1, we block the make-model clusters over standardized cluster-level means of primary outcome variables. Blocking is done with R’s blockTools package (Moore, 2012), which uses the optimal-greedy algorithm over the Mahalambois distance. We weight the five main outcome variables (log of price difference, occurrence of transaction, transaction price, advertising use, and demand index) twice as heavily as the cluster size. Our choice of weights is admittedly arbitrary, but we rationalize this as our objective is to balance over main outcome variables. Using these groupings, we will identify the first-stage assignment. In treatment clusters, we will randomly select 50% or 90% of new listings to receive the Price Calculator estimates, depending the saturation level the cluster was assigned to. None of the new listings in the “Control” clusters will receive the Price Calculator estimates. On the 9th week, all new listings will receive the Price Calculator.
In step 2, listing-level randomization is done based on the last digit of their user-ID on PakWheels. The choice of 5 or 9 digits is fixed across cluster and time, so as to limit the extent of potential interference and for logistical simplicity. The second-stage treatment digits are chosen based on a random number generator in R.
The intervention is designed in a way that limits the extent of treatment non-compliance as much as possible; for those randomly assigned treatment, the Price Calculator estimate is given automatically on the user interface during the post-making process. One exception is
that the sellers that use PakWheels’ mobile app can only receive Price Calculator estimates once they update their mobile app after the start of the intervention period. On the other hand, there is no need for an update if the user accesses PakWheels.com via the web (including internet browsers on mobile phones), so anyone assigned to treatment would receive it automatically. This may generate selection into treatment conditional on assignment, based on a) users’ preference for PakWheels’ app and b) their propensity to update the app. To this end, we will plan on identifying both intend-to-treat as well as treatment-on-treated effects. We will use the assignment variable as an instrument for treatment.
|
|
Field
Randomization Method
|
Before
As described in the Experimental Design section, the first-stage is cluster-randomized based on outcome means at the make-model level, and the second stage randomization is done by a random number generator in R.
|
After
As described in the Experimental Design section, the first-stage is randomization into three groups (control and two saturation levels) based on outcome means at the make-model level. The second stage randomization is done by a random number generator in R.
|
|
Field
Randomization Unit
|
Before
The first stage is clustered at the make-model level, and second stage is randomized based on the last digit of the user ID.
|
After
The first stage is at the make-model level, and second stage is the seller level, based on the last digit of the user ID.
|
|
Field
Planned Number of Clusters
|
Before
We expect to have 73 clusters (i.e. the make-models), but this is dependent on for how many make-models we would be able to offer Price Calculator estimates.
|
After
We expect to have up to 70 clusters (i.e. the make-models), but this is dependent on for how many make-models we would be able to offer Price Calculator estimates.
|
|
Field
Planned Number of Observations
|
Before
We expect to have a sample of 90,000 to 140,000, depending on the number of new posts over the treatment period that meet the sample inclusion criteria.
|
After
We expect to have a sample of 140,000 to 200,000, depending on the number of new posts over the treatment period that meet the sample inclusion criteria.
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
Approximately 25% (22,500-35,000) of posts will be directly treated, and another 25% will be in the spillover group. Remaining 50% will be the control group.
|
After
Supposing we have 200,000 posts during our experimental period, approximately 70,000 posts will be directly treated, and 30,000 will be in the spillover group. Remaining 100,000 will be the control group.
|
|
Field
Power calculation: Minimum Detectable Effect Size for Main Outcomes
|
Before
Through simulations using real data from PakWheels, we find that we are able to detect treatment and spillover effects of 0.05 times the standard deviation for most primary outcome groups, both direct and spillover, with over 80% probability. The exception is the spillover effect of the binary transaction outcome, where we have sufficiently high power for an effect size just above 0.1 SD. These effect sizes translate roughly into 4,900 PKR (32 USD) in absolute difference between listing price and Price Calculator estimate (level mean: 138,000 PKR), 5 percentage-points in transaction probability (mean: 0.42), and 49,000 PKR (320 USD) in transaction price (level mean: 1,735,707 PKR).
|
After
Through simulations using real data from PakWheels, we find that we are able to detect treatment and some spillover effects of 0.05 times the standard deviation for most primary outcome groups, both direct and spillover, with over 80% probability. The effect size of 0.05 SD translates into 11,594 PKR (65.73 USD at 176.4 PKR to USD) in absolute difference between listing price and Price Calculator estimate (level mean: 305,434 PKR), 2.44 percentage-points in transaction probability (mean: 0.394), and 55,473 PKR (314.47 USD) in transaction price (level mean: 1,893,626 PKR). Further detail is provided in the pre-analysis plan.
|
|
Field
Keyword(s)
|
Before
Firms And Productivity
|
After
Agriculture, Firms And Productivity
|
|
Field
Pi as first author
|
Before
No
|
After
Yes
|
|
Field
Building on Existing Work
|
Before
|
After
No
|