The impact of qualitative reviews in online markets: Empirical and experimental evidence on statistical discrimination

2026-03-30

2026-07-31

Whether target property was chosen

Each participant will be presented with 33 pairs of fictitious properties across three budget blocks (Low, Mid, High) and asked to select their most preferred property in each pair. Out of these 33 rounds, 28 are the primary experimental rounds containing the fully counterbalanced 2x2x2 attribute variations. The remaining 5 rounds are fixed filler/attention-check rounds. For the analysis, the data from the 28 experimental rounds will be reshaped into a "long format," resulting in 56 observations per participant (two competing properties per round). Our primary outcome is a binary indicator (0 or 1) for whether a specific property variant was selected by the participant.

General structure of the experiment

The experiment will run online on Prolific. After obtaining informed consent, participants will first report details about their most recent rental experience. The main task consists of 33 rounds divided into three budget blocks. In each round, participants are presented with two fictitious properties and must choose the one they would prefer to rent. To encourage truthful revelation of preferences and mitigate social desirability bias, participants are incentivised with bonus payments based on how closely their choices align with the modal choice (the option most frequently selected by other participants). The experiment concludes with a post-experimental survey comprising an Implicit Association Test to measure implicit biases and standard demographic questions.

Treatments

Our experimental design consists of three between-participant treatments. In all treatments, we use a within-subjects orthogonal design where participants evaluate 28 pairs of properties across three primary dimensions.
• Treatment 1 (Quantity): We vary the host race (minority/non-minority), host gender (man/woman), and review quantity (low/high, keeping the quality of reviews fixed).
• Treatment 2 (Positive Informativeness): We vary the host race, host gender, and the informativeness of reviews (low/high, keeping the number of reviews fixed) when all reviews are positive.
• Treatment 3 (Negative Informativeness): We vary the host race, host gender, and the informativeness of reviews (low/high, keeping the number of reviews fixed) when one of the reviews is negative.

Participants are randomly assigned to one of the three treatments. To prevent participants from seeing the exact same property twice while ensuring that all combinations of traits are tested against each other, the 28 experimental rounds are constructed using a predefined "property map". To achieve perfect counterbalancing, participants within each treatment are randomly assigned to one of 56 distinct participant types. This rotation ensures that property attributes are orthogonal to the round number and budget tier, minimising order effects.

Not available

Computerized randomization

Individual randomisation

No

1,200 individuals (400 per treatment arm)

Because participants make 28 valid experimental choices evaluated in pairs, the effective number of observations is 400 * 28 * 2 = 22,400 observations per treatment (67,200 total).

400 individuals per treatment

We base our sample size on the hardest to detect main effect identified in our pilot (the demographic penalties, which showed a roughly 3.0% difference in selection rates). We simulate data using the 28-round, long-format structure and the group means/standard deviations from the pilot. By running the full LPM on synthetic datasets across a grid of sample sizes, we determine the number of participants required to achieve 90% statistical power to detect a main effect coefficient of 0.032 at α = 0.05 to be N=400 per treatment arm.