Accounting for Heuristics in Reputation Systems: An Interdisciplinary Approach on Aggregation Processes

Last registered on June 21, 2023

Pre-Trial

Trial Information

General Information

Title
Accounting for Heuristics in Reputation Systems: An Interdisciplinary Approach on Aggregation Processes
RCT ID
AEARCTR-0003656
Initial registration date
December 09, 2018

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
December 20, 2018, 10:14 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
June 21, 2023, 2:52 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Region

Primary Investigator

Affiliation
Paderborn University

Other Primary Investigator(s)

PI Affiliation
Paderborn University

Additional Trial Information

Status
On going
Start date
2016-12-19
End date
2023-12-31
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Aggregation metrics in reputation systems are important for overcoming information overload. When using
these metrics, technical aggregation functions such as the arithmetic mean are implemented to measure the
valence of product ratings. However, it is unclear whether the implemented aggregation functions match the
inherent aggregation patterns of customers. In our experiment, we elicit customers’ aggregation heuristics
and contrast these with reference functions. Our findings indicate that, overall, the arithmetic mean performs
best in comparison with other aggregation functions. However, our analysis on an individual level reveals
heterogeneous aggregation patterns. Major clusters exhibit a binary bias (i.e., an over-weighting of moderate
ratings and under-weighting of extreme ratings) in combination with the arithmetic mean. Minor clusters
focus on 1-star ratings or negative (i.e., 1-star and 2-star) ratings. Thereby, inherent aggregation patterns
are neither affected by variation of provided information nor by individual characteristics such as experience,
risk attitudes, or demographics.
External Link(s)

Registration Citation

Citation
Djawadi, Behnud and Dirk van Straaten. 2023. "Accounting for Heuristics in Reputation Systems: An Interdisciplinary Approach on Aggregation Processes." AEA RCT Registry. June 21. https://doi.org/10.1257/rct.3656-2.5
Former Citation
Djawadi, Behnud and Dirk van Straaten. 2023. "Accounting for Heuristics in Reputation Systems: An Interdisciplinary Approach on Aggregation Processes." AEA RCT Registry. June 21. https://www.socialscienceregistry.org/trials/3656/history/183373
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Intervention (Hidden)
Intervention Start Date
2018-12-11
Intervention End Date
2018-12-12

Primary Outcomes

Primary Outcomes (end points)
Ranking of customer rating distributions
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We conduct a randomized controlled trial with a control group (baseline). In both groups, we run twelve periods containing three customer rating distributions each. For the purpose of incentivization we use real customer rating distributions in three periods whereby participants have the chance to win the associated products. These customer rating distributions have the unintended characteristic that at least one stochastically dominates at least one distribution out of this decision set. In the other periods subjects rank artificial customer rating distributions that have the feature of no (first-order) stochastic dominance.
Since it is crucial for our machine learning approach and its underlying assumptions,we plan to exclude decisions that contain stochastically dominated customer rating distributions for this part of analysis.
Experimental Design Details
Randomization Method
Randomization of the decision sets is done by a computer. E.g., decision set A is ranked by participant x in period k and by participant y in period f. In addition, the order the distributions are displayed (i.e., left, middle, right) is randomized by the computer.
Randomization Unit
individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
2 treatments: baseline (only customer rating distributions displayed) and information treatment (additionally, relative frequencies and the arithmetic mean displayed)
Sample size: planned number of observations
112 participants
Sample size (or number of clusters) by treatment arms
56 participants per treatment
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials