The Value of Recommender Systems

Last registered on November 22, 2022
View Trial History

Pre-Trial

Trial Information

General Information

Title
The Value of Recommender Systems
RCT ID
AEARCTR-0007545
Initial registration date
May 11, 2021

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
May 14, 2021, 9:35 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
November 22, 2022, 9:50 AM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country
United States of America
Region

Primary Investigator

Name
Duarte Goncalves
Affiliation
UCL
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Guy Aridor
PI Affiliation
Columbia University
Contact Investigator
PI Name
Ruoyan Kong
PI Affiliation
University of Minnesota - Twin Cities
Contact Investigator
PI Name
Joseph Konstan
PI Affiliation
University of Minnesota - Twin Cities
Contact Investigator
PI Name
Daniel Kluver
PI Affiliation
University of Minnesota - Twin Cities
Contact Investigator

Additional Trial Information

Status
Completed
Start date
2021-03-29
End date
2021-10-31
Keywords
Other
Additional Keywords
Recommender Systems, Search Engines
JEL code(s)
D82, D83, L15, L86, D12
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
In this project we run a longitudinal field experiment in order to decompose the mechanisms that drive the influence recommendation systems have on consumption choices in terms of their informational and product discovery value.

Targeting a class of goods for which recommendations are ubiquitous -- movies --, we randomize the set of recommended movies on a movie-recommendation platform and compare outcomes to a control group of movies that holds fixed individuals' idiosyncratic preferences.
We test for whether recommendations meaningfully affect consumption patterns and beliefs about products.
External Link(s)

Registration Citation

Citation
Aridor, Guy et al. 2022. "The Value of Recommender Systems." AEA RCT Registry. November 22. https://doi.org/10.1257/rct.7545-2.0
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
Intervention Start Date
2021-03-29
Intervention End Date
2021-10-31

Primary Outcomes

Primary Outcomes (end points)
Consumption choices (movie watching activity), elicited beliefs (expected rating and associated degree of uncertainty).
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We randomize the selection of recommended movies presented to users as recommended on an internet platform in order to study whether and how recommendations impact users' consumption patterns and their beliefs about both the recommended items and non-recommended items. Our study follows a within-subjects design. We have two treatment arms and a control group -- one set of movies that are excluded from both recommendation and belief elicitation (the control arm), one set of movies that we elicit beliefs about but exclude them from recommendation, and one set of movies that we both elicit beliefs for and include in platform recommendations. The sets of movies are fixed at enrollment and assigned to a treatment arm according to a matched pairs procedure using the predicted rating for the movies generated by the platform's recommendation algorithm. See PDF for more details.
Experimental Design Details
Randomization Method
The movies are assigned to the treatment arms when we target the participant for enrollment and the script is executed on the MovieLens servers. The movies we elicit beliefs about and recommend movies recommended are selected by an algorithm introducing computer-generated independent noise.
See PDF for details.
Randomization Unit
Our experiment is a within-subjects experiment where we randomize the set of movies participants are exposed to and recommended into two treatment arms and a control arm.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
Our experiment is a within-subjects experiment.
Sample size: planned number of observations
We target 4507 participants of the platform that match our eligibility conditions and ask them to participate in the experiment. We aim for 1100 participants to opt into the study with an average of 20 survey responses per participant.
Sample size (or number of clusters) by treatment arms
Our experiment is within-subjects and for each participant we allocate 250 movies to each treatment arm.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials

Documents

Document Name
Project Overview
Document Type
other
Document Description
This document outlines the main hypotheses, the experimental details, and the analysis plan.
File
Project Overview

MD5: eb378a1975dc7dca7bef7ecd4571c2e0

SHA1: 8d63d194a151459bea820ecfcc7a46d296db0d30

Uploaded At: April 13, 2021

IRB

Institutional Review Boards (IRBs)

IRB Name
Columbia University
IRB Approval Date
2020-10-12
IRB Approval Number
AAAT2659
IRB Name
University of Minnesota - Twin Cities
IRB Approval Date
2020-09-25
IRB Approval Number
STUDY00010960
Analysis Plan

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
October 31, 2021, 12:00 +00:00
Data Collection Complete
Yes
Data Collection Completion Date
October 31, 2021, 12:00 +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
1,452 enrollment — 1007 participants completed at least one survey and enrolled for at least one week
Was attrition correlated with treatment status?
No
Final Sample Size: Total Number of Observations
1,452 enrollment — 1007 participants completed at least one survey and enrolled for at least one week
Final Sample Size (or Number of Clusters) by Treatment Arms
N/a (within-subject)
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials