The Editor vs. the Algorithm: Targeting, Data and Externalities in Online News

Last registered on June 04, 2019
View Trial History

Pre-Trial

Trial Information

General Information

Title
The Editor vs. the Algorithm: Targeting, Data and Externalities in Online News
RCT ID
AEARCTR-0004264
Initial registration date
June 01, 2019

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
June 04, 2019, 12:14 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
June 04, 2019, 3:28 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country
Germany
Region

Primary Investigator

Name
Ananya Sen
Affiliation
Carnegie Mellon University
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Jorg Claussen
PI Affiliation
Contact Investigator
PI Name
Christian Peukert
PI Affiliation
Contact Investigator

Additional Trial Information

Status
Completed
Start date
2017-12-01
End date
2018-04-30
Keywords
Other
Additional Keywords
Algorithmic Recommendations, Returns to data, Filter Bubbles, Online News
JEL code(s)
Secondary IDs
Abstract
We run a large randomized eld experiment with a major news outlet in Germany to study the economics
of arti cial intelligence in the case of online news. We ask whether automated recommendation can
outperform a human editor in terms of user engagement, and how this relationship depends on the amount
of personal data that is available to the algorithm. In particular, we want to assess the economic returns to data for an algorithm.
We then investigate the unintended consequences of personalized recommendation, including potential changes in the distribution of news consumption across
article categories.
External Link(s)

Registration Citation

Citation
Claussen, Jorg, Christian Peukert and Ananya Sen. 2019. "The Editor vs. the Algorithm: Targeting, Data and Externalities in Online News." AEA RCT Registry. June 04. https://doi.org/10.1257/rct.4264-3.0
Former Citation
Claussen, Jorg, Christian Peukert and Ananya Sen. 2019. "The Editor vs. the Algorithm: Targeting, Data and Externalities in Online News." AEA RCT Registry. June 04. https://www.socialscienceregistry.org/trials/4264/history/47548
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Intervention Start Date
2017-12-05
Intervention End Date
2018-04-30

Primary Outcomes

Primary Outcomes (end points)
Clicks by the online readers of the webpage
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We team up with a major German news outlet to carry out a field experiment. The landing page of the news outlet's website is curated by a (human) editor. At each point in time, N articles are featured on the landing page. Their relative placement in one of the n slots may change over the course of a news cycle, and in response to user engagement.
In general, at a given point time, any user that arrives at the landing page sees the same content in the same placement. In the experiment, every time a users visits the landing page, she is randomly assigned to a control or treatment condition. If a user is assigned to a control condition then all the articles she observes on the landing page are the ones curated by the human editor. In the treatment condition, we customize the landing page by allowing a recommendation algorithm decide which of the N articles to be placed on a specific (fixed) slot n=4. The underlying model is trained on fine grained data about the past reading behavior of each individual user, starting with data from November 2017, with the objective of maximizing engagement of any user assigned to the treatment. The initial phase of the experiment has been successfully implemented and we have experimental data for a period of five months, from December 2017 to April 2018.
Experimental Design Details
Randomization Method
Randomization done with a software package used by the data science team of the partner company.
Randomization Unit
individual
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
Anyone who visits the homepage of the news outlet online is randomly assinged to a treatment or control group. Since this is a large news outlet, we would have several million individuals treated.
Sample size: planned number of observations
Randomization takes place in a continuous way, so it depends on how many users visit the page online. We expects millions of user-sessions.
Sample size (or number of clusters) by treatment arms
We have ended up with over 100 million user-sessions.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number
Analysis Plan

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials