Back to History Current Version

Timely Feedback and Toxic Communication

Last registered on October 06, 2025
View Trial History

Pre-Trial

Trial Information

General Information

Title
Timely Feedback and Toxic Communication
RCT ID
AEARCTR-0016553
Initial registration date
October 04, 2025

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
October 06, 2025, 3:20 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country
United States of America
Region

Primary Investigator

Name
Huen Tat Au-Yeung
Affiliation
Institute for International Economic Studies, Stockholm University
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Jinci Liu
PI Affiliation
Institute for International Economic Studies, Stockholm University
Contact Investigator

Additional Trial Information

Status
In development
Start date
2025-08-04
End date
2025-12-15
Keywords
Behavior, Crime, Violence, & Conflict, Firms & Productivity, Labor
Additional Keywords
JEL code(s)
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
We detect toxic messages on a global online collaboration platform on a regular basis. When a message is classified as toxic, we provide a reminder about the toxicity of the message to some of the senders.
External Link(s)

Registration Citation

Citation
Au-Yeung, Huen Tat and Jinci Liu. 2025. "Timely Feedback and Toxic Communication." AEA RCT Registry. October 06. https://doi.org/10.1257/rct.16553-1.0
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
Intervention (Hidden)
Every hour, we apply the Detoxify model (Hanu and Unitary, 2020) to identify messages with predicted toxicity above 0.5. Flagged messages are validated by GPT-4o to reduce false positives. Users who send their first toxic message during the four-week intervention period are randomly assigned to one of three arms:
(1) Private reminder: sent by email via platform notification setting (reminder is posted in a short-lived thread that tags the sender of the flagged message), visible only to the sender;
(2) Public reminder: posted in the same thread as the flagged message;
(3) Control: no reminder.
Intervention Start Date
2025-10-06
Intervention End Date
2025-11-03

Primary Outcomes

Primary Outcomes (end points)
Individual outcomes up to 6 weeks after the intervention: features of messages, activities.
Primary Outcomes (explanation)
Features of the messages: We focus on three features of messages: toxicity, positivity, and constructiveness. The toxicity classifier follows the same method as in the intervention, while positivity and constructiveness are identified with large language models.
Activities: We split activities for communication-related (comments) and productivity-related (lines of code, pull requests, commits).

Secondary Outcomes

Secondary Outcomes (end points)
Measures of collaboration and popularity, language of messages within the repository, up to 6 weeks after the intervention.
Willingness to subscribe (WTS) in the survey for the premium if toxicity detection is added.
Secondary Outcomes (explanation)
Team performance: measured at the repository level, including the number of stars, forks, and unique contributors.
WTS is given in intervals of >10%, 5-10%, and 1-5% for both increase and decrease, and no change. The midpoint is used for the intervals. For changes >10%, we use 12.5%

Experimental Design

Experimental Design
We detect potentially toxic messages using a state-of-the-art classifier and send reminders to a random subset of their senders. The reminders inform users that their messages may appear rude or disrespectful. We track changes in feedback composition and activity for up to six weeks after intervention.
Experimental Design Details
Every hour, we apply Detoxify to identify potentially toxic messages with a cutoff of 0.5 (Hanu and Unitary team, 2020). Messages flagged by Detoxify are further validated using GPT-4o to reduce incorrectly flagging non-toxic messages. When a user sends their first toxic message during the four-week intervention period, they are randomly assigned (by computer) to Private, Public, or Control. All subsequent messages by that user are tracked for six weeks to measure outcomes.
Randomization Method
Randomization is done automatically by the computer at the time a user sends their first toxic message.
Randomization Unit
The randomization is done at the user level.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
No cluster.
Sample size: planned number of observations
All users who send at least one public message classified as toxic during the intervention period.
Sample size (or number of clusters) by treatment arms
Equal distribution among three treatment arms.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
Swedish Ethical Review Authority
IRB Approval Date
2025-08-26
IRB Approval Number
2025-04872-01
Analysis Plan

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials