The impact of human-AI collaboration modes on worker experiences

Last registered on November 15, 2024
View Trial History

Pre-Trial

Trial Information

General Information

Title
The impact of human-AI collaboration modes on worker experiences
RCT ID
AEARCTR-0014729
Initial registration date
November 13, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
November 15, 2024, 1:52 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country
United States of America
Region

Primary Investigator

Name
Yuqing Ren
Affiliation
University of Minnesota
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Teng Ye
PI Affiliation
University of Minnesota
Contact Investigator
PI Name
Jundong Zhang
PI Affiliation
University of Minnesota
Contact Investigator

Additional Trial Information

Status
In development
Start date
2024-12-02
End date
2024-12-20
Keywords
Behavior, Gender, Lab
Additional Keywords
JEL code(s)
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
In this trial, we hope to study the impact of human AI collaboration modes on worker experiences and work quality. The research context is individual workers collaborating with a generative AI agent such as ChatGPT to perform professional writing tasks. The study will be posted as a survey on online platforms such as Prolific. We will study three collaboration modes of no AI, AI first, and human first, as well as the nature of the task (e.g., writing a report to a client vs. writing an email to co-workers) and the sentiment of the task (e.g., positive vs. negative). We will observe how these factors interact with one another to affect worker experiences and work quality. We will also examine heterogeneity among workers and how workers with different backgrounds react similarly or differently to the various task scenarios.
External Link(s)

Registration Citation

Citation
Ren, Yuqing, Teng Ye and Jundong Zhang. 2024. "The impact of human-AI collaboration modes on worker experiences." AEA RCT Registry. November 15. https://doi.org/10.1257/rct.14729-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Intervention (Hidden)
Intervention Start Date
2024-12-02
Intervention End Date
2024-12-20

Primary Outcomes

Primary Outcomes (end points)
We will measure both worker experiences and work quality. For workers experiences, we will ask workers about their experiences of working on the writing task, particularly the level of autonomy, self-efficacy, enjoyment, and work stress they experienced. For work quality, we will use computational tools to assess complexity and readability and have human experts assess the subjective measures of work quality.
Primary Outcomes (explanation)
We will construct some of our measures to explain why people may differ in their experiences with different collaboration modes. For instance, one measure is to assess the degree to which people are process-oriented or outcome-oriented in their work.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
It will be a 3x2x2 = 12 between-subjects design.

We will manipulate three factors: collaboration modes, nature of the writing task, and task sentiment. The three collaboration modes are performing the writing task without AI assistance, using AI to draft the writing and human revise, human drafting and AI revise. The two types of tasks are writing a report to a client and writing an email to co-workers. The two sentiments are positive and negative.
Experimental Design Details
Participants will be asked to imagine being a product manager at a tech company who is in charge of a AI drive-through project. Under the positive scenario, the pilot project with a restaurant chain is successful and has helped the restaurants increase revenue and gain positive customer reviews. Under the negative scenario, the pilot project with a restaurant chain is unsuccessful and has reduced revenue for the restaurants and led to negative customer reviews. Participants will be asked either to write a report to the client describing the outcome of the project and next steps or to write an email to the team members about the outcome of the project and next steps. While performing the writing tasks, participants will be randomly assigned to three conditions of using no AI assistance, using AI to draft the writing and then improve on it, or drafting the writing on their own and using AI to improve it.
Randomization Method
The survey will be built using Qualtrics and we will use the randomization built into Qualtrics.
Randomization Unit
Individual worker level
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
One cluster of participants recruited from Prolific
Sample size: planned number of observations
We hope to recruit 300-500 participants from Prolific.
Sample size (or number of clusters) by treatment arms
About 30-40 participants per condition
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
University of Minnesota Institutional Review Board
IRB Approval Date
2024-02-01
IRB Approval Number
STUDY00021153
Analysis Plan

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials