Generative AI & TeamWork: An experimental approach.

Last registered on May 13, 2024

Pre-Trial

Trial Information

General Information

Title
Generative AI & TeamWork: An experimental approach.
RCT ID
AEARCTR-0013603
Initial registration date
May 13, 2024

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
May 13, 2024, 12:47 PM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
Harvard Business School

Other Primary Investigator(s)

PI Affiliation
Harvard Business School
PI Affiliation
Harvard Business School

Additional Trial Information

Status
In development
Start date
2024-05-13
End date
2024-06-15
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
This study examines the effects of integrating a Generative AI (GenAI) tool (a company interface based on GPT4), on team dynamics and the division of labor in a large multinational consumer product company. Participants from Commercial and R&D functions will be randomly assigned to control and treatment groups to collaborate on real business problems during a one-day virtual hackathon. The control groups include individual controls (not using AI) and a Commercial+R&D team without AI. The treatment groups will have either Commercial or R&D paired with GenAI (T1), or both functions working together with GenAI (T2). Participants will work on compatible tasks using real background data from the Fabric Care, FemCare, Grooming, and Baby Care business units, with the goal of producing detailed one-page business solutions. The study aims to answer research questions on whether GPT-4 can substitute certain functions, change teamwork dynamics, and blur boundaries across functions. Crucially, it allows us to compare teams using AI and teams not using AI. The experiment will be conducted in three sessions across the US and Europe on May 13th-15th. The findings will provide insights into how GenAI tools like GenAI can be effectively integrated into collaborative team settings in knowledge-intensive domains within a multinational corporation.
External Link(s)

Registration Citation

Citation
Ayoubi, Charles, Fabrizio Dell'Acqua and Karim Lakhani. 2024. "Generative AI & TeamWork: An experimental approach.." AEA RCT Registry. May 13. https://doi.org/10.1257/rct.13603-1.0
Experimental Details

Interventions

Intervention(s)

1. The introduction of AI, a Generative AI (GenAI) tool, to specific team compositions:
- T1: Human Employee + AI
- T2: Both Commercial and R&D functions work together with AI.

2. The assignment of participants to different team compositions:
- Control groups:
- Individual controls (not using AI)
- Commercial + R&D team without AI
- Treatment groups:
- T1: Human Employee + AI
- T2: Commercial + R&D + AI

Within each group, team members will be randomized. One team member will be designated as the leader, responsible for submitting the solution and leading the usage of AI by sharing their screen. The other team member will follow and contribute to the task. Teams will work remotely and transcribe their conversations during the hackathon.

After completing their initial task, participants in the control groups (both individual and team) will undergo GenAI training. They will then re-do the task, allowing for a within-participant comparison of performance before and after the training.

The interventions aim to study the effects of integrating AI on team dynamics, division of labor, and the potential substitution of certain functions within the context of a large multinational consumer product company. Participants will collaborate on real business problems using actual background data from various business units (Fabric Care, FemCare, Grooming, and Baby Care) during a one-day virtual hackathon.
Intervention Start Date
2024-05-13
Intervention End Date
2024-05-15

Primary Outcomes

Primary Outcomes (end points)
Hackathon output.
Evaluations of the hackathon output.

Additionally, we are building outcomes from text:
GenAI prompts and responses: Collection of all the prompts given to the AI and the corresponding responses generated by the AI during the hackathon.
Transcripts of group interactions: Collection of the transcribed conversations among team members during the hackathon.
Primary Outcomes (explanation)
Hackathon output: Detailed business solutions to specific problems faced by the company's business units (Fabric Care, FemCare, Grooming, and Baby Care).
Evaluations of the output:

Three-level evaluations:
a. Workers trained by Business Unit (BU) leaders and rubrics on evaluating the solutions, possibly involving cross-BU evaluations.
b. Evaluation of the best solutions by BU leaders for final selection.
c. Complementary AI-generated evaluations of the solutions.

Secondary Outcomes

Secondary Outcomes (end points)
We will measure completion rates and timing for completing the task.

We are collecting survey question responses (see the Qualtrics flow).

Additionally, we are building outcomes from text:
GenAI prompts and responses: Collection of all the prompts given to the AI and the corresponding responses generated by the AI during the hackathon.
Transcripts of group interactions: Collection of the transcribed conversations among team members during the hackathon.
Secondary Outcomes (explanation)
We will measure completion rates and timing for completing the task.

We are collecting survey question responses (see the Qualtrics flow).


Additionally, we are building outcomes from text:
GenAI prompts and responses: Collection of all the prompts given to the AI and the corresponding responses generated by the AI during the hackathon.
Transcripts of group interactions: Collection of the transcribed conversations among team members during the hackathon.

Experimental Design

Experimental Design
This study employs a randomized controlled trial design to investigate the effects of integrating Generative AI (GenAI) on team dynamics, division of labor, and performance in a large multinational consumer product company. Participants from Commercial and R&D functions will be randomly assigned to control and treatment groups to collaborate on real business problems during a one-day virtual hackathon.

Groups:
- Control groups:
- Individual controls (not using AI)
- Commercial + R&D team without AI
- Treatment groups:
- T1: Human Employee + AI
- T2: Commercial + R&D + AI
Experimental Design Details
This study employs a randomized controlled trial design to investigate the effects of integrating Generative AI (GenAI) on team dynamics, division of labor, and performance in a large multinational consumer product company. Participants from Commercial and R&D functions will be randomly assigned to control and treatment groups to collaborate on real business problems during a one-day virtual hackathon.

Groups:
- Control groups:
- Individual controls (not using AI)
- Commercial + R&D team without AI
- Treatment groups:
- T1: Human Employee + AI
- T2: Commercial + R&D + AI

Participants will be randomly assigned to one of the following groups:
1. Individual controls (not using AI)
2. Commercial + R&D team without AI
3. Human Employee + AI (T1)
4. Commercial + R&D + AI (T2)

Within each group, team members will be randomized, with one designated as the leader responsible for submitting the solution and leading the usage of AI (if applicable). The treatment groups (T1 and T2) will have access to GenAI tools to assist them in collaborating and generating solutions to the business problems presented during the hackathon.

After completing the initial task, participants in the control groups will undergo GenAI training and re-do the task, allowing for a within-participant comparison of performance before and after the training.
Randomization Method
Random assignment of participants to the intervention groups is achieved through a computerized random number generator, ensuring that the allocation is both random and concealed until the point of assignment.
Randomization Unit
The randomization happens within each unit/geography (Europe/Americas). We have 4 business units and 2 geographies, and thus 8 randomization clusters. The probability of assignment is equal across clusters.
1. Individual level randomization:
- Participants are randomly assigned to either the individual control group (not using AI) or one of the team-based groups (Commercial + R&D team without AI, T1: Human Employee + AI, or T2: Commercial + R&D + AI).
2. Within-team randomization:
- For the team-based groups (Commercial + R&D team without AI, T1: Human Employee + AI, and T2: Commercial + R&D + AI), team members are randomized within each group.
- One team member is designated as the leader, responsible for submitting the solution and leading the usage of AI (if applicable).
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
We are randomizing 852 participants. Additionally, we are including additional participants (participants in band 4+) who are assigned to treated individual group. Finally, participants coming in last minute will be accepted in the control individual group. We will use the additional participants for exploratory analyses.
Sample size: planned number of observations
We are randomizing 852 participants. Additionally, we are including additional participants (participants in band 4+) who are assigned to treated individual group. Finally, participants coming in last minute will be accepted in the control individual group. We will use the additional participants for exploratory analyses.
Sample size (or number of clusters) by treatment arms
We are randomizing 852 participants. 364 are assigned to the paired control treatment (182 groups); 364 to the paired treatment (182 groups), 94 to the treated singleton; 30 to the control singleton.
Note that if one of the two paired members of a group does not show up, the other group member is automatically reassigned to the relevant singleton condition (either control or treatment based on the group assignment)
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
Harvard University-Area Committee on the Use of Human Subjects
IRB Approval Date
2024-04-09
IRB Approval Number
IRB23-0392

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials