RCT for AI Agent: Cybernetics and Human Capital Talent Growth

Last registered on January 10, 2025

View Trial History

Pre-Trial

Trial Information

General Information

Title

RCT for AI Agent: Cybernetics and Human Capital Talent Growth

RCT ID

AEARCTR-0015139

Initial registration date

January 09, 2025

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

January 10, 2025, 2:04 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name

Tianyang Han

Affiliation

HKU

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Ye Luo

PI Affiliation

HKU

Contact Investigator

PI Name

Yuxiao Wu

PI Affiliation

HKU

Contact Investigator

Additional Trial Information

Status

In development

Start date

2025-01-20

End date

2025-03-16

Keywords

Behavior, Education

Additional Keywords

AI (LLM)

JEL code(s)

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

The rapid growth of Large Language Models (LLMs) is gradually changing the way human learn and work. Nowadays, with more developed AI Agent, many tedious workflows can be largely simplified into simple human language instruction. With strong AI Agents that both master domain specific knowledge and provide automatic implementation, users can directly deliver a complete outcome for a given task without having to spend time and effort in mastering relevant knowledge.

While automation could directly boost productivity towards time-consuming work, its performance towards creative work stays not that satisfactory. AI researchers have been dreaming of creating robots to complete all work such that human could have time to think, to write, to compose and to create; but till now, the indirect influence of AI Agent application towards human capital knowledge level and creativity level has not receive sufficient attention or discussion:

1. With the help of AI Agent, users are no longer forced to immediately master relevant knowledge to complete time-consuming tasks on hand. Will users’ long-term knowledge level therefore be negatively affected? Will there be any heterogeneity within the effect (such as users’ background relevance and study ability)?
2. With the help of AI Agent, users can greatly save time from the time-consuming tasks. How will users’ performance towards creative work be affected? How will AI Agent’s effect on users’ knowledge level intervene in the procedure? Is the application of AI Agent eventually a good thing for human capital creativity?

We generate the following hypotheses:

Hypothesis 1: After applying AI Agent, users tend to have long-term underperformance on relevant domain knowledge, with the results stronger for more specialized/professional AI Agent design;
Hypothesis 2: After applying AI Agent, users will perform better on creative work, ceteris paribus;
Hypothesis 3: Knowledge level will endogenously affect creativity level: after applying AI Agent, users’ increase in creative work performance will be partially offset by the decrease of domain knowledge level, with the results stronger for more specialized AI Agent design;
Hypothesis 4: All of above effects will be heterogeneous based on users’ knowledge background and learning capability.

External Link(s)

Registration Citation

Citation

Han, Tianyang, Ye Luo and Yuxiao Wu. 2025. "RCT for AI Agent: Cybernetics and Human Capital Talent Growth." AEA RCT Registry. January 10. https://doi.org/10.1257/rct.15139-1.0

Sponsors & Partners

Experimental Details

Interventions

Intervention(s)

In this RCT, the intervention is designed to control students’ access to our well-prepared AI Agent tool in completing course assignments. Treatment group students can and are encouraged to use AI Agent to solve the knowledge-based task in their corresponding assignment, while control group students need to solve all questions in the assignment by themselves. Two types of AI Agents are randomly assigned to the treatment group each time (i.e., two subgroups in the entire treatment group):
• General AI Agent can complete an entire workflow but is not equipped with completely developed domain-knowledge tools.
• Specialized AI Agent, upgraded from general AI Agent with well-equipped domain-knowledge tools

Intervention Start Date

2025-02-14

Intervention End Date

2025-03-16

Primary Outcomes

Primary Outcomes (end points)

For the first treatment towards the second assignment, Group 1 is the treatment group while Group 2 is the control group. For the second treatment towards the third assignment, Group 2 is then the treatment group while Group 1 is the corresponding control group. The first lecture survey and first assignment are mainly for summary statistic exhibition and grouping and are therefore not used for further analysis.

From the second assignment, we shall collect creativity question scores from all students. From the second assignment survey, we shall collect:
• Time consumption on both questions;
• The contribution of AI Agent towards two questions (only from Group 1);
• The time saving of AI Agent towards two questions (only from Group 1).

From the third assignment, similarly, we shall collect creativity question score SC,3 from all students. From the third assignment survey, we shall collect:
• Time consumption on both questions;
• The contribution of AI Agent towards two questions (only from Group 2);
• The time saving of AI Agent towards two questions (only from Group 2).

From the final exam, we shall collect all exam question scores (Question 1 →Assignment 1, Question 2 →Assignment 2 Question 1, Question 3 → Assignment 3 Question 1) From the final exam survey, we shall collect the self-evaluation effect of AI Agent application towards knowledge learning.

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

Step 0 (Course Setup): Set three assignments throughout this course. The first assignment contains knowledge-driven questions. The second and third assignments each contain two questions, one about more theory knowledge work (for example, Fama-Macbeth regression, Machine Learning theory and application, etc.) and the other one containing more creative work. Also there will be a written final exam after all lectures and assignments. The 3-hour exam includes questions containing the theory knowledge from the three assignments. The final exam must be close-book.
• HW1: Factor model & Fama-Macbeth method (knowledge driven)
• HW2: Investment portfolio analysis (knowledge driven); Factor model & Fama-Macbeth method exploration (knowledge based on HW1; creative tasks)
• HW3: Machine learning practice (knowledge driven); Machine learning methodology explo- ration & creation (knowledge based on HW3; creative tasks)

Step 1: Prof. Ye LUO at the first lecture announces the existence of AI Agent availability towards both subclasses, introduce AI Agent briefly and announce that students will be taught about AI Agent and given accessible accounts in the class. After the first lecture, tutors will
distribute questionnaires to collect information from students. The information will serve as the pre-determined control variables in our future analysis. Questions will include (but may not limit to):
• Students’ previous experience towards LLM & AI Agent;
• Programming experience;
• Previous education/knowledge background;
• (Self-evaluation) Personal interest on research & in-depth thinking.

Step 2: For Assignment 1, no AI Agent will be provided and all the questions are knowledge-driven.We shall collect students’ performance in Assignment 1.

Step 3: After Assignment 1, conditional on students’ performance in Assignment 1, as well as students’ background match, we randomly split all the students into two groups. The first group (Group 1) will be trained before the announcement of Assignment 2. At the same time, each of them will get their AI Agent personal accounts. Two tutorial sessions will be held for Group 1. One sub-session will be taught standard AI Agent, while the other will be taught the specialized AI Agent. Students will not be told that two sub-sessions are provided different types of AI Agents. The second group (Group 2) will not get their AI Agent account or receive AI Agent training for
Assignment 2.

Step 4: For Assignment 2, we shall collect students’ performance for both groups. The AI Agent account will be suspended for Group 1 once the DDL of Assignment 2 approaches. A short survey will be distributed to all students, with self-evaluation questions including (but may not limit to):
• Time consumption on both questions;
• (Only for Group 1) The contribution of AI Agent towards two questions;
• (Only for Group 1) The time saving of AI Agent towards two questions;
• (Only for Group 2) Two placebo questions.

Step 5: After Assignment 2, Group 2 will be trained before the announcement of Assignment 3. At the same time, each of them will get their AI Agent personal accounts until the DDL of Assignment 3. Two tutorial sessions will be held for Group 2. One sub-session will be taught
standard AI Agent, while the other will be taught the specialized AI Agent. Students will not be told that two sub-sessions are provided different types of AI Agents. Group 1 at this time will not get their AI Agent account or receive AI Agent training for Assignment 3.

Step 6: For Assignment 3, we shall collect students’ performance for both groups. The AI Agent account will be suspended for Group 2 once the DDL of Assignment 3 approaches. A short survey will be distributed to all students, with self-evaluation questions including (but may not limit to):
• Time consumption on both questions;
• (Only for Group 2) The contribution of AI Agent towards two questions;
• (Only for Group 2) The time saving of AI Agent towards two questions;
• (Only for Group 1) Two placebo questions.

Step 7: The final exam will be close-book and will cover all knowledge contents covered in the three assignments. We shall record all students’ performance for further analysis. After final exam, tutors will distribute questionnaires to collect self-evaluation information from students. Questions will include (but may not limit to):
• The effect of AI Agent application towards knowledge learning.

Experimental Design Details

Not available

Randomization Method

Randomization done in office by a computer

Randomization Unit

Individual

Was the treatment clustered?