|
Field
Trial Status
|
Before
on_going
|
After
completed
|
|
Field
Abstract
|
Before
This study wants to explore the competition and collaboration between humans and Large Language Models(e.g. ChatGPT), as well as humans' attitudes towards LLMs. We are planning to run a randomized control trial on social media platform. Four accounts will be operated simultaneously from scratch, with the copywriting for three of the accounts being generated separately by humans, LLMs, and human-AI collaboration. The copy for the other account will be randomly selected from those generated by LLMs or human-AI collaboration, but we will indicate at the beginning of the post that the copy was generated or co-generated by LLMs. Afterwards, we will allocate traffic to the four accounts and observe the views, likes, reposts and comments acquired by the accounts. We also plan to conduct a similiar survey experiment, where copies are presented for participants to rate without or with the labeling of the writer. The distinction of survey from Weibo is that attention is more abundant and individul-data is collected. We will examine (1) In the context of copywriting, who will gain more traffic and perform better: humans, LLMs, or human-AI collaboration?(2) Compared to copywriting produced by humans, will social media users have different attitudes towards copywriting generated or co-generated by LLMs? (3) For people who cooperate with LLMs to write copywriting, will they adopt more LLMs later in their work? What is the effect of LLMs adoption on performance?
|
After
This study wants to explore the competition and collaboration between humans and Large Language Models(e.g. ChatGPT), as well as humans' attitudes towards LLMs. We are planning to run a randomized control trial on social media platform. Four accounts will be operated simultaneously from scratch, with the copywriting for three of the accounts being generated separately by humans, LLMs, and human-AI collaboration. The copy for the other account will be randomly selected from those generated by LLMs or human-AI collaboration, but we will indicate at the beginning of the post that the copy was generated or co-generated by LLMs. Afterwards, we will allocate traffic to the four accounts and observe the views, likes, reposts and comments acquired by the accounts. We also plan to conduct a similiar survey experiment, where copies are presented for participants to rate without or with the labeling of the writer. The distinction of survey from Weibo is that attention is more abundant and individul-data is collected. We will examine (1) In the context of copywriting, who will gain more traffic and perform better: humans, LLMs, or human-AI collaboration?(2) Compared to copywriting produced by humans, will social media users have different attitudes towards copywriting generated or co-generated by LLMs? For a separate study, we further study (3) For people who cooperate with LLMs to write copywriting, will they adopt more LLMs later in their work? What is the effect of LLMs adoption on performance?
|
|
Field
Last Published
|
Before
August 04, 2024 07:53 AM
|
After
August 30, 2025 11:20 PM
|
|
Field
Primary Outcomes (End Points)
|
Before
Weibo Experiment:
-consumer: the number of views, likes, reposts and comments for each post; the number of growing fans for each account; the number of purchasing if the post attaches a purchasing link;
- employee(writer): humans' perceptions and attitudes towards LLMs; the adoption of LLMs; the performance before and after the adoption of LLMs.
Survey Experiment: the scores given to each post
|
After
Weibo Experiment:
-consumer: the number of views, likes, reposts and comments for each post; the number of growing fans for each account; the number of purchasing if the post attaches a purchasing link;
- employee(writer): humans' perceptions and attitudes towards LLMs; the adoption of LLMs; the performance before and after the adoption of LLMs. (For the separate study)
Survey Experiment: the scores given to each post
|
|
Field
Primary Outcomes (Explanation)
|
Before
Weibo Experiment
-employee: their perceptions and attitudes towards LLMs will be elicited from surveys.
|
After
Weibo Experiment
-employee: their perceptions and attitudes towards LLMs will be elicited from surveys. (For the separate study)
|
|
Field
Experimental Design (Public)
|
Before
Weibo Experiment:
Each group corresponds to one social media account, and four accounts will be operated simultaneously from scratch.
- Pure Human Group: Humans generate copywriting independently.
- Pure AI Group: LLMs generate copywriting independently.
- Human-AI Collaboration Group: There is no specific limitation on the collaboration method. It can be multiple rounds of interaction between humans and LLMs, or humans modifying the copywriting generated by LLMs.
- Labeled Group: Randomly select copywriting from the Pure AI Group and Human-AI Collaboration Group for posting. When posting, indicate at the beginning that "This copywriting is generated by LLMs" or "This copywriting is co-generated by humans and LLMs".
These four accounts will be used to promote the same brand and product managed by a company. We plan to recruit employees from this company to generate or co-generate copywritings at the designated time and place. There will be "an initial questionnaire" to collect the basic information of these employees and especially to survey their usage of LLMs and other AI tools. "A following questionnaire" will be released two weeks after the copywriting generation and survey their later adoption and usage of LLMs. We will combine it with their performance evaluation data from the company to analyze the productivity effect of LLMs.
Among all the copywritings, part of them will be paired with product image or purchase link or both. We will then send these posts and allocate traffic for them. We will purchase the same exposure number for each post and observe the final number of views, likes, shares, and comments data. A questionnaire for social media users to survey their perceptions and attitudes towards these posts will be released through the four accounts when the experiment is finished.
Survey Experiment 1, 2 (Labeled or not)*3(present copies written by human, AI or human-AI collaboration) between-individual design:
Each participant are asked to rate three randomly chosen copies written by human/AI/human-AI collaboration, without or with the disclosure of the writer.
Survey Experiment 2, 2 (Labeled or not, this dimension is between-individual)*3(present copies written by human, AI or human-AI collaboration, this dimension is within-individual) design: Each participant are asked to rate copies without or with the disclosure. Each participant will be presented copies written by human, AI or human-AI collaboration simultaneously.
|
After
Weibo Experiment:
Each group corresponds to one social media account, and four accounts will be operated simultaneously from scratch.
- Pure Human Group: Humans generate copywriting independently.
- Pure AI Group: LLMs generate copywriting independently.
- Human-AI Collaboration Group: There is no specific limitation on the collaboration method. It can be multiple rounds of interaction between humans and LLMs, or humans modifying the copywriting generated by LLMs.
- Labeled Group: Randomly select copywriting from the Pure AI Group and Human-AI Collaboration Group for posting. When posting, indicate at the beginning that "This copywriting is generated by LLMs" or "This copywriting is co-generated by humans and LLMs".
These four accounts will be used to promote the same brand and product managed by a company. We plan to recruit employees from this company to generate or co-generate copywritings at the designated time and place. There will be "an initial questionnaire" to collect the basic information of these employees and especially to survey their usage of LLMs and other AI tools. "A following questionnaire" will be released two weeks after the copywriting generation and survey their later adoption and usage of LLMs. We will combine it with their performance evaluation data from the company to analyze the productivity effect of LLMs (For the separate study).
Among all the copywritings, part of them will be paired with product image or purchase link or both. We will then send these posts and allocate traffic for them. We will purchase the same exposure number for each post and observe the final number of views, likes, shares, and comments data. A questionnaire for social media users to survey their perceptions and attitudes towards these posts will be released through the four accounts when the experiment is finished.
Survey Experiment 1, 2 (Labeled or not)*3(present copies written by human, AI or human-AI collaboration) between-individual design:
Each participant are asked to rate three randomly chosen copies written by human/AI/human-AI collaboration, without or with the disclosure of the writer.
Survey Experiment 2, 2 (Labeled or not, this dimension is between-individual)*3(present copies written by human, AI or human-AI collaboration, this dimension is within-individual) design: Each participant are asked to rate copies without or with the disclosure. Each participant will be presented copies written by human, AI or human-AI collaboration simultaneously.
|