|
Field
Trial End Date
|
Before
June 30, 2024
|
After
August 30, 2024
|
|
Field
Last Published
|
Before
June 02, 2024 02:05 AM
|
After
August 04, 2024 07:53 AM
|
|
Field
Intervention End Date
|
Before
June 30, 2024
|
After
August 30, 2024
|
|
Field
Experimental Design (Public)
|
Before
Weibo Experiment:
Each group corresponds to one social media account, and four accounts will be operated simultaneously from scratch.
- Pure Human Group: Humans generate copywriting independently.
- Pure AI Group: LLMs generate copywriting independently.
- Human-AI Collaboration Group: There is no specific limitation on the collaboration method. It can be multiple rounds of interaction between humans and LLMs, or humans modifying the copywriting generated by LLMs.
- Labeled Group: Randomly select copywriting from the Pure AI Group and Human-AI Collaboration Group for posting. When posting, indicate at the beginning that "This copywriting is generated by LLMs" or "This copywriting is co-generated by humans and LLMs".
These four accounts will be used to promote the same brand and product managed by a company. We plan to recruit employees from this company to generate or co-generate copywritings at the designated time and place. There will be "an initial questionnaire" to collect the basic information of these employees and especially to survey their usage of LLMs and other AI tools. "A following questionnaire" will be released two weeks after the copywriting generation and survey their later adoption and usage of LLMs. We will combine it with their performance evaluation data from the company to analyze the productivity effect of LLMs.
Among all the copywritings, part of them will be paired with product image or purchase link or both. We will then send these posts and allocate traffic for them. We will purchase the same exposure number for each post and observe the final number of views, likes, shares, and comments data. A questionnaire for social media users to survey their perceptions and attitudes towards these posts will be released through the four accounts when the experiment is finished.
Survey Experiment, 2 (Labeled or not)*3(present copies written by human, AI or human-AI collaboration) between-individual design:
Each participant are asked to rate three randomly chosen copies written by human/AI/human-AI collaboration, without or with the disclosure of the writer.
|
After
Weibo Experiment:
Each group corresponds to one social media account, and four accounts will be operated simultaneously from scratch.
- Pure Human Group: Humans generate copywriting independently.
- Pure AI Group: LLMs generate copywriting independently.
- Human-AI Collaboration Group: There is no specific limitation on the collaboration method. It can be multiple rounds of interaction between humans and LLMs, or humans modifying the copywriting generated by LLMs.
- Labeled Group: Randomly select copywriting from the Pure AI Group and Human-AI Collaboration Group for posting. When posting, indicate at the beginning that "This copywriting is generated by LLMs" or "This copywriting is co-generated by humans and LLMs".
These four accounts will be used to promote the same brand and product managed by a company. We plan to recruit employees from this company to generate or co-generate copywritings at the designated time and place. There will be "an initial questionnaire" to collect the basic information of these employees and especially to survey their usage of LLMs and other AI tools. "A following questionnaire" will be released two weeks after the copywriting generation and survey their later adoption and usage of LLMs. We will combine it with their performance evaluation data from the company to analyze the productivity effect of LLMs.
Among all the copywritings, part of them will be paired with product image or purchase link or both. We will then send these posts and allocate traffic for them. We will purchase the same exposure number for each post and observe the final number of views, likes, shares, and comments data. A questionnaire for social media users to survey their perceptions and attitudes towards these posts will be released through the four accounts when the experiment is finished.
Survey Experiment 1, 2 (Labeled or not)*3(present copies written by human, AI or human-AI collaboration) between-individual design:
Each participant are asked to rate three randomly chosen copies written by human/AI/human-AI collaboration, without or with the disclosure of the writer.
Survey Experiment 2, 2 (Labeled or not, this dimension is between-individual)*3(present copies written by human, AI or human-AI collaboration, this dimension is within-individual) design: Each participant are asked to rate copies without or with the disclosure. Each participant will be presented copies written by human, AI or human-AI collaboration simultaneously.
|
|
Field
Planned Number of Observations
|
Before
Weibo experiment:
150 individuals for copy writing, with each individual generate or co-generate two posts; another 300 posts are generated by LLMs or randomly selected from the existing posts.
600 posts in total.
The number of planned exposure we purchase for each post is 10000-20000 users.
6000000-12000000 social media users exposed to copywritings for the above four accounts in total.
survey experiment:
Each participant rates three copies, 300-500 participants per treatment, 1800-3000participants in total.
|
After
Weibo experiment:
150 individuals for copy writing, with each individual generate or co-generate two posts; another 300 posts are generated by LLMs or randomly selected from the existing posts.
600 posts in total.
The number of planned exposure we purchase for each post is 10000-20000 users.
6000000-12000000 social media users exposed to copywritings for the above four accounts in total.
survey experiment 1:
Each participant rates three copies, 300-500 participants per treatment, 1800-3000participants in total.
Survey Experiment 2:
Each participant rates 4-6 copies, 250 participants per treatment, 500 participants in total.
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
Weibo experiment:
150 posts generated by humans independently; 150 posts generated by LLMs independently; 150 posts generated by human-AI collaboration; 150 posts randomly selected from posts generated/co-generated by LLMs and labeled "generated/ cogenerated by LLMs".
1500000-3000000 users exposed to copywritings for each account.
Survey experiment:
Each participant rates three copies, 300-500 participants per treatment; Each copy is rated by 7-11 participants on average.
|
After
Weibo experiment:
150 posts generated by humans independently; 150 posts generated by LLMs independently; 150 posts generated by human-AI collaboration; 150 posts randomly selected from posts generated/co-generated by LLMs and labeled "generated/ cogenerated by LLMs".
1500000-3000000 users exposed to copywritings for each account.
Survey experiment 1:
Each participant rates three copies, 300-500 participants per treatment; Each copy is rated by 7-11 participants on average.
Survey experiment 2:
Each participant rates 4-6 copies, 250 participants per treatment; Each copy is rated by 7-9 participants on average.
|