|
Field
Last Published
|
Before
August 04, 2026 03:21 PM
|
After
August 29, 2026 10:10 AM
|
|
Field
Planned Number of Clusters
|
Before
300 participants.
August 4, 2026 update: The original experiment recruited 297 participants. The new experiment recruiting those who work in creative jobs (including graphic designers, game developers, content creators, and copywriters) will include up to 180 participants, unless we cannot recruit this many participants with these jobs from Prolific. In that case, we will recruit as many as we can.
|
After
300 participants.
August 4, 2026 update: The original experiment recruited 297 participants. The new experiment recruiting those who work in creative jobs (including graphic designers, game developers, content creators, and copywriters) will include up to 180 participants, unless we cannot recruit this many participants with these jobs from Prolific. In that case, we will recruit as many as we can.
August 29, 2026 update: We will recruit an additional 550 participants for our third experiment.
|
|
Field
Planned Number of Observations
|
Before
300 participants, split across three treatment groups. The first treatment group sees the AI design a game after they design their first game. The second treatment group sees the AI design a game only after they design their second game. The final treatment group does not see the AI design a game at all.
We exclude (by asking to return the study) participants who give more than three incorrect answers to comprehension questions.
August 4, 2026 update: See above about the up to 180 new participants.
|
After
300 participants, split across three treatment groups. The first treatment group sees the AI design a game after they design their first game. The second treatment group sees the AI design a game only after they design their second game. The final treatment group does not see the AI design a game at all.
We exclude (by asking to return the study) participants who give more than three incorrect answers to comprehension questions.
August 4, 2026 update: See above about the up to 180 new participants.
August 29, 2026 update: See above about the 550 new participants.
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
In expectation: 100 see the AI design after they design their first game, 100 only after they design their second game, and 100 not at all.
August 4, 2026 update: Same randomization probabilities as the original experiment.
|
After
In expectation: 100 see the AI design after they design their first game, 100 only after they design their second game, and 100 not at all.
August 4, 2026 update: Same randomization probabilities as the original experiment.
August 29, 2026 update: Participants in the third experiment will have an equal chance of being assigned to each of the three treatments.
|
|
Field
Intervention (Hidden)
|
Before
Participants design three games, starting with a base game and using an AI coding agent to improve or redesign it. They do so over the course of three rounds for each game, where they send a message to the AI, and it re-codes the game in response to what they say. Throughout we collect participants WTP to keep designing the game rather than give up, as well as pre- and post-design predictions of their "customer's" WTP to play the game. We also elicit their WTP to keep the game (be sent a durable link to it) as well as to include it in a public gallery that other participants can see. Finally we elicit likert pride and enjoyment measures, as well as participants' beliefs about their ability relative to other participants.
In the customer phase (which happens after the design phase) participants express their WTP to play another participants' (or the AI's) game, relative to a basic game.
There are three treatment groups, assigned randomly. The first treatment group sees the AI design a game after they design their first game. The second treatment group sees the AI design a game only after they design their second game. The final treatment group does not see the AI design a game at all.
We also cross-randomize whether participants are told which games they are offered in the customer phase were designed by other participants or by the AI.
August 4, 2026 update: We will conduct a second experiment with up to 180 participants recruited from Prolific, screened on work experience as creatives (including graphic designers, game developers, content creators, and copywriters). The experiment is identical to the first, except these participants will not undergo the customer phase (a future round of Prolific participants will act as customers for these participants' games).
|
After
Participants design three games, starting with a base game and using an AI coding agent to improve or redesign it. They do so over the course of three rounds for each game, where they send a message to the AI, and it re-codes the game in response to what they say. Throughout we collect participants WTP to keep designing the game rather than give up, as well as pre- and post-design predictions of their "customer's" WTP to play the game. We also elicit their WTP to keep the game (be sent a durable link to it) as well as to include it in a public gallery that other participants can see. Finally we elicit likert pride and enjoyment measures, as well as participants' beliefs about their ability relative to other participants.
In the customer phase (which happens after the design phase) participants express their WTP to play another participants' (or the AI's) game, relative to a basic game.
There are three treatment groups, assigned randomly. The first treatment group sees the AI design a game after they design their first game. The second treatment group sees the AI design a game only after they design their second game. The final treatment group does not see the AI design a game at all.
We also cross-randomize whether participants are told which games they are offered in the customer phase were designed by other participants or by the AI.
August 4, 2026 update: We will conduct a second experiment with up to 180 participants recruited from Prolific, screened on work experience as creatives (including graphic designers, game developers, content creators, and copywriters). The experiment is identical to the first, except these participants will not undergo the customer phase (a future round of Prolific participants will act as customers for these participants' games).
August 29, 2026 update: We will conduct a third experiment with 550 participants recruited from Prolific. This experiment is largely similar to the first, except that these participants will not undergo the design phase. This experiment serves three purposes. First, these new participants will act as customers to the second experiment's participants' games (as well as the original experiment's participants' games), allowing us to truthfully incentivize and pay those participants. Second, we will ask participants to rate each game along 4 dimensions: fun, creativity, polish, and clarity. Finally, we introduce three treatments that the third experiment's participants are randomized into (across participant). The first corresponds to the arm in the first experiment where participants are not told whether the game they are potentially paying to play was designed by a human or entirely by the AI. In the second, we disclose whether each game was designed by the AI or not. In the third, we disclose whether each game was designed by the AI or not but (truthfully) inform participants that their WTP will not affect the bonus of the participant who designed the game.
|