Effect of A.I. Evaluators on Creative Output

Participants will begin by answering basic demographic questions in a Qualtrics survey. They will then answer questions about their feelings towards human evaluators and A.I. evaluators, and they'll provide details about their experiences in and knowledge of New York City.

Participants will be asked whether or not they want to create and submit short promotions ("promos", of 300 characters or less) to get people to visit New York City. They will either be told that their promos will be evaluated by "several people with marketing experience" or "an artificial intelligence (A.I.) program that is specifically designed for use in marketing." Each promo will be evaluated on a 0-100 scale based on how likely it is that someone will click on the promo to learn more. To incentivize participation, they will either be told they will "receive $0.50 (50 cents) for each promo they submit, so long as the promo is evaluated with a score above 33. If it is evaluated with a score of 33 or lower, no payment will be provided for that particular promo" or "receive $50 for each promo you submit that is among the 10 highest-scoring promos in the entire contest. Approximately one out of every ten participants is expected to win $50." In both cases, they will be told that they can submit up to 10 different promos to increase their potential payout.

After learning all this information, they'll be asked if they want to enter the contest. For those that say no, the survey will end and they'll receive the pre-contracted payment amount. For those that say yes, they'll have approximately one hour to submit their promos. Once they finish, they'll receive the pre-contracted payment amount and then, after the promos are evaluated, they'll receive additional payments given the parameters of their contest and the quality of their submitted promos.

2024-11-18

2024-12-02

The decision to enter the contest (yes/no response to the question asking if they want to participate). The number of promos submitted in the contest for those who enter.

The decision to enter will be based on their yes/no response to the question that explicitly asks if they want to enter the contest (after having learned all the parameter details of their specific contest). The number of promos will be based on the free-response entries they provide, with a limit of ten promos and with the caveat that they cannot submit the exact same promos multiple times.

Average quality of each promo, as evaluated on the 0-100 scale by (i) the A.I evaluator, (ii) the human evaluators, and (iii) all evaluators. We will also consider how participation (primary outcomes) and promo quality (secondary outcomes) vary based on the demographic characteristics of the participants (e.g., whether they from a demographic group that is traditionally underrepresented in creative endeavors or whether they exhibit ex ante high levels of algorithm aversion).

Participants will begin by answering basic demographic questions in a Qualtrics survey. They will then answer questions about their feelings towards human evaluators and A.I. evaluators, and they'll provide details about their experiences in and knowledge of New York City.

Participants will be asked whether or not they want to create and submit short promotions ("promos", of 300 characters or less) to get people to visit New York City. They will either be told that their promos will be evaluated by "several people with marketing experience" (human evaluator) or "an artificial intelligence (A.I.) program that is specifically designed for use in marketing" (A.I. evaluator). Each promo will be evaluated on a 0-100 scale based on how likely it is that someone will click on the promo to learn more. To incentivize participation, they will either be told they will "receive $0.50 (50 cents) for each promo they submit, so long as the promo is evaluated with a score above 33. If it is evaluated with a score of 33 or lower, no payment will be provided for that particular promo" (quantity-based incentives) or "receive $50 for each promo you submit that is among the 10 highest-scoring promos in the entire contest. Approximately one out of every ten participants is expected to win $50" (quality-based incentives). In both cases, they will be told that they can submit up to 10 different promos to increase their potential payout.

After learning all this information, they'll be asked if they want to enter the contest. For those that say no, the survey will end and they'll receive the pre-contracted payment amount. For those that say yes, they'll have approximately one hour to submit their promos. Once they finish, they'll receive the pre-contracted payment amount and then, after the promos are evaluated, they'll receive additional payments given the parameters of their contest and the quality of their submitted promos.

Randomization will occur via the block-randomization feature in Qualtrics. 25% of participants will be randomized into the A.I. evaluator and quantity-based incentives treatment; 25% of participants will be randomized into the A.I. evaluator and quality-based incentives treatment; 25% of participants will be randomized into the human evaluator and quantity-based incentives treatment; and 25% of participants will be randomized into the human evaluator and quality-based incentives treatment.

The only differences in content across the four treatment cells will be the language about the types of evaluators who will evaluate the promos and the types of incentives in place for participating in the contest.

Randomization via the Qualtrics block-randomization feature.

Individual labor market participant

No

1,000 labor market participants

1,000 labor market participants

250 labor market participants in A.I. evaluator x quantity-based incentives
250 labor market participants in A.I. evaluator x quality-based incentives
250 labor market participants in human evaluator x quantity-based incentives
250 labor market participants in human evaluator x quality-based incentives