The impact of ChatGPT on policy debates: a randomized control trial

My intervention consistis in testing the effect of ChatGPT on policy debates. I will do that through a randomized controlled trial at Esade Business School, involving 120 students. The treatment will consist in a 20 minute training in ChatGPT and the possibility of using the Chat to prepare three to four rounds of debates. Students in the control group will only be allowed to use conventional internet access to prepare the debates. I will be testing four main hypotheses: (a) whether ChatGPT improves debating skills of participants in the experiment; (b) whether it contributes to reduce inequality in debating skills; (c) whether it favors evidence-based policy positions and (d) whether it contributes to changing beliefs of debate participants. I will also test whether some of these effects are persistent after some time.

2023-03-20

2023-03-31

The main outcomes are:
1) Debating points: individual points of debating participants, a continuous variable from 0 to 100. These points will be assigned by expert judges. All debates will be audio recorded during the debate competition. Overall I expect to have about 240 recorded debates. Judges will be either former debating champions or debating/communication university teachers. Judges will know nothing about the experiment
2) Win/lose debate:
3) Belief update: variation between baseline and endline views on policy.

I plan to apply for access to student level administrative data from the university to look at additional outcomes, such as grades in other subjects. The hypothesis being that having been trained to use ChatGPT might affect students' usage of the tool for other subjects and might improve their performance.

To test the first hypothesis I will use two strategies. First, I will compare average results of people with and without chat. Second, I will compare outcomes within individuals, taking advantage that individuals will only be assigned to the ChatGPT condition after one round of debates. I will be testing the second hypothsis by checking whether ChatGPT has an asymetric impact depending on the initial debating skills (at baseline) of participants. I will use the first round of debates to divide participants into top (50%) and bottom (50%) performers. Then I will compare average debating points in rounds 2 and 3 for top and bottom performers among individuals who use ChatGPT and among individuals who do not use ChatGPT. This will tell me whether the effect of ChatGPT differs depending on baseline performance. All policy debates will be on policy topics with two clearly different sides: a "pro-evidece" side and an "anti-evidence" side. I will look at the interaction of using ChatGPT and being assigned the anti-evidence position. The goal is to test whether the effects of using ChatGPT are stronger when used to defend pro-evidence or anti-evidence arguments. Finally, I will be testing "the persuasive" effect of the Chat, whether it contributes to changing beliefs of debate participants. Do students exposed to defend a specific topic update their beliefs on that topic? In what direction? Is that conditioned by the use of the chat? I will do that by testing whether subjects move closer, after the debate, to the position that they were randomly assigned to defend. I will also study whether the effect is bigger for those using ChatGPT.

Randomization done in office by a computer. I will randomize students to Chat and NoChat conditions. I will also do a random match of debating couples for all rounds of debates. I will aslo do a random assignment of debate positions ("for" or "against") within couples. I will use basline information to balance randomization.

I will randomize at the individual level. I will use two blocks, for the two different days in whcih the debating competition will take place. Since individuals will do several rounds of debates, I will have up to four observations per individual.

No

120 students

I plan to obtain about 480 observations from four debating rounds of 120 students.

60 students treatment (use of ChatGPT)
60 students control (no use of ChatGPT)

The minimum detectable effect size for my main outcome is 0.374 of a standard deviation