Experimental Design Details
The study will be carried out as an incentivised survey experiment at a behavioural lab. The survey will be created and delivered using the otree software platform (https://www.otree.org – published under an MIT open-source license). The otree platform is already installed and available at the lab. GenAI answers to general (i.e. non-personal) hypothetical economic questions will be generated ahead of the experiment.
An Experiment Handbook was develop with detailed information about the procedures that will be followed by the lab manager to conduct the experiment. The Experiment Handbook also includes the instructions that will be shown to participants and samples of the experiment questions.
Participants will be asked to complete a survey that consists of 2 Parts. Participants will be randomly assigned to either a Control Group or one of 2 Treatment Groups. Group assignment affects only Part 1 of the survey, Part 2 is the same for everyone.
In Part 1, participants will be asked to answer multiple hypothetical questions (between 8 and 20). The questions mimic reasoning that people make every day when purchasing products or services. For some questions participants will need to type an answer, for others they will need to select an answer from a list of options (between 2 and 10 options). Each question has a correct answer, which is the best possible guess according to the laws of probability given the information available. If they select the correct answer, a monetary reward will be added to their total payment. If they select any other answer, nothing will be added or subtracted from their payment (i.e. they cannot lose previously earned rewards). After each question, participants will be asked to rate how confident they are of their answer using a Likert style scale. The questions and the order in which they are shown will be randomized for each participant from a pool of predesigned questions. After answering all questions in Part 1, participants will be shown a summary of their performance and their total payment.
Participants in the Control Group will be presented with the hypothetical questions as described above and will not be given the option to use GenAI technologies. Participants in Treatment Group 1 will be asked to answer the same questions, but they will be given the option to use GenAI at a cost. For each question, participants will be given the choice to exchange a predetermined proportion of their potential reward to see an answer produced by GenAI. The answers that GenAI produces are not guaranteed to be correct, and participants will be shown the probability that the GenAI answer is correct before they make their decision to use it or not. After seeing the GenAI answer, the participant must still explicitly state or choose an answer, and it does not have to be the same as the one provided by GenAI. The proportion of the reward to exchange to see the GenAI answer will range from 1% to 99% and will be randomized without any relationship to the question itself. For example, if a participant chooses to see the GenAI answer with a cost of 40%, and selects the right answer, 60% of the reward will be added to their payment. If they select a wrong answer, as before, nothing will get added or subtracted. Participants in Treatment Group 2 will also be given the option to use GenAI. However, for the first half of the questions (e.g. the first 5 of 10 questions) they will be able to use GenAI without any cost. In other words, no proportion will be deducted from their reward if they answer a question correctly, regardless of whether they chose to use GenAI or not. For the second half of the questions will be able to use GenAI at a cost, they will have to exchange a portion of the potential rewards with the same conditions as Treatment Group 1.
In Part 2, participants will be asked about their self-perceived ability for economic decision making, their every-day use of GenAI, and non-identifiable socio-demographics. All questions in this part are optional and answering them is not related to receiving payment or monetary rewards. The GenAI usage questions will ask participants if and how frequently they use GenAI technologies in their work and personal time, how long ago they started using it, if and how the complexity of the task they are tackling informs their decision to use GenAI, and how confident they are of the accuracy of GenAI generated answers. The socio-demographic questions will include gender, age bracket, occupation (student, in the work force, retired), field of study/work (as broad discipline options such as medicine, engineering), income bracket (or parental income bracket if none) and residence area (from broad geographic options in Glasgow, e.g. West End). All answers are collected anonymously, and thus it is not possible to link them to consent forms or payment profiles. Also, the socio-demographic questions can only be answered in broad categories which will ensure that individuals cannot be identified even if answers from all questions are combined. For example, age can only be answered in brackets of 5 years, and field of study/work only as a discipline rather than a specific programme.
The experiment was designed to last 50 minutes, and participants will be compensated for 1 hour of their time. Participants will be given 10 minutes to read the consent form and initial experiment instructions. Part 1 of the survey (which involves problem solving) was designed to be completed in 30 minutes and Part 2 can be finished in 5 to 10 minutes.
Multiple sessions of 24 participants will be run in the lab. During each session participants will be randomly assigned to one of the experimental groups such that there is an equal number of participants in each group by the end of the experiment. One pilot session will be run at the lab with 24 participants, prior to the main sessions, to test the procedure, survey design, data collection and analysis.
Responses to the questions will be saved as a dataset in secure stores provided by the University of Glasgow.
Responses will be analysed using within-subject and between-subject designs, looking at differences in the responses of individuals with and without treatment, and between control and treatment groups conditional on observed characteristics. Regression Analysis and Analysis of Variance (ANOVA) tests will be performed to test the statistical significance of results.