Experimental Design Details
Experimental design
The basic procedure
The experimental design encompasses four distinct phases, including the pre-intervention assessment, intervention, post-intervention assessment, and basic information.
During Phase 1 of the study, the initial questionnaire is distributed to the participants. The initial set of questionnaires comprises five questions, including students' career preferences, their perceptions of the availability of places in the labor market, their beliefs regarding the acquisition of new skills, their beliefs of the prevailing wage among the majority of individuals, and their personal beliefs on wages.
During Phase 2, the participants will be allocated into four groups using a random assignment method. The initial cohort will be provided with arbitrary information that is unrelated to AI/LLMs technology and will be designated as the placebo group. In the case of the second group, we implement our intervention by providing selected students neural, objective information regarding the development of LLMs. In the third group, participants will be provided with two practical examples showcasing the application of ChatGPT. Additionally, they will receive positive statements regarding the LLMs tools in the context of labor market outcomes. The information provided in group will also be provided to group 3. The final group will have same information to group 3, with the exception that the positive information will be substituted with negative information.
During Phase 3, all participants are requested to do the assessment which have descripted in Phase 1 once again and collect the post intervention results.
During Phase 4, participants will be requested to provide demographic information, including their gender, birthplace, working experience, parents' occupations, and education level, and whether their parents had experienced unemployment. Additionally, participants will be asked to provide school-related information, such as their faculties, academic majors, and academic performance. Furthermore, we also investigate participants' preferences in job selection, their willingness to invest in acquiring LLMs skills, their attitudes towards a newly proposed LLMs course, their perceptions of the extent to which LLMs will affect their chosen careers, and their prior experience with AI. Last, we will also collect participants' preferences for risk, job search time, and uncertainty.
Two incentives will be implemented in this study. First, we raffled off 10% of our total participants and who were then awarded a monetary incentive ranging from 50 to 100 CNY (equivalent to 8 to 16 Euros) for their involvement in the experiment. Second, in relation to question 4 of the questionnaires, participants will be provided with a reward of 10 CNY if their responses closely align with the actual average income.