Effects of GenAI on Rural Children Creativity

We will distribute tablets with search and/or GenAI functionalities to students in the after-school program, and evaluate their efficacy on student learning outcomes.

2025-03-17

2025-07-31

Our dependent variables include the level of learning core concepts and the level of innovation. After receiving the intervention, students are required to build science projects according to the course requirements. An example of a project is a model boat made from recycled materials. We will evaluate the extent to which students learn the core science concepts, and the innovation level of the students' works, using raters who are familiar with the domain. Raters will evaluate the following dimensions: (1) Completeness (To what extent do you believe this work is fully completed? 1 = Not at all, 10 =Very much), (2) Functionality (To what extent do you believe this work can achieve its intended functionality? 1 = Not at all, 10 =Very much), 3. Aesthetics (To what extent do you consider this work to be aesthetically pleasing? 1 = Not at all, 10 =Very much), (4) Functionality vs. Aesthetics (Do you think the makers of this product focus more on its functionality or aesthetics? 1 = Being only focusing on functionality, 7 = Being only focusing on aesthetics), (5) innovation (To what extent do you think the work is innovative? (1 = Not at all, 10 =Very much), (6) Overall rating (Please provide an overall rating for this work. 1 = Very poor, 10 =Excellent)

The primary outcomes are students’ learning outcomes as reflected by their in-classroom projects, including overall rating, ratings of completeness, functionality, aesthetics, and innovativeness. We ask 30-35 online crowdsourced raters to provide ratings for the six questions above, and take the average as the main outcome variables.

(1) We collect students’ midterm and final exam grades of 2024 Fall and 2025 Spring semesters. The secondary outcome variable is students’ academic performance of core school curriculums such as Chinese, Math, English, Science, and Arts.
(2) We collect students’ prompts when they interact with generative AI and search queries when they use the search engine, from which we create measures of students’ information seeking behavior.

(1) Students’ midterm and final exams are direct measures.
(2) From the generative AI conversation history, we measure frequencies of student and AI prompts, including conversation starters, on-topic and off-topic prompts, and language characteristics of student and AI prompts such as length, complexity, and sentiments. From the search engine history, we measure frequencies of student searches, including searches that are search starters, on-topic and off-topic searches, and language characteristics of student searches such as length, complexity, and sentiments.

Participants will be randomly assigned to a single-factor, three-level (Interactive device: Generative AI vs. search engines vs. control condition without access to tablets) between-subjects design.

Not available

The teachers are first clustered by city. Within each city, they are sorted by their self-reported years of teaching experience and grouped into triplets. Teachers in each triplet are randomly assigned to the generative AI, search engine, and control group, respectively. The randomization was done in the office by a computer. Our matched triplet design is a generalization of the efficient matched-pair design.

The randomization is conducted at the teacher level. The teachers are clustered by city.

Yes

In this experiment, clusters are defined at the group level, with each group consisting of 4–6 students (averaging 5 students per group). The Spring 2025 experiment involves 500 students from 17 schools. Based on this structure, the study consists of approximately 100 clusters. The distribution includes 34 clusters (168 students) in the AI condition, 34 clusters (172 students) in the search engine condition and 32 clusters (160 students) in the control group.

The Spring 2025 experiment involves students from 17 schools (teachers), with 12 schools assigned to the treatment group (6 to the search engine condition and 6 to the AI condition) and 5 schools assigned to the control condition. 500 students, with 160 in the control group and 340 in the treatment group (168 in the AI condition and 172 in the Baidu condition).

The Spring 2025 experiment involves students from 17 schools (teachers), with 12 schools assigned to the treatment group (6 to the search engine condition and 6 to the AI condition) and 5 schools assigned to the control condition. 500 students, with 160 in the control group and 340 in the treatment group (168 in the AI condition and 172 in the Baidu condition).

Using α=0.05 and β=0.20, we aim to detect a 0.33-standard deviation change (the minimum detectable effect size) in the main outcome variables. This requires us to have 144 students per experimental condition, and 432 students in total. We plan to recruit 500 students.