Experimental Evidence on the Learning Impact of Generative AI

This experiment investigates the causal impact of generative AI access on learning outcomes through a two-session laboratory study with undergraduate students. In Session One, participants are randomly assigned to either an AI-allowed or AI-forbidden condition and tasked with learning about one of three unfamiliar academic topics (Blockchain, Carbon Capture, or CRISPR). The session begins with participants completing a 5-question baseline knowledge assessment, followed by a 35-minute learning phase during which they research their assigned topic and complete a written analytical assessment of approximately 500 words. Participants in the treatment group have explicit permission to use generative AI tools during this learning phase, while those in the control group are instructed not to use these tools. After completing the written assessment, all participants take a post-learning assessment without access to external resources to measure immediate knowledge acquisition.

Session Two occurs approximately one week later and measures knowledge retention without any external resources. This session includes a follow-up written analytical task using a complementary prompt on the same topic and a 10-item multiple-choice test of factual knowledge. To incentivize effort, we implement a lottery-based compensation structure where participants earn tickets based on their performance on assessments, with each ticket giving them a chance to win monetary prizes.

The experimental design allows us to measure both the immediate effects of AI access on learning performance and its impact on longer-term knowledge retention. We collect detailed data on treatment compliance, assess written responses through blind evaluations by independent graders, and gather self-reported measures of AI experience and usage patterns. This allows us to examine not only whether AI access affects learning outcomes but also the potential mechanisms through which these effects might operate, such as changes in study behaviors, time allocation, and engagement with learning materials.

2025-04-21

2025-05-20

1. Written assessment quality in Session One (as measured by blind grader evaluations)

2. Post-learning assessment performance in Session One (multiple-choice test)

3. Follow-up written assessment quality in Session Two (as measured by blind grader evaluations)

4. Knowledge retention test performance in Session Two (multiple-choice test)

5. Learning gain (difference between post-learning and baseline assessments)

Knowledge retention is measured as the score on a 10-item multiple-choice test administered in Session Two, while learning gain represents the difference between the post-learning assessment and baseline knowledge assessment scores.

1. Time allocation during the learning phase

2. AI usage patterns (for treatment group)

3. Computational text metrics (lexical diversity, word count, syntactic complexity)

4. Retention rate (ratio of Session Two to Session One performance)

Time allocation measures will track how participants distribute their time across different activities (researching, writing, editing) during the learning phase. For the treatment group, we will analyze AI usage patterns including frequency and types of AI interactions. Computational text metrics will be derived from participants' written submissions using natural language processing methods.

This experiment uses a between-subjects design with random assignment. Undergraduate participants complete two sessions approximately one week apart. In Session One, participants learn about an unfamiliar academic topic and complete assessments. In Session Two, all participants complete retention assessments.

This experiment uses a between-subjects design with random assignment to either an AI-allowed or AI-forbidden condition. Undergraduate participants complete two sessions approximately one week apart. In Session One, participants learn about an unfamiliar academic topic and complete assessments, with only the treatment group permitted to use generative AI. In Session Two, all participants complete retention assessments without AI access.

Randomization done in office by a computer.

Individual participant

No

Not applicable (individual-level randomization)

~200-300 students

100-150 students in AI-allowed condition, 100-150 students in AI-forbidden condition

With our minimum target sample size of 200 participants (100 per treatment arm), significance level = 0.05, and 80% power, the minimum detectable effect size is 0.40 standard deviations for main between-group comparisons of learning outcomes. For knowledge retention measured after one week, the literature on generation effects in learning suggests potentially larger effects (0.64 SD), allowing us to detect meaningful differences even with some attrition. Power calculations are based on two-sided t-tests.

PAP