AI Assistance and Student Problem Solving: The Role of Timing and Accuracy

The intervention consists of providing participants with access to an AI-generated solution while completing a simple problem-solving task. The intervention varies along two dimensions: the timing of access to the AI-generated solution and the accuracy of the solution provided. Participants may receive access either immediately or delayed, and the AI-generated solution may be either correct or incorrect. Participants in the pure control group receive no AI assistance while solving the problem.

2026-04-20

2026-06-26

The primary outcomes are binary indicators of whether the participant submitted a correct answer in the first stage and in the second stage.

From the first stage of the task, the following are used as secondary outcomes:
⁃ AI usage behavior
⁃ Task completion time
⁃ An indicator for whether the participant entered a response prior to viewing the AI-generated solution
⁃ An indicator for whether the participant revised their answer after viewing the AI-generated solution
⁃ The time elapsed between the appearance of the AI-generated solution and the first subsequent edit to the answer
⁃ In cases where the participant’s answer differed from the AI-generated solution, an indicator for whether the participant subsequently revised their answer
In addition, participants’ reported confidence in their submitted answers in both stages is included as a secondary outcome.

The experimental design follows a 2×2 between-subjects framework with an additional pure control condition. Participants are randomly assigned to conditions that vary along two dimensions: the timing of AI access (immediate versus delayed) and the accuracy of the AI-provided solution (correct versus incorrect), or to a control condition with no AI access. Participants are randomly assigned at the individual level with equal probability across all five conditions. Participants complete a simple, incentivized problem-solving task. In the immediate access condition, participants can request an AI-generated solution at any point from the start of the task. In the delayed access condition, the AI-generated solution is not immediately accessible and becomes available during the course of the task, with the point of access standardized across all participants in this condition. Within each timing condition, participants are randomly assigned to receive either a correct or an incorrect AI-generated solution upon request. Participants in the control condition are not given any access to AI assistance and are required to solve the task independently. This results in five groups defined by the combination of access timing, solution accuracy, and the absence of AI access.

Group 1: Pure control (no AI access)
Group 2: Immediate access – correct solution
Group 3: Immediate access – incorrect solution
Group 4: Delayed access – correct solution
Group 5: Delayed access – incorrect solution


All participants are first presented with a problem and instructed to solve it within a fixed time limit. After completing the first task, all participants proceed to a second problem. In this second stage, all participants are treated identically regardless of their initial assignment. They are given a fixed time limit to solve the problem and are not provided with any AI assistance. Both stages are incentivized.

Not available

Randomization at the individual level via lottery. We implement stratified randomization by session to ensure balanced allocation of participants across treatment arms within each classroom.

individual level

No

Around 2500 high school students

Around 2500 high school students

Around 500 high school students in each of the 5 treatment arms.

We can detect a minimum effect size of approximately 8.8 percentage points in the accuracy rate between any two experimental groups with 500 participants per group (N = 2,500 total), assuming a baseline accuracy rate of 50%, α = 0.05, and power = 0.80. With 600 participants per group (N = 3,000 total), the corresponding minimum detectable effect is approximately 8.1 percentage points.