Conditions of AI Engagement in Learning: An Online Experiment

Participants complete a computer-administered two-phase task. In the study phase, they answer four multiple-choice questions, one from each of four academic domains, presented one per page in individually randomized order. In four of the five arms, an optional on-demand menu of AI-generated hints is available beside each question; participants choose freely whether and how much to consult it, and no correctness feedback is given. In the fifth arm no hints are available. In the test phase, all participants answer a further set of questions on related material with no assistance available.

Participants are randomly assigned with equal probability to one of five arms. The arms differ in the conditions under which the study phase is completed. Those conditions are specified in the non-public intervention field and in the pre-analysis plan, and will be released when the trial is complete.

All hint content is pre-generated by the research team using an AI language model, reviewed for accuracy, and embedded as fixed text in the survey instrument; no participant data are transmitted to any external AI system at any point. Correct answers earn a per-item bonus in both phases, in every arm.

2026-08-06

2027-08-06

Learning test score: total correct answers on the eight-item no-AI knowledge test (0-8; two questions per domain: one recall and one near-transfer). Main learning outcome for all primary confirmatory tests.

Mode 4 usage: the share of the four main-task questions on which the participants used Mode 4 at least once. Defined only for AI-available arms.

Mode 3 or 4 as first hint use: the share of the four main-task questions on which the participants used Mode 3 or 4 first. Defined only for AI-available arms and estimated at the question level.

Time duration: total seconds from question display to answer submission.

Immediate task accuracy: number of correct answers on the four main-task questions (0-4). Captures in-task performance, which may diverge from learning if AI use substitutes for rather than supports understanding.

Time measures: total seconds from question display to answer submission; time till first click; seconds between a hint click and the participant's next action (another click or answer submission). Time measures winsorized at the 99th percentile.

Perceived learning / pre-test confidence: participant's domain-by-domain prediction of whether they will answer each knowledge test question correctly, collected after the main task but before the test outcome is known. Serves as a proxy for perceived learning.

Hint-mode sequence: ordered list of mode buttons clicked within a question; analyzed for escalation patterns and repeat clicks.

Recall sub-score: correct answers on the four recall/recognition items referencing content from the original task question (0-4).

Near-transfer sub-score: correct answers on the four novel-scenario items applying the same underlying concept to a new context (0-4).

Mode 4 exclusive use: binary indicator per question (= 1 if Mode 4 was the only hint used on that question), averaged across questions. Captures the purest form of offloading -- answer elimination without any other hint engagement.

Number of distinct hint modes per question: count of unique mode buttons clicked, averaged across questions (0-4).

Between-subjects randomized experiment conducted online with approximately 2,000 participants recruited through Prolific, targeting approximately 400 per arm. Eligibility is restricted to United States residents aged 18–25 with at least some college education, prescreened by Prolific and re-verified within the survey instrument.

Participants first complete a baseline block comprising an instructed-response attention check and self-rated confidence in each of the four task domains. They are then randomly assigned with equal probability to one of five arms using the survey platform's randomizer. Randomization is at the individual level; assignment is stored as an embedded field and governs all subsequent survey branching.

One arm serves as a pure control in which no assistance is available. The remaining four arms, in all of which assistance is available, form a 2×2 between-subjects factorial over two manipulated task conditions. The identity of those two conditions is specified in the non-public design field and in the pre-analysis plan, and will be made public when the trial is complete. The control arm identifies the overall effect of assistance availability relative to working unaided; the factorial identifies the main effect of each manipulation within assistance-available arms, and their interaction.

Following the study phase, participants complete brief post-task ratings and report a domain-by-domain expectation of their own performance on the forthcoming test, then complete the unassisted test phase, followed by a validity check and a demographic block, both administered after all outcomes are measured.

Analysis is intent-to-treat. Person-level outcomes are estimated by OLS on arm indicators with the control condition as the reference category, using heteroskedasticity-robust standard errors, and reported both with and without a vector of baseline covariates. Question-level process outcomes are estimated with question fixed effects and standard errors clustered by participant. The factorial contrasts are estimated on assistance-available arms only, in both a saturated specification including the interaction and a restricted specification omitting it, always reported together. Pre-specified exclusions cover the attention check, the eligibility criteria, incomplete responses, duplicate submissions from a single platform identifier, and a post-outcome validity check; results including participants failing the validity check are reported as a robustness check.

Not available

After the baseline survey and task introduction, participants are randomly assigned with equal probability to one of five experimental arms using Qualtrics embedded data. The assignment is stored as an embedded field and used to condition all subsequent survey branching.

Individual participant

No

2000 individuals

2000 individuals

400 participants in each of five arms: 400 in the no-assistance control and 400 in each of the four assistance-available arms.