Learning to evaluate generative AI advice: a cluster-randomized trial of fading process scaffolding, judgment revision, and transfer

Last registered on August 31, 2026
View Trial History

Pre-Trial

Trial Information

General Information

Title
Learning to evaluate generative AI advice: a cluster-randomized trial of fading process scaffolding, judgment revision, and transfer
RCT ID
AEARCTR-0019544
Initial registration date
August 30, 2026

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
August 31, 2026, 8:52 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country
China
Region
Chongqing

Primary Investigator

Name
Senhao Xiong
Affiliation
khon kaen University
Contact Primary Investigator

Other Primary Investigator(s)

Additional Trial Information

Status
Completed
Start date
2026-03-02
End date
2026-08-26
Keywords
Education
Additional Keywords
generative artificial intelligence, appropriate reliance, instructional scaffolding, AI literacy, career decision-making, vocational education, cluster-randomized trial
JEL code(s)
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
This completed educational randomized controlled trial examined whether a fading process scaffold can improve how students evaluate generative-AI advice and whether the learned decision procedure transfers after support is withdrawn. The trial enrolled 480 second-year students in 24 intact classes across three higher vocational institutions in Chongqing, China. Randomization occurred at the class level within six school-by-teacher blocks. Within each four-class block, one class was assigned to each condition in a 2 x 2 factorial design crossing generative-AI access with TRACE process scaffolding: AI + TRACE, AI only, TRACE only, or neither. Each condition contained six classes and 120 randomized students. The intervention comprised four weekly 45-minute training sessions, followed by an assessment conducted while assigned support was still available. The two primary outcomes specified in the internal analysis plan were (H1) early unaided decision-process performance after both AI and TRACE were withdrawn and (H2) task-based appropriate reliance on identical version-locked AI advice after TRACE withdrawal. Secondary outcomes included support-present performance, delayed unaided transfer, judgment revision and final accuracy, selective access to standardized AI responses, and dynamic revision after new information. Class allocation was generated by computer using a Fisher-Yates permutation after class rosters were finalized. This registry entry is retrospective; the study was not prospectively registered.
External Link(s)

Registration Citation

Citation
Xiong, Senhao. 2026. "Learning to evaluate generative AI advice: a cluster-randomized trial of fading process scaffolding, judgment revision, and transfer." AEA RCT Registry. August 31. https://doi.org/10.1257/rct.19544-1.0
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
The trial crossed two instructional factors: access to institutionally managed generative AI and access to TRACE, a structured decision-process scaffold. TRACE externalized five operations: Target the decision; Review evidence and uncertainty; Assess alternatives and consequences; Clarify criteria and trade-offs; and Execute, evaluate, and update. TRACE support was faded across sessions from full prompts to guided, reduced, minimal, and final-cue forms. Four conditions were implemented: (A) AI + TRACE, (B) AI only, (C) TRACE only, and (D) neither. All four groups followed the same career-decision curriculum, worked with parallel scenarios, received the same scheduled instructional time, and used the same core response structures. During S1-S4 and T1, both AI-access conditions used the same institutionally managed qwen3.5-plus service under a common system prompt and common access configuration. Retrieval/web search and external tools were disabled, no condition-specific model configuration was used, and no model-version change was recorded during the training period. T2B and T2C used separately version-locked assessment outputs and involved no live generation.
Intervention (Hidden)
Intervention Start Date
2026-03-09
Intervention End Date
2026-04-01

Primary Outcomes

Primary Outcomes (end points)
H1 - Early unaided transfer: total score on a parallel scenario-based decision-process task administered five days after T1 with both AI and TRACE removed. Six dimensions were each scored from 0 to 4, producing a total score from 0 to 24.
H2 - Appropriate reliance on standardized AI advice: item-level binary appropriate-reliance outcome on a 10-item challenge administered after TRACE withdrawal. Both AI-trained groups received identical version-locked recommendations. An item met the appropriate-reliance criterion only when the principal action fit the advice status, the final judgment fit the case, and the action-specific explanation met the item-specific requirement.
Primary Outcomes (explanation)
H1 was designed to assess whether learners could initiate the trained decision procedure without either AI advice or TRACE support. The 0-24 total score aggregated six observable decision-process dimensions: problem/constraints, evidence/uncertainty, alternatives/consequences, criteria/trade-offs, justification/action, and updating.
H2 was designed to assess conditional reliance on AI advice of varying quality while holding the advice itself constant across the focal AI + TRACE and AI-only conditions. The 10 items represented correct/aligned advice, factual error, constraint omission, value mismatch, and insufficient information. Item-specific permissible-action, final-judgment, and explanation keys were finalized before recruitment; the final key was fixed on 11 February 2026 and was not modified in response to formal trial data.

Secondary Outcomes

Secondary Outcomes (end points)
Support-present performance at T1 (0-24 decision-process total); delayed unaided transfer at T3 (0-24); T2B initial and final judgment accuracy, correction of initially incorrect judgments, harmful revision of initially correct judgments, action fit, final-judgment fit, and explanation adequacy; T2C selective allocation of three standardized-response reveals; and T2D dynamic revision after new information. A supplementary interview strand was also collected as supporting qualitative context.
Secondary Outcomes (explanation)
Secondary outcomes were used to distinguish support-present performance, post-advice judgment revision, delayed transfer, selective use of standardized AI information, and dynamic updating. These outcomes were not part of the two-primary-outcome family specified in the internal analysis plan unless explicitly stated otherwise.

Experimental Design

Experimental Design
Teacher-blocked 2 x 2 class-cluster randomized educational trial conducted in three higher vocational institutions in Chongqing, China. The factors were generative-AI access (yes/no) and TRACE process scaffolding (yes/no). The sample comprised 24 intact second-year classes and 480 students, with 20 registered students per class. Six teachers each contributed a four-class school-by-teacher block; within each block, one class was assigned to each of the four conditions. Every condition therefore included six randomized classes and 120 randomized students. Class rosters and individual consent were completed before allocation was released. All groups received the same curriculum, scenario structure, scheduled instructional time, and common teacher guidance; the intended contrasts were AI access and TRACE support. Students were required to attend the institutionally scheduled career-planning course, but consent for research use of their data was separate from course participation and had no effect on grades, credits, teacher evaluation, access to instructional resources, or other academic rights. The study is being registered retrospectively after completion.
Experimental Design Details
Randomization Method
After class rosters were finalized, the allocation sequence was generated by computer using a Fisher-Yates permutation within each of six school-by-teacher blocks. One class within each four-class block was assigned to each of the four conditions. Condition assignments were not released until enrollment was complete.
Randomization Unit
Intact class. Randomization was conducted within school-by-teacher blocks.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
24 intact classes
Sample size: planned number of observations
480 students
Sample size (or number of clusters) by treatment arms
AI + TRACE: 6 classes, 120 students; AI only: 6 classes, 120 students; TRACE only: 6 classes, 120 students; Neither: 6 classes, 120 students.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
No prospective power calculation determined the sample size. All 24 eligible intact classes available within the scheduled curriculum and implementation period were enrolled. Any minimum-detectable-effect calculations reported in the manuscript are post hoc design-sensitivity analyses and were not used to determine N.
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
Khon Kaen University Ethics Committee for Human Research, Panel 3
IRB Approval Date
2026-02-10
IRB Approval Number
Reference No. HE683660; Institutional Review Board No. IRB00012791
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
Yes
Intervention Completion Date
April 01, 2026, 12:00 AM +00:00
Data Collection Complete
Yes
Data Collection Completion Date
May 10, 2026, 12:00 AM +00:00
Final Sample Size: Number of Clusters (Unit of Randomization)
24 randomized classes; all 24 clusters remained in the trial.
Was attrition correlated with treatment status?
Yes
Final Sample Size: Total Number of Observations
480 students were randomized. Outcome-specific completion was T0 480, T1 455, T2A 453, T3 419, T2D 448, T2B 437, and T2C 436.
Final Sample Size (or Number of Clusters) by Treatment Arms
All four randomized arms retained 6 classes and 120 randomized students. Participant completion varied by assessment; for T2B, AI + TRACE contributed 114 of 120 students and AI only 103 of 120.
Data Publication

Data Publication

Is public data available?
No

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Program Files

Program Files
No
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials