Agentic AI in the Classroom: Frontier or European AI tutors? A pre-analysis plan

The frontier arm is Claude Opus 5.5. The European arm is Mistral Medium 3.5. Both reach the server through OpenRouter, get the same prompt, the same checklist of elements for each question, and the same sentence cap. Universities in Norway face a real choice here: a European model keeps data and supply closer to home, a frontier model may perform better. We do not know the size of that trade-off for grading.

Each student is assigned one grader for the entire semester and uses that grader again and again. Nobody knows whether one is better. Repeated feedback from the same model may shape how a student writes, what they practice, and how much they learn, so a difference could build up over the term, even if a single grade looks alike.

2026-10-06

2026-12-06

Three outcomes, each with its own design.

\textbf{Outcome 1: the AI tutor's effect on student scores.} Does a student assigned to Opus score differently later than one assigned to Mistral? Measured only on what students do on the platform: the scores on their later mocks. No submitted work and no exam result is used. Shadow grading gives every essay a grade from both models, so the two arms are compared on scores from the same model, not on scores from each student's own arm. The outcome is the student's mean score in percent over all completed mocks after the first, with the essay bands taken from Opus in both arms. The same analysis with the bands from Mistral Medium is reported beside it. Randomised at the student level.

\textbf{Outcome 2: the precision of each AI grader against a human grader.} This outcome is studied only if the human grading is carried out. On the same essays, how often does each model place the essay in the same rubric band as a trained human? The benchmark is how often two humans agree with each other. A model is as good as a human if it agrees with a human as often as a second human does. Shadow grading gives both model bands on every essay.

\textbf{Outcome 3: use of the AI tutor.} Does a student assigned to Opus use the AI tutor more than one assigned to Mistral? Use is the number of messages the student sends to the Socratic tutor plus the number of mocks the student starts, counted from the day after the arm is assigned until 19 November. A message or a mock counts whether or not the model answered, so a refused call does not lower the count. The two parts are also reported one by one. The arm is assigned at a student's first use, so a student who never uses the tutor has no arm. The outcome is therefore use after the first day, among students who used the tutor at least once.

Each student gets one arm for both parts: Opus for grading and tutoring, or Mistral for both

Not available

Each student in the course is assigned to one of two arms the first time they use the AI tutor on the course platform. The assignment is made by the server with one cryptographic random bit, drawn at the moment of first use. It is stored in a database table and never changed. The two arms have equal probability. No stratification and no blocking is used. The student keeps the same arm for the whole semester, in both parts of the tutor (the essay grader and the Socratic tutor).

Treatment and control. The two arms are two versions of the same AI tutor that differ only in the language model behind it. Arm A uses a frontier model (Claude Opus 5.5). Arm B uses a European model (Mistral Medium 3.5). Everything else is held fixed: the prompt, the rubric and checklist, the sentence caps, the output format, the interface, and the rule the server uses to turn the model's verdict into a grade. Students are not told which arm they are in, and the model's identity is never shown.

There is no no-tutor control group. Every student has access to the tutor. The comparison is between the two models, and the primary hypothesis is equivalence: that the arm has no effect on later mock scores. In that sense the trial is an A/B comparison of two active treatments, and arm B can be read as the control condition if the registry needs one named.

Two further points may help. First, every essay is graded by both models in the background. The student sees only the grade from their own arm, and the other grade is stored and never shown. This gives a within-essay paired comparison alongside the between-student randomisation. Second, students in arm A who reach a spending cap are served the arm B model for the rest of the term. The assigned arm does not change, and the analysis is intention-to-treat.

individual

No

none

53 students

equal size