Transforming Economics Examinations: A Token-Based System for Enhanced Learning and Student Success

Every student in the class will have the chance to use tokens to access standardized forms of assistance, such as conceptual hints during one of the course’s exams.
If they participate, students will:
• Be asked to complete reflective journals and focus group interviews regarding using the token system during the exam.
• Permit us to use their data on Canvas, their token use data (e.g., when tokens were used, which exam questions they were applied to, and what type of hint or support was selected), and their responses to the reflective journals and focus groups. This combined dataset will allow us to assess token access and use’s performance and learning effects.
A reflective journal will be a Qualtrics survey implemented online and should take them 5-7 minutes. They may take them in the privacy of their home or another setting of their choosing. A focus group interview should take them 5-7 minutes and will be implemented in person on a prearranged date and time. We expect the participants to attend the focus groups and complete the reflective journals by the deadline, and we will give them about a week to do so in each case. These assignments will not count toward their Econ 1010 grade.

2025-09-22

2026-05-15

(1) direct performance effect through treatment-control comparisons in each midterm (i.e., midterm exam score), and (2) learning effect by comparing the final exam performance on specific modules between students who previously used tokens for that module and those who did not.

(1) Direct performance effect estimated using both exam-specific regressions and within-student fixed effects models to compare token and no-token conditions across midterms, controlling for midterm difficulty and student fixed effects.
(2) Learning effect estimated on final exam module-specific scores using regression models with module fixed effects, assessing whether prior token exposure for a module predicts higher performance on that module in the final.
(3) Additional analyses include treatment–covariate interactions (e.g., baseline GPA, diagnostic scores, demographic/background variables) to explore heterogeneous treatment effects.
(4) Models incorporate pre-specified covariates (academic records, demographic/background data, standardized test scores, and other IRB-approved sources) and apply multiple testing adjustments, clustered standard errors, and robustness checks.

Our evaluation strategy integrates quantitative and qualitative methods to assess the token-
based examination support system. Traditional randomized crossover designs require
participants to receive identical treatments in different sequences with washout periods (to
eliminate prior effects) between treatments (Jones and Kenward, 2014). We employ a
"modified" randomized crossover design where tests on different modules effectively eliminate
carryover effects while maintaining experimental rigor. This adaptation is necessary in
educational settings where identical tests cannot be repeated. Using our teaching database, we
will calibrate test difficulties across modules.
Design Structure:
1. Midterm 1 (Module 1)
- Group A: With tokens
- Group B: Without tokens
2. Midterm 2 (Module 2)
- Group A: Without tokens
- Group B: With tokens
3. Final Exam (Module 1+2)
- All students: Without tokens
Our design identifies two causal effects using student-level observations: (1) direct
performance effect through treatment-control comparisons in each midterm, and (2) learning
effect by comparing final exam performance on specific modules between students who
previously used tokens for that module and those who did not. While order effects cannot be
identified in a single experiment, repeated implementations across courses would enable such
identification. Using standardized scores ensures fair evaluation while controlling for difficulty variations.

Not available

Randomization will be done in office by a computer

Individual students

No

280

280

140 per group.
Since the experiment will be conducted in two semesters, we plan to have: 40 per group in Fall 2025 semester and 100 per group in the Spring 2026 semester.

Our power analysis indicates detecting a 5-point difference requires at least 184 students (92 per group) for learning effects and 30 students for direct performance effects (α=0.05, power=0.8, sd=12).