Minimum detectable effect size for main outcomes (accounting for sample
design and clustering)
H1) Trend Test on Pos>Zero>Neg we code as a variable with 1,0,-1. We estimate mixed-effects regressions with participant-level random effects and interactions between the trend variable and the treatment conditions.
The power analysis is based on standard deviations from a previous classroom experiment (SDs ranging from 25 to 50)
and high within-subject correlations across environments (approximately 0.8–0.9).
Assuming 240 participants, 80% power, and a 5% significance level, the approximate minimum detectable effects are:
Trend: 2.5–3; Trend × treatment interaction: 5–6; Three-way interaction: 10–12, measured on the 0–150 decision scale.
Overall, the design is well powered to detect small trend effects and moderate treatment differences in trends.
H3) Rank preservation: To test this hypothesis, participants are ranked separately within each environment, and Kendall’s coefficient
of concordance W is computed across the three rankings. Kendall’s W ranges from 0 (no rank agreement) to 1 (perfect rank preservation).
The power analysis for Kendall’s W uses a simulation-based approach with: N=240, three repeated environments, 5,000 simulation replications, and a two-sided significance level of 5%.
The simulations assume latent correlations across environments ranging from weak to strong dependence. The minimum detectable concordance effect with 80% power corresponds approximately to ρ≈0.15 which reflects relatively weak rank stability.
Pilot data from a previous classroom experiment suggest substantially stronger associations, with observed pairwise correlations around 0.8 or higher. Under such conditions, statistical power is expected to be substantially above 80%, implying that the planned sample size is sufficient to detect even moderate deviations from perfect rank preservation.