Synthetic Experiments for Decision-Making Under Uncertainty: Validating AI Agent-Based Research Methods Against Human Experimental Data

The synthetic experiments replicate the experimental design of a randomized controlled trial previously conducted with human subjects (pre-registered separately). The experiment with human subjects tested whether causal reasoning trainig (Intervention 1 via causal resoning game) and ChatGPT access (Intervention 2) produce complementary effects on strategic decision-making performance.

2025-11-04

2026-03-31

Performance score = (Awareness + Usage) / 2

Performance within experiment is measured using a two-attribute rubric. Awareness captures the quality of strategies ensuring target audiences notice and recall merchandise. Usage captures strategies driving actual purchasing behavior. Each attribute is scored 1-5. The overall performance score averages the two attributes. The same rubric is applied to both human and synthetic responses under blind scoring protocols. Distance to Expert Solution provides a rubric-independent performance measure. We calculate semantic similarity between each response and a gold-standard expert solution using embedding based cosine distance.

For each synthetic experiment $j \in \{1, \ldots, 16\}$, we estimate the same regression model as the human benchmark:
\begin{equation}
Y_i^{(j)} = \beta_0^{(j)} + \beta_1^{(j)} \text{Causal}_i + \beta_2^{(j)} \text{ChatGPT}_i + \beta_3^{(j)} (\text{Causal}_i \times \text{ChatGPT}_i) + \epsilon_i^{(j)}
\end{equation}
We will use the Chow test to assess structural equality and parameter stability between each synthetic experiment and the human benchmark. The Chow test evaluates whether the coefficients from the synthetic experiments are statistically different from those in the human experiment, testing the null hypothesis:
\begin{equation}
H_0: \beta_k^{(j)} = \beta_k^{\text{human}} \text{ for all } k \in \{0, 1, 2, 3\}
\end{equation}
Methods that fail to reject the null hypothesis at conventional significance levels will be considered high-fidelity approximations of human experimental behavior.
To assess whether synthetic responses span similar ranges and distributions as human responses, we will calculate the Jensen-Shannon divergence (JSD) between synthetic and human outcome distributions:
\begin{equation}
\text{JSD}^{(j)} = \text{JSD}\left(P_{\text{synthetic}}^{(j)} \| P_{\text{human}}\right)
\end{equation}
Lower JSD values indicate better distributional alignment. This metric captures whether synthetic agents produce the full spectrum of human-like responses rather than converging to modal patterns.

Causal reasoning score

Causal Reasoning Score assesses three attributes on 1-5 scales. The overall score averages the three attributes. This measure tests whether synthetic agents exhibit realistic causal reasoning patterns that match human responses. We also use a secondary measure which is the self-reported causal reasoning score as reported by both human and synthetic participants. Process Measures include response length and linguistic complexity, ChatGPT usage intensity (prompt count and conversation depth for GPT-access conditions), and solution originality (deviation from modal responses).

The synthetic experiments replicate the experimental design of a randomized controlled trial previously conducted with human subjects (pre-registered separately). The experiment with human subjects tested whether causal reasoning training and ChatGPT access produce complementary effects on strategic decision-making performance using a 2×2 factorial design with 800 participants. Using the results from the experiment with human subjects as ground truth, these 16 synthetic experiments test the same hypotheses and attempt to replicate the same results. The overarching goal is to test the extent to which synthetic agents approximate the responses of human subjects and therefore produce similar experimental outcomes. Key Design Principle: The experimental design and execution are identical across all 17 experiments (16 synthetic + 1 human). The ONLY difference is the participant pool: human subjects versus synthetic personas generated through different methods.

Not available

Simple randomization assigns personas to the four experimental conditions (n=200 per cell). Stratification balances on key demographic variables. Randomization seeds are logged for reproducibility.

Syntetic persona

No

0 clusters

800 synthetic personas

200 synthetic personas for treatment arm Non-Causal, No ChatGPT.
200 synthetic personas for treatment arm Non-Causal, ChatGPT.
200 synthetic personas for treatment arm Causal, No ChatGPT.
200 synthetic personas for treatment arm Causal, ChatGPT.

Each synthetic experiment uses N=800 (200 per cell), matching human sample. Power calculations from the human RCT apply. Verbatim from Human RCT: "With N = 800 (200 participants per cell), we have more than adequate power for all planned analyses. For the main effects assuming Cohen’s d = 0.3 (typical for educational interventions), the power exceeds 97%, using two-way ANOVA with α = 0.05. Pairwise comparisons between conditions maintain over 90% power for detecting small-to-medium effects (Cohen’s d = 0.3)."