Evaluating an AI-Powered Research Development Tool for Academic Productivity and Well-being

This is a two-arm, parallel-group, single-blind randomized controlled trial (RCT). Participants — PhD students and junior economists within five years of their doctorate — will be randomly allocated to either a control group or a treatment group in a 1:1 ratio, using stratified randomization by career stage. The intervention will last for 24 months. Data will be collected through baseline and endline surveys and continuous behavioral logging from the AI platform. The study aims to evaluate whether the productivity effects of AI access vary systematically with the researcher's level of accumulated tacit knowledge.

2026-05-11

2028-06-30

1. Change in objective academic progression milestones over the 24-month period, specifically measured by: (a) rate of desk rejections, (b) invitations to Revise and Resubmit (R&R), and (c) total number of working papers publicly circulated (e.g., NBER, CEPR, SSRN, or university working paper series)
2. Heterogeneity of the treatment effect on objective publication milestones by career stage: whether the impact of AI access on reducing desk rejections and increasing R&Rs, and the amount of research, differs systematically between PhD students and junior economists.
3. Change in the number of papers submitted for publication.
4. Change in self-reported job satisfaction.
5. Change in self-reported work-life balance satisfaction.

1. Change in algorithmic paper quality (0-100 score). At the endline, both treatment and control group working papers will be processed blindly through the 'Editor Agent' prompt of the ResearchAI tool to calculate a standardized quality score. This serves as a mechanistic measure of algorithmic compliance rather than a measure of ultimate scientific validity.

This is a two-arm, parallel-group, single-blind randomized controlled trial (RCT) evaluated over a 24-month period. The study investigates the causal impact of a structured, AI-powered Research Development Tool on the academic productivity of early-career researchers. Participants will be stratified by career stage (PhD students vs. junior economists within five years of their doctorate) and randomized in a 1:1 ratio into either a treatment group or a waitlist control group.

The treatment group will receive immediate access to the AI tool, which provides expert-level feedback on working papers. The waitlist control group will conduct their research as usual (which may include ad-hoc use of general-purpose LLMs) and will receive access to the treatment tool only after the 12-month intervention concludes. Our primary objective is to measure changes in real-world publication milestones (e.g., desk rejection rates, Revise & Resubmit invitations, and working paper circulation). Secondarily, we will test whether the treatment effect varies systematically by the researcher's level of accumulated tacit knowledge (career stage), identifying whether AI access compresses or amplifies existing inequalities in research output.

Not available

Individual participants will be randomly assigned to one of the two arms using a computerized stratified random assignment. Stratification is based on one binary variable: Career Stage (PhD Student vs. Junior Economist).

Individual participant

No

Not applicable (individual-level randomization).

Our primary recruitment target is 504 participants (252 per arm), which provides 80% power to detect a moderate effect (d = 0.50) on the main ATE and 80% between treatment and career stage. This is our reference scenario given that existing literature on AI and cognitively demanding tasks (Brynjolfsson et al., 2023; Noy & Zhang, 2023) reports effects in the small-to-medium range. Gender interactions are pre-specified as exploratory: detecting a triple interaction (treatment × career stage × gender) requires approximately four times the sample needed for the main interaction.

Arm 1 (Control Group - waitlist/status quo): 252 participants
Arm 2 (Treatment Group - RefereeAI Suite): 252 participants

Power calculations assume a two-arm, individual-level randomized trial with no clustering (α=0.05, Power = 80%). Crucially, our primary hypothesis investigates whether the AI tool democratises tacit knowledge—meaning we are testing an interaction effect (Treatment × Career Stage) rather than just an Average Treatment Effect (ATE). Our baseline recruitment target of 504 participants is powered under an Optimistic Perspective. Assuming around 50/50 stratified split between PhD students and junior economists, the Minimum Detectable Effect Size (MDES) for the interaction under different enrollment scenarios is as follows: - Optimistic Target (Moderate Effect, MDES = 0.50 standard deviations): Requires 504 total participants (252 per arm). This is our baseline operational target. (Note: detecting the main ATE alone at this magnitude would require just 126 total participants). - Reference Scenario (Small-to-Moderate Effect, MDES = 0.30 SD): Requires 1,396 total participants (698 per arm). If recruitment exceeds our baseline target, this is our extended goal. - Conservative Scenario (Small Effect, MDES = 0.20 SD): Requires 3,140 total participants (1,570 per arm). Operational Constraints on Power: These calculations depend on the demographic balance of our strata. Statistical power for an interaction is constrained by the size of the smallest cell, because the variance function p(1−p) is maximized at p=0.5. If our real-world recruitment pool deviates from a 50/50 split, the required sample size increases significantly to maintain the same MDES. For example, to detect an effect of d=0.50, a 30/70 demographic split increases the required sample from 504 to 524 participants to preserve the necessary minimum cell size. To transparently pre-register our power constraints, we outline the expected sample requirements under varying degrees of demographic imbalance (d=0.5): - Optimal Split (50/50): Requires 504 total participants (252/arm). The minimum cell size is 126. This is our baseline operational target. - Mild Imbalance (60/40 or 40/60): Requires 524 total participants (262/arm). - Moderate Imbalance (70/30 or 30/70): Requires 598 total participants (299/arm). - Severe Imbalance (80/20 or 20/80): Requires 786 total participants (393/arm). - Extreme Imbalance (90/10 or 10/90): Requires 1,396 total participants (698/arm). If the true effect of the AI tool is smaller, the required sample sizes increase drastically (d=0.3). - Optimal Split (50/50): Requires 1,396 total participants (698/arm). - Moderate Imbalance (70/30 or 30/70): Requires 1,662 total participants (831/arm). - Severe Imbalance (80/20 or 20/80): Requires 2,182 total participants (1,091/arm).