Causal Impact of Content-Aligned Expertise on Grant Peer-Review Predictive Performance

Last registered on December 26, 2025
View Trial History

Pre-Trial

Trial Information

General Information

Title
Causal Impact of Content-Aligned Expertise on Grant Peer-Review Predictive Performance
RCT ID
AEARCTR-0017412
Initial registration date
December 05, 2025

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
December 26, 2025, 1:43 AM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Primary Investigator

Name
Chiara Franzoni
Affiliation
Polytechnic University Milan
Contact Primary Investigator

Other Primary Investigator(s)

Additional Trial Information

Status
In development
Start date
2025-12-09
End date
2026-02-28
Keywords
Education, Firms & Productivity, Other
Additional Keywords
grant peer review; expertise; predictive validity; forecasting; correctness; content alignment
JEL code(s)
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
This experimental study uses randomized matching to identify the causal effect of reviewer content expertise on the predictive validity of grant-proposal evaluations. The principle of “deference to expertise” assumes that reviewers whose expertise is more closely aligned with a proposal’s content should be more likely to predict the expected social value of proposals than reviewers with weaker content alignment. Identifying the causal impact of expertise alignment with observational data is challenging, because expertise is the primary criterion for review assignment generating endogeneity in content-alignment of assignments. We overcome this challenge by experimentally randomizing reviewers into high- or low-alignment conditions, enabling identification of the causal impact of content expertise on predictive performance.
External Link(s)

Registration Citation

Citation
Franzoni, Chiara. 2025. "Causal Impact of Content-Aligned Expertise on Grant Peer-Review Predictive Performance ." AEA RCT Registry. December 26. https://doi.org/10.1257/rct.17412-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
This experimental study uses random assignment of proposals to reviewers to identify the causal effect of reviewer content alignment on predictions evaluations. For each reviewer–proposal pair, we compute an ex-ante measure of content alignment between each reviewers’ prior publications and the text of the proposals. Random assignment of proposals to reviewers therefore induces exogenous variation in reviewer content expertise across evaluations. Because expertise measures are constructed entirely from pre-treatment information, observed differences in outcomes across levels of expertise can be causally attributed to this variation.
Intervention Start Date
2025-12-15
Intervention End Date
2026-01-31

Primary Outcomes

Primary Outcomes (end points)
Prediction errors
Primary Outcomes (explanation)
Prediction errors are measures as the absolute or quadratic difference between the predicted outcome and actual project realizations (publications, citations, mentions in press, media and policy, clinical collaborations and innovation disclosure) of project r expressed in absolute number or quantiles.

Secondary Outcomes

Secondary Outcomes (end points)
Aggregation benefit (Wisdom-of-the-crowd)
Secondary Outcomes (explanation)
We further compare crowds of experts/non-experts and crowds of variable sizes against best and worst experts. The aggregation benefit or crowd wisdom is computed as the difference between Mean Absolute Error and Absolute Error of the Mean of group predictions.

Experimental Design

Experimental Design
We pre-screen participants to check the alignment between their background expertise and each proposal content. We use this information to identify eligible reviewers as those who are minimally competent and have at least one high and one low content alignment match. We then randomly assign them to one of two conditions: High-alignment (they evaluate a proposal selected at random from their high-alignment options); Low-alignment (they evaluate a proposal selected at random from their low-alignment options).

After receiving their assigned proposal, reviewers provide predictions on the project’s future research impact. The causal effect of content alignment on prediction accuracy is estimated by comparing prediction accuracy across the two randomly assigned groups, while accounting for proposal-level differences.
Experimental Design Details
Not available
Randomization Method
Randomization is done in office, using a random generator provided by a common software (e.g., Excel or Python).
Randomization Unit
Randomization units are individual-proposal matches. Each individual-proposal match can result in a low-content alignment match or high- content alignment match.
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
Minimum 490 observations.
The ideal sample we hope to achieve is about twice as big.
Sample size: planned number of observations
Minimum 490 observations. We hope to achieve a sample about twice as big. However, the final number of observations is difficult to predict, as it may be substantially affected by attrition, given that we need to recruit very qualified participants (scientists who publish in the biomedical literature). Even if these participants have signed-up to volunteer in the study, it is customary to in academia to turn-down peer review jobs for time-constraints.
Sample size (or number of clusters) by treatment arms
We expect the proportion in treatment to be between 0.4-0.5, and thus aim for a minimum of 230 in each treatment arm.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
With a two-sided significance level of 0.05, a 0.4 statistical power of 0.80, we expect to need at least 490 subjects to detect an MDE of 0.1 This calculation assumes a standard deviation of about 0.27 (based on prior observational studies in similar areas). This estimates assumes a high attrition (about 48%), which is to be expected in peer-review assignments and in consideration of the the substantial time effort needed and the highly-qualified profile of participants. The ideal sample we hope to achieve is at least twice as big.
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
NBER IRB
IRB Approval Date
2025-12-04
IRB Approval Number
IRB Ref#25_212
Analysis Plan