Bias in Grant Peer Review: A Field Experiment

Last registered on October 06, 2025
View Trial History

Pre-Trial

Trial Information

General Information

Title
Bias in Grant Peer Review: A Field Experiment
RCT ID
AEARCTR-0016800
Initial registration date
September 29, 2025

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
October 01, 2025, 8:02 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
October 06, 2025, 3:05 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country
Luxembourg
Region

Primary Investigator

Name
Cindy Lopes Bento
Affiliation
KU Leuven
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
L Katariya
PI Affiliation
Organization
PI Name
C Bento
PI Affiliation
Organization
PI Name
C Grimpe
PI Affiliation
Organization

Additional Trial Information

Status
In development
Start date
2025-04-23
End date
2025-12-08
Keywords
Behavior, Finance & Microfinance, Other
Additional Keywords
peer review, research proposal, funding agency
JEL code(s)
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Growing empirical evidence reveals systematic risk aversion in science funding agencies. In response, alternatives to the traditional peer-review process have been sought. Previous research shows that evaluation formats can have a strong impact on the evaluation scores which determine the fundability of the idea. The traditional peer-review system relies on reviewers providing holistic scores across a set of loosely defined criteria. For instance, in the NIH peer-review process, reviewers first evaluate proposals along 5 dimensions and afterwards provide a final overall score which is used to decide the final funding decision. In the absence of any formal way to aggregate scores of different dimensions into the final score, the dimensions of risk and return get conflated. Using a field-experiment we will investigate whether changes to measurement methods can help to uncover risk aversion and other biases.
External Link(s)

Registration Citation

Citation
Bento, C et al. 2025. "Bias in Grant Peer Review: A Field Experiment." AEA RCT Registry. October 06. https://doi.org/10.1257/rct.16800-2.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
New format of reviewer form.
Intervention (Hidden)
Treatment will include use of a modified version of the standard review form. The standard form (status quo) asks reviewers to provide written assessments as well as scores for various criteria as well as an overall score and overall assessment. The form used in the treatment asks the reviewers to complete the evaluation in two separate steps. In step 1, reviewers evaluate the proposal in a subjective expected utility (SEU) framework. It involves first identifying the project outcomes, then assessing the utility of each outcome and finally assessing the likelihood of the outcomes happening. In step 2, reviewers follow an evaluation process identical to the standard form. Only after completing step 1 can they move to step 2 but can no longer change their evaluations of step 1.
Intervention Start Date
2025-09-19
Intervention End Date
2025-12-08

Primary Outcomes

Primary Outcomes (end points)
Our key outcome variables of interest are the scores assigned to a proposal by a reviewer.
Primary Outcomes (explanation)
In the status quo, reviewers will assess a proposal on four criteria. Each criteria will be assessed on a 4-point Likert scale: (1) ‘Fair/Poor’, (2) ‘Good’, (3) ‘Very Good’, and (4) ‘Excellent’. In addition, reviewers will provide an overall assessment for the proposal by choosing one of the following: C, B, A or A+. In the treatment, in addition to the scores just described, reviewers will also provide scores for utility and likelihood of each outcome that they will identify. Utility will be scored on a scale from 0 to 10 and likelihood on a scale of 0–100% .
The total expected utility (TEU) will be calculated a the sum of the products of the utility and likelihood scores of each outcome.
We will also construct a 'fundability' variable based on the recommendation of the reviewers. In status quo, we will rank-order the proposals based on the average recommendation scores of the three reviewers. In treatment, we will rank-order the proposals based on the TEU values. Then, we will use a hypothetical funding line to decide which proposals will get funded or rejected. The funding line can be constructed in at least 2 different ways: 1) total no. of proposals that got funded in reality, and 2) total amount of funding disbursed in reality. Using one of these funding lines, we can obtain a binary measure of ‘Fundability’ of the proposals if they fall above or below this line.

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The experiment empirically tests for risk aversion in the peer review process.
Experimental Design Details
Blocked (or multi-site) design where each research proposal is reviewed by reviewers who either use the standard form (status quo) or the modified form (treatment). For each proposal, three reviewers are in status quo and one reviewer is in the treatment.
Randomization Method
Sequential randomization: the time of entry in the experiment decides assignment to control vs. treatment. The last reviewer to enter is assigned to the treatment group, while the first three reviewers are assigned to the control group.
Randomization Unit
Reviewer
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
532 reviewer-proposal pairs
Sample size: planned number of observations
532 reviewer-proposal pairs
Sample size (or number of clusters) by treatment arms
133 X 3 = 399 control, 133 X1 = 133 treatment
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
Supporting Documents and Materials
IRB

Institutional Review Boards (IRBs)

IRB Name
IRB Approval Date
IRB Approval Number
Analysis Plan

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials