Many Economists

Last registered on September 21, 2026
View Trial History

Pre-Trial

Trial Information

General Information

Title
Many Economists
RCT ID
AEARCTR-0019741
Initial registration date
September 15, 2026

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
September 21, 2026, 9:38 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Country
United States of America
Region

Primary Investigator

Name
Nick Huntington-Klein
Affiliation
Seattle University
Contact Primary Investigator

Other Primary Investigator(s)

PI Name
Claus Portner
PI Affiliation
Seattle University
Contact Investigator

Additional Trial Information

Status
Completed
Start date
2022-10-17
End date
2023-12-31
Keywords
Labor, Other
Additional Keywords
applied microeconomics, metascience, replication
JEL code(s)
C00
Secondary IDs
10.17605/OSF.IO/CJ9YX
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
This study has a large number of teams attempt the same research task (in applied microeconomics and causal inference) with a goal of seeing how research choices and outputs vary between those researchers. Multiple rounds of work, with differing degrees of restrictions on what researchers can do and how much freedom they have to produce their work, which allows us to isolate the sources of variation between reserachers.
External Link(s)
Original OSF preregistration

Registration Citation

Citation
Huntington-Klein, Nick and Claus Portner. 2026. "Many Economists." AEA RCT Registry. September 21. https://doi.org/10.1257/rct.19741-1.0
Sponsors & Partners
Experimental Details

Interventions

Intervention(s)
The research task will be manipulated, with the original research task being assigned in stage 1, the research task with the research question specified being assigned in stage 3, and the research task with the research question specified and the data pre-cleaned assigned in stage 5.

The peer review assignments will be assigned in stages 2, 4, and 6.
Intervention (Hidden)
Intervention Start Date
2022-10-17
Intervention End Date
2023-12-31

Primary Outcomes

Primary Outcomes (end points)
- Researcher characteristics from the attached sign-up form
- For each round of the replication, responses to the Research Justification Survey
- The successful completion (or failure and attrition) of each researcher in each stage
- Programming language(s) used in the turned-in code
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The main outline of the study is repeated-measures. At six different opportunities, participants will submit their results and take a survey. Analysis will compare these results across the six opportunities. 1. Initial research task 2. Optional revision following peer review 3. Research task with more precisely-defined research question 4. Optional revision following peer review 5. Research task with more precisely-defined research question and pre-cleaned data 6. Optional revision following peer review Within this structure, the peer review stages (2, 4, 6) are randomized and there are two-group comparisons, where 2/3 of the participants will be assigned to treatment (performing a peer review and having peer review performed on their work), and the rest to control. Among those assigned to treatment, they will be assigned to random pairs, and those pairs will review each others' work. There will then be between-group comparisons, evaluating the work of those who did peer review against those who did not, and also comparing the different randomly-assigned pairs of peer reviewers against each other.
Experimental Design Details
Randomization Method
Randomization will be done in office by a computer.
Randomization Unit
Individual researcher
Was the treatment clustered?
No

Experiment Characteristics

Sample size: planned number of clusters
The study aims for a minimum of 90 replicators who complete all stages of the replication. To achieve this, there was a goal of recruiting at least 200 replicators into the project, to account for potential attrition.
Sample size: planned number of observations
270 research-task outputs
Sample size (or number of clusters) by treatment arms
This is a within design so all researchers will be exposed to all treatments. This means 90 researchers per arm.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
This 90-completed-replication figure is based on a simulation-based power analysis. In simulated data where each research stage has 5% less between-researcher variation than the previous stage (which is likely a conservative estimate of the effect), the linear relationship between research stage and the squared deviation from the mean was statistically significant at the 95% level 90% of the time with 90 included researchers, implying 90% power for the test. 90 researchers also provides 85% power for comparison of any two stages in which there is a 15% decline in variance between those two stages. A 15% decline between stages seems reasonable given the findings in the many-analysts literature.
Supporting Documents and Materials

Documents

Document Name
Research Task with Question and Data Specified
Document Type
other
Document Description
The Task 3 instructions
File
Research Task with Question and Data Specified

MD5: 401230a697791a9c55d306d878f943a4

SHA1: 8e0c27208a9f6ac63cefcf769694009080bc0438

Uploaded At: September 15, 2026

Document Name
Research task
Document Type
other
Document Description
Round 1 instructions
File
Research task

MD5: bf15687b5a0da8a6b94834c8d18367e2

SHA1: d7cb024af7a3073044c5c1bcc8c797af0cf7fa24

Uploaded At: September 15, 2026

Document Name
Referee instructions
Document Type
other
Document Description
Peer-review instructions
File
Referee instructions

MD5: 14515d9a6e687551b0a00b6fa4af3a33

SHA1: 2da4fcff788af5684d4463f466386435112adfd4

Uploaded At: September 15, 2026

Document Name
State-level data documentation
Document Type
Document Description
Documentation for the provided state-level immigration policy data
File
State-level data documentation

MD5: b0d0df406d50d599f4bf859639cbabdf

SHA1: d3de249a9b71e4d6e48c023dc5e4039eb92fffa0

Uploaded At: September 15, 2026

Document Name
Outreach text
Document Type
other
Document Description
Text used in recruitment
File
Outreach text

MD5: 8e060465e7c414fa63b942f37b0dbe31

SHA1: 5e4cf0de52d7da3fa6e1257b55ce0125c709b0d4

Uploaded At: September 15, 2026

Document Name
List of Societies
Document Type
other
Document Description
List of contacts for economics societies to contact by newsletter
File
List of Societies

MD5: 692da4e4dceb4dad3901ff186246fb59

SHA1: edde62b26c0ce024da9e0108ff8b65410fb04a5d

Uploaded At: September 15, 2026

Document Name
Sign up form
Document Type
other
Document Description
Signup form text
File
Sign up form

MD5: ecc069ee96bc603fa00b390d22fa21c4

SHA1: 4c1badfdf463befe391273e3cf4f7d82a726c7a8

Uploaded At: September 15, 2026

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
Seattle University IRB
IRB Approval Date
2023-08-23
IRB Approval Number
N/A
Analysis Plan

Analysis Plan Documents

Preregistration Port Analysis.docx

MD5: 72e71c1c93074742d874c8a87e4e682c

SHA1: b6d5b31d1540462d0f3382b717eaccaef9196929

Uploaded At: September 15, 2026

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials