Improving results reporting in research

2023-10-11

2023-12-31

First-stage outcomes:
- Email open rate
- Page visit rate

Primary outcome:
- Index of research results reported (described in more detail in attached PAP)

The experiment is clustered at the study-level, but interventions will be at the individual-study pair level. All studies are randomized into one of three arms: two treatments and a control. More details on the experiment can be found in the embargoed experimental design field and the attached PAP.

This study will test the effectiveness of two types of nudges, "salience of missing information" and a lottery, at prompting collaborators to update the post-trial fields on their registration. The treatments are randomized at the trial entry level, and each individual in the sample will receive one set of emails (initial and, if needed, follow-up) per registration for which they are listed as a "collaborator", with the email content varying depending on whether the registration has been assigned to control, treatment 1, or treatment 2. As individuals on multiple trials will receive multiple emails and, most likely, multiple treatments, we will test both spillovers and dosage effects by including terms in our main specification for counts of control, treatment 1, and treatment 2 emails received per trial, as well as the number of collaborators on a trial.

Randomization is stratified by a categorical variable with three values: 1) “single trial PIs”: trials where no collaborator is listed on any other trial; 2) “multi-trial PIs”: trials where at least one collaborator is on at least one other trial, but no collaborators are on five or more other trials; and 3) “many-trial PIs”: trials where at least one collaborator is on at least five other trials.

Within the strata the randomization probability is set to weight all three arms evenly so as to produce three arms per stratum with near-equal numbers of trial entries.

We start with all collaborators on all trials listed on the AEA RCT Registry. We then apply 4 ex-ante sample restrictions. Specifically, the trials must:

- Be at least one year post the listed trial end date as of the sample extraction date
- Have some empty post-trial fields, with the set of relevant fields described in the pre-analysis plan
- Not have any collaborators who meet any of the following “centrality”-related criteria:
- be listed on >=20 registry entries
- be a collaborator on trials with >=20 other collaborators

Collaborators on trials that meet the following criteria will still receive treatment, but we will
exclude them from the main analysis:
- Any collaborator on the trial was in the 63-person sample for our associated qualitative survey of registry users

Done by a computer (in R)

Research study level

Yes

4,717 studies.

6,312 study-author pairs

Approximately 1,572 studies (clusters) per each of the three treatment arms