Improving results reporting in research

This study tests the effectiveness of email nudges at encouraging AEA RCT Registry users to update post-trial fields after the study has ended. The interventions will be email-based nudges and will be tested against a control email. Randomization is at the registration level, but the treatment will be at the "collaborator" level, with "collaborator" defined as anyone who has an AEA RCT Registry account and has been signed up as an editing collaborator on the registry entry. That is, each collaborator on a study will receive the same treatment (email nudge). Because of this, individuals in our sample that are collaborators on more than one trial will get multiple treatments (emails), one for each trial, and will most likely receive different treatments (see the "experimental details" section for more).

Each treatment will consist of up to four emails, depending on how (to what extent) registrants respond to the first email. Specifically, all collaborators in the sample will receive the first email. If a collaborator visits their registration and makes any change to the trial entry within one week of the visit, they (and the other collaborators on the registration) will not receive any further emails. The follow-up emails will be identical to the original email received and will happen at periods of roughly two weeks from when the previous email was sent. If collaborators do not act on any of the four emails, interventions will end once the last (third) follow-up email has been sent.

We will send the emails through Qualtrics, which caps the number of emails that can be sent within one week. We will hence split the sample into two groups – those that are part of the “main registration network” and those that are not. The main network is defined as the largest connected set of collaborators in the bipartite graph defined by registrations and collaborators in our sample.

The treatment arms are as follows:
Control: collaborators will receive a standard email very similar to the automated currently sent to any users that are approaching the end date for their trial (the normal emails will be stopped at least two weeks ahead of intervention start). The control email provides a reminder that the registrant’s trial has ended and provides a link to the follow-up page that allows for entering/editing post-trial fields.

Treatment 1 (Salience of missing information): collaborators will receive an email that lists the post-trial fields for which the entry does not have any information (with the list in a large box in the middle of the email).

Treatment 2 (Incentive/lottery): collaborators will receive an email that informs them that if they complete their post-trial fields within two weeks they will be entered into a lottery to win a $100 Amazon gift card (also placed in a large box in the middle of the email)
Additionally, both treatment 1 and treatment 2 emails share the following common elements:
- homogenous formatting and text beyond the differences described above
- a large red button that takes registrants to the page where they can edit their fields.
- Two reminders to registrants:
- completing a registration helps other researchers both conduct their own research and to find and cite the collaborator's
- registrants who don't update their PTR fields within two years of trial end date are far less likely to do so at all

2023-10-11

2023-12-31

First-stage outcomes:
- Email open rate
- Page visit rate

Primary outcome:
- Index of research results reported (described in more detail in attached PAP)

The experiment is clustered at the study-level, but interventions will be at the individual-study pair level. All studies are randomized into one of three arms: two treatments and a control. More details on the experiment can be found in the embargoed experimental design field and the attached PAP.

This study will test the effectiveness of two types of nudges, "salience of missing information" and a lottery, at prompting collaborators to update the post-trial fields on their registration. The treatments are randomized at the trial entry level, and each individual in the sample will receive one set of emails (initial and, if needed, follow-up) per registration for which they are listed as a "collaborator", with the email content varying depending on whether the registration has been assigned to control, treatment 1, or treatment 2. As individuals on multiple trials will receive multiple emails and, most likely, multiple treatments, we will test both spillovers and dosage effects by including terms in our main specification for counts of control, treatment 1, and treatment 2 emails received per trial, as well as the number of collaborators on a trial.

Randomization is stratified by a categorical variable with three values: 1) “single trial PIs”: trials where no collaborator is listed on any other trial; 2) “multi-trial PIs”: trials where at least one collaborator is on at least one other trial, but no collaborators are on five or more other trials; and 3) “many-trial PIs”: trials where at least one collaborator is on at least five other trials.

Within the strata the randomization probability is set to weight all three arms evenly so as to produce three arms per stratum with near-equal numbers of trial entries.

We start with all collaborators on all trials listed on the AEA RCT Registry. We then apply 4 ex-ante sample restrictions. Specifically, the trials must:

- Be at least one year post the listed trial end date as of the sample extraction date
- Have some empty post-trial fields, with the set of relevant fields described in the pre-analysis plan
- Not have any collaborators who meet any of the following “centrality”-related criteria:
- be listed on >=20 registry entries
- be a collaborator on trials with >=20 other collaborators

Collaborators on trials that meet the following criteria will still receive treatment, but we will
exclude them from the main analysis:
- Any collaborator on the trial was in the 63-person sample for our associated qualitative survey of registry users

Done by a computer (in R)

Research study level

Yes

4,717 studies.

6,312 study-author pairs

Approximately 1,572 studies (clusters) per each of the three treatment arms