Experimental Design Details
We will conduct a randomized experiment to test the effect of the above-described incentive strategy on the ROC. The experiment will last up to 18 weeks beginning on Sept 19, 2023. Each week will be designated either an incentive week or control week, and all reviewer invitations initiated during a given week will be in the same treatment status. Randomization by weeks is necessary because the journal is unable to randomize at the individual or submission level, and a week-on/week-off approach ensures that the treatment and control weeks are uniformly spread across the experimental period. Review invitation letters will be switched on Tuesday of each week in order to avoid switches on weekends or holidays. The starting condition (incentive or control) will be chosen at random prior to study initiation.
During control weeks, the standard reviewer invitation letter will be sent out automatically by the Editorial Manager software anytime a handling editor invites a reviewer to review a manuscript. During intervention weeks, the incentive letter will automatically be sent. Reviewers who are invited using the incentive letter will be mailed a cheque in the amount of USD$250 by the journal once their review is submitted. Data regarding review acceptance, time to acceptance, time to review submission, and review quality, will be collected automatically by the Editorial Manager software. Review quality will be adjudicated by the handling editor using a standardized 100-point scale.
For the primary analysis, all control weeks will be pooled, and all intervention weeks will be pooled, thereby creating two groups of reviewer invitations. Group assignment will be determined solely by the letter that was used to invite the reviewer, irrespective of when the review is submitted, or when other reviewers for the same manuscript are invited.
While reviewers often review multiple versions of a manuscript, including the initial submission as well as any subsequent revisions, only first reviews will be eligible for the stipend. Follow-up reviews of manuscripts that have been revised and resubmitted will not be eligible. This is to ensure that a reviewer is only paid once per manuscript, and that only the first review is compensated. The control condition will be the standard reviewer invitation letter, slightly modified to inform the prospective reviewer that anonymized data from the Editorial Manager system may be used for research
Number of rounds--The budget can accommodate payments of USD$250 for up to 270 reviewers. The number of treatment weeks that can be accommodated by the budget will depend on the number of manuscripts submitted to the journal, the corresponding number of reviewer invitations sent, and the share of invited reviewers who accept the invitation and submit a report (which is likely impacted by the incentive payment and therefore not fully predictable).
We anticipate running the experiment between 10 and 16 weeks, depending on the treatment effectiveness and submission numbers, with the possibility of increasing the duration to 18 weeks if the submission numbers are substantially lower than expected during this period. We will assess the budget after each treatment week and not introduce additional treatment weeks once the remaining budget is potentially insufficient to pay the reviewer incentives in another week of treatment.
Because we anticipate a low reviewer response rate over the holidays, the period between December 12, 2023 and Jan 1, 2024 will be a blackout period during which the standard (control) reviewer letter will be used, but no data will be collected.
All information regarding the date that invitations are sent, the reviewer response, the time of report submission, and other key performance metrics is captured directly in Editorial Manager.