|
Field
Trial End Date
|
Before
May 31, 2021
|
After
December 31, 2021
|
|
Field
Last Published
|
Before
December 23, 2020 11:21 PM
|
After
May 17, 2021 08:26 PM
|
|
Field
Intervention End Date
|
Before
March 31, 2021
|
After
May 19, 2021
|
|
Field
Primary Outcomes (End Points)
|
Before
1. Estimated effect of language editing on writing quality.
2. Estimated effect of language editing on paper quality.
3. Estimated effect of writing quality on paper quality.
|
After
1. Estimated effect of language editing on writing quality.
2. Estimated effect of language editing on paper quality.
3. If we find significant effects for primary outcome 1, we will also estimate the effect of writing quality on paper quality.
|
|
Field
Primary Outcomes (Explanation)
|
Before
Explanation for primary outcome 1. Academic economists will rate the writing quality of a paper on an 11-point scale ranging from 0 (very bad) to 10 (very good).
To estimate the effect of language editing on writing quality, we will regress writing quality on an indicator showing whether the paper is edited or not. To increase precision of our estimates, we will also include the standard controls (see below).
Explanation for primary outcome 2. Academic economists will rate the quality of a paper on an 11-point scale ranging from 0 (very bad) to 10 (very good).
To estimate the effect of language editing on perceptions of paper quality, we will regress paper quality on an indicator showing whether the paper is edited or not. To increase precision of our estimates, we will also include the standard controls.
Explanation for primary outcome 3. The same academic economists will also rate the quality of the writing on an 11-point scale ranging from 0 (very bad) to 10 (very good).
We estimate the effect of writing quality on perceptions of paper quality with an instrumental variable approach. In the first stage, we predict writing quality with a dummy variable indicating whether the paper is edited or not (the first stage is identical to the analysis for primary outcome 1). In the second stage, we estimate the effect of writing quality (as predicted in the first stage) on perceived paper quality. To increase precision of our estimates, we will include the standard controls in both stages.
These are the “standard” control variables:
• Rater fixed effects (i.e. academic judge or writing expert)
• Paper fixed effects
|
After
Explanation for primary outcome 1. Academic economists will rate the writing quality of a paper on an 11-point scale ranging from 0 (very bad) to 10 (very good).
To estimate the effect of language editing on writing quality, we will regress writing quality on an indicator showing whether the paper is edited or not. To increase precision of our estimates, we will also include paper fixed effects.
Explanation for primary outcome 2. Academic economists will rate the quality of a paper on an 11-point scale ranging from 0 (very bad) to 10 (very good).
To estimate the effect of language editing on perceptions of paper quality, we will regress paper quality on an indicator showing whether the paper is edited or not. To increase precision of our estimates, we will also include paper fixed effects.
Explanation for primary outcome 3. If we find significant effects of language editing on writing quality, we will also estimate the effect of writing quality on paper quality with an instrumental variable approach. In the first stage, we predict writing quality with a dummy variable indicating whether the paper is edited or not (the first stage is identical to the analysis for primary outcome 1). In the second stage, we estimate the effect of writing quality (as predicted in the first stage) on paper quality. To increase precision of our estimates, we will include paper fixed effects in both stages.
|
|
Field
Randomization Method
|
Before
Randomization for main analysis sample:
We have 30 papers and 24 academic raters for our main analysis.
The sampling procedure works in two steps. First, we assign “paper numbers” to “rater numbers”. Second, we randomly assign papers to paper numbers, and raters to rater numbers. For example, this random assignment might assign the paper “The effect of X on Y” to be paper number 14 and rater “Professor X at University Y” to be rater number 21. This assignment will be done with a computer.
Details for the first step. We assign “paper numbers” (1–30) and “rater numbers” (1–24) as follows:
• Rater numbers 1–8 see paper numbers 1–10. Rater numbers 1–4 see the edited versions of the odd-numbered papers. Rater numbers 5–8 see the edited versions of the even-numbered papers.
• Rater numbers 9–16 see paper numbers 11–20. Rater numbers 9–12 see the edited versions of the odd-numbered papers. Rater numbers 13–16 see the edited versions of the even-numbered papers.
• Rater numbers 17–24 see paper numbers 21–30. Rater numbers 17–20 see the edited versions of the odd-numbered papers. Rater numbers 21–24 see the edited versions of the even-numbered papers.
These paper numbers are presented in random order to each rater. Appendix B shows the paper-to-rater assignment for each of the 24 raters. In this assignment, each paper gets gated by 8 raters (who see 4 edited and 4 non-edited). And each rater sees 10 papers (5 edited and 5 non-edited).
Randomization for language expert sample:
We have 30 papers and 15 language expert raters. The assignment follows a similar procedure to the assignment for our main analysis sample. Appendix B shows the assigned number for each of the 15 raters. In this assignment, each paper gets rated by 4 raters (who see 2 edited and 2 non edited) and each rater sees 8 papers (4 edited and 4 non-edited).
|
After
Randomization for main analysis sample:
We split the papers by topic into a micro-economics group (20 papers) and a macro-economic group (10 papers).
We randomly assign each paper within a group one “paper number”, so that papers in the micro group will be assigned numbers 1-20 and papers in the macro group will be assigned numbers 21-30.
After that, we create the following six paper blocks:
1. Papers 1-10 (micro), papers 1-5 original version, papers 6-10 edited version
2. Papers 1-10 (micro), papers 1-5 edited version, papers 6-10 original version
3. Papers 11-20 (micro), papers 11-15 original version, papers 16-20 edited version
4. Papers 11-20 (micro), papers 11-15 edited version, papers 16-20 original version
5. Papers 21-30 (macro), papers 21-25 original version, papers 26-30 edited version
6. Papers 21-30 (macro), papers 21-25 edited version, papers 26-30 original version
Randomization for our language expert sample:
We randomly assign each paper one “language paper number”. This is a different randomization, the “language paper numbers” are not the same as the “paper numbers”.
After that, we create the following six blocks:
1. Papers 1-10, papers 1-5 original version, papers 6-10 edited version
2. Papers 1-10, papers 1-5 edited version, papers 6-10 original version
3. Papers 11-20, papers 11-15 original version, papers 16-20 edited version
4. Papers 11-20, papers 11-15 edited version, papers 16-20 original version
5. Papers 21-30, papers 21-25 original version, papers 26-30 edited version
6. Papers 21-30, papers 21-25 edited version, papers 26-30 original version
This assignment of papers to paper numbers will be done with a computer.
The papers will be presented in random order to each rater. This randomization will be done within the survey by Qualtrics.
Assignment of raters to paper blocks
For economists:
We create two lists of academic economists from Australian universities or research institutes: one for micro economists and one for macro economists.
We invite economists in both groups via email to evaluate 10 academic papers in their discipline.
Those who agree to evaluate ten papers are assigned to a paper block, and we will send them a survey containing links to the relevant papers. More specifically, micro-economists will be assigned to paper blocks 1-4 (which contain micro papers) and macro-economists will be assigned to paper blocks 5-6 (which contain macro papers).
The order of the paper block assignment will be determined by the order the economists agree to participate. For example, the first micro-economist will be assigned to paper block 1, the second micro-economist will be assigned to paper block 2, etc. Similarly, the first macro-economist will be assigned to paper block 5 and the second macro-economist will be assigned to paper block 6.
We will deviate from this assignment-procedure to avoid that economists from the same institution are asked to judge different versions of the same paper within a short time. Economists from the same institution are more likely to talk to each other about the task and therefore might realize that we have included different versions of the same paper in the experiment. This may raise suspicions, which we want to avoid. We will solve this problem by swapping the assignment to paper blocks with economists from other institution. For example, if two economists from the University of Melbourne would have been assigned to paper-blocks 1 and 2, and one economist from the University of Sydney would have been assigned to paper-block 3, we would swap the paper-block assignments of the second University of Melbourne economist with the University of Sydney economist.
For language experts:
We will invite language experts to evaluate 10 academic papers.
If they agree, we will send them a survey containing links to the relevant papers.
The order of the paper block assignment will be determined by the order the language experts agree to participate. The expert who first agrees to participate will be assigned to paper block 1, the second expert will be assigned to paper block 2, etc.
As for the academic economists, we will deviate from this assignment procedure to avoid that experts from the same institution are asked to judge different versions of the same paper.
|
|
Field
Planned Number of Observations
|
Before
For our main analysis, we expect to have 240 paper-version-rater observations.
For the analysis by language experts, we expect to have 120 paper-version-rater observations.
|
After
For our main analysis, we expect to have 300 paper-version-rater observations.
For the analysis by language experts, we expect to have 180 paper-version-rater observations.
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
Main analysis:
Treatment group: 30 edited paper versions (= 120 paper-version-rater observations)
Control group: 30 non-edited paper versions (= 120 paper-version-rater observations)
Language expert analysis:
Treatment group: 30 edited paper versions (= 60 paper-version-rater observations)
Control group: 30 non-edited paper versions (= 60 paper-version-rater observations)
|
After
Main analysis:
Treatment group: 30 edited paper versions (= 150 paper-version-rater observations)
Control group: 30 non-edited paper versions (= 150 paper-version-rater observations)
Language expert analysis:
Treatment group: 30 edited paper versions (= 90 paper-version-rater observations)
Control group: 30 non-edited paper versions (= 90 paper-version-rater observations)
|
|
Field
Keyword(s)
|
Before
Other
|
After
Other
|
|
Field
Intervention (Hidden)
|
Before
Scientists often find it challenging to convey complex ideas in writing so that they can be easily understood. Conventional wisdom holds that overcoming this challenge will cause others, like referees and conference organisers, to evaluate one’s paper more positively.
In this project, we will test whether this wisdom is true and estimate the effect of writing quality. The intervention will consist of offering free language editing for papers of PhD students in economics. This language editing is done by two editors who work for a plain language consultancy and follows the procedure outlined in the language editing guide shown in Appendix A.
This study follows a within-paper design. Each paper has two versions, the edited and the non-edited version. Both versions will be evaluated by academic economists and language experts. Our estimates of the treatment effects are the difference in evaluations of edited papers and non-edited papers (conditional on control variables).
|
After
******Update: This experiment consists of two parts. In part 1, we collect and edit academic papers. In part 2, we ask experts to rate the original and edited version of these papers. This version of the trial registration contains updated information about part 2 which we document before the start of part 2. ******
Scientists often find it challenging to convey complex ideas in writing so that they can be easily understood. Conventional wisdom holds that overcoming this challenge will cause others, like referees and conference organisers, to evaluate one’s paper more positively.
In this project, we will test whether this wisdom is true and estimate the effect of writing quality. The intervention will consist of offering free language editing for papers of PhD students in economics. This language editing is done by two editors who work for a plain language consultancy and follows the procedure outlined in the language editing guide shown in Appendix A.
This study follows a within-paper design. Each paper has two versions, the edited and the non-edited version. Both versions will be evaluated by academic economists and language experts. Our estimates of the treatment effects are the difference in evaluations of edited papers and non-edited papers (conditional on paper fixed effects).
|
|
Field
Secondary Outcomes (End Points)
|
Before
We have seven secondary outcomes, which show the effect of language editing on raters’ assessment of the following.
1. The chance of getting the paper accepted at a conference.
2. Conditional on submission, the chance of the paper getting desk rejected.
3. The probability of getting published in a good journal.
4. The writing quality as judged by writing experts.
5. How easy the paper is to understand as judged by writing experts.
6. How easy the paper is engaging as judged by writing experts.
7. How well-structured the paper is as judged by writing experts.
We have three secondary outcomes, which show the effect of writing quality on the raters’ assessment of the following.
8. The chance of getting the paper accepted at a conference.
9. The chance of sending the paper to referees.
10. The probability of getting published in a good journal.
|
After
We have three secondary outcomes, which show the effect of language editing on economists’ assessment of the following.
1. The chance of getting the paper accepted at a conference.
2. Conditional on submission, the chance of the paper getting desk rejected.
3. The probability of the paper getting published in a good journal.
If we find significant effects of paper language editing on writing quality, we will also estimate the effect of writing quality on economists’ assessment of the following.
4. The probability of getting the paper accepted at a conference.
5. The probability of getting desk rejected.
6. The probability of getting published in a good journal.
We have five secondary outcomes which are based on the evaluations of language experts. These show the effect of language editing on:
7. The writing quality.
8. How easy the paper allows the reader to find the key messages.
9. To what extent the paper is free of spelling and grammar mistakes.
10. How easy to read the paper is.
11. How concise the paper is.
|
|
Field
Secondary Outcomes (Explanation)
|
Before
Secondary outcomes that look at the effects of language editing (1–7) will be estimated with regressions of the outcome variable on an indicator showing whether the paper is edited and the set of standard controls.
Secondary outcomes that look at the effect of writing quality (8–10) will be estimated using instrumental variable regressions which include the set of standard controls. This is the same approach as for primary outcome 3 (see above).
Secondary outcomes 5–7 are taken from a survey to language experts. All other outcomes are taken from a survey to academic raters.
Secondary outcomes 1 and 8 will use this survey question:
How likely would you be to accept this paper at a general economics conference (such as the Australian Conference of Economists)? Answer options: 0 to 100 percent.
Secondary outcomes 2 and 9 will use this survey question:
Imagine you were an editor of a general economics journal that is rated A on the ABDC list.
How likely would you be to desk reject the paper? Answer options: 0 to 100 percent.
Secondary outcomes 3 and 10 will use this survey question:
How likely is it that this paper will get published in an A or A* journal according to the ABDC list? Answer options 0 to 100 percent.
Secondary outcomes 4 will use this survey question:
Overall, the quality of the writing is. Answer options: 0 (very bad) to 10 (very good).
Secondary outcomes 5-7 will use these survey questions.
Please state how much you agree with each of the following statements. Answer options: 1 (Strongly disagree) to 5 (Strongly agree).
• The paper is easy to understand (outcome 5).
• The paper is engaging (outcome 6).
• The paper is well-structured (outcome 7).
|
After
Secondary outcomes 1-6 are based on survey responses of economists.
Secondary outcomes 7-11 are based on survey responses of language experts.
Secondary outcomes that look at the effects of language editing (1–3, 7-11) will be estimated with regressions of the outcome variable on an indicator showing whether the paper is edited and paper fixed effects.
Secondary outcomes that look at the effect of writing quality (4–6) will be estimated using instrumental variable regressions which include paper fixed effects. This is the same approach as for primary outcome 3 (see above). We will only estimate these effects if writing quality is significantly affected by language editing.
|