Factors that influence perception of research quality

In this survey experiment, we survey roughly 5000 academic economists to measure factors that influence their assessment of research quality. We are particularly interested in factors that can be chosen by the researcher, and interested in any demographic effects. In particular, we will explore the effect of 1) spelling errors, 2) grammar errors, 3) the choice of writing program (e.g., LaTeX vs. Microsoft Word) and 4) the interaction effects from subject characteristics (experience with editorial experiences, age, personal use of writing, non-native speaker). To do this, subjects will be assigned to one of 6 treatments (cover pages) that vary according to whether they are written in Microsoft Word or LaTeX, whether they have spelling errors, and whether they have grammar errors. After exposure to one of the six treatments, subject perception of research quality will be assessed using a series of Likert scales.

2025-02-28

2025-03-31

After exposure to one of the six treatments (cover pages), subjects will be asked to assess the quality of the research described in the abstract. In particular, they will be asked, the degree to which they think the idea is i) interesting, ii) important, iii) well written and coherent, iv) likely to be published, and v) well cited. For each of these outcomes, subjects will describe their views on a four-point Likert scale (ranging from strongly disagree to strongly agree). Using the same Likert scale, subjects will also be asked their hypothetical recommendation to a handling editor should they be a reviewer for the research, and they best guess of the "overall quality" of the research. From this data, we will generate an outcome variable equal to unity if subjects "agree" or "strongly agree" and zero otherwise. We will also estimate effects defining the outcome variable as unity for subjects who "strongly agree" only.

Beyond the estimation of average treatment effects (e.g., the average effect of reading an abstract written in Microsoft Word, with spelling errors, or will grammar errors), we will explore a series of heterogeneous effects based on age, editorial experience, university affiliation, gender, race, ethnicity, and JEL code they are primarily associated with. We will also estimate effects after dropping people in the spelling error treatment (grammar error treatment) who did not say that they found a spelling error (grammar error). Relatedly, we will also estimate effects after dropping people who did not correctly guess whether the abstract was written in Microsoft Word or LaTeX.

Additional we will analyze the effects based on ethnicity more willing to forgive on spelling errors or grammatical errors for publication. We will also explore the effect of the user's primary (stated) writing program and their willingness to accept the paper for publication written in the opposing writing software (i.e. are LaTeX users more likely to reject a Word submitted paper). Finally, we will explore the effect of interacted treatments. For example, we will test whether the effect of latex is larger in the spelling error treatment.

Subjects (academic economists) will be recruited to participate in this survey experiment using their professional email addresses. They will first answer a brief (3 minute) survey about themselves (age, gender, editorial experience, for example). Then, subjects will be assigned to one of six treatments (cover pages) that vary according to whether they i) were written in Microsoft Word or LaTeX, ii) have a spelling error, or iii) have a grammar error. Subjects will then be asked to assess the quality of the underlying research, as described above.

Not available

Subjects will be randomly assigned to one of six treatments using Qualtric's randomization tool.

Randomization will be assigned at the individual level.

No

NA

We will survey roughly 5,000 economists (we are limited based on the amount of money we had to pay a research assistant to collect the email addresses). We don't know what the response rate will be.

NA