Peer Review Clinical Trial
Official title:
Testing for the Presence of Authorship Bias in Peer Review
No consensus exists among biomedical journals on the subject of blinding during the peer
review process. Some journals attempt to remove all identifiers of authorship from potential
manuscripts before delivering to referees for peer review while others prefer to leave the
authorship transparent. Although peer review remains the gold standard for manuscript
evaluation, the lack of a standardized blinding process between editorial offices may be a
source of publication bias and make comparisons of manuscripts published in different
journals more difficult to interpret.
Proponents of a blinded peer review system believe that knowledge of authorship may leave
referees vulnerable to biases about those authors' previous research, perceived expertise,
institution, nationality, or gender. This shifts judgment away from the merits of the
scientific work and introduces pre-conceived notions about the identity and background of
the author into the review process. Conversely, supporters of transparent authorship argue
that knowledge of author identity makes it easier for the referee to provide more
appropriate critiques. Open author identification allows referees to compare the current
manuscript to previously published work by the author, and to recognize or identify
potentially important conflicts of interest.
The investigators therefore propose an experimental study to address the question of whether
blinding affects the likelihood a reviewer will recommend acceptance of a study being peer
reviewed. The investigators plan to work in partnership with a journal to have a fabricated
manuscript peer reviewed by a large number of reviewers; the authors will send this "test"
manuscript out either in blinded form (authors' names/institutions not shown to reviewers)
or in open form, with the names of several well-known, well-regarded authors and their
institutions visible to peer reviewers. The manuscripts will otherwise be identical.
The partnering journal, Clinical Orthopaedics and Related Research allows both open and
blinded peer review, regularly employing both approaches (authors can choose what kind of
review they prefer when they submit to CORR), and so the reviewers are unlikely to be
troubled by seeing a manuscript in either format.
The primary goal of this study is to determine whether the identification of a manuscript
with a prominent group of authors (open authorship) will affect the likelihood that
reviewers will recommend the manuscript for publication.
Additionally, the investigators plan to purposely place several errors in the manuscript,
and tally how often reviewers detect these mistakes, to determine whether reviewers read the
work of prominent authors less critically.
The investigators will also compare the reviewers' grading of the (identical) methods
section, to determine whether the identification of a manuscript with prominent authors
results in higher reviewer grades for methodological rigor.
This study tests the following specific hypotheses:
1. The visibility of prominent author names and institutions on test manuscripts will be
associated with increased likelihood that the manuscript will be recommended for
publication by peer reviewers. (Primary Study Endpoint)
2. The visibility of prominent author names and institutions will be associated with a
decreased likelihood that purposefully placed "errors" in the experimental manuscript
will be detected by the reviewers.
3. The visibility of prominent author names and institutions will be associated with
increased scores given by reviewers for the methods section, despite the fact that the
methods sections of the experimental manuscript will be identical.
NOTE:
All elements of the research protocol were approved by the University of Washington Human
Subjects Review Committee, including the study's opt-out design for reviewer recruitment,
since it was considered important that reviewers not know that the paper they evaluated was
part of a study. To minimize the potential for harm, the authors chose as the topic of the
fabricated test manuscript an intervention that could not be immediately put into practice
at reviewers' own centers (a particular team-training process in the operating room). In
addition, participating reviewers were informed which paper was the test manuscript at the
completion of data collection, so that they did not pursue this sort of team training based
on the results of the fabricated study.
To reduce the risk of an observer (Hawthorne) effect, it was important to minimize the
likelihood that participant reviewers would learn of the study while it was in progress. To
this end, the investigators elected not to pre-register the study in a trial registry.
n/a
Allocation: Randomized, Intervention Model: Parallel Assignment, Masking: Double Blind (Subject, Outcomes Assessor), Primary Purpose: Health Services Research
| Status | Clinical Trial | Phase | |
|---|---|---|---|
| Completed |
NCT03119376 -
Web-based Tool to Improve the Assessment of Reporting (COBPeer)
|
N/A |