The challenges of double-blind peer review in an era of increasing research transparency

Last year, we presented a vision to make Prosthetics and Orthotics International the journal of choice for all those who wish to access and contribute to the ever-increasing body of knowledge in our field. ¹ For Editors-in-Chief, this entails not only day-to-day management of manuscript submission, peer review, and publication processes, but also consideration of evolving principles and practices regarding the reporting of research. In this editorial, we discuss the challenges we experience in maintaining a double-blind peer review process while also encouraging research transparency.

Types of peer review processes

As we described in a previous editorial, ² peer review of manuscripts is a process of subjecting scholarly work to the scrutiny of other experts in the field. Reviewers provide feedback to authors that ideally helps to improve how the work is presented, and they provide editors with recommendations as to whether the work should be considered for publication in its current form. Peer review is thus intended to help improve the quality of submitted manuscripts and identify those manuscripts that are most suited for publication.

Peer review can be either double-blind (i.e. where an attempt is made to conceal the identity of both reviewers and authors), single-blind (i.e. where the identity of authors is revealed to reviewers but not vice versa), or open (i.e. the identity of reviewers and authors are known to each other). Open peer review may even involve publication of peer review feedback and author responses, along with the published manuscript (or provided to the authors in the event of a manuscript rejection). ³ The main argument for blinding a reviewer to the authors’ identity is that the reviewer should assess manuscripts on their own merit, rather than on the author’s reputation or prior record. Likewise, the principal argument for blinding authors to a reviewer’s identity is that a reviewer should be able to provide critique, and criticism when appropriate, without fear of repercussion from the authors. Proponents of open review argue that not blinding authors or reviewers increases transparency and accountability, provides for better inclusivity in research processes, and recognizes the reviewer’s contribution to published research literature. ⁴

Single-blind peer review is the most common approach used in biomedical journals, ⁵ in part because it is perceived to offer the best chance of an honest review without potential repercussions to the reviewer. However, given that the reviewer is aware of the identity of the authors, it may be subject to a reviewer’s personal biases (e.g. decisions which are consciously or unconsciously influenced by sex, race, nationality, seniority, reputation and/or affiliation of the author). It may also result in reviewers being unnecessarily harsh since they remain unknown to the authors. Double-blind peer review is also common among biomedical journals ⁵ and may attenuate some of these issues (e.g. reviewer bias) but not others (e.g. unnecessarily harsh reviews). While less common, open review has gained some traction with publishers such as PLoS, BMJ, and Biomed Central. However, contradictory perceptions exist regarding open review: while it may encourage better quality reviews because it is less secretive, it may also encourage less honest, less critical, or less rigorous reviews for the same reason. ⁵

Prosthetics and Orthotics International currently uses a double-blind peer review process. Despite double-blinding, reviewers may still recognize authors through other mechanisms such as writing style, subject matter, and self-citation. ⁵ Maintaining anonymity in double-blind peer review can be challenging for a small field such as prosthetics and orthotics, particularly given that researchers tend to publish in very narrow sub-specialties. In some cases, it can be relatively easy for a reviewer to identify the author, particularly if the work reported in a manuscript builds on prior published work or has been presented at a major conference. ⁶ Double-blinding can also be challenging when authors have pre-registered their study or published their protocols prior to commencing the research. ⁶ Blinding reviewers to authors’ identities in such situations means that reviewers are unable to identify possible publication and reporting bias by checking the pre-registered information against the submitted manuscript.

We are also beginning to see authors submitting manuscripts to Prosthetics and Orthotics International that have been previously posted to a preprint server. Preprint servers make manuscripts immediately and freely available (open access) because they do not require that manuscripts be subjected to peer review prior to posting. Preprints are easily citable because they are assigned their own unique digital object identifier (DOI). ⁷ This makes them useful when authors need to reference a manuscript in a grant or other article. SAGE allows authors to submit manuscripts to most of their journals, even if they have been previously made available on a preprint server. ⁸ Prosthetics and Orthotics International currently requires authors to disclose if they have previously posted a submitted manuscript to a preprint server by providing the DOI. While this information is not passed on to reviewers, the existence of preprints hinders our ability to maintain author anonymity during peer review.

Importance of transparency in research reporting

Further challenging the feasibility of blinding authors’ identities is the increasing desire for transparency in research reporting. Initially fueled by concerns regarding publication bias and an inability to replicate published research, ⁹ publishers and editors have increasingly sought to improve the ways in which research is reported. This has led to mandates requiring prospective registration of clinical trials and efforts to make study and review protocols available prior to conducting the study. As previously reported, Prosthetics and Orthotics International endorses the International Committee on Medical Journal Editors (ICMJE) requirement that researchers prospectively register their clinical trials in a public registry and encourages registration of systematic reviews. ¹⁰ Protocol registration and publication is intended to reduce publication bias (by eliminating selective publication and reporting of research outcomes) and to prevent unnecessary duplication of research. However, registering and/or publishing a protocol creates an early scientific record that identifies the authors, particularly given details of the aims or method which can be matched to the manuscript under peer review.

While improving the transparency of research reporting is an important goal, it increasingly challenges our ability to blind authors. ¹¹ For example, while citing a published protocol may demonstrate that the methodology has already received peer scrutiny, doing so identifies the authors. At present, Prosthetics and Orthotics International requests information about protocol registration at the time of submission, but blinds reviewers to these details. We instead rely on editors, who can access details that are otherwise blinded, to scrutinize and compare the registered protocol with that being reported in the manuscript and to assess whether there is any publication or reporting bias. Blinding reviewers to prior work, such as cited protocols, is more difficult given that authors often rely on referencing the more detailed protocol and provide only a summary in their methods. In addition, registered protocols and the manuscript under peer review often have very similar titles or keywords, making them easy to link. Like other Editors-in-Chief, we grapple with the concern that blinding reviewers to registered (and possibly published) protocols might hinder efforts to promote research transparency. ¹¹ In contrast, reviewers who have access to the protocol through a trial registry or prior publication, can read the submitted manuscript in the context of what has been proposed originally, thus providing greater scrutiny and assisting editors in the process of holding authors accountable for the research that they report.

Conclusions

Clearly, maintaining the anonymity of authors during peer review can be difficult. We anticipate that it will only become more challenging as we strive to increase the transparency of research published in Prosthetics and Orthotics International. The challenge facing Editors-in-Chief and Editorial Board members is how to promote research transparency in accordance with contemporary publication standards, while maintaining anonymity of authors and facilitating an objective and fair process of peer review.

These issues will certainly require further consideration, particularly as we weigh the merits of alternative forms of peer review (i.e. single-blind or open peer review) for Prosthetics and Orthotics International. ¹¹ Another possibility that may be worth considering is making the double-blind peer review process optional, as journals like Nature Research have done. With such a policy, the decision of whether or not to blind a manuscript would fall to the authors. Some authors might choose double-blind peer review because of concerns of bias, while others may opt for single-blind peer review in order to capitalize on the benefits of publishing a protocol in advance of disseminating results of a study.

As we weigh these options, we welcome feedback from our readers, authors, reviewers, and editors to make Prosthetics and Orthotics International the journal of choice for all who wish to consume or contribute to our ever-growing body of knowledge.