Peer review in Paramedicine: Owning imperfections and striving to optimise

Peer review is like democracy: a system full of problems but the least worst we have.

– Richard Smith

In 1665, the English journal Philosophical Transactions of the Royal Society implemented what is credited as being the first structured peer review process in scientific journal publishing. ¹ Throughout the ensuing 450 or so years, peer review has been subject to frequent criticism regarding its effectiveness as a means to ensure originality, validity, quality and appropriateness in what a given journal publishes. ² Over that time the nature of peer review has evolved considerably, with contemporary models of peer review being quite different to those one would have been exposed to when submitting to Philosophical Transactions four centuries ago. ³ In the post-World War 2 ‘modern era’ of publishing, multiple models and iterations of peer review have emerged, with distinctive differences present within and between disciplines. ³ Single-blind review, double-blind, open, transparent, collaborative, dynamic and post-publication review are but some of the approaches seen in contemporary publishing; all have arguable benefits and disbenefits, all are prone to error, and none have been proven objectively to be superior to another. ¹ Arguments largely lie in the eye of the beholder, and are often grounded in personal experience and preference rather than empirical fact. This editorial seeks to communicate to authors, reviewers, and the wider paramedicine academic community our philosophy toward, and approach to peer review, and to outline our efforts as an editorial entity to optimise the process and own its (and our) imperfections.

Whilst evolution has occurred and initiatives seeking to overcome the inherent limitations of peer review implemented, the merit and relevance of traditional peer review models continue to be argued, and the cries of ‘peer review is broken’ continue to echo within scientific and academic communities, particularly in the ‘blogosphere’. ⁴ Whilst it remains a core element in the publishing processes of thousands of journals, the objective reality is that little is known theoretically or empirically about peer review.^2,5 A systematic review of 19 studies by Jefferson et al. ² from 2002 reported that despite its widespread use, peer review was ‘largely untested and its effects uncertain’, whilst another concluded that until the objectives of peer review can be clearly defined it will be ‘almost impossible to assess or improve its effectiveness’. ⁶

On a more personal note, our editorial team have all been, and continue to be, authors themselves, and their individual experiences no doubt mirror those of researchers seeking to submit their work to Paramedicine. All have recollections of peer-review engagement that through their individual lens’ has spanned the continuum of helpfulness and quality, and most would report having encountered the widely recounted, but fortunately disproven, phenomenon of ‘Reviewer 2’.^7,8

At Paramedicine, we believe peer review is not broken. Imperfect? Yes. Fallible? Yes. But broken? No. To suggest it is broken infers it was previously ‘unbroken’; we argue it has never reached perfection, and most likely never will. We acknowledge its imperfections but argue in favour of its relevance and essentiality in our mission to publish quality research with potential to impact and create progressive dialogue within, the international paramedicine profession.

So, what is peer review and what is its purpose? There is no singular accepted definition or purpose, which may indicate the difficulty in navigating its tensions and opportunities. At Paramedicine, we view peer review as a collaborative dialogue between the triumvirate of author, reviewer, and editor. We seek to position it as collegial rather than adversarial, and desire authors and reviewers, who are in truth are one and the same at one time or another, to embrace it as such. We see the purpose of peer review as existing on two levels. At an individual level, we see it as an examination of a submission's strengths as much as its perceived weaknesses, and as a stimulus to provoke author reflection. In a 2008 editorial for Medical Education, Kevin Eva wrote that ‘the reviewer is always right’. ⁹ In reflecting on peer review, Eva enunciated a perspective in which whatever a reviewer writes is inherently ‘right’, in the sense that if questions have been posed and issues raised, there is reflection to be undertaken by an author, and that even when adamant no such issues exist, that reflection will almost certainly lead to changes representing an improvement to some degree.

On a higher level, we see peer review as a measure of accountability to the journal's readership and the profession more broadly. We feel a strong sense of obligation to ensure the research published in Paramedicine is as valid and robust as possible, as it is research of high calibre that will ultimately have the strongest potential for impact and likelihood to inform and guide advancement of the profession.

At Paramedicine, we have implemented a peer review process tailored to contribute to our vision and mission, enhance authors’ experiences, and meet our obligation to the profession. Nuance in peer review is essential, acknowledging the differing publishing philosophies and ethos that exist within and between scientific and clinical disciplines. We have adopted a double-blinded peer-review model; whilst not empirically superior to other models, a 2018 Pubons (Clarivate Analytics) global study involving over 11,000 respondents reported authors were most likely to submit to, and conduct peer review for, journals using a double-blind model. ¹⁰ A double-blind model was assessed as being most appropriate given what might be described as the ‘maturing’ status of paramedicine research and peer review capacity.

Central to our peer review process is a multi-level approach to review that actively involves reviewers and the editorial leadership team, but in which key editorial decision responsibility sits with the editorial team. The reviewers play an essential role in informing the decisions of the editorial team through their feedback and subsequent recommendations. A frequent criticism of peer review has been that reviewers may be too demanding or not demanding enough and that editorial staff are too passive in moderating or contextualising reviewer recommendations. Further, authors often mistakenly believe that satisfying reviewers is all that is required and that the publishing outcome rests uncomfortably in the hands of two unknown reviewers whose expertise in the content area, research method, or in peer review itself, may not be guaranteed.

At Paramedicine, all submissions receive a ‘first pass’ preliminary review by the editorial leadership group which focuses on a high-level assessment of alignment to the journal's mission, vision and scope, perceived value to the broader profession, methodological appropriateness, and adherence to the mandatory submission requirements. A frequent frustration expressed by authors across disciplines is that the rationale for a so-called ‘desk rejection’ is restricted all too often to brief generic replies providing no contextualisation or specific feedback that could assist authors in further developing and improving their work. Authors whose papers do not pass the preliminary review at Paramedicine receive detailed feedback as part of our commitment to transparent decision making and authentic engagement.

Following a favourable a preliminary review, the manuscript is distributed to a Deputy Editor for the next level of assessment and potentially progression to peer review. Following their own assessment, the Deputy Editor may withhold a paper from peer review and engage the Editor-in-Chief in collaborative discussion regarding progression should they identify worthy concerns. Collaborative discussion may result in a paper being declined at this point, in which case both the Editor-in-Chief's and the Deputy Editor's assessments will be communicated to authors in a transparent and hopefully productive manner. Should the paper raise no concerns for the Deputy Editor, the paper progresses to peer review having already undergone two levels of assessment and review.

Wherever possible peer reviewers are assigned after careful consideration to their suitability regarding content or methodological expertise. Frequently, one reviewer will be drawn from our Associate Editorial Board consisting of 31 PhD-qualified researchers. The other is selected from the broader research community within or outside paramedicine, dependent on the nature of the manuscript and its methods. All Associate Editorial Board members have undertaken training in peer review ethics and process, and external reviewers are encouraged to do the same. Reviewers are provided resources to help guide the structure, content, and tone of their reviews, and are asked to include a decision recommendation to the Editorial Team.

In line with COPE guidance (https://publicationethics.org/peerreview) regarding management of reviews deemed to be of poor quality or when uncertainty exists regarding a reviewer's position or assessment, the editorial team maintains the right to engage with the reviewer to collegially discuss the nature of the review and seek clarification. Essential to note is that it is not the decision recommendation made by the reviewer that may be discussed or challenged, but the nature and quality of the review itself. We believe transparent feedback to reviewers regarding Editorial Team perceptions of quality and/or usefulness is a vital but often neglected part of a journal's quality improvement and assurance process. In doing so, the author's experience and quality of review are enhanced, and the reviewer's development is supported if necessary. In a discipline such as paramedicine, where research capacity and journal publishing are in a period of infancy relative to longer-established scientific and clinical disciplines, development of reviewer capacity must be a function embraced by an editorial team.

The result of the peer review process at Paramedicine is that the authors receive comprehensive feedback from four ‘peers’; two reviewers, and two members of the Editorial Team. A manuscript's outcome therefore rests upon a collaborative, collegial, productive, and transparent process grounded in international guidance for ethical peer review, not as is often mistakenly assumed in the hands of two reviewers of ill-defined capacity and or quality. We believe this multi-level approach contains checks and balances that serve to optimise the validity and robustness of the review process and facilitate its purpose as described earlier being achieved.

One of the most notable challenges for any journal is securing peer reviewers in a time when an increasing number of journals seek reviewers from within a finite number of researchers. ¹¹ We recognise the need to secure reviewers by providing a positive experience when they engage, and to increase capacity within the profession by ‘growing our own’. In December 2022, Paramedicine launched its Peer Review Quality initiative as part of its strategic commitment to the active development of peer review quality within the profession. The central pillar of that initiative is a Peer Review Mentoring Program (PRMP), an international project designed to build capacity and quality in paramedicine peer review. The PRMP provides an opportunity for novice reviewers to engage with the journal and develop their capacity, and for experienced reviewers the opportunity to engage in reflective continuing professional development to refine and enhance their skills. A survey from 2016 reported most reviewers have not received formal training in peer review, and that three-quarters of all reviewers, and 89% of early career researchers, desired such training. ¹²

Further to growing one's own, we argue that more can be done to enshrine engagement in peer review during the formative years of research training, where it can be positioned within an emerging researcher's mindset not as a burden, but as an act of academic citizenship and reciprocity that makes a meaningful contribution to the profession whilst yielding personal benefits too.^13,14 We encourage research supervisors to engage higher degree research students in peer review as a matter of course, initially in co-performed models before progressing to independent reviewing.

In summary, peer review remains an essential component despite uncertainty regarding its effect. Our objective at Paramedicine is to acknowledge and attend to its weaknesses, leverage its strengths, and capitalise on opportunities to invest in peer review, in doing so optimising it within the context of our journal's philosophy and the profession of paramedicine. The journal will continue to actively pursue transparent and productive quality improvement in its peer review processes and seek to provide stewardship regarding building of peer review capacity within individuals and the profession internationally.