The critical role of peer reviewers: Challenges and future steps

Abstract

The critical role of peer reviewers in the publishing process is examined. Examples of typical challenges are provided, including the relative lack of rewards for this important task. Particular attention is paid to the need to consider the diversity of the peer reviewers recruited and impediments to the selection beyond Areas of Competence, often due to the small available pool. Finally, recommendations for improvement are suggested.

Peer review – where scientists from the same field of authors with competency on a subject evaluate a submission – is the most standardised and widespread procedure by which scientific journals assess the quality and priority of submitted manuscripts (Pierno et al., 2017). Thus, selecting peer reviewers who will provide high-quality reviews and overseeing the fairness of the reviews are central tasks of journal editors. Peer-review related choices and actions have direct consequences for the framework within which the submitted work is interpreted and evaluated (Black et al., 1998).

 Despite the vital role played by peer reviewers in science journal publishing, their roles and functions are poorly defined (Glonti et al., 2019). Additionally, most peer reviewers perform these tasks as a voluntary “service to the scientific community”, and the explicit recognition of these contributions as unpaid, “essential” work remains limited (Riley & Jones , 2016).

The choice of reviewers and the editor's ability to give proper weight to the reviewers’ assessments are crucial mechanisms in the journal's ambitions toward quality and fairness. Thus, journals need to be aware of how they work as filters for restricting or allowing more geographical, cultural, epistemic, and gender-based heterogeneity. Diversity and inclusion ambitions vary among journals. Some have a scope concerned with a particular region or a specific research tradition or method. In this piece, we list some challenges and recommendations to increase editorial offices’ awareness of how the scientific review process involves exclusion and inclusion mechanisms. We hope that our points of view will also be helpful to a broader audience in the scientific community, including users of research, readers, authors, and their colleagues.

Reviewer selection and recruitment is a widespread challenge among scientific journals. The work is unpaid, and academics are often busy with competing demands that complicate adherence to tight journal deadlines. Good reviewers already multitask and are seldom prepared to prioritise a review request above other commitments. Once an academic has reached a certain point in their career, there is minimal incentive to be a reviewer, which provides little “added value” to their university. The obvious educating and (self-)scholarly developmental functions of peer review could be more emphasised in conceptions of the task. In this piece we turn attention to the diversity aspects in the peer-review process. Journal editors’ consideration of reviewers’ diversity aspects is important, as a reviewer who represents a somewhat different perspective can bring about the kind of dialogue and dialectic reasoning that results in better thinking (Myerson, 1994). Reviewers’ comments can greatly improve an author’s ability to communicate their research results, possibly to a more multidisciplinary audience. To that end, this position statement identifies several key challenges facing reviewers and editors in the context of gatekeeping mechanisms affecting diversity in science publishing. The text is structured as follows: We begin by listing some typical challenges and circumstances that encourage diversity in reviewer processes and general traits in science publishing culture that stifle diversity in publishing. Finally, we formulate some recommendations.

Examples of challenges associated with peer review

Anonymity

The double anonymity of a review guarantees that reviewers do not know who the person behind the work is, and the authors do not know who the reviewer is. In the power constellation between reviewer and author, anonymity can be more important for those whose work is being evaluated than those who provide judgments. It can ensure fair treatment of authors regardless of their background. Independent of the anonymity question, some journals publish peer reviews or have open public review processes where people can participate. One example of the latter is the open research platform F1000 Research. Van Rooyen et al. (1999) have shown that when reviews are not anonymous, there is no significant effect on the quality of the review, the recommendation regarding publication, or the time taken to review. Still, it significantly increases the likelihood of reviewers declining to review. Increasingly, journals have transitioned from a double-blind to a single-blind peer-review process: reviewers know the authors’ identities, but the peer reviewers’ identities will be unknown to the authors (Maggio & Driessen, 2020). As anonymity for the author is often, in practice, difficult to ensure, particularly in a national or regional journal, or if the subject of the text is very specific, this is often one reason for single-blind peer review. However, the peer-review process is unblinded for some journals, such as the British Medical Journal. Ultimately, the blinding of the peer-review process may incentivise some reviewers, particularly those who provide harsher critiques and may, in turn, affect the peer review process.

Limited pool of reviewers

As in many other fields of research, in addiction journals, there is a limited pool of reviewers. The more we select by demographics etc., the more we narrow down the already small pool of possible referees. Selection criteria that add to or go beyond subject expertise seem like a luxury and would be the exception rather than the norm. Among solicited reviewers, no response to or non-acceptance of peer-review invitations are frequent, particularly among senior reviewers. Occasionally, the reviewer selection is arbitrary. Amid our editorial team, it boils down to how idealistic versus practical we should and can be. To be more idealistic would require journals to encourage editorial boards, for example, to have a certain percentage (quota) of reviewers from low/middle-income countries and financial incentives to promote this. Ideally, a journal's impact factor – a metric that quantifies the number of times other sources cite a particular article over a period of time – could be weighted to reflect this.

No systemic self-reflection practices

Depending on the flow of submissions, an editorial office may develop a stringent perspective of a good or poor-quality paper. The editorial board has advocated regular, critical self-reflection to avoid a routine, self-serving and self-reproducing system (Wiley, 2021). For a fair review process, journals should seek variations in methods and the epistemic backgrounds of the articles they publish and their peer reviewers. In many editorial offices, there is bound to be a lack of systematic or recurrent audits of the representation of different authors from different career phases, places, and educational backgrounds. While some of this diversity data can be acquired from publishers, all journals should guarantee diversity of contents and authors within the limits of their goals.

Reviewer selection and epistemic considerations

Editors have a role in monitoring their roster of reviewers. This includes insight into the epistemic adherence of reviewers. For example, an editor who goes against the reviewer's recommendation can risk this reviewer not volunteering anymore. Therefore, in the instructions to reviewers, it should be clearly stated that the editor has the final word. However, the epistemic adherence of editors is an equally important factor. To that end, editors must understand the perspective or viewpoint likely to be employed by the scholars chosen to comment on the work submitted to the journal. Reviewers’ biases are, to a varying degree, difficult to detect from reviews, and this may influence an editor's selection for further involvement. On the other hand, editors’ alertness and critical awareness can influence the fairness of a review process.

Motives and incentives for peer-review activities

Reviewing is not necessarily “unpaid” work, as most employed reviewers receive salaries from a university or external funders. However, since academic scholars usually have a high workload, reviewing competes with the other tasks we are expected to perform. Consequently, the task of reviewing “steals” some of the time meant for spending on those other tasks and activities. To that end, it seems as if there is a culture of acceptance within the university environment concerning the task of reviewing; meaning that it is acceptable to do this during regular work hours. Still, the usefulness of this culture should be re-evaluated at some point, given that publishing companies make a big turnover while some of the work done is paid for with public means. There is now a debate as to whether reviewer should be paid, especially as the open-access model (where institutions are expected to pay for publication) becomes more widespread, so that the readership can be widened. Other motives for being a reviewer, which may become less relevant as one's academic career progresses, may include (Balster, 2017):

Awareness of colleagues’ work lines, particularly for early career reviewers (occasional conflict of interest?)

Quid pro quo: “They will review mine if I review theirs.”

Education for the motivated.

Interest in becoming an Editor by shadowing the work of more senior colleagues.

Reviewer recognition

Another challenge is the limited reviewer recognition in academia. A major consideration for editors is finding appealing incentives for reviewers to promote peer reviewing. Incentives have included listings with peer-reviewer and research repository databases, such as Publons (Cuellar, 2018) or ORCID (2021), as well as letters of recognition from the respective journal. Reviewer recognition may be more pertinent in academic spheres, where universities may factor the extent of an individual's journal peer-review activities into teaching-based promotions and annual activity reports. Alternatively, some journals offer continuing medical education (CME) credits, which are necessary for the maintenance of credentials for peer reviewers in medical sciences.

Predatory publications and conferences

The race for publication is well established, and the publishing industry is estimated to generate hundreds of millions of dollars. Predatory author services emulating legitimate ones have mushroomed and should be assessed using the same criteria as predatory journals (el-Guebaly et al., 2021). Regarding reviewer diversity, predatory publication venues may preferentially target peer reviewers from more disadvantaged settings, who may not be familiar with or aware of predatory publications, which may, in turn, impact the quality of the review generated.

New business models

There are different groups of journals and business models. The Multidisciplinary Digital Publishing Institute (MDPI) is predicated upon encouraging authors to pay for open access while not being seen as predatory. In return for this payment, authors are guaranteed quick publication and a high probability of acceptance. Typically, administrators select reviewers from a database of reviewers expected to turn around their reviews in 7–10 days. Then a large editorial board and one of the editors will decide whether to accept or reject the paper (hopefully, they will be selected for methodological and subject-based expertise). In itself, a fast turnaround is a desirable goal. Still, as the growth in this group has been very rapid, it is questionable whether review quality can be maintained within such a model. Different MDPI journals cover many scientific subjects, and one of the business models is to create a series of special issues. As a result of this approach and their open-access nature, these journals have high impact factors and SCImago ratings as they score well within journal metrics. The open-access model requiring authors to pay for publication and the fact that this model is highly linked to journal metrics means that this business model is likely to restrict diversity in publishing. Petrou (2020) and Crosetto (2021) provide critiques of the MDPI model. However, it should be acknowledged that many international authors are using this route, most notably those from China and the Far East, who previously may have found it efficient to access the more traditional addiction and other scientific journals. Ultimately, the high speed of turnaround expected by these journals may in turn impact reviewer selection, by favouring those reviewers who can comply with these expedited timelines (staff in the more junior stages of their careers with higher teaching loads may find it harder to have sufficient time to complete reviews). In addition, under the MDPI model, reviewers are given discounted future open-access costs of publishing within an MDPI journal, which may further incentivise reviews by academics considering publishing within that journal.

Research assessment exercises and journal metrics

The Research Excellence Framework (Higher Funding Council of England, 2021) in the UK is an exercise occurring in five to seven-year cycles, whereby the UK research quality of academic institutions is assessed, and each university is awarded funds from the Department of Education as a result of the evaluation. The largest weighting is applied to peer-reviewed publications. Every academic who is deemed to have significant research responsibilities (assessed by the academic institution) has to be entered into the exercise. A minimum of one article and a maximum of five papers can be entered for each author. Some form of objective selection criteria is to be applied. Academic institutions have become increasingly reliant on journal metrics, depending upon impact factors and citation data. Diversity is not included in how metrics are calculated. However, peer reviewers may be increasingly drawn towards reviewing for higher-impact journals, and this may, in turn, lead to a disproportionate representation of reviewers over journals with lower impact factors or research metrics. Furthermore, this might also be perceived as a problem in the context of a paper that argues for more diversity yet relies upon academic influence and prestige to encourage particular types of peer reviewers.

In the UK, for example, academics are encouraged to publish in SCImago quartile one journals, and professional advancement/promotion will depend on publishing in these journals. If the trend of linking academic advancement to publication record continues, it will likely become increasingly challenging to publish in these journals. In addictions research, it is equally possible that a better place to target a paper to an audience interested in the findings could be in quartile two or three journals, or even practitioner-focused journals, which often do not have impact factors or SCImago ratings.

For addiction journals, this problem is particularly acute. Addiction journals come under the SCImago categorisations, Psychiatry and Mental Health (SCImago, 2021). The highest-ranking journal in this section is World Psychiatry, with a rating of 9.239. Addiction is the highest-rated addiction-related journal at number 21, with a SCImago rating of 2.745. In addition, Addiction is the only addiction-themed journal with a SCImago rating of more than 2.

Other countries also have their own science journal ranking systems, according to which universities get their state funding. Finland, for example has three classification levels for categorising journals: 1 = basic; 2 = leading; 3 = top journals (Julkaisufoorumi, 2022). Most thematic and specialised journals, such as addiction journals, are in non-ranked categories (0) and or in category 1. Smaller national journals by international standards can be in category 2, because they are the leading national journal in that field. Category 3 is only for larger, high-impact generic disciplinary journals.

Epistemic diversity and systemic issues

In publishing, additional systemic issues must be addressed to encourage diversity in peer-review activities. While there are many potential issues, one of these involves the length of papers accepted in high-impact journals. Given that qualitative research by its nature requires more space, it follows then that quantitative research is automatically better suited for publication in high-impact journals due to journal word-length requirements. Consequently, there may be an over-representation of quantitative peer reviewers and an under-representation of qualitative peer reviewers, impacting diversity.

Recommendations

We suggest the following recommendations in line with practical ideas to improve the reviewing experience and diversity in the reviewing process, in order to enhance the communication of research (Schroter et al., 2018).

Support of reviewers

The lack of reviewer recognition negatively impacts the publication process overall (D’Abramo, 2017) since scientists often decline invitations to peer review and quality is not always ensured (Warne, 2016). An ad hoc educational process is available but requires time and resources. Encouraging a more diverse selection of reviewers would require significant input by the publisher and agreed-on criteria to reward reviewers from specific regions or countries. One step could be to weigh the reviewers’ diversity as part of the international journal ranking (e.g., impact factors). International boards can take similar steps regarding the incentivisation of diverse reviewer panels. Evidence of robust inclusion of this perspective in practice in terms of sincere attempts to recruit reviewers following a more diversity-aware agenda could be noticed by publishers on the journals’ websites.

International boards

As stated, getting reviewers is a challenge, notwithstanding diversity. International journals review their editorial boards during annual journal meetings and excuse members deemed inactive (mainly because they have not responded to reviewer requests) (Foster, 2021). Understandably, when editors look for reviewers, the first port of call is often people they know or on whom they know they can rely. Regrettably, this can be a barrier to diversity. Editors may benefit from looking outside their networks and making an effort to keep an open mind when selecting reviewers.

Reviewer training and raising awareness of predatory journals

Reviewer training can improve the quality of the peer-review process and potentially improve the diversity of reviewers (Hesselberg et al., 2020). However, awareness must also be raised about predatory practices in publishing, which can adversely impact peer reviewers, possibly in time becoming a barrier to promotion (el-Guebaly et al., 2021).

Awareness of the evolution of other disciplines

In their review, Walker and Rocha da Silva (2015) highlight sea changes in peer review linked to the internet and open-access publishing. Peer review may become more non-selective, as practiced in open-access publications. With the emergence of formal post-publication review, reader commentaries may provide additional peer-review channels. However, there remain major differences in the peer-review process of various disciplines.

Valuing reviewers

Bianchi et al. (2019) proposed an F3-index that measures the reviewer's contribution to editorial processes of scholarly journals, similar to a journal's impact factor or CiteScore. Suggested quantifiable context-specific dimensions are: Report Delivery Time, Length of Report and Alignment of Recommendations to Editorial Decisions.

Conclusions

To promote diversity among reviewers and reviews, we need to inspect some significant barriers to encouraging diversity in publication. Journal impact factors, research metrics and research assessment exercises, such as the Research Excellence Framework, are ongoing issues. In reality, they pay lip service to diversity and inadvertently act as a barrier to it. Arguably, there is minimal insight into the inherent biases in the journal ranking/metric processes. This article has listed some circumstances which render it more difficult for journals to recruit reviewers who represent more diverse backgrounds or just reviewers who are likely to appreciate diverse perspectives, different epistemic traditions, and sociocultural contexts. To that end, we hope that this position statement has addressed the need to support journals, authors, and reviewers in promoting a realistic goal of greater diversity. We also hope that this article could start a wider discussion about improving the review process, which should include authors, reviewers, editors, university administrators, and those responsible for research funding in different countries.