Publishing more than reviewing? Some ethical musings on the sustainability of the peer review process

Abstract

Based on our editorial experience, and acknowledging the regular editor grievances about reviewer disengagement at professional meeting and conferences, in this article we argue that the review system is in need of significant repair. We argue that this has emerged because an audit culture in academia and individual incentives (like reduced teaching loads or publication bonuses) have eroded the willingness of individuals to engage in the collective enterprise of peer-reviewing each others’ work on a quid pro quo basis. In response to this, we emphasise why it is unethical for potential reviewers to disengage from the review process, and outline the implications for our profession if colleagues publish more than they review. Designed as a political intervention in response to reviewer disengagement, we aim to ‘politicise’ the review process and its consequences for the sustainability of the scholarly community. We propose three pathways towards greater reviewer engagement: (i) senior scholars setting the right kind of ‘reviewer’ example; (ii) journals introducing recognition awards to foster a healthy reviewer progression path and (iii) universities and accreditation bodies moving to explicitly recognise reviewing in workload models and evaluations. While all three proposals have merit, the latter point is especially powerful in fostering reviewer engagement as it aligns individual and institutional goals in ‘measurable’ ways. In this way, ironically, the audit culture can be subverted to address the imbalance between individual and collective goals.

Introduction

Dear XXXX ¹

Sorry I am buried right now. My projects are coming due over the next few weeks and I am almost constantly reading and commenting on them now.

Dear XXXX

Sorry, I have already done too many reviews this year.

Dear XXXX

Not available

We received these emails from senior scholars on sending a ‘personalised’ journal review invitation to her/him. In the first case, this was following the email sender’s request for feedback – generously provided despite being ‘buried’ ourselves – on a potential journal submission. In the second case, we received this response following the sender’s accepted publication in a journal. The final email is a common response to requests for reviewing, no explanation, just a refusal to review. We venture a guess that many colleagues with editorial experience in organization theory or management studies regularly receive emails of this kind – emails indicative of a problem in the peer-review system. We argue this problem resides often in individuals prioritising their own research agenda over contributing to and sustaining the community through the provision of peer review (see e.g. Gallagher, 2013; Treviño, 2008). In saying so, we note that these decisions are often influenced by systemic incentive issues emerging in academia generally.

In this article, we propose the thesis that the review system is in need of significant repair. In saying so, we go beyond existing problems in the peer-review process – such as frailties concerning human judgement or the exploitation of academics (Jordan, 2020; see also Macdonald and Kam, 2011) – and focus on the audit culture in academia that fans hypercompetition (Edwards and Roy, 2017) and unsustainable individual incentives (e.g. publication bonuses or reduced teaching loads when hitting ‘A’ publications, see Aguinis et al., 2020). These factors are eroding the willingness of individuals to engage in the collective enterprise of peer-reviewing each others’ work on a quid pro quo basis (Hibbert et al., 2021; Jordan, 2020). By sharp contrast, those who entertain the ‘give-and-take-attitude’ and generously provide feedback to authors have their time available for writing reduced. That is, to the extent that active reviewers display prosocial and collegial behaviour (see e.g. Brewis, 2018) and agree to review for colleagues, they potentially face reduced time for their own research. This, in turn, can result in those reviewers lagging behind in terms of publication outputs, which is a real currency in modern academia (Butler and Spoelstra, 2012; Dean and Forray, 2018; Willmott, 2011).

In response to these developments, we emphasise why it is unethical for potential reviewers to disengage from the review process, and outline the implications for our profession if colleagues publish more than they review. We consider our article as a political intervention (Gabriel, 2016) to the mis-developments that are afoot in terms of reviewer disengagement. Specifically, we paraphrase Chelli and Cunliffe’s (2020) notion of ‘politicisation of knowledge and its consequences for conducting research’ (p. 1) to the ‘politicisation of the review process’ and its consequences for the sustainability of the scholarly community. Given the ethical valence encapsulated in the dual focus that guides our article, this step is, we believe, long overdue and in need for a perhaps provocative reckoning with the status quo. It is long overdue because, with a few exceptions (Treviño, 2008), many editorials in management studies (DeSimone et al., 2021; Ragins, 2017) and ethics journals (Borkowski and Welsh, 1998) continue to focus on the ‘how to review’ submissions, thereby ignoring why academics should review. This important point was also missed in a recent (commendable) attempt at setting reviewing competencies (Köhler et al., 2020). In addition, highlighting the political dimensions of reviewer disengagements and the implications of reviewers refusing to participate in the process justifies a provocative reckoning with the status quo now. Our claim around reviewer disengagement is based on (i) data revealing that in some disciplines, 20% of the researchers perform 69–94% of the reviews (Kovanis et al., 2016), (ii) reports of editors having to issue up to 12 invitations to get two reviewers in some journals (Hazen et al., 2016) and (iii) growth in journal submissions which is not matched at all with a growth in reviewer numbers in some journals (Dean and Forray, 2018). While we recognise that there are personal and professional circumstances that result in not every review invitation being accepted, our arguments are aimed at those potential reviewers who can merely (and persistently) muster self-interested excuses (the first introductory quote), or those who decline and ‘provide no reason at all’, those that ‘simply don’t respond’ and worse still, those who ‘don’t review, period’, even though some of them ‘are among the most well-known and distinguished in the Academy’ (Treviño, 2008: 8, see third introductory quote).

In what follows, we develop two ethical critiques in more detail as part of the concern about increased reviewer disengagement. First, we note that the uneven distribution of reviewer workloads – that is, when a small number of reviewers review the major bulk of submissions – suggests a violation of principles concerning social justice. Social justice is defined as ‘a state of affairs . . . in which (a) benefits and burdens in society are dispersed in accordance with some allocation principle . . .; (b) procedures, norms and rules that govern political and other forms of decision making preserve the basic rights, liberties and entitlements of individuals and groups; and (c) human beings . . . are treated with dignity and respect not only by authorities but also by other . . . fellow citizens’ (Jost and Kay, 2010: 1122). A social justice perspective is entirely appropriate in our essay as it represents an antidote, inter alia, to arbitrary or unnecessary suffering (Jost and Kay, 2010). Suffering here concerns both those authors whose work is slow to be reviewed, and those few reviewers who agree to review the major volume of submissions. For the latter cohort, it produces potential disparities in terms of career progressions. Whereas some colleagues (who never review) may more rapidly progress their careers on the back of a ‘never-give-only-take-attitude’, those who engage in the review process may be disadvantaged, especially in what is increasingly an audit culture in academia, where only publishing is counted (Aguinis et al., 2020; Argento et al., 2020). If the only thing that matters to university administrators and managers is the number of published articles we produce, then this can create competition among members of the academic community for much-sought-after publication spaces. This, in turn, can undermine the very essence of supporting more collective efforts, such as participating in the review process to support colleagues.

The second point we wish to highlight here is that when there is a limited pool of reviewers (i.e. the small volume of reviewers who do most of the reviewing most of the time), the full possible range of expertise that theoretically exists is practically not available for editors to make their decisions (Jordan, 2020). This point alerts us to the possibility that, as a result of less expertise, it becomes less likely for academics to detect potential deficiencies of a theoretical, empirical, methodological or analytical kind and to offer advice on how/where these deficiencies can be addressed. ² If these problems remain undetected and potential enhancements are not made, the result may be unreliable research or poorly conceived published articles. In such cases, the prospect of work practices (e.g. advice relating to recruitment or important leadership skills) being informed by such research raises the risk of harm for individuals and organisations (Lindebaum, 2013; Lindebaum et al., 2018). Likewise, a good review should be, in our view, also about offering developmental and constructive avenues to enhance and improve the manuscript or research. Finally, a limited pool of reviewers can entail (too) much influence on their part in deciding what is ‘good and relevant research’. That is, only drawing on a small cadre of reviewers could result in limited perspectives being accepted by the journal and, therefore, could entail the constriction of knowledge in the discipline.

Below we turn now to elaborating in more detail on the two questions that guide our essay. This is followed by casting a look into the future with a view to propose ways to counter the growing problem of reviewer disengagement.

Why is it unethical for potential reviewers to disengage from peer-review?

With reference to the possible suffering that can occur when social justice principles are violated, we ask ‘who’ is actually suffering as a result of reviewer disengagement? We count our colleagues and authors among them, the various communities of researchers that benefit from timely review of their work and new studies to inform their own work, and of course, individuals and organisations whose problems can be resolved through high-quality peer reviewed research. Given that the COVID 19 pandemic only accelerates and enhances the pressures on academics who may have to manage increased teaching workloads, as well as their responsibility as researchers and reviewers, it is easy to offer the excuse ‘I’m too busy to review’. We reiterate that reviewers can have legitimate reasons both personally and professionally to sometimes (but not persistently) decline invitations. However, as everyone is busy, is it appropriate to simply focus on ourselves all the time to safeguard our time against the incursion of review invitations? In this regard, Treviño (2008) notes that if we ‘widen the lens to take consequences to others into account’, then the ‘decision to review is quite straightforward’ (p. 9). She continues to argue – and we fully concur – that ‘if we want others to provide quality reviews of our work, we should be willing to do the same for them’ (p. 9). Taking into account the presence of others who are affected by and through our (in)actions raises also the moral dimension of the review process. Morality inherently implies a concern for others (Solomon, 1993). Moral disengagement, however, is defined as the ‘process of cognitive restructuring that allows individuals to disassociate with their internal moral standards and behave unethically without feeling distress’ (Newman et al., 2020: 535). Based on this definition, those potential reviewers who have voluntarily disengaged from the community service of reviewing could be seen as immoral due to the suffering that their inaction inflicts on authors and reviewers.

In relation to the second ethical point that a limited pool of reviewer expertise can diminish the quality of research and hence potentially can cause harm at work, we agree that technical clarifications on how to review certain kinds of submissions can be valuable (see e.g. DeSimone et al., 2021). However, the fact remains that not detecting deficiencies in manuscripts, or failing to advise authors on how to improve their theoretical, empirical, methodological or analytical contributions, poses a fundamental challenge to the broad purpose of academic endeavours; that is, to advance and accumulate knowledge with a view to address important human needs (Suddaby, 2014; Tsui, 2016). But not only that; if we take a step back and entertain for a moment the question what happens if these deficiencies are not detected, or when opportunities for improvement are not offered in the peer-review process (due to lack of broader expertise), then we are confronted with the ethical fallout of only a limited expertise being available for the review process. A relevant example is a recent extended post-publication commentary (Prochilo et al., 2019) on a range of ‘seminal’ studies on organisational neuroscience (Waldman et al., 2011, 2017). Prochilo et al. (2019) demonstrated that these studies lack transparency to be clearly understood, evaluated or replicated – in addition to the misuse of inferential tests that lead to misleading conclusions. Against the backdrop of mounting concern amongst neuroscientists themselves about the validity of the ways in which neuroscientific data are applied (Button et al., 2013; Thibault and Raz, 2017; Wexler and Thibault, 2019), Button et al. (2013) caution about the ‘ethical dimensions’ of ‘unreliable research [which] is inefficient and wasteful’ (p. 365). Lastly, we cannot help but think about article retractions in leading management and leadership journals in the previous decade, where post-publication scrutiny ascertained errors in the statistical analysis which then undermined the conclusions drawn from these projects (see Atwater et al., 2014; Spoelstra et al., 2016: for a discussion). Our question is, would an expanded reviewer pool, specifically in terms of methodological expertise, have contributed to these issues being discovered earlier? At the very least, it seems safe to conclude that if those who uncovered the errors post-publication were involved in the initial review, these retractions may not have been necessary.

What are implications for our profession if colleagues publish more than they review?

The struggles that we and many other editors (Driggers, 2015; Treviño, 2008) experience in enlisting the support of reviewers signifies, for us, a shift from concerns for others to more self-interested attitudes and behaviours that threaten to undermine the function of the peer-review process. It is here that the ethical issues become functional. We consider the function of peer-review to be that broadly of a quality control check/improvement process, where peer-review ensures that academic conventions in the production of knowledge are complied with and the potential of research is realised. When we say ‘academic conventions’, we apply a pluralist and inclusive perspective to explicitly recognise that these ‘conventions’ can only be consistently applied in their ‘home’ paradigm. Thus, in quantitative research, these conventions typically concern issues of reliability and validity (Cohen, 1988); in qualitative research, scholars speak of trustworthiness of findings (e.g. Cloutier and Ravasi, 2021; Plakoyiannaki et al., 2017); and in critical theory, academic ‘rigour’ can imply internal coherence in the logic of a normative argument that in crucial ways seeks to challenge predominant orthodoxies (Lindebaum and Wright, 2021).

However, fulfilling the functions of quality control and improvement comes under pressure when too many academics disengage from the review process. Self-interested behaviour, such as refusing to review as a consistent response to advance one’s own career, has negative consequences for our scholarly community. Most authors appreciate the benefits of receiving a well-crafted peer-review. Given that acceptance rates at journals are generally low, a benchmark for most academics is this: if some academics are publishing more than they are reviewing (cf. first and third introductory quote), then they are contributing to an unsustainable imbalance.

Taking account of our own experiences, and the many published commentaries that sometimes feel like a cry for help from editors so that reviewers engage more, we wonder about the cracks in the system – both their depth and width. We are reminded of Merton (1968), who once observed that:

‘when the net balance of the aggregated of consequences of an existing social structure is clearly dysfunctional, there develops a strong and insistent pressure for change. It is possible . . . that beyond a given point, this pressure will inevitable result in more or less predetermined directions for social change’ (p. 94, italics in original).

If we consider previous editorials (Campbell and Conlon, 2021; Treviño, 2008), and recent studies on reviewer (dis)engagement (Dean and Forray, 2018; Kovanis et al., 2016) as emerging proof that the review system is becoming problematic (because too few reviewers undertake too much of the review volume), then the need for change may be right here and right now.

Looking ahead and a call for action

In our article, we expressly delineated our arguments in terms of a specific group of academics – not the reviewers who regularly review (cf. second opening quote), not the ones who on occasion have to decline because of rival commitment (both private and professional), but those scholars, especially senior ones, who fail to have reasonable justifications for declining review invitations (cf. third opening quote), those who do not respond at all to review invitations or those who simply do not review to ensure their immediate self-interest of publishing is not jeopardised (see first opening quote).

Our article had a dual focus on firstly emphasising why it is unethical for potential reviewers to disengage from the review process, and secondly, on outlining the implications for our profession if colleagues publish more than they review. Our aim was to shine a light on the politicisation of the review process and its consequences for the sustainability of the scholarly community. Politicising the review process is required in response to reviewer disengagement for two reasons. First, we argue that there is a violation of principles around social justice, which can entail arbitrary or unnecessary suffering (Jost and Kay, 2010) amongst colleagues. As outlined, suffering here concerns both those authors whose work struggles to be reviewed, and those few reviewers who agree to review the major volume of submissions (and thereby have less time at hand to advance their own writing). Second, a shrinking reviewer pool is likely to reduce our own legitimacy as social scientists when said pool is less likely (i) to detect cases of unreliable research and (ii) to help improve published research through the review process. In this day and age, when populist politicians and conspiracy theorists only too eagerly proliferate their ‘alternative news and facts’, any report about ‘poor’ research, or research that is not well thought-through, only seems to provide ammunition to them for rejecting science (including social science), while at the same time unsubstantiated versions of reality can be advanced. In addition, there is always a risk that a small reviewer pool bears excessive influence on what gets published.

So what can be done? Should journals remunerate reviewers for their work? We respond in the negative here, because classic psychological research confirms that extrinsic motivation (i.e. contingent rewards, like financial incentives for review) often reduce and replace intrinsic motivation, where one desires to perform primarily because of personal motives and interests (Deci et al., 1999). This well-established finding in psychological studies – where extrinsic incentives ‘crowd out’ intrinsic motivation – has been observed in a variety of fields, such as the provision of public goods and volunteering (Frey and Jegen, 2001) and academia itself (Gabriel, 2019). A controversial result is that individuals can feel alienated from their work when they perceive it is controlled by external incentives (Frey and Jegen, 2001). In what follows below, we offer three ideas on how to improve the situation.

First, scholars, especially senior ones, need to stop setting the wrong example in relation to what Harley (2019) refers to as the ‘heroic’ publishing machine. We note Harley’s (2019) observation about the countless times he has seen senior scholars boasting about their publication track record at conferences and in keynote speeches, rather than their efforts to develop colleagues through engaging in the peer-review process. Therefore, to reverse this development, senior scholars should start setting the right example in their engagements with students and junior colleagues within their institutions and at professional meetings and conferences. That is, to promote more prominently the idea that participating in the review process is a way to acquaint oneself with future research, to learn about the publishing process by interacting with authors, editors and other reviewers and to influence the spectrum of what is to be considered good and relevant research. Similarly, universities need to realise that the audit system that now appears to dominate academic management (Welch and Li, 2021) may not be sustainable, as it emphasises individual performance (often with a very narrow focus on published works) over more collectively oriented goals (Aguinis et al., 2020; Campbell and Conlon, 2021; Edwards and Roy, 2017). It, therefore, does not support a holistic approach to furthering knowledge; there can be no academic advancement without reviewing published works. Ironically, we note that there can be no auditing and ranking without reviewed publications too! ³

Second, in trying to encourage more engagement with reviewing, we ask ourselves: what could be considered a reward in academia that is related to reviewing? While intrinsic rewards relate to extending knowledge and contributing to the scholarly community (Campbell and Conlon, 2021), these reasons have obviously not been enough to engage the majority of potential reviewers. Our view is that being a member of an editorial board may increase the likelihood that academics will review submissions instead of refusing to do so. This provides recognition that can be useful to colleagues as they advance their careers as a reputational currency. Yet, we do not advocate the exponential expansion of editorial boards for the sake of easing the reviewer shortage. Instead, some journals, such as the Academy of Management Learning and Education, are using a reviewer progression system. Such a progression system is based on bestowing reviewer awards upon junior reviewers, who can be even at PhD or Post-doc level in terms of career stage. This junior award co-exists with the standard ‘best reviewer award’, and together they feed into and maintain the value of board membership, while encouraging more academics to accept ad hoc reviews. Academics could start off doing ad hoc reviews, and then build up over time to becoming a full editorial board member. If journals provide a clear and transparent career progression path for academics – perhaps even to the point of becoming and associate editor – they may be more inclined to start the journey. This path towards external recognition can then be meaningfully complemented with an exercise in internal recognition. That is, if universities themselves, as well as those institutions to whom universities submit data for accreditation purposes (e.g. AACSB), also recognise their shortcomings in valuing reviewing as an integral contribution to the scholarly community by explicitly recognising it in workload models, ⁴ then we believe there may be scope to start reverse the process of reviewer disengagement. That internal recognition can be gained (or not) in annual performance reviews, when academic managers or department heads assess if their staff have performed any reviews in the previous year. If the number is below expectations, then goals can be set or reiterated to increase the number of reviews performed in the next year. This proposal has the benefit of being amenable to enforcement by employers thanks to, ironically, reviewing becoming a ‘measurable’ item in auditing processes. If applied effectively, it means that excuses enlisted to refuse to perform peer review (like ‘I have X and Y targets to meet’) lose their foundation, as peer review will simply be one target alongside others (e.g. X being publishing, and Y being peer-review).

Our hope is that purely self-interested behaviour will decline over time when both institutions and academics understand better that their and our own success are dependent upon reviewers providing quality reviews. The explicit alignment of institutional and individual goals is, we believe, a powerful proposal to this end. After all, both academic institutions and we need our work to be reviewed.

In sum, we have argued that the combination of (i) senior scholars setting the right kind of example, (ii) journals introducing a recognition process to foster a healthy reviewer progression path and (iii) universities and accreditation bodies moving to explicitly recognise reviewing in workload model and as part of scholarly activities can encourage attitudinal and behavioural changes towards greater reviewer engagement. In this way, the (otherwise much lamented) ‘audit culture’ can actually become a vehicle through which goal setting in relation to peer-review becomes ‘measurable’ over time. Once reviewing is more explicitly recognised in workload models and by accreditation bodies, we can imagine increased momentum towards more reviewer engagement. No community can be built and sustained on the back of the type of individualism that shines through in the introductory quotes, or indeed by shifting the entire load to a small number of individuals. So next time the invitation arrives to a review a paper, we should ‘think twice’ (Driggers, 2015) before we decline it. There is, we should never forget, a person behind the submitted manuscript who invested enormous amounts of time and energy in writing this article, a colleague that may have just submitted her/his first paper, a person whose career and livelihood depends on getting a paper accepted sooner rather than later, or a person who has already reviewed 10 manuscripts this year in support of other colleagues’ scholarship. Think also about the risk that a flawed or underdeveloped piece of research may slip into publication without our broader expertise have been applied to it, and that this research actually does harm in the context of organisational practise. None of that is desirable if we care about our profession and our colleagues. What will you do differently in future, and what will you tell your institution to do differently in future?