How good is good enough…and why wasn’t my paper published?

A little over 18 months ago, in June 2021, I was fortunate to be offered the opportunity to serve as the inaugural Editor in Chief (EiC) of Paramedicine, the international scientific journal of the Australasian College of Paramedicine and Paramedic Chiefs of Canada. I leapt at the chance, sensing a rare opportunity to make a meaningful contribution to academic publishing, dissemination of evidence, and change within paramedicine research. I'd had some exposure to such a role, serving as the Interim EiC for the previous iteration of the journal during a period of major transition, but was (and still am) acutely aware of being a relative novice in the publishing space. So, I did what I feel every astute leader should do, that is, consult broadly, and surround yourself with smarter people possessing the brightest minds, and try not to get in their way too much. Four international deputy editors were engaged, collectively bringing extensive international research and editorial expertise and as a leadership team, we set about planning a new era and identity for a journal with an almost 20-year history of publishing in paramedicine. ¹ A daunting task for sure, but one we have pursued with thoroughness and vigour, grounded in collaboration, consultation and engagement with stakeholders around us. Soon to follow was an international associate editorial board, consisting of 28 PhD-qualified paramedics and those researching paramedicine, representing 10 countries.

Clear to us from the earliest of discussions, informed by extensive consultation, was a desire to actively contribute to uplift in paramedicine publishing and research. We did not want to be just another journal in the field, now with a new name, but essentially doing what this journal had done before. ‘Groundhog day’ was not on our agenda. We would not continue to be an easy target for publication, where acceptance of submissions was perceived by some to be almost a foregone conclusion. Informed by our own investigations and emboldened by a common desire to serve the profession in the most faithful way possible, we became resolute in our commitment to raise the bar in terms of quality in what would be published.

Having now completed a year of publishing in this new era for the journal, our commitment to pursuing and maintaining high publishing standards remains embedded in all that we do. Governance-wise we have established a ‘best-practice’ editorial process compliant with COPE recommendations, ² and uplifted the editorial qualifications of our entire team through structured editorial continuing professional development. We have invested heavily in peer-review quality and enhancement, including peer-review mentoring to support existing reviewers and engage a new generation of emerging reviewers. ³ We have sought to provide guidance and leadership to authors through editorials on various areas of research methodology and process, hoping to encourage reflection on current standards and reinvigorate discussion and debate on the importance of academic discourse, conceptual and methodological diversity and rigour.^4–6

Change, though, is never easy. Change brings with it an inevitable period of disruption, in which long-held perceptions and beliefs are called into question. Discomfort is experienced. Clarity is sought and expectations realigned. Dogma is challenged and ‘sacred cows’ sometimes slaughtered. Whilst potentially painful and difficult for a time, this is in fact ‘positive disruption’, essential for professional growth and development, vital to long-term sustainability of any organisation or enterprise, and a facilitator of innovation and creativity. ⁷

Pleasingly, our commitment to quality and efforts to raise standards have prompted overwhelmingly positive feedback, but there has also been criticism. Some strands of negative criticism, uncommon though they may be, include suggestion that Paramedicine has ‘become too hard to publish in’, ‘too obsessed with methods rather than the results and conclusions’ and is ‘no longer supportive of junior and student researchers’.

At this juncture, I feel it necessary to unpack each of these to increase clarity regarding our vision and mission – reflecting on the first year, I think it would be fair to say that our messaging and intentions, focussed as they were, could perhaps at times have been clearer.

Regarding being too hard to publish in, I would admit that the bar is substantially higher in this new era than in previous iterations and that acceptance may be less common compared to an individual's previous experience; but we, as an editorial team, assert that our standard and expectation of quality is consistent with those long accepted in similar neighbouring healthcare disciplines, and quite simply, is just where it should now be. It might be about perspective: I would argue the current expectation is not unusually high, more so that those past expectations may have been too low, static for too long and infrequently challenged – those may even have been necessary early in our development as a profession as we sought to develop research capacity and engagement. Some may argue paramedicine is still an emerging research profession, not yet mature enough in research capacity and development – even if there was some truth in that, it does not in my view satisfy an argument for perpetuation of a lesser standard of conceptual or methodological rigour. I further argue that paramedicine as a profession has evolved such that we are now at a critical point of capacity, quality and knowledge that necessitates new expectations.

This segues neatly into the perception some may hold that Paramedicine places too much focus on methodological rigour, a perception that leads to the inevitable question of ‘how good is good enough?’ It is indeed a curious opinion, one that we suggest is perhaps more a compliment than a criticism. There is of course no empirical benchmark on how ‘good’ a paper needs to be to be published in this or any other journal – each has its own vision and mission to guide editorial decisions, and these will be made within the context and nuance of the discipline, in partnership with reviewers. ³ Our position at Paramedicine, and one that I suspect would be commonplace elsewhere, is that findings and conclusions arising from a paper are only as good as the methodology that gave rise to them. Findings and conclusions arising from fundamentally flawed (or poorly described) research, interesting and novel though they may be, have little likelihood of achieving impact and being translated into practice, and if they were, may lead to erroneous practice to the detriment of patients, paramedics, or the systems in which they work, or paramedicine more broadly. My editorial colleague Dr Walter Tavares insightfully proposes this to be ‘hurtful science’ – in his own words, ‘the appearance of science, in the spaces and places that science lives, but which due to quality issues, hurts more than it helps’.

Our editorial team of course recognise that all research has inherent weaknesses, and therefore weigh the strengths of a paper against its weaknesses. We consider its competitiveness compared to other similar submissions, examine its congruence with our vision and mission and assess its significance to profession more broadly. We also consider the quality and transparency of the reporting within a submission, referring to widely accepted reporting standards clearly elucidated through resources such as the standardised reporting guidelines available via the EQUATOR network (https://www.equator-network.org/). At Paramedicine, our default position is to invest in submissions, and where unmitigable weaknesses or fatal limitations do not exist, work with authors to strengthen their work where possible within the realms of what is reasonable. Perfect research has never been sought. It would never be found because it does not exist.

We have seen brilliant work submitted to the journal this past year, research capable of substantial impact. We have been intrigued by excellent research that challenges who we are and what we do, and which has the potential to advance and transform the profession. There is abundant evidence of increasing diversity and complexity in methods and methodologies, in paradigms being applied, and in questions being asked. It is clear to us that paramedicine research is very much on the rise, and as an editorial team, we feel privileged to serve as a conduit of this work to the broader profession.

Unfortunately, there are more than a few papers that on appraisal are irrevocably conceptually and/or methodologically flawed, many of which concerningly are outputs of higher degree by research (HDR) studies supervised by post-doctoral academics. For transparency and clarity, the more common issues we encounter include, but are not limited to:

Failure to connect with literature beyond that specific to paramedicine, resulting in missed opportunities for broader academic conversations involving neighbouring professions. There remains a tendency to view paramedicine as an isolated island, separate from other health professions, instead perhaps as one island within an archipelago of interconnected health professions. We offer guidance on this in a recent editorial by Tavares et al. ⁴

Systematic and scoping reviews that fail to adhere to contemporary recognised methodological guidance, and best-practice reporting guidelines, and for which no a-priori review protocol is publicly available. We offer guidance on this in a recent editorial by Simpson et al. ⁶

Research submissions that are silent on theoretical foundations and/or conceptual frameworks, and that appear to lack congruence across methods and methodology. We offer guidance on this in a recent editorial by Brydges et al. ⁵

Survey research using instruments that on face value have not been subjected to any robust attempts to establish elements of validity and reliability, and which make no reference to a priori sample size considerations.

Underpowered research that is silent on sample size considerations, and which report findings and make recommendations untempered by an awareness of the implications of being underpowered including susceptibility to Type II error.

Small sample size studies described as pilot or feasibility studies, but which contain no rationale for why a pilot or feasibility design was used, or no criteria that needed to be met other than that the sample size was small, and which it appears may have been retrospectively labelled as such.

Retrospective analytical observational studies that report results without controlling for confounders using statistical adjustment methods such as multivariate regression modelling where appropriate.

Analytical research that has used statistical methods unsuitable to the nature of the data being analysed, for which there is reasonable concern that statistical consultation has not occurred.

Non-engagement with, or a lack of adherence to, widely accepted reporting guidelines, resulting in major reporting omissions that erode the perceived integrity of the submission.

A third strand of criticism centred around support of novice or emerging researchers and those undertaking HDR programs, suggesting Paramedicine does not give voice to such work. This is simply untrue – we are proud that 38% of published primary research papers in 2023 were the result of student-led HDR work (Honours, Masters or PhD). Paramedicine does not distinguish student research upon submission, so each paper is taken on its anonymised merits – our philosophy is that student research should be held to the same rigour as non-student, investigator-led research. It is true that HDR research often faces feasibility considerations including time and funding constraints, and that they are more likely to result in underpowered work. That in and of itself does not exclude it from consideration for publication in our journal. There is of course a threshold, intangible as that might be in a quantitative sense, at which an underpowered study becomes data-sparse to the degree to which its significance erodes, and the strength of resulting evidence falls beneath the threshold of what the editors and reviewers collectively see as viable within the context of an individual paper. We argue that there is a stronger role for research supervisors to play in ensuring HDR research is of sufficient rigour by, principally by ensuring: (1) the supervisory team has appropriate conceptual and methodological expertise; (2) the research is rigorous in design and held to high academic standards before it commences; (3) it is executed with active and engaged supervision and governance; (4) it is undertaken by a student whom wherever possible has undertaken formalised training in the selected methodology and (5) all supervisors have read, actively contributed to final manuscript versions such that the established requirements of authorship are satisfied, and approved the final submission. Our view is that implicitly indicating to students that lower standards in design and rigour are acceptable, or that peer-reviewed publishing is simple, will ultimately be detrimental to a student as they progress through their research journey.

I was recently asked by a colleague why their paper was not published in Paramedicine. It is a fair question that I think we have all wanted to ask of an Editor at some time in our publishing career. Personally, I have had several papers rejected from Paramedicine over the past year following double-blind peer review (the editorial decisions for which I am not involved and the submission anonymised). After each rejection, in the midst of feeling that sense of despondence I am sure we have all felt, I found myself asking the same questions I am sure we have all asked. Was our work bad? Had we done something fundamentally wrong in a conceptual or methodological sense? Why is their opinion of my work different from my own? Why do not they understand the paper as we do? Could the reviewer possibly be right? My co-authors and I certainly thought the work was valuable, original and offered new knowledge that could make an impact. We often ask these questions without considering what we as authors of the submission may have contributed to the review we received, and forgetting that the concept of ‘good enough’ is not dichotomous in nature – there is a continuum of ‘good’ and of ‘quality’, rather than an objective benchmark that you either satisfy or do not. At this point I am reminded of a thought-provoking editorial by Kevin Eva, titled ‘The Reviewer Is Always Right’, in which he insightfully suggests ‘the onus is on us as authors to find a way to pitch the message in a way that minimises confusion and, in turn, maximises impact’. ⁸ Where our work sat on that afore-mentioned continuum would I am sure have been a key determinant in the decision to decline, but it would not have been the only determinant. As touched on earlier, acceptance is not just about quality, and rejection does not infer unpublishability – it is as much about alignment with a journal's vision and mission, it is competitiveness with other like papers currently being reviewed, the novelty of the findings, the value it adds to what is already known, and its significance and potential for impact. It is also about how much work would need to be done to bring the paper to a point at which it would be suitable for publication – editors oversee many papers and reviewers are consistently difficult to secure. Economical use of reviewers is another consideration, and progressing papers that will need large amounts of work to peer-review can ultimately be counter-productive for the journal, the reviewers and the authors themselves.

We as an editorial team are however not without fallibility; we know that sometimes misjudgements are made and that some editorial decisions may not be considered optimal. I have on occasion made decisions to decline that perhaps could have gone the other way; there is evidence that editorial decisions are subject to the same ‘noise’ that affects other areas of human decision making. ⁹ In acknowledgement of that, we have implemented robust processes to minimise the ‘noise’ and optimise our editorial decision-making. ³ The editorial team at Paramedicine will always be open to feedback, as we see this as essential to our own reflective process and the broader development of the journal.

As the journal enters a new year of publishing, we as an editorial team remain steadfast in our commitment to quality and excellence. Paramedicine will unapologetically continue to advocate for conceptually sound and methodologically robust research and agitate for an increase in publication expectations and standards. There will continue to be positive disruption, debate and collegial discourse, and there will be further criticism. This scrutiny and dialogue are welcomed by the editorial team – it sounds like exactly what a journal should be trying to achieve.