Abstract
Background
Innovation has in common the promise of benefit for patients; however, past experience has shown that this promise is not always delivered. Instead, low-value innovation might encourage treatment variation and dilute the available body of evidence. This study aims to investigate (1) whether the peer-review process is capable of filtering out low-value innovation appropriately, and (2) whether low-value surgical innovation would be preferred more often than nonoperative innovation by peer-reviewers in the treatment of proximal humeral fractures in the elderly.
Materials and methods
Two duplicated sham scientific abstracts, respectively introducing a low-value surgical innovation and a valuable nonsurgical innovation, were submitted to nineteen peer-reviewed scientific meetings worldwide for orthopedic trauma surgery with submission deadlines between 01/01/2022 and 31/12/2022. Decision regarding abstract acceptance was compared.
Results
There was a high acceptance rate for the abstract introducing low-value surgical innovation (12 out of 19 (63.2 %)), which was higher than that of a nonoperative duplicate (10 out of 19 (52.6 %)), but this difference was not statistically significant (p = 0.5). The majority of the ten meetings that accepted both abstracts placed both in equivalent programmatic tiers (oral presentation (4) and poster presentation (2)). In three meetings, the surgical abstract received superior program placement (oral presentation). In one case, it was the opposite.
Conclusion
There is a high acceptance rate for low-value surgical innovation among peer-reviewed scientific meetings. However, we can not conclude that low-value surgical innovation is preferred more often than nonoperative innovation by peer-reviewers as the differences in acceptance rate were small and not statistically significant. The peer-review process may be suitable as value-based medicine emerges. Scientists should be encouraged to pursue value-based innovation.
Keywords
Introduction
Evidence over the last two decades has not been able to demonstrate superiority of surgery over nonoperative treatment for the majority of proximal humeral fractures (PHF), one of the most common injuries in the elderly.1–5 It is suggested for this age group that nonoperative treatment has functional outcomes similar to those of surgery, but with lower risks of complications and subsequent operation.5,6 Nevertheless, a continuous increase in published surgical techniques and implants introduces greater choices in management,6–9 and complicates consensus building on optimal treatment leading to substantial variation in management.7,10 While continued investigations of an unsolved clinical problem should be encouraged, low-value research can encourage treatment variation and dilute the available body of evidence.
Technological advances in orthopaedics usually have in common the promise of benefit for patients; however, past experience has shown that this promise is not always delivered.
A recent analysis of reports published over the last 20 years, during which time new technologies and techniques have become available, of patients with shoulder arthritis treated with arthroplasty did not provide evidence that surgical outcomes have improved over this timeframe with the advent of new technologies. 11 While these publications passed through the established scientific peer-review process during that large time period, the incremental value to the patient of each new technology remained questionable.
Aside from the medical–industrial complex, 12 the authors suspect that the established peer-reviewing system might also play a significant role in functioning as a gatekeeper for published evidence catalyzing future trends in innovation .
The authors hypothesize that (1) there is a high acceptance rate for low-value surgical innovation among peer-reviewed scientific meetings as this system might be incapable of filtering out low-value content appropriately, and that (2) low-value surgical innovation will be preferred more frequently by peer-reviewers over valuable nonoperative innovation in a surgical specialty. We define value in this study as health outcome per unit cost. 13 To date, there is no study investigating this issue.
The purposes of this comparative study were to investigate these hypotheses using the model of the treatment of proximal humeral fractures in the elderly through two sham scientific abstracts submitted to peer-reviewed scientific meetings.
Materials and methods
Sham abstracts
This study was authorized by the local ethical committee (no. 10065_BO_K_2021). As no human participants, animals or any materials from a natural setting were involved, no consent was obtained.
We designed two sham scientific abstracts to investigate the peer-review process of scientific meetings. Both abstracts were designed to have levels of evidence, patient demographics, sample size, time period of inclusion, follow-up period, patient-reported outcomes (PROs), wording, and total word count of a maximum of 200 words in order to meet the requirements of as many scientific meetings as possible.
Both abstracts claimed to introduce surgical and nonsurgical innovation with similar PROs in the treatment of displaced PHF in the elderly in the form of an uncontrolled case series. Value was defined as health outcome per unit cost according to the equation of value by Porter. 13 The value proposition of each abstract was manipulated by keeping health outcome constant and altering cost, the denominator of the value equation.
The abstract introducing the surgical innovation was designed to be inferior in terms of creating additional patient value. Instead of providing superior PROs than previously described for the procedure of open reduction and internal plate fixation, this innovation resulted in the same PROs, but with much higher costs, high complication rates, and revision surgery rates. High costs were a result of the disproportional application of expensive trendy key technologies like 3D-printing, anatomic pre-contoured plate osteosynthesis, patient-specific instrumentation (PSI), and minimally invasive surgery. According to a certified national medical device manufacturer specializing in patient-specific implants (K-Implant GmbH, Garbsen, Germany), 14 the pure material and labor costs alone of such a surgical, patient-specific treatment would be an estimated $ 6,000 to $ 9,000 per patient.
By comparison, the nonsurgical innovation also resulted in similar PROs as previously described for the procedure of nonoperative treatment in an arm sling with much lower com-plication and subsequent surgery rates. However, the nonsurgical innovation was designed to reduce costs, the denominator of Porter`s equation, and thus increase value for patients.
13
Lower costs were a result of a self-made version of commercially available arm slings, which cost between $100 to $130 (Figure 1).
15
We thus have presented a multitude of additional costs for the initial surgical care innovation compared to the nonsurgical innovation. No funding source played any role in this investigation. Plagiarism analysis confirming a high match rate of 91.8% between the surgical abstract (left side) and the nonsurgical abstract (right side).
Expected PROs were extrapolated from the largest randomized clinical trial on displaced PHFs treated mainly with open reduction and plate fixation to date. 16 Differences in complication and revision rates were selected based on known relative risk ratios of the latest comprehensive Cochrane review on the treatment of PHF at the time of study design. 4
Firstly, the surgical abstract (Figure 1) was constructed by the first author (S.R.) and subsequently linguistically edited by one blinded assistant professor of orthopaedic surgery (D.Z.) with language proficiency level C2 according to the Common European Framework of Reference for Languages (CEFR), not privy to the study intention at that time. Subsequently, based on the surgical abstract, a nonsurgical duplicate (Figure 1) was created by the first author. The maximal amount of duplicated wording content was intended and verified through the online plagiarism software https://www.copyleaks.com/. 17 Figure 1 shows both abstracts after plagiarism analysis confirming a high match rate of 91.8%. Both abstracts are provided as supplementary file (supplementary file).
Scientific meetings
The most relevant peer-reviewed scientific meetings of the native German speaking D-A-CH region (Germany, Austria, Switzerland) as well as international English-speaking scientific meetings with submission deadlines between 01/01/2022 and 31/12/2022 were screened by one senior resident of orthopedic trauma surgery trained with special focus in upper extremity surgery (S.R.), and included in this investigation. Figure 2 shows details of inclusion in a flow chart. Table 1 lists all included meetings. Flow chart showing details of inclusion. *ASES = Annual Meeting of American Shoulder and Elbow Surgeons 2022, SESA = Biennial Closed Conference of the Shoulder and Elbow Society of Australia 2022, ÖGU/ÖGOuT (Official German abbreviation) = Annual Meeting of the Austrian Society for Traumatology and Austrian Society for Orthopedics and Traumatology. Listing of all included scientific meetings. N.d. = not determinable. aData based on scientific booklet. bData based on scientific booklet and publicly accessible program committee report. Approx. = approximately.
In order to keep the proportion of homogeneity as high as possible, English language was chosen in cases of free choice of language. Only in one of the meetings (no. 6), the abstracts had to be translated into German before submission due to restrictive language requirements (Table 1). Minor changes regarding abstract formatting and title length without altering the content had to be made for meetings no. 10, 12, 14, 15, 19, and 20 to meet the formal submission criteria (Table 1). Accidental incomplete submission occurred for one meeting (no. 12), which is why only the surgical abstract was analyzable (Table 1). In all cases of acceptance, the abstracts were retracted. In addition, all offices were surveyed via email in May 2024 on the total number of accepted abstracts (oral presentation and poster), and the acceptance rate (Table 1). In case of no response after 6 weeks, the online web program and booklets were searched and, if available, the total number of accepted abstracts was obtained from them (Table 1).
Statistical analyses
Descriptive statistics were used to compare acceptance rates and differences in program placement (oral vs poster presentation) of both abstracts. Fisher exact tests were performed to compare the proportions of accepted surgical and nonoperative abstracts. An accepted oral presentation was defined as a superior programmatic placement to a poster presentation. In one scientific meeting (No. 14) video presentation was defined as oral presentation. For the analyses, SPSS 26 (IBM, Armonk, New York) was used.
In order to calculate the sample size, the Chi-squared Goodness-of-fit test for contingency tables was performed with G*Power 3.1.9.4 (Heinrich-Heine-University, Düsseldorf, Germany). As the authors’ initial working hypothesis assumes a significant difference in the acceptance rate, a strong effect size (w) of 0.5 was chosen. With only one degree of freedom (df), and an alpha error of 0.05, a total sample size of 32 is required for a power of 0.8 (1-β error). This would be given. As the two abstracts are submitted to each of the 19 evaluable meetings, the total sample size is 38.
Results
Mean time interval to decision was similar for both abstracts (101.8 days for the surgical abstract vs 102.4 days for the nonoperative duplicate). However, there was a high acceptance rate for the abstract introducing surgical innovation (13 out of 20 (65 %)), which was higher than that of the nonoperative duplicate (10 out of 19 (52.6 %)), but this difference was not statistically significant (p = 0.5). Similar distributions were found after exclusion of the scientific meeting of the British Trauma Society, which had incomplete submission (12 out of 19 (63.2 %) versus 10 out of 19 (52.6 %)) (p = 0.7).
The majority of the ten meetings that accepted both abstracts placed both at equivalent programmatic tiers (oral presentation (4) and poster presentation (2)). In three meetings, the surgical abstract was preferred over the nonoperative duplicate in superior program placement. In one case, it was the opposite. Figure 3 shows details of results for each meeting. Details of results for each scientific meeting. n.e. = not evaluated. n.m. = not measured. This figure has been designed using license free, and editable images from Flaticon.com.
Discussion
This is the first study investigating the robustness of the peer-review process to low-value innovation using the model of the treatment of proximal humeral fractures in the elderly. The principal finding that there is a high acceptance rate for low-value innovation among peer-reviewed scientific meetings might underline our first hypothesis that peer-review system might be incapable of filtering out low-value content appropriately. In addition, it might justify the need for a discourse in the scientific community, which has so far been touched on only sporadically within the context of journal publication requirements.18,19 Aside from journals, scientific meetings would benefit from clear and uniform criteria for valuable innovation as they often function as the first screen for the presentation of current and future trends to a scientific audience. Of the 20 meetings examined, only one, the Annual Meeting of the Orthopaedic Research Society (ORS), suggested that defined criteria might be significant in the reviewing process. On the abstract submission site, it is required to provide context in terms of study significance and clinical relevance. For these purposes, submitters are referred to the enhanced review criteria for evaluation of scientific and technical merit of applications submitted to the National Institutes of Health (NIH) for grants. 20 Requirements for the topic “innovation” are also listed among other points defined as novel approaches, instrumentation or interventions that challenge and seek to shift current research or clinical practice paradigms, 20 even though the current study was based upon reviews made just a year following that published criteria and perhaps too soon for this influence to be incorporated into the review process.
In theories of technological advancement, innovation is seen as an endogenous process, where previously accumulated knowledge enables future progress. This view is captured famously in Isaac Newton’s observation that if he had seen further, it was by ‘standing on the shoulders of giants’. 21 Today, researchers are faced with the burden of an exponentially growing volume of new scientific and technological knowledge in the last decades. 21 This has been assumed recently as one of several potential factors hindering the basic conditions for disruptive innovation. 21
In the present study, we introduced two abstracts with differing value propositions, due to comparable clinical outcomes with higher associated cost in one compared to the other. As both introduced innovations were fictitious, the actual associated costs are based only on estimates, which has to be considered as a limitation. Ultimately, neither the nonoperative innovation abstract nor the surgical innovation abstract did introduce improved clinical outcomes compared to the literature. However, the nonsurgical innovation was associated with lower complications, revision rates, and costs, with which it results in more value according to the equation of value by Porter. Taking the present study as a concrete example, with the comprehensive knowledge of the last 20 years, the abstract introducing low-value, derivative innovation should have been rejected, making room for more valuable work; this was unfortunately done in only 35 % of cases. Instead, the majority of scientific meetings in our study accepted these abstracts. The abstract acceptance rates of major orthopaedic scientific meetings are not generally disclosed; however, the available evidence would suggest that approximate acceptance rates of 37% to 55% may be expected.22–24 Given this, the acceptance rates of 53% and 65% in our study were relatively high. Giving low-value innovations the platform of presentation might affirm scientists to keep focusing on a narrower scope of existing knowledge, and dilute what recently has been named as the effective stock of knowledge. 21 We speculate that this vicious circle might be one potential reason for the significant knowledge gap that has been highlighted on the topic of PHF. 25 According to a scoping review, there is an enormous numerical imbalance in scientific literature on PHF between, on the one hand, a narrow slice of less than 5% of papers concerning nonsurgical treatment, and on the other hand, more than 60% of papers concerning surgical treatment including countless uncontrolled cases series introducing new surgical implants or slightly modified procedures as intended in our sham abstract.5,25 This appears perplexing. No randomized clinical trial (3%) or comprehensive Cochrane review, what one might call the effective stock of knowledge, has been able to demonstrate superiority of surgical treatment during that same period.8,25
We have demonstrated a statistically unsignificant difference between the acceptance rates of the abstracts for the low-value surgical innovation and the valuable nonsurgical innovation, but the difference was smaller than expected and smaller than previous findings led us to assume. This is why, we have to reject our second initial hypothesis low-value surgical innovation would be preferred more often than nonoperative innovation by peer-reviewers in the treatment of proximal humeral fractures in the elderly.
In the majority of scientific meetings where both abstracts were accepted, both were placed in equivalent programmatic tiers. The renowned annual meeting of EFORT, which is the largest platform for European Orthopaedics and Traumatology, comprising 41 national member societies from 39 member countries as well as 13 associate scientific members, even preferred the nonoperative innovation over the low-value surgical innovation in superior program placement. While the high acceptance rate for abstracts detailing the low-value derivative surgical innovation is disappointing, researchers may be encouraged that innovations with favorable outcomes that lower cost can be similarly well-received by the scientific community. These findings should keep scientists encouraged to pursue value-based innovation in emerging times of value-based healthcare.
This study has several limitations to consider. A small sample size of peer-reviewed scientific meetings was included. This was large enough for the authors’ initial working hypothesis. However, this hypothesis must be rejected and the observed statistically unsignificant difference between the acceptance rates can not be supported by sufficient power and allow a robust conclusion since the total sample size is too small, as revealed by a post hoc power analysis. In addition, the selection of meetings might have been subject to selection bias. Since abstracts were submitted to a variety of society meetings across countries and languages, it is possible that societal, cultural, or language biases influenced acceptance or rejection of submissions. In addition, a range of very different sized meetings with various, largely unclear acceptance rates were included (Table 1). Thus, the acceptance rate of our sham abstracts could logically also depend on the size of the congress and the volume of abstracts that could be included.
Furthermore, it is unclear how many peer reviewers were involved in the review process of each scientific meeting, and whether both abstracts were reviewed by the same reviewers.
In addition, our conclusions pertain to peer-reviewed scientific meetings and cannot be extrapolated to peer-reviewed scientific journals. As it is known that accepted scientific content at peer-reviewed annual meetings is not necessarily accompanied by subsequent publication in peer-reviewed journals. The peer review process might be likely less rigorous for the hosts and their recruited reviewers of scientific meetings than for editors of scientific publications. Hosts of scientific meetings might select topics that will highlight interest, increase attendance, and facilitate discussion. While this also occurs in the medical literature, peer review in scientific meetings might be deliberately less consistent and critical. This must be considered as a major limitation. However, during study design, the authors did not see a viable possibility of creating a whole sham manuscript with similar highly duplicated content as provided in this study. Moreover, scientific meetings frequently serve as a gatekeeper for frontiers research, with a recent study demonstrating that 73% of podium presentations and 65% of poster presentations at the American Academy of Orthopaedic Surgeons Annual Meetings from 2014 to 2017 progressed to eventual peer-reviewed publication. 26 Moreover, our study was designed to test the premise that reviewers of scientific meetings would prefer and prioritize value-based evidence in the treatment of PHF, but this is merely an assumption. If the expressed aim of scientific meetings were to maximize audience engagement regardless of the value of the content presented, then our research design would have been flawed. The authors do not believe this is or should be the case. Lastly, PROs and relative risk ratios of complication and revision rates used in this study were taken from a Cochrane review from 2015. This was the most recent available data at the time of study preparation and initiation. There has since been an updated Cochrane review from 2022, reporting similar PROs with only minor differences. 5
Conclusion
There is a high acceptance rate for low-value surgical innovation among peer-reviewed scientific meetings. However, we can not conclude that low-value surgical innovation is preferred more often than nonoperative innovation by peer-reviewers as the differences in acceptance rates were small and not statistically significant. The peer-review process may be suitable as value-based medicine emerges. Scientists should be encouraged to pursue value-based innovation.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: We acknowledge support by the German Research Foundation (DFG) and the Open Access Publication Fund of Hannover Medical School (MHH).
