Abstract
An aspect of overuse is who decides which practices are evaluated for overuse and which of the studies on overuse are published in the medical literature. We sought to examine the frequency with which studies in medical journals questioned an established practice. As a secondary objective, we sought to determine if there was variance among medical specialties. We conducted a retrospective, cross-sectional review of the published literature in 14 medical specialty journals. We included studies from one issue in three high-impact journals (November/December 2020) for each specialty. We assessed whether the study reported on a medical practice, whether it reported on an existing practice, whether the author expressed uncertainty regarding the practice, whether the study was a randomized design, and if the authors encouraged further testing in randomized studies. For all medical specialties combined, we found that 37% (n = 98) questioned existing practices, and 15% (n = 40) either tested the practice in a randomized trial or encouraged future randomized testing of the practice. The medical specialties that questioned their practices the most were gastroenterology (61%; n = 10/18), obstetrics/gynecology (52%; n = 11/21), and cardiovascular (50%; n = 5/10). These findings indicate that, although research is being conducted to examine current medical practices, few studies advocate for randomized testing of these practices, and even fewer actually test them in a randomized fashion. Additionally, the variation across medical specialties suggests areas in which to look for potential practices that are low-value, duplicative, and/or wasteful.
Rising medical costs in the US have led to increasing interest in identifying low-value, wasteful, or duplicative medical practices that provide little benefit to the patient but collectively impart substantial healthcare costs. The decision of which practices are evaluated for overuse can influence which practices are later determined to be low-value.
The medical specialties that evaluated and tested existing practices in their field the most were gastroenterology, obstetrics/gynecology, and cardiovascular. Conversely, clinical neurology, rheumatology, and endocrinology were least likely to test and question existing practices.
Medical disciplines with the least introspection may be areas to examine potential low-value, duplicative, and/or wasteful practices.
Introduction
Rising medical costs in the US have led to increasing interest in identifying low-value, wasteful, or duplicative medical practices that provide little benefit to the patient but collectively impart substantial healthcare costs. One estimate suggests that low-value services among the entire Medicare population could cost as much as $8.5 billion. 1 Our research—and work done by other researchers—have identified numerous low-value care practices across all medical specialties.2,3
Identifying practices that have already been flagged as wasteful, duplicative, or low-value is one aspect of reducing healthcare costs. Still, another element of overuse is who decides which practices are evaluated for overuse. The Choosing Wisely Campaign has collected lists of low-value practices by medical specialty, but it relies on professional organizations to report which practices should be listed as low-value. As a result, several concerns were raised about the decisions over which services are included as part of the Choosing Wisely Campaign, including the reporting organizations being more apt to report low-value care from other medical disciplines than from their own field. 4 Further, there is wide variability in the impact on healthcare costs in which services are reported (eg, perioperative fasting vs surgery as initial treatment for cancer when effective neoadjuvant is available), as physicians and professional organizations, not payers of medical care, determine services deemed low-value. 5
Most, if not all, low-value, duplicative, or wasteful services are identified through published studies on the topic, but this depends on which services are studied, how they are reviewed, and which studies are accepted for publication in medical specialty journals. The question then can be asked, do certain medical disciplines question their own practices more than others (ie, do they evaluate existing practices, and do they seek this evaluation in the form of high-quality study design)? As such, we sought to look across medical specialties to see if there are disciplines where this introspection is more or less common in the published medical literature.
Methods
Article Selection
From a list of medical disciplines, we selected the top three medical journals that publish original research, based on impact factor, for each specialty. We used the Scimago Journal and Country Rank (https://www.scimagojr.com/journalrank.php?area=2700), by subject category under “medicine” for determining the journal’s impact factor. We selected the current issue (as of November 24, 2020), which was usually a December 2020 issue but sometimes a November 2020 issue, and made a list of all research articles, original research, research letters, brief reports, commentaries, editorials, and reviews included in that issue of the journal. We excluded letters to the editors, which are almost always critical of a previously published article and do not provide any new findings.
Data Abstraction
For each article that reported new findings or that conducted a formal analysis of findings in the literature (eg, original research, research letters, brief reports, and reviews), we abstracted these data: the overall results of the study (pos = met its endpoint; neg = did not meet its endpoint; or equivocal = both positive and negative results), funding source, study design, whether the article focused on a specific medical practice, and if so, whether the practice was novel or established, whether the practice was particular to that specialty, whether the authors of the study were critical of the practice, whether the authors expressed uncertainty or conflicting expert opinion about the practice if it was established, and whether the study was randomized or if the authors of the article promoted further research of the practice in higher-quality studies (eg, randomized controlled trials). We defined established medical practices as those for which we could find evidence of their use outside of clinical trials. For the majority of the studies, we were able to locate this information in the article’s introduction, discussion, or accompanying editorial. For other practices, we searched Google Scholar to see if observational/non-interventional studies had documented the practice’s use.
For each article that only summarized findings in the literature and did not report new results (eg, commentaries and editorials), we abstracted these data: whether the article was about a specific medical practice, whether the author of the commentary/editorial supports the practice, whether the author of the article promoted further research and evaluation of the practice, and whether the article was related to an original research article or review in the same issue of the journal.
We created a variable indicating whether the study exhibited introspection of the respective practice. Studies/articles we considered were those that concerned an established or existing medical practice specific to their field and were evaluated because of uncertainty regarding the practice. A flowchart of how introspection was determined is presented in Figure 1. Flowchart of the questions to determine introspection in published original research articles, research letters, brief reports, and reviews in high-impact medical specialty journals (November/December 2020).
Statistical Analysis
We reported results in frequencies and percentages. Our primary focus was to evaluate these frequencies across all medical specialties, but as a pilot or secondary objective, we examined these by medical specialty. All analyses were performed in R statistical software, version 3.6.1. Because we used publicly available data, which did not include personally identifiable information, an institutional review board’s approval was not required.
Results
Questioning of existing practices in original articles, brief reports, research letters, and reviews reporting on a medical practice published in high-impact journals (November/December 2020).
aQuestioning of existing practices was determined as studies/articles examining an established or existing medical practice specific to their field, and where the authors expressed uncertainty regarding the practice.
For all medical specialties combined, we found that 37% (n = 98) questioned existing practices. The medical specialties that most questioned practices they performed were gastroenterology (67%; n = 12/18), obstetrics/gynecology (52%; n = 11/21), cardiology (50%; n = 5/10), pulmonary and critical care (50%; n = 3/6), and neurosurgery (45%; n = 15/33). Figure 2 shows the percentage of studies reporting on a medical practice and the percentage of studies raising questions or doubts about practices they perform. Variation in the percentage of original studies that questioned specialty-specific practices (Nov/Dec 2020).
Questioning of existing practices in original articles, brief reports, research letters, and reviews reporting on a medical practice published in high-impact journals (November/December 2020) by medical specialty.
aQuestioning of existing practices was determined as studies/articles examining an established or existing medical practice specific to their field, and where the authors expressed uncertainty regarding the practice.
Questioning of existing practices in commentaries and editorials reporting on a medical practice published in high-impact journals (November/December 2020), by medical specialty and study results linked to commentary/editorial.
aQuestioning of existing practices was determined as studies/articles examining an established or existing specific medical practice that was specific to their field, and where the authors expressed uncertainty regarding the practice.
Discussion
We found that only a minority of published studies in high-impact medical journals (37%) questioned a medical practice that was established at the time of the study. Further, only 15% of studies concerning a medical practice (40% of studies questioning a medical practice) either tested the practice in a randomized study or encouraged trialists to test the practice in a randomized study. Our results, which are slightly higher than previous estimates, 6 apply to discipline-specific journals rather than general medical journals. Similar to these prior studies, we found that only a small percentage of studies that questioned existing practices were funded by industry (3%), even though as many as half of all clinical trials registered on clinicaltrials.gov are sponsored by industry. We also found that almost half of studies questioning an existing practice examined a procedure, rather than a medication or another type of practice. Industry has a vested interest in supporting research involving novel medical practices, particularly those involving drugs, but there is little incentive for them to fund studies on practices that have already been implemented. 7
We found notable variation in how often a specialty was willing to question practices they perform. Gastroenterology, obstetrics/gynecology, cardiovascular, and pulmonary and critical care were specialties with a high percentage of articles that not only questioned a specific medical practice but also either tested these practices in a randomized study or encouraged further testing in a randomized study. Conversely, clinical neurology, rheumatology, nephrology, urology, and endocrinology had a lower percentage of introspection and encouragement of randomized studies. We previously reported that for these specialties, there was a low percentage of practices that were considered medical reversals, which may be due to reduced desire to appraise longstanding practices in these specialties. 2
For specialties that tend to not question the practices they perform, there is a good chance that there are undiscovered examples of overused medical practices. By identifying these areas and specialties where there is less examination of established medical practices, one might find ways to reduce low-value care. For researchers interested in low-value care, these specialties may provide a good opportunity, as low hanging fruit may yet be unaddressed.
We found that studies questioning the effectiveness of utilized practices tended to reflect procedural practices, including radiation treatment and screening, and less of practices regarding lifestyle practices, drugs, and devices. There may be various reasons the latter studies were less likely to question specialty-specific practices. Drugs and devices tended to be novel and funded by industry. In contrast, studies on lifestyle practices, including diet and exercise, often took positive results of observational studies as reason to implement them into practice before randomized studies were done to confirm efficacy. Perhaps these practices are so readily accepted because people feel little to no harm or downsides in implementing them. The situation of randomized studies refuting observational evidence in diet and exercise has occurred multiple times in recent years, and in some cases, they were shown to be harmful in randomized studies.8,9
We also found studies that questioned specialty-specific practices tended to report negative or null results compared to studies that did not. This may signal publication bias, where positive studies are more likely to be published unless there is uncertainty regarding the practice.10,11 Preferential publication may indirectly promote low-value care through fewer negative or null studies on these same practices being published. 12 The current status of publication bias is unknown, in part because meta-analyses often do not evaluate this type of bias. 13
Limitations
There are two key limitations to our work. First, we utilized only one month’s worth of publications, which limits our capacity to conduct cross-specialty comparisons. To lessen bias, we chose which month to analyze at random. Also, because of the COVID-19 pandemic, an unusually high number of studies were reported on SARS-CoV-2. SARS-CoV-2 studies were published broadly across medical specialties. These studies often did not report on medical practices, and if they did, the practices were novel. Because of this, the results of our analyses are not generalizable to the literature at-large. Second, our analysis relied on the subjective judgment as to whether or not the study questioned existing specialty-specific practices. We attempted to use clear determination rules, and each article was reviewed by two independent reviewers (AH and KP), with a third (VP) reviewing any discrepancies. As such, others may code these articles differently. We encourage these efforts.
Conclusion
In summary, our results suggest that although research is being conducted to examine current medical practices, few studies encourage randomized testing of these practices, and even fewer actually test them in a randomized fashion. Further, the variation across medical specialties suggests areas in which to examine potential practices that are low-value, duplicative, and/or wasteful.
Footnotes
Author Contributions
VP, AH, and KP conceptualized study design; AH and KP reviewed and abstracted data; VP reviewed and confirmed abstracted data; AH wrote the first draft of the manuscript; and all authors reviewed and revised subsequent and finalized draft of the manuscript.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Laura and John Arnold Foundation.
Data Sharing Statement
All data are publicly available in the published literature.
