Abstract
Background:
Spin is defined as the use of specific reporting strategies to highlight the beneficial effect of a treatment despite nonsignificant results. The presence of spin in peer-reviewed literature can negatively impact clinical and research practices. The purpose of this study was to identify the quantity and types of spin present in primary studies and systematic reviews using suture tape augmentation for ankle instability as a model.
Methods:
This study was conducted per Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Each abstract was assessed for the presence of the 15 most common types of spin. Extracted data included study title, authors, publication year, journal, level of evidence, study design, funding, reported adherence to PRISMA guidelines, and PROSPERO registration. Full texts of systematic reviews were used in the assessment of study quality per A Measurement Tool to Assess Systematic Reviews Version 2 (AMSTAR 2).
Results:
Nineteen studies were included in the final sample. At least 1 type of spin was identified in each study except one (18 of 19, 94.7%). The most common type of spin observed was type 3 (“selective reporting or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention”) (6 of 19, 31.6%), The second most reported category of spin was type 4 (“the conclusion claims safety based on non-statistically significant results with a wide confidence interval”) (4 of 19, 21.1%). Among systematic reviews, we identified type 5 (“the conclusion claims the beneficial effect of the experimental treatment despite a high risk of bias in primary studies”) in 4 out of 6 (66.7%) of the articles that were included. No significant associations were found between study characteristics and type of spin.
Conclusion:
In this exploration of the introduction of a new technology, we identified spin to be highly present in the abstracts of primary studies and systematic reviews concerning suture tape augmentation for ankle instability. Steps should be taken by scientific journals to ensure that spin is minimized in the abstract to accurately reflect the quality of the intervention.
Introduction
Chronic ankle instability (CAI) is characterized by repeated ankle sprains due to episodes of instability of the ankle joint for at least 1 year.16,17 It has been reported that between 32% and 74% of individuals with a history of an ankle sprain subsequently develop symptoms of chronic instability.1,21 CAI is usually treated with nonoperative modalities, but surgery is typically recommended if patients continue to have symptoms of instability after a minimum of 3 to 6 months of conservative treatment. 35 Currently, anatomic ligament repair through the technique proposed by Broström 6 in 1966 and modified by Gould et al 15 in 1980 is the gold standard for treatment.15,29 However, the Broström repair may not be as effective for high-risk patients, especially those with generalized ligamentous laxity, 28 high body mass index,18,39 prior unsuccessful repair, 26 or high-demand athletes.23,32 With these limitations, alternative techniques have emerged in hopes of producing better outcomes.
One of these alternative techniques is suture tape augmentation, in which the existing ligament is reinforced to provide additional stability while healing occurs. 37 The rationale for suture tape augmentation for CAI is early postoperative mobilization, faster rehabilitation, and decreased risk of reinjury.7,11,22,36,38 However, overtensioning the suture tape may alter the biological integrity of the ligament complex and restrict motion at the subtalar joint. 10 It is important to critically examine the quality of the research on suture tape augmentation for CAI and understand the associated complications.
Clinical studies evaluating novel techniques such as suture tape augmentation are at risk of spin because much of the evidence consists of small, preliminary studies of a novel technology with a high risk of bias. Spin, defined by Boutron et al 5 as “use of specific reporting strategies, from whatever motive, to highlight that the experimental treatment is beneficial, despite a statistically nonsignificant difference for the primary outcome, or to distract the reader from statistically nonsignificant results.” The abstracts of clinical research studies are particularly vulnerable to bias due to constraints such as strict word counts and selective reporting of results. Previous studies have found a high incidence of spin in the abstracts of certain orthopaedic topics such as lower extremity and spine interventions.3,9,25,34 Abstracts play a critical role in the overall perception of a study’s findings, as many physicians incorporate research findings into practice based solely on the abstract. 4 It is therefore critical to gauge the incidence of spin in clinical papers examining the efficacy of newer, experimental treatments. The focus of this study was to identify the quantity and types of spin present in the clinical studies reporting on outcomes of suture tape augmentation for CAI as an example of a likely more systemic problem. This study’s secondary objective was to characterize the studies in which spin was identified to determine whether identifiable patterns exist among studies with spin.
Methods
This study was conducted per Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines using a predetermined protocol. 27 A single author (A.T.) conducted a search of the PubMed, Scopus, and SportDiscus databases using the following search terms: “suture augmentation” AND “ankle instability”; “suture tape” AND “ankle instability.” The search results were aggregated and deduplicated in EndNote X9 (Clarivate, Philadelphia, PA). Two authors (A.T. and M.H.) independently screened the identified studies for inclusion, and the reference section of the included articles were scanned to ensure all eligible articles were obtained.
Eligibility
Primary studies, systematic reviews, and meta-analyses reporting outcomes of suture tape augmentation for ankle instability published in a peer-reviewed journal were eligible for inclusion. Databases were queried from inception to November 16, 2022, on which the searches were conducted. Exclusion criteria were studies that were not peer-reviewed, not published in English, retracted or withdrawn, included nonhuman or cadaver subjects, published without an abstract, or did not have full text available.
Training
Two authors (A.T. and M.H.) were trained to identify common study designs and characteristics to identify and characterize spin using the method proposed by Yavchitz et al 40 (Table 1), and to assess study quality among the systematic reviews and meta-analyses using version 2 of A Measurement Tool to Assess Systematic Reviews (AMSTAR 2). AMSTAR 2 is a 16-point questionnaire that quantifies the quality of a systematic review based on criteria such as whether authors report the use of a predetermined protocol, funding sources, and conflicts of interest, and/or adequately characterize studies included in the review. 33 AMSTAR 2 has undergone rigorous assessment itself and has demonstrated high interrater reliability and construct validity. 24
Frequency of Each Spin Type and Categories in the Included Studies.
Data Extraction
Two authors (A.T. and M.H.) extracted data independently, resolving any disagreements with group discussion if needed. General data that were extracted included study title, author, publication year, journal, level of evidence, study design, funding source, reported adherence to PRISMA guidelines, and preregistration of the study protocol with International Prospective Register of Systematic Reviews (PROSPERO). The title and abstract of each included study were assessed for the presence of the 15 most common types of spin, with full texts reviewed during cases of disagreement or for clarification during assessment. Each of the 15 types of spin underlie one of 3 categories: misleading interpretation, misleading reporting, and inappropriate extrapolation. The full texts of systematic reviews and meta-analyses were used in the assessment of study quality per AMSTAR 2. Study quality was further categorized based on the AMSTAR 2 assessment into critically low, low, moderate, and high categories. 33 The 2022 Scopus CiteScore (Elsevier, Amsterdam, the Netherlands) was obtained for Scopus Indexed Journals, and the 2022 Clarivate Impact Factor was obtained for all eligible journals.
Statistical Analysis
We used descriptive statistics to characterize the frequency of spin occurring in the included studies overall and with respect to the 15 most common types. Because of the small sample sizes involved, study characteristics (study type, funding, PRISMA adherence, PROSPERO registration) and AMSTAR 2 confidence rating were associated with each spin type using Fisher exact test. The Wilcoxon rank-sum test evaluated the relationship between each spin type and Clarivate Impact Factor, Scopus CiteScore, and level of evidence. A P value <.05 was used to determine statistical significance. Not all spin categories were applicable to each study type, nor was the AMSTAR 2 questionnaire applicable to nonsystematic reviews. Studies with missing data in these categories were treated as nonentities. Stata, 17.0 BE (StataCorp, LLC, College Station, TX), was used to perform all statistical analyses.
Results
The initial search terms yielded 366 articles, of which 142 were removed as duplicates. The title and abstract screening resulted in the removal of 122 articles, and full-text screening resulted in the removal of 1 article because of the lack of full-text availability (Figure 1). Nineteen articles were included in the final sample, of which 13 were primary studies (13 of 19, 68.4%) and 6 were systematic reviews and/or meta-analyses (6 of 19, 31.6%). One study was of level I evidence (1 of 19, 5.3%), 4 were level II evidence (4 of 19, 21.1%), 4 were level III evidence (4 of 19, 21.1%), and 10 were level IV evidence (10 of 19, 52.6%). The average 2022 Clarivate Impact Factor was 2.506 (range: 0.279-4.342), and the average 2022 Scopus CiteScore among the Scopus Indexed Journals was 4.119 (range: 0.4-8.5). Six of the articles reported receiving some form of external funding (6 of 19, 31.6%), 2 of which were industry-funded (2 of 6, 33.3%). All of the systematic reviews (6 of 6, 100%) adhered to PRISMA guidelines, but only 1 (1 of 6, 16.7%) had registered with PROSPERO (University of York, York, UK).
PRISMA flowchart for study selection.
At least 1 type of spin was identified in the abstracts of each study except for one (18 of 19, 94.7%). The median number of spin categories identified in each study was 1. The most common type of spin observed was type 3 (“selective reporting or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention”) (6 of 19, 31.6%). The second most reported category of spin was type 4 (“the conclusion claims safety based on non-statistically significant results with a wide confidence interval”) (4 of 19, 21.1%). Of the systematic reviews, spin type 5 (“the conclusion claims the beneficial effect of the experimental treatment despite a high risk of bias in primary studies”) was the most common (4 of 6, 66.7%). Full scoring of the included studies is demonstrated in Table 1.
Based on AMSTAR 2 assessment of the systematic reviews and meta-analyses, 1 study (16.7%) received a low confidence rating because of demonstrating 1 critical flaw, and the remaining 5 studies (83.3%) received a critically low confidence rating as they contained more than 1 critical flaw (83.3%). No studies were rated as having moderate or high confidence, which would have required sufficiency in all critical domains. Complete AMSTAR 2 assessment is demonstrated in Table 2.
AMSTAR 2 Assessment of Included Systematic Reviews.
Abbreviations: PICO, population, intervention, comparison, and outcomes; RoB, risk of bias.
Only applicable to studies conducting meta-analysis.
There were no statistically significant relationships between any of the spin categories and other study characteristics or AMSTAR 2 confidence ratings. There were also no significant relationships between the AMSTAR 2 confidence ratings and other study characteristics. No association was found between AMSTAR 2 confidence ratings and AMSTAR 2 overall scores for the 6 systematic reviews.
Discussion
We found in this study of a novel new technology, the use of suture tape augmentation for ankle instability, that spin frequently occurred in the abstracts of clinical studies and systematic reviews. At least 1 type of spin was identified in the abstracts of each study except for one. The most common type of spin observed was type 3 (“selective reporting or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention”). The second most reported category of spin was type 4 (“the conclusion claims safety based on non-statistically significant results with a wide confidence interval”) (4 of 19, 21.1%). Among systematic reviews, we identified type 5 spin (“the conclusion claims the beneficial effect of the experimental treatment despite a high risk of bias in primary studies”) in 4 of 6, or 66.7% of the articles that were included. There was no statistically significant association between the type of spin and study characteristics.
Spin may be widely present in the orthopaedic literature. Kim et al 20 found that at least 1 form of spin was found in all 17 included studies concerning superior capsular reconstruction, and Carr et al 8 found that 65.1% of included studies on the topic of Achilles tendon ruptures contained some type of spin in the abstract. In other subspecialties, the rate of spin is variable. A study examining randomized controlled trials (RCTs) in the otolaryngology field reported that 70% of the abstracts contained spin, 12 whereas another article reported that 44.3% of the abstracts in emergency medicine RCTs contained spin. 30
The most frequently cited category of spin was type 3 (“selective reporting or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention”) (6 of 19, 31.6%). Sarhan et al 31 presents an example of type 3 spin by claiming that “ankle ligament reconstruction with addition of polyester tape augmentation is an effective technique in treating chronic ankle instability with satisfactory surgical outcome” using only the American Orthopaedic Foot & Ankle Society (AOFAS) and free online Foot and Ankle Ability Measure (FAAM) scores in their analysis. Only the AOFAS score demonstrated a significant improvement compared to the preoperative scores, whereas the actually validated FAAM score change was insignificant. Furthermore, the results in the full text reported that 6.7% of the patient cohort had a positive anterior drawer test postoperatively, which was not reported in the abstract. Although it may be true that the suture tape augmentation intervention can achieve positive outcomes, the conclusion of the abstract overemphasizes the one significant finding in the study to claim the overall effectiveness of the experimental intervention to treat chronic ankle instability.
Spin type 5 (“the conclusion claims the beneficial effect of the experimental treatment despite a high risk of bias in primary studies”) was only scored among systematic reviews because of the item concerning “primary studies.” Our review found that 4 of the 6 (66.7%) systematic reviews included contained spin in this category, a particularly important finding given that many physicians and other health care providers may use systematic reviews as their best source of evidence on a topic due to time constraints and the plethora of clinical studies. 14 A study by Jones et al 19 found that spin type 5 was the second most frequently identified category in the abstracts of systematic reviews reporting on treatments for proximal humerus fractures. This finding is particularly relevant given that suture tape augmentation is an alternative approach for the treatment of ankle instability, and systematic reviews may be more susceptible to spin in this category when the experimental treatment is relatively new. Physicians should be aware of the risk of bias in primary studies when interpreting the results and conclusions of systematic reviews in these cases. Although the quality of the included studies is not necessarily the fault of the reviewers, it is important for the systematic review to disclose the high risk of bias in the abstract to highlight the strength of the conclusions. In doing so, physicians can be better equipped to make informed treatment decisions in clinical practice.
The results of our review indicate that spin in the included studies generally favors suture tape augmentation as an effective intervention to treat ankle instability. This “positive” spin is important to identify, as positive-outcome bias can influence a reviewer’s perception of the article. A randomized controlled trial by Emerson et al 13 presented identical manuscripts to peer reviewers that differed only in that one claimed a positive outcome of the intervention and the other concluded no difference. The study found that reviewers awarded higher methods scores to the positive-outcome manuscripts and were more likely to recommend the manuscript for publication than the no-difference version. It may be the case that the recent popularity of an experimental treatment such as suture tape augmentation encourages authors to publish findings that support the use of this approach, but it is important to consider that many of these findings are preliminary. Further randomized, comparative research is needed to make definitive conclusions about which treatment technique is associated with superior outcomes.
There are many reasons for why abstracts may be especially susceptible to spin. Most journals have word count limits for the abstract section, which may cause authors to leave out key aspects of the study and result in selective reporting of the most significant or compelling results. Additionally, the abstract plays a critical role in the perception of the entire article. Many readers may not read the full text if the abstract is not enticing enough, 2 which may pressure authors to overemphasize or claim a strong direction of effect when there is not necessarily enough evidence to do so. A foundational study by Boutron et al 5 found that physicians were more likely to read the full text of an article when spin in the abstract was present. Furthermore, spin in the abstract made it more likely that physicians would rate the intervention as favorable despite insignificant results.
Although identifying spin in the clinical literature for suture tape augmentation for ankle instability is critical in assessing the vulnerabilities of an article’s claim, it does not necessarily indicate that the conclusions made are completely baseless. Rather, it suggests that abstracts may not be the best overall reflection of a study’s results and as such, the full text should be reviewed whenever possible. The majority of the literature included in this study was of level IV evidence, which is to be expected in that suture tape augmentation is still a relatively new intervention. Because of this, readers should be aware of the weaknesses that may accompany such preliminary evidence and carefully consider if the claim that suture tape augmentation is effective in treating ankle instability is accurately supported by the article’s results and methodology. Indeed, the same principle should be applied to all new technology used in surgery.
Given that spin is highly present in the literature and our finding that the strength of the conclusions in abstracts do not accurately reflect the overall results, it is important to consider recommendations for improvement. Researchers should ensure that their conclusions consider both statistically and nonstatistically significant results, as well as bias in primary studies or the article itself. Additionally, it is important for both authors and readers to be aware of spin in the literature given the significant impact of an abstract on the overall perception of the article. It would be beneficial for journals to implement guidelines for minimizing spin in the submission checklist and train reviewers to identify spin in the review process.
This study was not without limitations. Although we attempted to minimize the risk of bias by using the standard definition of spin created by Yavchitz et al 40 and performing the data extraction independently and in duplicate, the classification of spin is still inherently subjective. The approach to identifying spin outlined by Yavchitz et al 40 was originally intended for systematic reviews and meta-analyses, but the majority of categories are still applicable to primary clinical studies. Additionally, when evaluating spin in the abstracts, we only referenced the full texts when further clarification was necessary, so it is possible that the spin in the abstract may not be reflective of the amount of spin in the full text. We chose this method with the knowledge that many physicians base their clinical decisions solely on the abstract of an article. 4
Conclusion
Spin is highly present in the abstracts of primary studies and systematic reviews concerning the relatively novel technology of suture tape augmentation for ankle instability. At least 1 type of spin favoring suture tape augmentation was identified in the abstracts of each study except for 1 of 19. Steps should be taken to ensure that spin is minimized in the abstract to accurately reflect the quality of all interventions, especially new technologies that may be prone to overenthusiastic reporting of positive results
Footnotes
Ethical Approval
Ethical approval was not sought for the present study because of no human subject participation.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. ICMJE forms for all authors are available online.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
