The Continuous Fragility Index of Statistically Significant Findings in Randomized Controlled Trials That Compare Interventions for Anterior Shoulder Instability

Abstract

Background:

Evidence-based care relies on robust research. The fragility index (FI) is used to assess the robustness of statistically significant findings in randomized controlled trials (RCTs). While the traditional FI is limited to dichotomous outcomes, a novel tool, the continuous fragility index (CFI), allows for the assessment of the robustness of continuous outcomes.

Purpose:

To calculate the CFI of statistically significant continuous outcomes in RCTs evaluating interventions for managing anterior shoulder instability (ASI).

Study Design:

Meta-analysis; Level of evidence, 2.

Methods:

A search was conducted across the MEDLINE, Embase, and CENTRAL databases for RCTs assessing management strategies for ASI from inception to October 6, 2022. Studies that reported a statistically significant difference between study groups in ≥1 continuous outcome were included. The CFI was calculated and applied to all available RCTs reporting interventions for ASI. Multivariable linear regression was performed between the CFI and various study characteristics as predictors.

Results:

There were 27 RCTs, with a total of 1846 shoulders, included. The median sample size was 61 shoulders (IQR, 43). The median CFI across 27 RCTs was 8.2 (IQR, 17.2; 95% CI, 3.6-15.4). The median CFI was 7.9 (IQR, 21; 95% CI, 1-22) for 11 studies comparing surgical methods, 22.6 (IQR, 16; 95% CI, 8.2-30.4) for 6 studies comparing nonsurgical reduction interventions, 2.8 for 3 studies comparing immobilization methods, and 2.4 for 3 studies comparing surgical versus nonsurgical interventions. Significantly, 22 of 57 included outcomes (38.6%) from studies with completed follow-up data had a loss to follow-up exceeding their CFI. Multivariable regression demonstrated that there was a statistically significant positive correlation between a trial’s sample size and the CFI of its outcomes (r = 0.23 [95% CI, 0.13-0.33]; P < .001).

Conclusion:

More than a third of continuous outcomes in ASI trials had a CFI less than the reported loss to follow-up. This carries the significant risk of reversing trial findings and should be considered when evaluating available RCT data. We recommend including the FI, CFI, and loss to follow-up in the abstracts of future RCTs.

Keywords

shoulder instability shoulder general statistics fragility index continuous fragility index statistical significance

The shoulder is highly susceptible to instability and dislocations.^1,11 Numerous trials have been conducted to evaluate various aspects of the surgical and nonsurgical management of anterior shoulder instability (ASI).^5,34,48,54 While well-conducted randomized controlled trials (RCTs) provide the best available evidence when making treatment decisions, it is important to understand how fragile or robust the results of a trial are when interpreting findings to develop a degree of confidence in the reported estimate of effect.

The traditional threshold of P < .05 used in most RCTs for statistical significance has been criticized for being arbitrary and for encouraging a categorical interpretation of evidence.^4,40,41,59 Several measures have been subsequently developed to supplement the P value threshold, most notably, the fragility index (FI), which was first conceptualized by Feinstein¹⁹ in 1990 and popularized for use in trials by Walsh et al⁵⁶ in 2014. In summary, the FI is the number of events in 1 trial arm that must be changed to nonevents, with the other arm unchanged, so that a statistically significant difference between the 2 trial arms becomes a nonsignificant one. For example, in an outcome with an FI value of 1, it would only take 1 patient’s outcome to change from an event to a nonevent for the findings to no longer be statistically significant. Therefore, fragility can identify significant outcomes relying on only a few patients for significance.

The FI can also provide insight into whether a trial’s loss to follow-up can affect its overall findings.⁵⁶ In trials in which the number of patients lost to follow-up exceeds the FI of the results, the number of patients with an unknown outcome that may possibly be a nonevent is enough to threaten reversing the significance of the trial’s results. Previous FI reviews have reported the percentage of such studies in different fields as ranging from 12.5% to >50%.^32,49,56 If identified as a widespread issue, this phenomenon can be mitigated by adequately powered studies and accurate estimates of loss to follow-up.

A key limitation of the FI is that it can only be applied to dichotomous outcomes, overlooking vast continuous data that can be equally important in shaping clinical practice. Continuous outcomes include various objective outcomes (eg, range of motion, strength, procedure time) and patient-reported outcomes. Previous reviews that have evaluated the FI report excluding between 25% and 82% of assessed studies because of a lack of dichotomous outcomes.^8,25,51 Caldwell et al⁸ recently introduced the continuous fragility index (CFI), allowing for the assessment of the fragility of continuous outcomes.

While a recent systematic review assessing the FI of ASI RCTs has been published,¹³ it was limited to dichotomous outcomes such as redislocations, ability to return to play, noncompliance, and need for revision surgery. As such, the purpose of the present review was to evaluate the fragility of significant continuous outcomes in RCTs that compare interventions for ASI to provide readers with an understanding of the quality of available RCT data by which treatment decisions are made.

Methods

Search Strategy and Screening

We conducted an electronic search of the MEDLINE, Embase, and CENTRAL databases from inception through October 6, 2022. The search terms included the following: (RCT or randomized) and ((shoulder and (instability or dislocation or subluxation)) or Bankart lesion) (see the Appendix, available in the online version of this article). We automatically removed duplicate records with reference management software (EndNote X9; Clarivate). Using predetermined eligibility criteria, 2 reviewers (M.A. and M.S.) independently screened the title and abstract of each study using systematic review management software (Rayyan 0.9; Qatar Computing Research Institute). We advanced disagreements at the title/abstract review stage to the full-text review stage to prevent any premature exclusion of studies. Discrepancies between the 2 reviewers at the full-text stage were resolved by consulting the senior author (M.K.).

Eligibility Criteria

The inclusion criteria of this study were as follows: (1) 2-arm RCT assessing the management of ASI; (2) statistically significant finding on ≥1 primary, coprimary, or secondary continuous outcome such as pain, range of motion, questionnaire scores, duration of procedure, and dislocation displacement; (3) details on the sample size, mean values, and spread statistics (standard deviation, confidence interval, range, etc) for the significant outcome(s) for both arms; (4) human study; and (5) English language. The exclusion criteria were as follows: (1) multidirectional or posterior instability studies; (2) unspecified dislocation direction (anterior, posterior, etc); (3) fracture studies; (4) studies investigating perioperative interventions; (5) multiple publications for the same trial with different follow-up times (we included only the latest publication); (6) studies with >2 intervention groups or arms; (7) ASI prevention studies; and (8) letters, reviews, meta-analyses, protocols, biomechanical studies, cadaveric studies, and abstract-only publications.

Data Extraction

Again, 2 reviewers (M.A. and M.S.) independently extracted relevant information from each study using a collaborative data extraction spreadsheet (Google Sheets; Google) designed a priori. Extracted data included study characteristics (author(s), study design, publication year, etc), patient characteristics (age, sex, etc), intervention, length of follow-up, number of shoulders analyzed in each study group, and number of shoulders lost to follow-up. For each statistically significant continuous outcome in a study such as pain or range of motion, we extracted the sample size, mean values, and spread statistics of the outcome for both study groups. We categorized outcomes as either primary, secondary, or unspecified. In studies that did not specify a primary outcome but conducted a sample size calculation, we considered the outcome used to power the study as the primary outcome and the rest of the outcomes as secondary. We tallied the total number of outcomes, primary outcomes, and secondary outcomes. Overall, 4 of 27 studies (14.8%) did not report loss to follow-up,^10,18,28,45 which we excluded from calculations involving loss to follow-up. Outcomes were further categorized as either objective (eg, duration of procedure, range of motion) or patient reported (eg, questionnaires, pain measures). We determined the level of evidence for each study by using the classification system set out by Wright et al.⁶⁰

Calculation of CFI

To calculate the CFI, the full data sets of the measurements of 2 patient groups were required. First, the group with the higher mean was noted. Then, the data point closest to and greater than the mean in the higher-mean group was moved to the lower-mean group. This was repeated until the P value obtained from the Welch t test no longer demonstrated a statistically significant difference between the 2 groups.⁸ The CFI was defined as the number of data points moved. This CFI calculation is visualized in Figure 1.

Figure 1.

Calculating the continuous fragility index (CFI). The outcomes for 2 theoretical patient groups are illustrated, with each outcome represented as a diamond on the number line. (A) Before any manipulation, there is a statistically significant difference between the groups (P < .0001). We first identify the data points closest to and greater than the mean in the intervention group. These data points are moved 1 unit at a time in a sequential manner to the control group until the 2 groups are no longer statistically different (resulting in B). In this case, 5 diamonds (C) must be moved. Therefore, the robustness of the initial difference can be described with a CFI value of 5.

The method outlined above requires the full data set of the outcome measure of interest. However, published RCTs often only report summary estimates such as the mean and standard deviation. Therefore, an approximation method has been described by Caldwell et al⁸ that utilizes various descriptive statistics (ie, mean, standard deviation, and sample size) to generate simulated full data sets, assuming a normal distribution, which can then be used to calculate the CFI. To minimize the random error associated with generating random data sets, this model implements multiple iterations that are then averaged accordingly.

All CFI values in this study were calculated using an online calculator that utilized the approximation method described by Caldwell et al.^8,9 Model inputs included sample size, mean, and standard deviation of both groups in a study. Adjustable model parameters included the number of iterations used to produce the mean CFI, which was set to 5, and the tolerance. The model’s tolerance is the proximity of the random data set’s mean and standard deviation to that of the trial of interest’s respective values; this was set at 0.01.

Overall, 9 included studies used spread statistics other than the standard deviation such as the range, confidence interval, and interquartile range.^∥ Following Caldwell et al’s⁸ methodology for calculating the CFI, we estimated the standard deviation for those outcomes using appropriate conversion tools.^26,57,58

Statistical Analysis

We calculated descriptive statistics across all studies. For a given study with multiple eligible outcomes for analysis, we calculated the CFI of each eligible outcome and averaged them to obtain a study-specific CFI per previous methodology.^16,46 We also calculated the primary-only study-specific CFI for each study, if applicable, by only considering the fragility of primary and coprimary outcomes. We grouped the included studies based on the type of interventions compared and presented statistics for each. We categorized all outcomes as either objective or patient reported, presented statistics for both, and determined whether there was a statistically significant difference between the 2 outcome types using the Mann-Whitney U test. We performed multivariable linear regression in which the outcome variable was the study-specific CFI and the predictor variables were the (1) sample size, (2) journal impact factor, (3) percentage of participants lost to follow-up, (4) duration of follow-up in months, and (5) publication date.

Results

Study Selection and Characteristics

We identified a total of 1877 records across the MEDLINE, Embase, and CENTRAL databases (Figure 2), with 1080 remaining for title and abstract screening after removing duplicates. After applying the eligibility criteria, we identified 27 studies for inclusion.^¶

Figure 2.

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart.

We also calculated the overall CFI for the following study types (Table 1): (1) studies comparing surgical methods (11 studies),^# (2) studies comparing nonsurgical reduction interventions (6 studies),^{2,3,21,36,38,52} (3) studies comparing immobilization methods (3 studies),^24,37,43 and (4) studies comparing surgical versus nonsurgical interventions (3 studies).^7,35,47 We grouped the remaining 4 studies into “other.”^17,28,33,39

Table 1

Characteristics and CFI Values of Included Studies^a

					Mean CFI
First Author (Year)	Comparison	No. of Shoulders Lost to Follow-up	No. of Outcomes^b	Sample Size	All Outcomes	Primary Outcomes
Surgical methods (n = 11)
Netto⁴⁴ (2012)	ABR vs OBR	8	1	42	0.0	0.0
Bottoni⁶ (2006)	ABR vs OBR	3	1	61	22.0	NA
Castagna¹⁰ (2009)	AC vs AC and posterior plication	NR	1	40	15.4	NA
Desai¹⁴ (2021)	ABR with vs without curettage	12	1	80	3.8	NA
Fabbriciani¹⁸ (2004)	ABR vs OBR	NR	1	60	15.8	NA
Mohtadi⁴² (2014)	ABR vs OBR	34	1	162	27.2	NA
Norlin⁴⁵ (1994)	BR with Mitek anchors vs with bone sutures	NR	1	40	0.0	NA
Salomonsson⁵³ (2009)	Putti-Platt procedure vs BR	4	3	66	7.9	7.9
Zarezade⁶³ (2014)	Bristow repair vs BR	3	1	37	3.6	NA
Robinson⁵⁰ (2008)	Arthroscopic lavage vs lavage and BR	5	1	83	30.2	NA
Yapp⁶² (2020)	Arthroscopic lavage vs lavage and BR	23	2	65	1.0	NA
Median (IQR)				61 (40)	7.9 (21)	3.4
Nonsurgical reduction interventions (n = 6)
Akcimen² (2020)	Original vs modified ER reduction	0	1	62	8.2	NA
Amar³ (2012)	Milch vs Stimson technique	0	1	60	19.6	NA
Li³⁶ (2023)	Modified Milch vs Hippocratic technique	0	10	126	30.4	29.6
Ghane²¹ (2014)	Scapular manipulation vs traction-countertraction	0	3	97	25.7	NA
Maity³⁸ (2012)	FARES vs Eachempati method	11	3	149	27.5	33.4
Sahin⁵² (2011)	Scapular manipulation vs Kocher technique	0	1	61	11.4	NA
Median (IQR)				79.5 (65)	22.6 (16)	31.5
Immobilization methods (n = 3)
Heidari²⁴ (2014)	IR vs ER immobilization	5	1	97	8.0	NA
Liavaag³⁷ (2009)	IR vs ER immobilization	4	1	51	2.8	2.8
Momenzadeh⁴³ (2015)	IR vs ER immobilization	5	1	20	1.2	NA
Median (IQR)				51	2.8	2.8
Surgical vs nonsurgical interventions (n = 3)
Bottoni⁷ (2002)	ABR vs immobilization	3	2	21	3.6	NA
Kirkley³⁵ (2005)	Surgery vs immobilization	9	4	40	2.4	NA
Pougès⁴⁷ (2021)	ABR vs immobilization	2	5	38	0.8	NA
Median (IQR)				38	2.4	NA
Other (n = 4)
Eshoj¹⁷ (2020)	Neuromuscular vs standard care exercise	4	6	56	0.6	1
Hurley²⁸ (2020)	Latarjet procedure with tranexamic acid vs with placebo	NR	3	100	14.7	15.6
Kim³³ (2003)	Immobilization vs rehabilitation	0	3	62	8.7	NA
Martinez-Rico³⁹ (2018)	Rehabilitation vs rehabilitation with telephone help	1	4	70	9.9	24
Median (IQR)				66 (26)	9.3 (7.6)	19.8 (9.4)
Overall median (IQR)				61 (43)	8.2 (17.2)	11.8 (24.9)

ABR, arthroscopic Bankart repair; AC, anterior capsulorrhaphy; BR, Bankart repair; CFI, continuous fragility index; ER, external rotation; FARES, fast, reliable, safe; IR, internal rotation; NA, not applicable; NR, not reported; OBR, open Bankart repair.

Number of significant continuous outcomes that were included in the calculation of the study’s CFI. This included primary, secondary, and unspecified outcomes.

The 27 included studies had a total of 1846 patients (82% male; mean age, 29.4 ± 7.0 years) before any loss to follow-up. The median follow-up time was 24 months (IQR, 30.05). We only included outcomes at final follow-up for analysis. The median sample size was 61 shoulders (IQR, 43). The median percentage of loss to follow-up per study was 6.1% (IQR, 125%), and the median number of shoulders lost to follow-up per study was 5.9 (IQR, 8). All included studies were of level 1 evidence, except 3 studies that were level 2 because of loss to follow-up exceeding 20% of the sample size.^35,43,62

There was a total of 63 individual outcomes included in the present analysis, of which 17 were primary outcomes, 33 were secondary outcomes, and the rest were unspecified outcomes. We grouped individual outcomes that occurred commonly (≥3 times) across the included studies: Western Ontario Shoulder Instability Index (13 outcomes), pain on a visual analog scale (6 outcomes), reduction time (6 outcomes), and Constant-Murley score (4 outcomes) (Table 2).

Table 2

CFI of Individual Outcomes Occurring ≥3 Times^a

	No. of Outcomes	Median	IQR	95% CI
Western Ontario Shoulder Instability Index	13	1.0	2.7	0-3
Visual analog scale	6	13.5	13.8	9.8-33.4
Reduction time	6	26.9	8.4	8.2-56
Constant-Murley score	4	9.5	13.8	0-15.8

CFI, continuous fragility index.

Continuous Fragility Index

The overall median CFI for included studies was 8.2 (IQR, 17.2; 95% CI, 3.6-15.4; range, 0-30.4) (Table 1). The median primary-only CFI was 11.8 (IQR, 24.9; 95% CI, 1-29.6), although this only involved 8 studies that had an eligible primary outcome. Regarding different study types, the median CFI was 7.9 (IQR, 21; 95% CI, 1-22) for 11 studies comparing surgical methods, 22.6 (IQR, 16; 95% CI, 8.2-30.4) for 6 studies comparing nonsurgical reduction interventions, 2.8 for 3 studies comparing immobilization methods, and 2.4 for 3 studies comparing surgical versus nonsurgical interventions (Table 1). The number of shoulders lost to follow-up was more than the study-specific CFI in 9 studies.^{14,17,35,37,42-44,47,62} Of these, the number of shoulders lost to follow-up was, on average, 6.8 (95% CI, 2.0-11.6) more than the study-specific CFI. A histogram of the frequency of study-specific CFI values is plotted in Figure 3.

Figure 3.

Histogram of study-specific continuous fragility index (CFI) values.

The median CFI was 26.9 (IQR, 8.4; 95% CI, 8.2-56) for reduction time outcomes, 13.5 (IQR, 13.8; 95% CI, 9.8-33.4) for visual analog scale outcomes, 9.5 (IQR, 13.8; 95% CI, 0-15.8) for Constant-Murley score outcomes, and 1.0 (IQR, 2.7; 95% CI, 0-3) for Western Ontario Shoulder Instability Index outcomes (Table 2).

We identified 26 objective outcomes, with a median CFI of 14.3 (IQR, 37), and 37 patient-reported outcomes, with a median CFI of 3.6 (IQR, 30). The wide interquartile ranges in CFI values across both groups led to a nonsignificant difference between these outcome types (Mann-Whitney U test; P = .116).

Collectively, the sample size, journal impact factor, percentage lost to follow-up, follow-up duration in months, and publication date statistically significantly predicted the study-specific CFI in multivariable linear regression (F_1,18 = 17.9; P < .001; R² = 0.5). Individually, the trial’s sample size was the only statistically significant predictor in the model (r = 0.23 [95% CI, 0.13-0.33]; t = 4.23; P < .001).

Discussion

The important finding of this study is that the median CFI for RCTs evaluating ASI was 8.2 (IQR, 17.2; 95% CI, 3.6-15.4; range, 0-30.4). In other words, the median number of patients that must be moved from one treatment arm to the other to change a study’s outcome was 8.2. ASI RCTs evaluated here were, on average, comparable with, if not more robust than, available measures of the robustness of the sports orthopaedics literature, which reports a median CFI of 7 (IQR, 11.1).^8,30 While these summary statistics provide a useful gauge of the robustness of the ASI literature as a whole, the wide ranges and confidence intervals of median CFI values must be noted, indicating that not all study outcomes are equally robust.

Loss to follow-up is an important source of risk in RCTs that can seriously harm a trial’s validity.¹⁵ The CFI can demonstrate the importance of loss to follow-up by highlighting its ability to affect the statistical significance of outcomes that are fragile enough. In the case of the ASI literature, this study found that 38.6% of included outcomes had a CFI lower than their trial’s loss to follow-up, with the loss to follow-up being, on average, 6.8 (95% CI, 2.0-11.6) shoulders more than the study’s CFI. Therefore, if data from patients lost to follow-up were to be included, there is a risk that the statistical significance of these outcomes may reverse. Compared with our rate of 38.6%, there are 2 previous orthopaedic CFI reviews that report this percentage as 7% and 27.8%, respectively,^22,23 indicating that this may be a larger problem for the ASI literature. Furthermore, lost data are more likely to be different than available data in the same group of patients,¹⁵ making this effect more concerning. Overall, this highlights the importance of more robust results that are more resistant to this effect. As well, considering that 20% of orthopaedic trials do not report on loss to follow-up⁵⁵ and that about 15% of studies included in this review did not, it is important that reporting loss to follow-up improves in the future as well as critical focus by authors to maximize patients’ follow-up.

One way to obtain higher CFI values is to increase sample sizes. The CFI algorithm demonstrates a tightly linear relationship between the sample size and CFI if all other variables are held constant.⁸ We also found that sample size was a statistically significant predictor of the study-specific CFI in multivariable regression. Similarly, a statistically significant association between the sample size and study-specific CFI (P = .008) was found in Caldwell et al.’s application of the CFI to a previous sports orthopaedics systematic review with a beta coefficient of 0.332.^8,30 Sample size may also be able to explain differences in the CFI between study types. For example, RCTs comparing surgical arms had a median sample size of 61 (IQR, 40) and a median CFI of 7.9 (IQR, 21), whereas nonsurgical RCTs had a median sample size of 79.5 (IQR, 65) and a median CFI of 22.6 (IQR, 16). Still, it is particularly more difficult to recruit a greater number of patients for surgical RCTs,^20,29 which comprised approximately 40% of the included studies in this review. One strategy for increasing the sample size in surgical trials is multicenter collaboration.¹² However, only 1 of the 11 surgical studies included in the present review utilized >1 center in its trial.⁶³ Increasing the number of multicenter surgical trials for the management of ASI is an aim for potential future improvement.

In studies using statistical tests less conservative than the Welch t test (used in the CFI calculation), a CFI of zero may result. This indicates that simply using the Welch t test changed the result to nonsignificant without manipulating the data. In fact, 8 of the 63 included outcomes (12.7%) had a CFI of zero, despite being reported as statistically significant. Ensuring that significance does not hinge on using a less conservative test is another way to improve the robustness of available outcomes reported as statistically significant in the ASI literature.

An FI of 3 has been shown to represent being in the top 25th percentile of statistical robustness across previous trials.⁵⁶ No similar threshold value has been derived for the CFI to date. While the CFI and FI both utilize an iterative process to determine the minimum data modifications required to change statistical significance, their different methodologies render a comparison invalid.⁸ Manipulating dichotomous data has a larger effect on dichotomous tests such as the chi-square test compared with manipulating continuous data sets on tests such as the t test.⁸ This leads to generally larger magnitudes of the CFI compared with the FI. For example, a sports orthopaedics systematic review by Khan et al³⁰ reported a mean FI of 2, while Caldwell et al’s⁸ analysis of the same studies included in that review produced a significantly greater mean CFI of 9 (P < .0001). The present study’s findings further support the importance of complementing the FI with the CFI when possible for an accurate assessment of a study’s statistical robustness.

There have been 5 published studies using the CFI at the time of writing this study, with reported median CFIs of 3, 5, 6, 7 and 9.^{8,22,23,27,61} Therefore, the present review’s median CFI of 8.2 ranks at the higher end of robustness compared with other specialty areas. As an example, Caldwell et al⁸ found a median CFI of 7 for sports orthopaedics RCTs. By this measure, the ASI RCTs evaluated here are, on average, comparable with, if not more robust than, available measures of the robustness of the sports orthopaedics literature. Still a novel metric, studies assessing the CFI in both the orthopaedic and the nonorthopaedic literature will aid researchers in designing robust RCTs that clinicians can have full certainty in. We recommend including the FI, CFI, and loss to follow-up in the abstracts of all future RCTs to allow clinicians to better interpret trial outcomes and evaluate the effect of loss to follow-up on the results.

This study is not without its limitations. First, this study did not have access to outcome data for the included trials. Therefore, the mean, standard deviation, and sample size of each outcome were used to calculate the CFI assuming a normally distributed data set. This may have led to inaccuracies for skewed data sets. The effect was minimized by using multiple iterations (n = 5) when calculating the CFI. Second, not all studies presented the standard deviation, which was required for calculating the CFI. As such, validated conversion tools were used for obtaining the standard deviation required for the CFI calculation.^26,57,58 Third, this study only considered significant outcomes for inclusion and analysis. However, statistically nonsignificant outcomes also constitute important evidence that can guide clinical practice. Determining their fragility can help identify interventions initially ruled as nonsignificant but fragile, guiding further trials for more definitive evidence. To this end, a reverse FI was first used by Khan et al³¹ to communicate the robustness of a conclusion of nonsignificance, which involves manipulating events until significance is achieved. A reverse CFI for continuous nonsignificant outcomes can also be calculated by modifying the methodology from Caldwell et al,⁸ as described and used by Gupta et al.²³ Future reviews may focus on this area in the ASI literature.

Conclusion

Supplemental Material

sj-pdf-1-ajs-10.1177_03635465231202522 – Supplemental material for The Continuous Fragility Index of Statistically Significant Findings in Randomized Controlled Trials That Compare Interventions for Anterior Shoulder Instability

Supplemental material, sj-pdf-1-ajs-10.1177_03635465231202522 for The Continuous Fragility Index of Statistically Significant Findings in Randomized Controlled Trials That Compare Interventions for Anterior Shoulder Instability by Mohammed Al-Asadi, Michelle Sherren, Hassaan Abdel Khalik, Timothy Leroux, Olufemi R. Ayeni, Kim Madden and Moin Khan in The American Journal of Sports Medicine

Footnotes

Submitted March 10, 2023; accepted July 31, 2023.

One or more of the authors has declared the following potential conflict of interest or source of funding: O.R.A. has received speaking fees from Conmed and is a tier 2 Canada Research Chair in Joint Preservation Surgery. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.

An online CME course associated with this article is available for 1 AMA PRA Category 1 Credit™ at https://education.sportsmed.org/Public/Catalog/Home.aspx?CourseSearch=1&Criteria=9&Option=25. In accordance with the standards of the Accreditation Council for Continuing Medical Education (ACCME), it is the policy of The American Orthopaedic Society for Sports Medicine that authors, editors, and planners disclose to the learners all financial relationships during the past 12 months with any commercial interest (A ‘commercial interest’ is any entity producing, marketing, re-selling, or distributing health care goods or services consumed by, or used on, patients). Any and all disclosures are provided in the online journal CME area which is provided to all participants before they actually take the CME activity. In accordance with AOSSM policy, authors, editors, and planners’ participation in this educational activity will be predicated upon timely submission and review of AOSSM disclosure. Noncompliance will result in an author/editor or planner to be stricken from participating in this CME activity.

ORCID iDs

Mohammed Al-Asadi

Moin Khan

∥

References 6, 7, 10, 17, 39, 45, 50, 53, .

¶

References 2, 3, 6, 7, 10, 14, 17, 18, 21, 24, 28, 33, 35 -39, 42 -45, 47,50, 52, 53, 62, .

#

References 6, 10, 14, 18, 42, 44, 45, 50, 53, 62, .

References

Abrams

Akbarnia

. Shoulder dislocations overview. In: StatPearls. StatPearls Publishing; 2022. Accessed January 12, 2023. http://www.ncbi.nlm.nih.gov/books/NBK459125/

Akcimen

Bedel

. Comparison between new modified external rotation method and external rotation method for reduction of ASD. Am J Emerg Med. 2020;38(5):874-878.

Amar

Maman

Khashan

, et al. Milch versus Stimson technique for nonsedated reduction of anterior shoulder dislocation: a prospective randomized trial and analysis of factors affecting success. J Shoulder Elbow Surg. 2012;21(11):1443-1449.

Amrhein

Greenland

McShane

. Scientists rise up against statistical significance. Nature. 2019;567(7748):305-307.

Belk

Wharton

Houck

, et al. Shoulder stabilization versus immobilization for first-time anterior shoulder dislocation: a systematic review and meta-analysis of level 1 randomized controlled trials. Am J Sports Med. 2023;51(6):1634-1643.

Bottoni

Smith

Berkowitz

Towle

Moore

. Arthroscopic versus open shoulder stabilization for recurrent anterior instability: a prospective randomized clinical trial. Am J Sports Med. 2006;34(11):1730-1737.

Bottoni

Wilckens

DeBerardino

, et al. A prospective, randomized evaluation of arthroscopic stabilization versus nonoperative treatment in patients with acute, traumatic, first-time shoulder dislocations. Am J Sports Med. 2002;30(4):576-580.

Caldwell

J-ME

Youssefzadeh

Limpisvasti

. A method for calculating the fragility index of continuous outcomes. J Clin Epidemiol. 2021;136:20-25.

Caldwell

J-ME

Youssefzadeh

Limpisvasti

. Continuous fragility index calculator. Accessed December 24, 2022. https://jmcaldwell.shinyapps.io/CFIApp/

10.

Castagna

Borroni

Delle Rose

, et al. Effects of posterior-inferior capsular plications in range of motion in arthroscopic anterior Bankart repair: a prospective randomized clinical study. Knee Surg Sports Traumatol Arthrosc. 2009;17(2):188-194.

11.

Chang

L-R

Anand

Varacallo

. Anatomy, shoulder and upper limb, glenohumeral joint. In: StatPearls. StatPearls Publishing; 2022. Accessed January 12, 2023. http://www.ncbi.nlm.nih.gov/books/NBK537018/

12.

Chung

Song

; WRIST Study Group. A guide to organizing a multicenter clinical trial. Plast Reconstr Surg. 2010;126(2):515-523.

13.

Davey

Hurley

Doyle

, et al. The fragility index of statistically significant findings from randomized controlled trials comparing the management strategies of anterior shoulder instability. Am J Sports Med. 2023;51(8):2186-2192.

14.

Desai

Singh

Mata

. Arthroscopic Bankart repair with and without curettage of the glenoid edge: a prospective, randomized, controlled study. Arthroscopy. 2021;37(3):837-842.

15.

Dettori

. Loss to follow-up. Evid Based Spine Care J. 2011;2(1):7-10.

16.

Ehlers

Curley

Fackler

Minhas

Chang

. The statistical fragility of hamstring versus patellar tendon autografts for anterior cruciate ligament reconstruction: a systematic review of comparative studies. Am J Sports Med. 2021;49(10):2827-2833.

17.

Eshoj

Rasmussen

Frich

, et al. Neuromuscular exercises improve shoulder function more than standard care exercises in patients with a traumatic anterior shoulder dislocation: a randomized controlled trial. Orthop J Sports Med. 2020;8(1):232596711989610.

18.

Fabbriciani

Milano

Demontis

, et al. Arthroscopic versus open treatment of Bankart lesion of the shoulder: a prospective randomized study. Arthroscopy. 2004;20(5):456-462.

19.

Feinstein

. The unit fragility index: an additional appraisal of “statistical significance” for a contrast of two proportions. J Clin Epidemiol. 1990;43(2):201-209.

20.

Ferreira

. Surgical randomized controlled trials: reflection of the difficulties. Acta Cir Bras. 2004;19(suppl 1):2-3.

21.

Ghane

M-R

Hoseini

S-H

Javadzadeh

H-R

Mahmoudi

Saburi

. Comparison between traction-countertraction and modified scapular manipulation for reduction of shoulder dislocation. Chin J Traumatol. 2014;17(2):93-98.

22.

Gupta

Movsik

al Farii

. Statistical fragility of ketamine infusion during scoliosis surgery to reduce opioid tolerance and postoperative pain. World Neurosurg. 2022;164:135-142.

23.

Gupta

Ortiz-Babilonia

, et al. The statistical fragility of platelet-rich plasma as treatment for plantar fasciitis: a systematic review and simulated fragility analysis. Foot Ankle Orthop. 2022;7(4):247301142211440.

24.

Heidari

Asadollahi

Vafaee

, et al. Immobilization in external rotation combined with abduction reduces the risk of recurrence after primary anterior shoulder dislocation. J Shoulder Elbow Surg. 2014;23(6):759-766.

25.

Herndon

McCormick

Gazgalis

, et al. Fragility index as a measure of randomized clinical trial quality in adult reconstruction: a systematic review. Arthroplast Today. 2021;11:239-251.

26.

Higgins

Thomas

Chandler

, et al. Chapter 6: choosing effect measures and computing estimates of effect. In: Higgins

Thomas

(eds). Cochrane Handbook for Systematic Reviews of Interventions. Wiley; 2008. Accessed December 5, 2022. www.training.cochrane.org/handbook

27.

. The fragility index for assessing the robustness of the statistically significant results of experimental clinical studies. J Gen Intern Med. 2022;37(1):206-211.

28.

Hurley

Lim Fat

Pauzenberger

Mullett

. Tranexamic acid for the Latarjet procedure: a randomized controlled trial. J Shoulder Elbow Surg. 2020;29(5):882-885.

29.

Indrayan

Mishra

. The importance of small samples in medical research. J Postgrad Med. 2021;67(4):219-223.

30.

Khan

Evaniew

Gichuru

, et al. The fragility of statistically significant findings from randomized trials in sports surgery: a systematic survey. Am J Sports Med. 2017;45(9):2164-2170.

31.

Khan

Fonarow

Friede

, et al. Application of the reverse fragility index to statistically nonsignificant randomized clinical trial results. JAMA Netw Open. 2020;3(8):e2012469.

32.

Khan

Ochani

Shaikh

, et al. Fragility index in cardiovascular randomized controlled trials. Circ Cardiovasc Qual Outcomes. 2019;12(12):e005755.

33.

Kim

S-H

K-I

Jung

M-W

, et al. Accelerated rehabilitation after arthroscopic Bankart repair for selected cases: a prospective randomized clinical study. Arthroscopy. 2003;19(7):722-731.

34.

King

Cowling

. Management of first time shoulder dislocation. J Arthrosc Joint Surg. 2018;5(2):86-89.

35.

Kirkley

Werstine

Ratjek

Griffin

. Prospective randomized clinical trial comparing the effectiveness of immediate arthroscopic stabilization versus immobilization and rehabilitation in first traumatic anterior dislocations of the shoulder: long-term evaluation. Arthroscopy. 2005;21(1):55-63.

36.

Wen

Wang

Dong

. Effect of using the modified Milch technique on quality of life in patients with anterior dislocation of the shoulder joint. Altern Ther Health Med. 2023;29(1):144-149.

37.

Liavaag

Stiris

Lindland

, et al. Do Bankart lesions heal better in shoulders immobilized in external rotation? A randomized single-blind study of 55 patients examined with MRI. Acta Orthop. 2009;80(5):579-584.

38.

Maity

Roy

Mondal

. A prospective randomised clinical trial comparing FARES method with the Eachempati external rotation method for reduction of acute anterior dislocation of shoulder. Injury. 2012;43(7):1066-1070.

39.

Martinez-Rico

Lizaur-Utrilla

Sebastia-Forcada

Vizcaya-Moreno

de Juan-Herrero

. The impact of a phone assistance nursing program on adherence to home exercises and final outcomes in patients who underwent shoulder instability surgery: a randomized controlled study. Orthop Nurs. 2018;37(6):372-378.

40.

McShane

Gal

. Statistical significance and the dichotomization of evidence. J Am Stat Assoc. 2017;112(519):885-895.

41.

McShane

Gal

Gelman

Robert

Tackett

. Abandon statistical significance. Am Stat. 2019;73(suppl 1):235-245.

42.

Mohtadi

NGH

Chan

Hollinshead

, et al. A randomized clinical trial comparing open and arthroscopic stabilization for recurrent traumatic anterior shoulder instability: two-year follow-up with disease-specific quality-of-life outcomes. J Bone Joint Surg Am. 2014;96(5):353-360.

43.

Momenzadeh

Pourmokhtari

Sefidbakht

Vosoughi

. Does the position of shoulder immobilization after reduced anterior glenohumeral dislocation affect coaptation of a Bankart lesion? An arthrographic comparison. J Orthop Traumatol. 2015;16(4):317-321.

44.

Netto

Tamaoki

MJS

Lenza

, et al. Treatment of Bankart lesions in traumatic anterior instability of the shoulder: a randomized controlled trial comparing arthroscopy and open techniques. Arthroscopy. 2012;28(7):900-908.

45.

Norlin

. Use of Mitek anchoring for Bankart repair: a comparative, randomized, prospective study with traditional bone sutures. J Shoulder Elbow Surg. 1994;3(6):381-385.

46.

Parisien

Ehlers

Cusano

, et al. The statistical fragility of platelet-rich plasma in rotator cuff surgery: a systematic review and meta-analysis. Am J Sports Med. 2021;49(12):3437-3442.

47.

Pougès

Hardy

Vervoort

, et al. Arthroscopic Bankart repair versus immobilization for first episode of anterior shoulder dislocation before the age of 25: a randomized controlled trial. Am J Sports Med. 2021;49(5):1166-1174.

48.

Rees

Shah

Edwards

, et al. Treatment of first-time traumatic anterior shoulder dislocation: the UK TASH-D cohort study. Health Technol Assess. 2019;23(18):1-104.

49.

Ridgeon

Young

Bellomo

, et al. The fragility index in multicenter randomized controlled critical care trials. Crit Care Med. 2016;44(7):1278-1284.

50.

Robinson

Jenkins

White

Ker

Will

. Primary arthroscopic stabilization for a first-time anterior dislocation of the shoulder: a randomized, double-blind trial. J Bone Joint Surg Am. 2008;90(4):708-721.

51.

Ruzbarsky

Rauck

Manzi

, et al. The fragility of findings of randomized controlled trials in shoulder and elbow surgery. J Shoulder Elbow Surg. 2019;28(12):2409-2417.

52.

Sahin

Oztürk

Ozkan

Atıcı

Ozkaya

. A comparison of the scapular manipulation and Kocher’s technique for acute anterior dislocation of the shoulder. Eklem Hastalik Cerrahisi. 2011;22(1):28-32.

53.

Salomonsson

Abbaszadegan

Revay

Lillkrona

. The Bankart repair versus the Putti-Platt procedure: a randomized study with WOSI score at 10-year follow-up in 62 patients. Acta Orthop. 2009;80(3):351-356.

54.

Schliemann

Minkus

Seybold

Scheibel

. Conservative management of first-time traumatic anterior shoulder dislocation. Obere Extrem. 2021;16(1):2-7.

55.

Somerson

Bartush

Shroff

Bhandari

Zelle

. Loss to follow-up in orthopaedic clinical trials: a systematic review. Int Orthop. 2016;40(11):2213-2219.

56.

Walsh

Srinathan

McAuley

, et al. The statistical significance of randomized controlled trial results is frequently fragile: a case for a fragility index. J Clin Epidemiol. 2014;67(6):622-628.

57.

Walter

Yao

. Effect sizes can be calculated for studies reporting ranges for outcome variables in systematic reviews. J Clin Epidemiol. 2007;60(8):849-852.

58.

Wan

Wang

Liu

Tong

. Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Med Res Methodol. 2014;14(1):135.

59.

Wasserstein

Lazar

. The ASA statement on p-values: context, process, and purpose. Am Stat. 2016;70(2):129-133.

60.

Wright

Swiontkowski

Heckman

. Introducing levels of evidence to the journal. J Bone Joint Surg Am. 2003;85(1):1-3.

61.

Ortiz-Babilonia

Gupta

, et al. The statistical fragility of platelet-rich plasma as treatment for chronic noninsertional Achilles tendinopathy: a systematic review and meta-analysis. Foot Ankle Orthop. 2022;7(3):247301142211197.

62.

Yapp

Nicholson

Robinson

. Primary arthroscopic stabilization for a first-time anterior dislocation of the shoulder: long-term follow-up of a randomized, double-blinded trial. J Bone Joint Surg Am. 2020;102(6):460-467.

63.

Zarezade

Rozati

Banadaki

Dehghani

Shekarchizade

. Comparison of Bristow procedure and Bankart arthroscopic method as the treatment of recurrent shoulder instability. Adv Biomed Res. 2014;3(1):256.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.06 MB