Abstract
Background:
Orthopaedic research may involve multiple observations from the same patient because of bilateral joint involvement, multiple disease sites, or recurrent disease episodes. These situations violate statistical independence and need to be accounted for via appropriate statistical techniques. Failing to account for nonindependence may lead to biased and overly precise effect estimates.
Purpose:
To determine the degree to which orthopaedic sports medicine studies analyze dependent observations and the proportion of these failing to account for nonindependence.
Study Design:
Cross-sectional study.
Methods:
Clinical studies published in The American Journal of Sports Medicine from 2012 to 2017 were reviewed. Studies reporting nonindependent observations because of multiple extremity involvement or multiple disease episodes were identified. Methods to account for nonindependence were recorded. Studies violating the assumption of independence were identified and stratified by study design, level of evidence, body part involved, and inclusion of a statistician coauthor. Univariate logistic regression was used to determine whether these factors were associated with violations of statistical independence.
Results:
After screening 1016 articles, 886 clinical studies were reviewed. A total of 135 (15%) studies analyzed dependent observations, and 111 (82%) of these failed to account for nonindependence. Relative to the knee, studies of the hip (odds ratio [OR], 0.21; P = .02) and the thigh or leg (OR, 0.03; P = .004) were less likely to violate statistical independence. Study design (P = .03) was also associated with violations of statistical independence. Among studies that analyzed dependent observations, the median proportion of dependent observations relative to the total number of observations in each study was 0.07 (interquartile range, 0.04-0.12).
Conclusion:
The analysis of dependent observations is common in the orthopaedic sports literature, but most studies do not adjust for nonindependence in these situations. Investigators should be aware of incorrect inferences arising from nonindependence and how to statistically adjust for dependent data.
In many orthopaedic studies, the extremity is the unit of analysis rather than the patient. 20 As a result, orthopaedic investigations often include multiple observations from the same patient because of bilateral joint or extremity involvement, multiple lesions within a joint or extremity, or recurrent disease episodes. 16,20 This is particularly common in the sports medicine literature because of the high incidence of bilateral 19 or recurrent 18 injuries.
Most common analytical methods used in clinical research assume that all observations within a data set are independent. 10,20 However, in samples with multiple nonindependent observations from the same patient, the analysis must account for intrapatient correlations between exposures and outcomes. 20 Analyzing nonindependent data as if they were independent can lead to biased or overly precise effect estimates. 20 For example, if a data set containing information on patients undergoing anterior cruciate ligament reconstruction has a subset of patients who underwent bilateral procedures, and the outcomes of both knees for each bilateral patient are similar, then an unadjusted analysis will effectively “double count” those knees. In other words, the results of an unadjusted analysis will underestimate the standard error, yielding an overly precise effect estimate. An adjusted analysis, on the other hand, will correctly account for correlations between the bilateral knees and provide an accurate effect estimate.
Statistical methods such as generalized estimating equations (GEEs) 8 and mixed- or random-effects models 17 can be utilized to account for nonindependence. Although these methods are applicable to any specialty that regularly analyzes multiple observations, 6 the frequency with which these are used in the orthopaedic sports literature is not well known. Two studies in 2006 and 2010 by Bryant et al 5 and Park et al, 14 respectively, surveyed the inclusion of bilateral injuries in the orthopaedic literature. These studies found that a significant proportion of orthopaedic clinical studies reported nonindependent observations and that few appropriately adjusted for nonindependence. These studies evaluated small samples of articles, were conducted more than 5 years ago, and were not focused on the sports medicine literature. Importantly, these studies limited their focus to bilaterality, even though there are other relevant causes of dependent observations. For example, multiple lesions from the same joint or extremity (eg, multiple discrete chondral defects in a knee) and multiple injury episodes (eg, recurrent concussions) are common sources of dependent observations in the sports medicine literature. Last, these studies did not analyze study factors that may be associated with inappropriate analyses.
The objective of this study was to survey recent orthopaedic sports medicine articles from a single high-impact journal to identify a sample of studies that analyzed multiple observations from the same patient because of multiple extremity involvement, multiple lesions within the same extremity, or multiple injury episodes. We then sought to determine if the authors attempted to adjust for nonindependence between those observations and, if so, the statistical methodology used. We hypothesized that studies reporting data involving nonindependent observations would be common, given that many orthopaedic sports conditions (eg, anterior cruciate ligament injuries, femoroacetabular impingement, and concussions) are known to occur bilaterally or recurrently. 11,18,19 We further hypothesized that statistical adjustments for nonindependence would be underutilized and that the degree to which studies inappropriately analyzed dependent observations would be associated with the level of evidence and body part under analysis.
Methods
Search Strategy
The search strategy was designed with the assistance of a research librarian. Articles published in The American Journal of Sports Medicine from March 2012 to March 2017 were analyzed. Articles of interest were clinical studies, defined as those that presented original research on a disease or intervention and reported patient data with any outcome measure. These included randomized controlled trials, cohort studies, case-control studies, cross-sectional studies, and case series. Meta-analyses, systematic reviews, controlled laboratory studies, descriptive laboratory studies, descriptive epidemiological studies, economic studies, case reports, editorials, corrigenda, and articles without abstracts were excluded from the initial search algorithm using publication type and title or abstract filters.
All retrieved studies initially underwent an abstract review by 2 independent reviewers (D.G.L., T.T.) with the assistance of Rayyan systematic review software (Qatar Computing Research Institute). 13 Studies were excluded during the abstract review if they (1) met any of the above exclusion criteria but were not filtered by the initial search algorithm; (2) involved wrong population of interest (eg, surveys of medical providers); (3) were reliability or validation studies assessing observer agreement, test-retest reliability, construct validity, minimal clinically important difference, or measures of test performance (eg, sensitivity, specificity, negative predictive value, positive predictive value); (4) were epidemiological studies assessing the incidence or prevalence of a given condition or procedure in a representative population; or (5) were basic science studies conducting biomechanical or histological analyses without patient outcomes.
Study Classification and Data Extraction
Studies that passed the initial abstract review subsequently underwent a full-text review by 2 independent reviewers (D.G.L., T.T.). Studies were first categorized by how they reported the numbers of observations and patients in the abstract as either (1) only patients reported, (2) only observations reported, (3) neither patients nor observations reported, or (4) patients and observations both reported. The latter category was for studies that made an explicit distinction between the number of observations and of patients (eg, “90 hips in 90 patients”) or if studies excluded multiple observations as noted in the abstract.
Extracted data for all studies included the study design, level of evidence, 7 specific body part involved (eg, hip, knee, shoulder, foot/ankle), and inclusion of a coauthor with statistical expertise. Studies involving multiple body parts were categorized as “multiple,” while studies assessing the axial skeleton or hand were categorized as “other” because of the low number of studies. Statistical coauthorship was a binary (yes/no) variable determined by having either (1) an author affiliation with a department of biostatistics or epidemiology or (2) an author with an advanced degree in biostatistics or epidemiology (eg, MPH, MS, or PhD). These criteria are consistent with other studies in the orthopaedic literature that have assessed statistical expertise. 4,12
Each study was then scrutinized to ascertain whether the number of observations analyzed and number of patients included were equal. Exposures and outcomes were first evaluated to determine whether any observation-specific data were reported or analyzed. Studies that only reported and analyzed data on a per-patient basis were determined to have met criteria for independence and did not undergo a further review. Articles that reported the same number of observations and patients were classified as equal and were not reviewed further. Articles that only referred to the number of patients and did not suggest the inclusion of multiple observations were assumed to be equal for the purposes of this study. Articles that reported different numbers of observations and patients, or those that only reported the number of observations without reporting the number of patients, were classified as potentially unequal and underwent additional data extraction.
Extracted data included the number of observations analyzed, the number of patients included (if reported), and the type of analysis performed. The type of analysis was defined hierarchically as either descriptive (eg, proportions, means, medians), hypothesis testing (eg, t test, rank-sum test, analysis of variance), or regression modeling (eg, linear regression, logistic regression, Cox proportional hazards regression). Studies that only conducted descriptive analyses were not analyzed further. If a study conducted hypothesis testing or regression modeling on the per-observation level, it was subsequently assessed for the use of statistical methods to account for nonindependence. These methods were classified as either (1) nonindependent observations considered to be independent, including those that did not specifically define how the observations were analyzed; (2) a single observation (mean or 1 side) per patient chosen for analysis; (3) multiple observations analyzed separately from those with single observations; or (4) multiple observations analyzed via statistical methodologies accounting for nonindependence (eg, GEEs, mixed- or random-effects models, or other strategies).
Studies that met the following 4 criteria were considered to violate the statistical assumption of independence: (1) included multiple observations from the same patient, (2) conducted inferential hypothesis testing and/or regression modeling, (3) analyzed data on a per-observation basis, and (4) analyzed dependent observations as independent observations. Of note, articles that compared affected and unaffected limbs in the same patient were not included because paired analysis was not the topic of interest in this study. Similarly, articles that analyzed repeated observations from the same patient over time (eg, observations of the same patient at 1, 2, and 6 months postoperatively) were not included because repeated-measures analysis was not the topic of interest in this study.
Statistical Analysis
Univariate logistic regression was used to determine whether study design, level of evidence, body part under study, and inclusion of a coauthor with statistical expertise were associated with violations of statistical independence. The total sample for logistic regression analysis consisted of all studies that used dependent data and conducted statistical analyses beyond descriptive statistics. Cases were considered those studies that did not adjust for data dependency in their statistical analysis. Studies without dependent data were not included in the logistic regression analysis. A 2-sided type I error rate of 0.05 was used to indicate statistical significance. All calculations were performed using STATA 15.0 (StataCorp).
Results
The initial search strategy yielded 1016 articles. Of these, 130 were excluded after an abstract review. Thus, a total of 886 clinical studies underwent a full-text review (Figure 1). Of these, 751 studies were subsequently excluded because they either did not contain dependent observations (n = 717), were ambiguous about whether dependent observations were included (n = 24), or included dependent observations but did not conduct inferential analyses beyond descriptive statistics (n = 10). The 24 ambiguous studies only reported the number of observations (eg, 100 ankles) without reporting the corresponding number of patients from which the observations came, thereby precluding a determination of whether dependent observations were present in the data set.
Flow diagram of literature search.
Ultimately, 135 studies (15%) were found to contain dependent observations and to have conducted statistical analyses beyond descriptive statistics on the data (Table 1). These were considered to constitute all studies analyzing dependent observations over the study period. Among all 135 studies that analyzed dependent observations, the median proportion of dependent observations relative to the total number of observations in each study was 0.07 (interquartile range, 0.04-0.12). In other words, in our sample of studies that conducted analyses on dependent data, the number of observations exceeded the number of patients in each study by a median of 7%, with half of the studies falling within a range of 4% to 12%.
Characteristics of Included Studies (N = 135)
Of the 135 studies analyzing dependent observations, 111 (82%) failed to account for data dependency by treating their data as if they were independent. Among these 111 studies, 78 (70%) performed hypothesis testing without regression modeling, while the remaining 33 (30%) performed hypothesis testing with regression modeling. None of the 111 studies explicitly mentioned in their methods or discussion whether they considered intrapatient correlation or attempted to account for dependence. Six additional studies (4%) reported dependent observations but chose 1 measurement from each patient (eg, mean or 1 side) for statistical analyses, thereby avoiding statistical nonindependence at the loss of statistical power. Three of these only analyzed unilateral patients, 1 compared unilateral patients to bilateral patients, 1 picked a random side among bilateral patients, and 1 took a mean value of bilateral measurements. Eighteen studies (14%) utilized GEEs or random- or mixed-effects models to correctly account for data dependence. For example, in a cross-sectional study evaluating cam-type deformities in young male soccer players versus controls, Agricola et al 2 utilized a GEE to calculate differences in range of motion between cases and controls that appropriately accounted for correlations between bilateral hips. Similarly, in a retrospective cohort study evaluating the effect of nonsteroidal anti-inflammatory drugs on heterotopic ossification after hip arthroscopic surgery, Beckmann et al 3 utilized random-effects regression models to account for the nonindependence of patients who underwent multiple surgeries. Last, in a retrospective cohort study evaluating whether the magnetic resonance imaging grade predicts return-to-running time in athletes with femoral neck stress fractures, Ramey et al 15 used a linear mixed model to control for patients who had repeat fractures.
Results of the univariate logistic regression analysis are summarized in Table 2. Relative to studies of the knee, studies of the hip (odds ratio [OR], 0.21 [95% CI, 0.06-0.76]; P = .02) and the thigh or leg (OR, 0.03 [95% CI, 0.002-0.32]; P = .004) were less likely to be analyzed incorrectly. Although study design was overall found to be associated with violations of statistical independence (P = .03), no differences were noted when each individual study design was compared relative to randomized controlled trials. Lower levels of evidence were not associated with an increased risk of unadjusted analyses (P = .51). Similarly, having a coauthor with statistical knowledge was not associated with a decreased risk of unadjusted analyses (P = .36).
Univariate Logistic Regression Analysis of Factors Associated With Potential Violations of Statistical Independence
Excluding the level of evidence variable because of collinearity with the study design (variance inflation factor, 4.4), a multivariable logistic regression model demonstrated that study design remained a predictor of violations (P = .02) and that studies of the thigh or leg (P = .004) remained more likely to be analyzed correctly relative to studies of the knee. Because all case-control studies and studies of the foot and ankle were analyzed incorrectly and therefore dropped from the multivariable analysis (n = 23), the multivariable model had a limited sample of 112 studies.
Discussion
In this literature review of 886 clinical studies published in The American Journal of Sports Medicine, we found that 111 of 135 studies (82%) analyzing dependent observations failed to account for intrapatient correlation. None of the 111 studies that failed to account for nonindependence explicitly stated why they did not adjust for intrapatient correlation. It may be assumed that the authors either (1) were not aware of the issue, as it may not be well understood even among biostatisticians if they are not accustomed to regularly analyzing dependent data, or (2) made the decision to not account for dependence; however, in the latter case, it would still be appropriate to mention their rationale regarding the analysis of dependent data. Studies of the knee and hip were the most common among our sample. Interestingly, studies of the knee were more likely to analyze dependent data incorrectly (93% vs 73%, respectively). This may be because hips are more intuitively considered to affect each other, whereas knees are more often considered independent.
Our findings are consistent with prior studies of the orthopaedic literature. A 2006 study by Bryant et al 5 found that 60 of 76 studies (79%) in 7 high-impact orthopaedic journals failed to account for nonindependent observations. A subsequent study in 2010 by Park et al 14 found that 125 of 151 (83%) studies in the Journal of Bone & Joint Surgery possibly violated statistical independence because of the failure to adjust for bilaterality. Thus, our finding that 82% of studies failed to adjust for correlated outcomes is closely aligned with these prior findings and suggests that correlated data continue to be an issue for orthopaedic researchers. In context with these 2 prior studies, our study’s strengths are (1) its broader definition of nonindependence to include other common causes of intrapatient correlation, such as multiple injury sites and multiple injury events in addition to bilaterality, and (2) its large sample size (886 studies underwent a full-text review). Our study is also unique from the 2 prior studies in that it focused on the sports medicine literature and used a logistic regression model to identify study factors (eg, body part under study and study design) associated with violations of statistical independence.
Concerns about statistical nonindependence are not restricted to orthopaedics. Any study assessing observation-specific data (eg, individual limbs, individual organ function, individual disease events) rather than patient-specific data (eg, quality of life, functional scores) should account for intrapatient correlation. Studies in the neuroscience literature 1 and ophthalmological literature 9 have found that the majority of published studies in their respective fields do not account for correlated data. Given that problems with correlated data extend across the medical literature, this is an issue that both researchers and journal editors and reviewers should be cognizant of when producing or reviewing orthopaedic studies.
To decide the appropriate statistical method for a given analysis, investigators need to be able to first determine the question of interest. Some situations may require a patient-based approach, particularly if the question of interest relates to overall function or quality of life. However, if exposures or outcomes are specific to an extremity or injury episode, then an observation-based analysis is required. If investigators include more than 1 joint-specific or injury-specific observation from the same patient, then they should use appropriate techniques to account for nonindependence. 20 Common techniques include GEEs 8 and mixed- or random-effects models. 17 These techniques appropriately account for nonindependence, thereby providing nonbiased effect estimates in the setting of intrapatient correlation. Given the complexities of conducting such analyses, it may be helpful in these situations to consult someone with advanced training in biostatistics.
Our study has limitations. Importantly, there are other causes of nonindependence that we did not assess, such as paired observations (eg, affected limb vs unaffected limb) or repeated measures over time. These are conceptually distinct entities with unique analytic requirements and are outside the scope of the present study. Second, although the proportion of studies failing to account for correlated data was high, the proportion of correlated observations within those studies was low. The consequences of failing to adjust for statistical nonindependence will be most significant in studies with larger proportions of correlated observations or a high degree of intrapatient correlation. Therefore, it is possible that the studies that violated statistical independence in our sample may not have had significantly different findings had they appropriately adjusted for statistical dependence. Additionally, our method of classifying studies with an author with presumed statistical expertise is subject to error. Authors with advanced degrees and/or affiliations with a biostatistics or epidemiology department may still lack the knowledge of how to appropriately manage nonindependent data. Despite the limitations of this approach, it has been used multiple times in the orthopaedic literature as a feasible way of determining statistical expertise among an author list. 4,12
Conclusion
We found that the analysis of dependent observations is common in the orthopaedic sports medicine literature, but most studies do not adjust for nonindependence. Investigators should be aware of incorrect inferences arising from nonindependence and should use appropriate methods to account for intrapatient correlations.
Footnotes
One or more of the authors has declared the following potential conflict of interest or source of funding: M.S.K. is a consultant for OrthoPediatrics, Smith & Nephew, and Stryker. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.
Ethical approval was not sought for the present study.