Abstract
Background:
Clinically relevant threshold values associated with patient-reported outcome measures after orthopaedic procedures such as anterior cruciate ligament reconstruction (ACLR) are important for relating these scores to meaningful postoperative improvement.
Purpose/Hypothesis:
The purpose of this study was to determine the Patient Acceptable Symptom State (PASS) for the Patient-Reported Outcomes Measurement Information System Computer Adaptive Test (PROMIS-CAT) after ACLR. It was hypothesized that preoperative sport participation would have an impact on PASS achievement.
Study Design:
Case series; Level of evidence, 4.
Methods:
Included were consecutive patients who underwent primary assisted ACLR between January 4 and August 1, 2016. Patients were administered the PROMIS-CAT Physical Function (PF) and Pain Interference domains preoperatively and at a minimum 2 years postoperatively, with external anchor questions used to determine the PASS. Receiver operating characteristic (ROC) curves were constructed for the entire study population as well as separately for athletes and nonathletes to determine PROMIS PASS thresholds for each population. A previously published PROMIS-PF minimal clinically important difference was used to evaluate postoperative improvement. A post hoc multivariate nominal logistic multivariate analysis was constructed to assess the effects of preoperative patient characteristics on the likelihood of attaining both the minimal clinically important difference and PASS.
Results:
In total, 112 patients were included in the study, with 79 (71%) having recreational or higher levels of athletic participation. The PASS for the study population was 56.0 (area under the ROC curve, 0.86) and was unaffected by baseline PROMIS-PF scores but was affected by preoperative athletic participation (56.0 for athletes, 49.0 for nonathletes). A post hoc analysis found 57 patients (51%) achieved the PASS for the PROMIS-PF (cutoff, 56.0), but when the athlete and nonathlete thresholds were applied to their respective patient groups, 66% of athletes and 64% of nonathletes achieved the PASS postoperatively. The multivariate analysis found that sport participation (odds ratio, 6.2; P = .001) but not age, sex, body mass index, or preoperative PROMIS affected the likelihood of achieving the PASS on the PROMIS-PF.
Conclusion:
Preoperative athletic participation significantly affected the ability to achieve PASS.
Primary anterior cruciate ligament (ACL) reconstruction (ACLR) is the standard of care for acute ACL tears, resulting in generally high rates of improved knee stability, satisfaction, and return to sport in athletic populations. 15,31 In addition to objective and radiographic metrics evaluating knee stability and construct status, patient-reported outcome measures (PROMs) have become invaluable for both counseling patients preoperatively and evaluating postoperative recovery. 37 ACLR is among the most studied orthopaedic procedures, with multiple injury-specific and generalized health surveys currently in clinical practice. 23,24 In addition to statistical significance, recent reports have emphasized clinically relevant outcome thresholds such as the minimal clinically important difference (MCID), Patient Acceptable Symptom State (PASS), and substantial clinical benefit (SCB) when evaluating PROMs. 27 The PASS is a valuable threshold for identifying postoperative outcomes given that it is associated with when a patient “feels good” and is highly correlated with overall satisfaction with surgery. 22
With regard to ACLR, PASS and MCID thresholds have been identified for multiple joint-specific questionnaires, including the International Knee Documentation Committee (IKDC) Subjective Form, Knee injury and Osteoarthritis Outcome Score (KOOS), Western Ontario and McMaster Universities Osteoarthritis Index, and Lysholm knee scale (Table 1). Several recent studies have defined PASS thresholds after ACLR for the IKDC and KOOS subscales, with 1 report documenting PASS rates of >95% at the 2-year follow-up for these surveys. 5,25
Previously Reported PASS and MCID Thresholds for Common Joint-Specific Outcome Measures After Anterior Cruciate Ligament Reconstruction a
a Dashes indicate areas not reported. ADL, Activities of Daily Living; IKDC, International Knee Documentation Committee; KOOS, Knee injury and Osteoarthritis Outcome Score; MCID, minimal clinically important difference; PASS, Patient Acceptable Symptom State; QoL, Quality of Life; Sport, Sports and Recreation; WOMAC, Western Ontario and McMaster Universities Osteoarthritis Index.
Significant limitations of existing ACLR outcome measures relate to intrinsic ceiling effects, survey fatigue, and the associated burden of routine survey administration, collection, and documentation. The Patient-Reported Outcomes Measurement Information System (PROMIS) was developed by the US National Institutes of Health in 2003 in part to overcome the intrinsic limitations of ceiling and floor effects as well as ease both patient and administrative workload through computer adaptive test (CAT). 7,8 PROMIS is now frequently used throughout the orthopaedic community, with multiple studies evaluating its utility in the spine, arthroplasty, foot and ankle, and sports literature. 2,12,19,22 In a recent report by Scott et al, 30 the PROMIS metric was validated against legacy ACLR outcome measures, including the KOOS–Sports and Recreation subscale, KOOS–Activities of Daily Living subscale, and the Short Form–36 Health Survey Physical Function.
The purpose of this study was to determine the PASS after ACLR for PROMIS-CAT. As athletes will likely require a higher level of function to “feel good,” we hypothesized that preoperative sport participation would affect postoperative PASS thresholds.
Methods
Patient Selection
This institutional review board–approved retrospective review of prospectively collected data was performed on 172 consecutive patients who underwent primary ACLR between January 4 and August 1, 2016. A power analysis (G*Power v3.1.9.4, Dusseldorf) 16 required 108 participants using the previously reported PROMIS Physical Function (PF) MCID of 4.6 with a standard deviation of 10 to achieve adequate power (1 – β > 0.8; P = .05) for this study. 11 The study population represented a multisurgeon cohort of 5 fellowship-trained sports medicine surgeons (R.B., J.G., M.D.M., S.M.) at a high-volume academic institution. Adult patients (age >18 years) were included if they received primary ACLR and completed preoperative and minimum 2-year postoperative PROMIS, athletic participation, and PASS questionnaires. In total, 39 patients were excluded for refusal to participate, revision ACLR, multiligamentous injury requiring surgical treatment, or prior ipsilateral knee surgery, while 21 patients were lost to long-term follow-up (Table 2). All patients in this study received bone–patellar tendon–bone grafts with surgeon-dependent choice for autograft or allograft. Electronic medical records were reviewed to obtain demographic information as well as data regarding intraoperative findings and postoperative complications. Patients who participated in collegiate or professional sports were defined as “high-level” athletes. Patients who did not regularly participate in sports but endorsed symptoms of knee instability in the setting of ACL injury were also indicated for ACLR. Previously described surgical technique and standard of care rehabilitation protocols were used for all patients. 1,11
Patients Excluded From the Study
Patient-Reported Outcomes
All study patients were administered the PROMIS-CAT assessment for the lower extremity PF, Pain Interference (PI), and Depression domains preoperatively and a minimum of 2 years postoperatively. Surveys were administered over the telephone or electronically through a secure server (REDCap). PROMIS scoring is based on a normalized T score with a standard deviation of 10. In addition, the ACL–Return to Sport after Injury short form 35 was administered as well as the following PASS anchor question: “Taking into account all the activities you have during your daily life, your level of pain, and also your functional impairment, do you consider that your current state is satisfactory?” 33,35 Patients who responded in the affirmative were deemed to have met the PASS threshold.
Statistical Analysis
Statistical analysis was conducted with the JMP platform. Variables were characterized as continuous (ie, age, body mass index [BMI]), ordinal (sport level), or categorical (ie, sex, graft type). Bivariate analyses were used to compare continuous data, whereas univariate analyses were used to compare continuous with categorical data. Pearson χ2 and Fisher exact tests were used to compare categorical data. To determine PASS thresholds, a receiver operating characteristic (ROC) curve was constructed in which optimized specificity and sensitivity profiles (Youden index) were obtained by correlating the binary outcome of the PASS question with continuous PROMIS scores. 10,22 Patients lost to follow-up were excluded in the final analysis. The area under the ROC curve (AUC) was calculated for each ROC, with AUC values >0.7 considered acceptable models. 21 We stratified the analysis based on preoperative pain and functional scores (above or below the 50th percentile) as well as athletic participation to determine the impact of these variables on threshold determination. The previously established MCID of 4.6 was used to evaluate clinically relevant improvement. 11 Nominal logistic multivariate analyses were constructed to identify preoperative predictors of achieving the PASS or MCID. A P value <.05 was considered statistically significant.
Results
Patient Demographics
Of the 133 patients meeting inclusion criteria, 21 were lost to follow-up, leaving 112 (84%) with a mean follow-up of 34.2 ± 2.2 months available for analysis. There were 55 women (49%), with the study cohort having a mean age of 30.4 years (range, 18-65 years) and BMI of 26.7 kg/m2 (range, 17.4-48.8 kg/m2) (Table 3). Postoperatively, 45 of 79 (57%) athletes had returned to their preinjury level of sport, with 25 of 34 (74%) of those unable to return citing their knee symptoms as the reason for their inability to return to sport. At the most recent follow-up, 12 patients (11%) required revision procedures (arthrofibrosis, 7; hardware removal, 1; septic arthritis, 1; symptomatic meniscal tear, 1; graft retear, 2).
Demographic, Injury, and Surgical Characteristics of the Study Cohort (N = 112) a
a Values are presented as mean ± standard deviation or number (%). ACLR, anterior cruciate ligament reconstruction; BMI, body mass index; BTB, bone–patellar tendon–bone graft; RTS, return to sport.
b Some patients had a combination of meniscal and chondral injuries.
Patient-Reported Outcome Scores
The study cohort demonstrated significant postoperative improvement in both PROMIS-PF and PROMIS-PI (P < .0001) (Figure 1), with a mean improvement of 16.3 ± 12.0 (range, 14.7-44.6) for the PROMIS-PF and 9.4 ± 10.9 for the PROMIS-PI (range, 26.3-33.0). Athletes had superior postoperative PF and pain scores as well as improvement compared to nonathletes, but there were no differences in rates of attaining the MCID for the PROMIS-PF (Table 4). Of the patients, 93 (83%) affirmatively endorsed the PASS question, “Taking into account all the activities you have during your daily life, your level of pain, and also your functional impairment, do you consider that your current state is satisfactory?” Of these, 24 of 33 nonathletes (73%) and 69 of 79 athletes (87%; P = .09) achieved the PASS based on survey responses. All 45 athletes who returned to sport, as well as the 9 patients whose inability to return to sport was unrelated to the knee, responded affirmatively to the PASS question, while 10 of 25 (40%) of athletes who were unable to return to sport due to their knee responded negatively to the PASS question.
The study cohort demonstrated significant improvement from preoperative to postoperative Patient-Reported Outcomes Measurement Information System (PROMIS) Physical Function and Pain Interference scores. *P < .0001.
Comparison of PROMIS-PF and PROMIS-PI Scores Between Athletes and Nonathletes a
a Values are presented as mean ± standard deviation or number (%). Boldface P values indicate statistically significant difference between groups (P < .05). MCID, minimal clinically important difference; PF, Physical Function; PI, Pain Interference; PROMIS, Patient-Reported Outcomes Measurement Information System.
PASS Determination
For the entire cohort, the PASS threshold for the PROMIS-PF was determined to be 56.0 (AUC, 0.86) after optimization of the specificity-sensitivity profile of the ROC curve (Table 5). The PASS threshold was not significantly affected by baseline preoperative PROMIS-PF scores (56.0 vs 54.7), but when the cohorts were segregated based on athletic participation, nonathletes had a lower PASS threshold (49.0) compared to athletes (56.0). The PASS thresholds were similar when comparing recreational and high-level athletes (56.0 and 57.0, respectively) and not affected by age, sex, or BMI. The PASS threshold for the PROMIS-PI (49.7; AUC, 0.82) was not affected by sport participation, but patients with higher postoperative pain had a lower threshold to achieve the PASS (56.6 vs 46.6).
PASS Thresholds for PROMIS-PF and PROMIS-PI Determined Through Sensitivity-Specificity Optimization of Receiver Operating Characteristic Curves
a AUC, area under the receiver operating characteristic curve; PF, Physical Function; PI, Pain Interference; PROMIS, Patient-Reported Outcomes Measurement Information System.
Application of PASS Thresholds
A post hoc analysis applying the full cohort PASS-PF threshold (56.0) to the study population found that 57 patients (51%) achieved the PASS-PF threshold postoperatively (Table 6). Using this threshold, 52 of 79 athletes (66%) achieved the PASS compared to 5 of 33 nonathletes (15%; P < .0001). Of the athletes who were able to return to their preoperative level of competition, 84% achieved the PASS, whereas athletes unable to return had a PASS rate of 41% (P = .001). When applying the athlete- and nonathlete-specific thresholds of 56.0 and 49.0, respectively, there were no proportional differences between patients who achieved the PASS (21/33 [64%] of nonathletes vs 52/79 athletes [66%]; P = .84). When preoperative scores were evaluated, 6 (5%) patients had achieved the PASS-PF while 24 (21%) achieved the PASS-PI before their procedure.
Application of the PASS-PF Threshold to Different Study Cohorts a
a Values are presented as number (%). Athletes had higher rates of achieving PASS compared to nonathletes (P < .0001). Athletes who returned to sport had higher rates achieving PASS compared to those who did not (P = .001). PASS, Patient Acceptable Symptom State; PF, Physical Function; RTS, return to sport.
A nominal logistic multivariate analysis was constructed to evaluate preoperative demographic and surgical variables affecting the ability to achieve the PASS or MCID (Table 7). Preoperative sport participation was significantly associated with attaining the PASS while lower preoperative PROMIS-PF scores were significantly associated with achieving the MCID.
Multivariate Analysis of Factors Associated With Achieving the PASS or MCID a
a Boldface P values indicate statistical significance (P < .05). BMI, body mass index; BTB, bone-patellar tendon-bone graft; MCID, minimal clinically important difference; OR, odds ratio; PASS, Patient Acceptable Symptom State; PF, Physical Function; PROMIS, Patient-Reported Outcomes Measurement Information System.
b Odds ratios for continuous variables reflect cumulative odds for every integer increase in the variable (ie, for every integer increase in preoperative PROMIS-PF, there is a 1.1% decrease in the likelihood of attaining the MCID-PF, or patients with a preoperative PROMIS-PF score of 40 were 11 times less likely to attain the MCID-PF than patients with a preoperative score of 50).
Discussion
In a review of 112 consecutive patients with minimum 2-year outcomes undergoing primary ACLR, we found that athletes required a higher PASS-PF threshold (56.0) compared to nonathletes (49.0). This is important clinically, as athletes require a higher PF to achieve satisfactory outcomes than nonathletes, and if the athlete PASS-PF threshold is applied to nonathletes, it will result in a higher proportion of clinical failures for the nonathlete population. In addition, the post hoc application of the calculated PASS thresholds resulted in proportionally lower rates of PASS achievement (51%) compared to the patient responses to the anchor-based PASS question (83%). Most patients who did not achieve this PASS threshold were nonathletes, and when we applied the calculated nonathlete PASS threshold of 49.0, we found that nonathletes had similar rates of achieving the PASS compared to athletes (64% vs 66% for athletes and nonathletes, respectively). Furthermore, the sensitivity of PASS thresholds for both athletes (75%) and nonathletes (75%) was improved compared to the PASS cutoff from the full cohort (61%). The difference between the athlete and nonathlete PASS exceeds the previously calculated MCID (4.6), supporting the clinical relevance of this difference. The impact of athletic participation on achieving the PASS was reinforced by a multivariate regression analysis identifying athletic participation as the only significant predictor of achieving the PASS postoperatively.
Clinically relevant outcome thresholds, including the PASS, MCID, and SCB, have been used throughout the orthopaedic literature to enhance the utility of PROMs. With regard to the ACLR literature, the PASS for the IKDC and KOOS subscales was previously reported by Muller and colleagues. 25 The authors performed a study of 250 patients in whom PASS thresholds of 75.9 (sensitivity, 83%; specificity, 96%) and 75.0 (sensitivity, 87%; specificity, 88%) were calculated for the IKDC and the KOOS–Sports and Recreation subscale, respectively. While they did not apply this threshold to their study population, the proportion of positive PASS responders compared favorably to our study (89.2% vs 83%), with the resulting discrepancy possibly explained by differences in follow-up rates between the studies (31.5% vs 84%). They did not identify proportional differences in PASS responses with regard to sport activity level (no sport vs light vs moderate vs heavy), but only 4 of 250 (1.6%) reported no sporting activity compared to 30% in the present study.
In a novel report by Vega et al, 34 the PASS question itself was recently validated against KOOS and Hospital for Special Surgery Pediatric Functional Activity Brief Scale (Pedi-FABS)–based models of clinical success. Similar to our study, they reported 249 of 299 (83.3%) patients satisfying the PASS anchor question and found substantial agreement between successful outcomes defined by the PASS and either the KOOS or Pedi-FABS. Furthermore, while they did not calculate separate PASS thresholds, 84% of self-reporting athletes answered yes to the PASS question in their study compared to 78% of nonathletes, which was similar to the 87% and 73% reported in the present study. The primary limitation of the PASS question in their study was decreased specificity when compared to KOOS or Pedi-FABS benchmarks of clinical success. This resulted in a large number of false-positive results, in which patients who responded positively to the PASS questionnaire were deemed clinical failures by the KOOS or Pedi-FABS definitions of success. Our study produced similar findings—that is, a greater number of patients responded positively to the PASS questionnaire (83%) compared to the number of patients who achieved the statistically derived PASS threshold (51%). This was ameliorated partly by the application of separate PASS thresholds for athletes and nonathletes that found success rates of 66% and 64%, respectively, which are similar to other PASS rates after arthroscopic procedures in the hip and knee. 9,10,14,22
Even with the modified PASS thresholds, however, there remains a discrepancy between the 83% of patients who were satisfied with their postoperative symptom state and the populations that achieved the statistically derived benchmarks. In addition, 60% (15/25) of athletes who were unable to return to sport due to their knee symptoms achieved the PASS based on survey responses, compared to the 24% (6/25) of athletes in this cohort who achieved the PASS statistically. This highlights the finding that return to sport is not necessarily the ultimate arbiter of postoperative satisfaction or success after ACLR, as numerous physical and psychosocial factors influence recovery and final outcomes. 13,17 Furthermore, these findings represent inherent limitations of the PASS metric, which has sensitivity and specificity profiles (75% vs 100% for athletes and nonathletes) that must be appreciated when counseling patients, as a cohort of patients satisfied with their outcome may not achieve the PASS threshold.
With the increasing utility of PROMs in evaluating orthopaedic patients, a number of different thresholds have been assigned to confer “clinical relevance.” Of these, the MCID is among the most common, but the heterogeneity of its determination and its dependence on preoperative scores represent significant limitations. 11,20,36 The present study found that the MCID-PF was affected only by preoperative PROMIS scores when using a previously reported 0.5 standard deviation–based MCID of 4.6, which was consistent with a recently published anchor-based MCID of 4.5 11,38 Thus, patients who have exceedingly low preoperative scores may achieve clinical success according to the MCID, but they may have inferior outcomes relative to their specific population and may not consider their result a success. Conversely, patients who have higher preoperative outcome scores may not achieve the MCID despite having a satisfactory outcome. The SCB addresses this limitation by focusing on the threshold for which patients achieve substantial improvement from their preoperative state, but it is similarly limited by a dependence on preoperative scores and presents a threshold that may not be achievable for most patients. 27 The PASS ameliorates these challenges as it only requires a postoperative score to define where most patients view their symptoms as acceptable and, when combined with PROMIS, normalizes these scores to those of the general population. Predictably, the nonathlete population in the present study had a PASS threshold (49) that mirrored the general population mean (50), while athletes required a higher level of function to have a satisfactory outcome.
While the PASS, along with the MCID and SCB, are valuable metrics for interpreting outcomes for patient populations, care must be taken when applying these thresholds to individuals. If patients have an outcome score that falls outside of the predetermined PASS threshold, there may be a discrepancy between whether the PASS was achieved and their ultimate happiness with their outcome. Setting appropriate thresholds that more accurately reflect the desired surgical population (ie, athletes vs nonathletes) can partially address this limitation, but it does not entirely solve the problem. By normalizing patient postoperative outcome scores against their preoperative scores, the maximal outcome improvement (MOI) threshold is an intriguing metric that may be more applicable at the individual patient level. 32 In a recent study, Beck and colleagues 3 defined the MOI for the American Shoulder and Elbow Surgeons and Single Assessment Numeric Evaluation outcome scores for patients undergoing arthroscopic rotator cuff repair and concluded that workers’ compensation and hand dominance were the primary preoperative factors influencing achievement of the MOI. Additional metrics such as the forgotten joint score and the perception of being “completely better” have been used to address the high ceiling effects of legacy measures and warrant further investigation in the ACLR literature. 4,29
Limitations
While strengths of this study include a multisurgeon cohort of consecutive patients undergoing primary ACLR with a minimum 2-year follow-up, the present study is not without limitations. We used a questionnaire that allowed patients to self-identify as being athletes or nonathletes, and the level of competition was obtained from preoperative clinic notes and was not standardized. The use of the Tegner activity scale would have provided more objective criteria to athletic participation that may have allowed a better definition of threshold values for more elite athletes. Furthermore, most athletes in the study population were recreational or high school athletes, and the PASS threshold for success may be higher for collegiate and professional athletes.
While PASS-PF threshold determination was not affected by age, sex, or preoperative score on a multivariate analysis, this may have been affected by the specific type of sport which was not analyzed in this study. In addition, this study did not include pediatric or adolescent patients, which may further limit the generalizability of the present findings. From a surgical perspective, all patients in this study received either bone–patellar tendon–bone autograft or allograft reconstructions, and the findings identified here may be less applicable to patients receiving alternate grafts or for revision ACLR. Despite these limitations, this study provides an important metric for evaluating postoperative success after ACLR. Using the tiered approach to providing value-based care for orthopaedic procedures described by Bernstein et al, 6 achieving the PASS represents the primary outcome goal, with the MCID and SCB representing minimum and superior outcome goals, respectively. Over the coming years, it will become increasingly important for orthopaedic surgeons to develop a more unified approach when evaluating outcome scores, and future studies, economic analyses, and consensus agreements will be required to better define the comparative value of orthopaedic procedures in the modern era.
Conclusion
PASS thresholds for the PROMIS-PF and -PI outcome scores were determined in a series of consecutive patients undergoing primary ACLR. Preoperative athletic participation significantly affected the ability to achieve PASS, and we recommend using separate athlete and nonathlete thresholds when evaluating postoperative outcomes after ACLR.
Footnotes
Final revision submitted March 14, 2023; accepted April 13, 2023.
One or more of the authors has declared the following potential conflict of interest or source of funding: R.B. has received education payments from Prodigy Surgical Distribution. J.F.B. has received consulting fees from Active Life, DJO, Endo Pharmaceuticals, Medartis, Nextremity Solutions, Novastep, Stryker, Subiomed, Vilex, Vivex, and Wright Medical; has received royalties from Nextremity; and has stock/stock options in Carbon22 and Nextremity Solutions. M.D.M. has received consulting fees from Arthrex, nonconsulting fees from Arthrex, and hospitality payments from Prodigy Surgical Distribution and has stock/stock options in Concentric Analgesics, Karma, and Telephus. S.M. has received consulting and nonconsulting fees from Arthrex. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.
Ethical approval for this study was obtained from the University of Rochester (No. MOD00003131).