Abstract
Background:
Patient-reported outcome measures (PROMs) contribute to evaluating and improving the quality of patient care. Patient outcomes after rotator cuff repair (RCR) have been researched; however, the relationship between PROM response rates and individual and health care correlates has not been thoroughly investigated.
Purpose:
To examine differences in individual and health care factors among patients who had undergone RCR based on their PROM response rates.
Study Design:
Case-control study; Level of evidence, 3.
Methods:
Shoulder-specific and general PROMs were solicited via email and text message of all patients who underwent RCR between 2016 and 2020. Three subgroups were classified: (1) complete responders completed all 1-year postoperative PROMs, (2) partial responders answered enough questions to produce ≥1 usable score, and (3) nonresponders did not respond to a single measure. Correlates were assessed using analysis of variance and chi-square tests. Adjusted multinomial logistic regression models identified predictors of 1-year PROM response.
Results:
Of 2195 patients included at the 1-year follow-up, 34% were complete responders; 11%, partial responders; and 55%, nonresponders. Patients had a mean age of 61.8 years, 63% were men, and 90% were White. Pre- and postoperative PROM scores were similar across responder groups. In stepwise selection, 1-year responses (complete or partial) were associated with older age, later year of surgery, White race, and having workers’ compensation insurance. The strongest predictor of PROM response was having workers’ compensation insurance.
Conclusion:
Patients with workers’ compensation insurance compared with other insurance types responded to PROMs at disproportionately higher rates. This could distort postoperative PROM scores in the population studied because there are known differences among patients with this insurance status.
Patient-reported outcome measure (PROM) surveys are used in orthopaedic and sports surgery to quantify functional improvement and pain reduction over the course of care. Collecting complete and accurate PROM data is crucial to monitor the results of treatment, improve the quality of care, and produce meaningful clinical research. 20 Postsurgical PROMs are important pieces of information that providers can use to assess surgical results and to engage in quality improvement activities. 20 Relatedly, PROMs are an integral tool, as one component of value, to assess outcomes of care in conjunction with metrics such as cost, readmission, and revision rates. 20 Despite these benefits, when response rates are low or vary by factors associated with differences in outcomes, data may not be representative of the patient population, leading to generalizability bias and limiting clinical and research utility.16,19,30 PROM response rates for orthopaedic treatments have been shown to vary, ranging from 24% to 75% over follow-up periods typically ranging from 1 week to 2 years.6,17,18,27
Rotator cuff repairs (RCRs) have a prevalence of 165 per 100,000 person-years in the United States and are increasing at a rate of 1.2% per year. 44 This increase is primarily driven by the increasing number of adults aged 50 to 64 in the population. 44 Unfortunately, only a handful of studies have investigated PROM response rates specifically in this population.21,24 Additionally, when response rates in shoulder procedures are studied, RCR data are often pooled with other shoulder surgical procedures,18,28,33,42 limiting our ability to draw conclusions related to this specific clinical population.
It is important to have an accurate understanding of the results of RCR. Ideally, patients completing PROMs would be representative of the entire population undergoing RCR in order to avoid inadvertent skewing of the results and incorrectly representing true population outcomes. It is also important to assess differences between subgroups of patients who complete all of the PROMs administered compared with patients who complete some but not all PROMs, in order to identify strategies to improve response rates. To determine whether PROM response rates vary by demographic, clinical, or health care system characteristics, we performed a retrospective comparative study within a large, multiclinic, university-affiliated sports medicine practice between January 2016 and December 2020.
The purposes of the study were to (1) determine the percentage of nonresponders, partial responders, and complete responders in an RCR population, (2) compare demographic, clinical, and health care system factors among those 3 categories, and (3) define the magnitude of association between response rates and various correlates.
Methods
Study Design and Participants
This retrospective study of prospectively collected PROMs was performed at a single institution and approved by the University at Buffalo’s institutional review board. Data from all patients who underwent RCR by multiple providers (n = 11) between January 1, 2016, and December 31, 2020, were included. All patients provided informed consent to be enrolled in the PROMs data capture system and could opt out at any time.
Inclusion and Exclusion Criteria
All patient records with the following Current Procedural Terminology codes for RCR surgeries, both arthroscopic and open, were initially included in the data set: 23410, 23412, 23420, and 29827. On average 25 patients per year were not enrolled in our patient-reported outcome system and thus were excluded from this study.
Demographic and Health Care Data Collected
The following demographic variables were extracted from electronic medical records: age, sex, race, ethnicity, language, body mass index, tobacco use, zip code, health insurance, laterality, and location (ambulatory surgery vs hospital setting). The median household income for each zip code was obtained from state-specific income statistics from the US Census Bureau. 40
PROM Collection Process
Our orthopaedics and sports medicine department uses an automated, practice-wide system to collect PROMs with the Outcomes-Based Electronic Research Database (OBERD), a health- and web-based software. 39 Patients received automated PROM surveys via email and text at specified intervals, depending on their reason for visit, diagnosis, and procedure. Specific to this study for RCR, patients received 1 of 5 iterations of order sets (a grouping of PROMs that included 6-8 measures each on a specified timeline). All patients received email and/or text message reminders based on their opt-in status for Short Message Service reminders. The emails and text messages were sent to patients when the time period opened and then again 5 days, 3 days, and 1 day before the close date (ie, availability window). For the preoperative availability window, the PROMs were opened 90 days before surgery and closed the day of the surgical procedure. OBERD uses a 28-day month standard structure for calculating days in order sets. The 1-year postoperative time point had an availability window of 8 to 14 months. PROM data were collected and stored in our institution’s OBERD and HIPAA (Health Insurance Portability and Accountability Act)–compliant online data capture software program and were extracted directly from that database for purposes of this study.
PROMs Collected
The preoperative and postoperative PROMs analyzed in this study included measures of pain, function, treatment outcomes, overall well-being, and mental health. Specifically, they included the American Shoulder and Elbow Surgeons score (ASES), measuring pain and performance in activities of daily living; ASES Computer Adaptive Testing, measuring pain and performance in activities of daily living; a Patient Acceptable Symptom State question, measuring patient satisfaction with treatment; the Patient-Reported Outcomes Measurement Information System Global scale, measuring general health care–related quality of life; the Disabilities of the Arm, Shoulder and Hand (DASH) questionnaire and QuickDASH (shortened version), measuring function; the Single Assessment Numeric Evaluation, measuring function; the Simple Shoulder Test, measuring functional limitations; visual analog scales, measuring shoulder pain and function; and the Veterans RAND 12-Item Health Survey, measuring health care–related quality of life. Some PROMs such as ASES Computer Adaptive Testing and the QuickDASH were introduced midway through the study period in an attempt to improve patient compliance by offering shorter, adaptive measures. 2 Therefore, we decided to implement PROMs with fewer questions while keeping the integrity of the questionnaire and insights into the patients’ improvements from 2017 onward.
Statistical Analysis by Study Aim
To determine percentages of responder groups among patients who had undergone RCR (aim 1), we classified patients into 3 subgroups based on whether they completed all, some, or no PROMs. Patients were considered complete responders if they completed all 1-year postoperative PROMs, partial responders if they answered enough questions to produce ≥1 usable PROM score, and nonresponders if they did not respond to a single PROM in the order set.
To compare demographic, clinical, and health care system factors among those subgroups (aim 2), we calculated descriptive statistics to describe patient demographic and health care characteristics for the overall study sample and stratified by response groups (nonresponders, partial responders, and complete responders). Univariate statistics were calculated, and the normality of all continuous variables was assessed. The majority of the continuous variables followed a normal distribution; therefore, parametric tests were used to compare continuous variables, and no additional transformation was performed. For demographic data and PROMs collected at the time of surgery, we used a 1-way analysis of variance to compare continuous variables and a chi-square test to compare binary and categorical variables. Additionally, the percentage of missing clinical PROMs at the time of surgery was calculated and compared using a chi-square test.
Finally, to determine the relative magnitude of the associations between correlates and responder subgroup (aim 3), we fit a multinomial logistic regression model. We included demographic variables that were significantly different between response groups to predict the likelihood of partially or fully completing PROMs. Next, a binomial logistic regression model with stepwise conditional selection was made to predict any response (both partial and complete) from nonresponders to identify which variable was most predictive of responding at the 1-year postoperative follow-up. For this exploratory model, we adjusted our models with all demographic characteristics that had a P value of <.01. Last, mean change in continuous PROMs was calculated for the partial and complete responder categories and compared. P < .05 was considered statistically significant, and all analyses were performed using SPSS Version 28 (IBM Corp). 14
Results
A total of 2478 patients who underwent RCR were identified between January 1, 2016, and December 31, 2020. A total of 2195 (88.6%) patients were included in the analytic sample, after excluding n = 125 who missed the surgical outcome enrollent window and n = 158 duplicates. Within the included analytic sample, 740 patients (34%) were classified as complete responders; 240 patients (11%), as partial responders; and 1215 (55%), as nonresponders at 1 year after RCR (Figure 1).
Flowchart of study participants and categorization of responder subgroups for 1-year postoperative patient-reported outcome measures. CPT, Current Procedural Terminology code; Post-op, postoperative.
Demographic and health care system characteristics, both overall and by response group, are presented in Table 1. The mean age of the entire study sample study sample was 61.8 ± 10.1 years, and most of the participants were men (63.4%) and were White (90.0%). Mean body mass index was 30.1 ± 6.0. Median household income was $64,608 ± $14,652, and the majority of patients (67.5%) had private health insurance. When participants were stratified by 1-year responder subgroup, statistically significant differences were seen in patient age, year of surgery, race, health insurance type, and clinical setting.
Patient Characteristics by 1-Year Postoperative Response Subgroups (N = 2195) a
Values are expressed as n (%) or mean ± SD. Bold indicates statistically significant, p-value <.05.
To assess whether PROM scores were different between responder subgroups, we compared preoperative and postoperative PROMs by 1-year responder subgroups (Table 2); for PROM-specific response rates and sample sizes, see Appendix Table A1 (available in the online version of this article). Notably, scores for all subgroups improved over time. We found no statistically significant differences at P < .05 in preoperative PROMs by 1-year responder subgroups or nearly all postoperative PROMs. The only exception was the postoperative function score (measured via the visual analog scale), which differed when we compared the partial and complete responder subgroups. Complete responders had slightly higher postoperative function compared with partial responders (7.26 ± 2.54 and 6.73 ± 2.91, respectively; P = .046), a marginally statistically significant difference.
Preoperative and Postoperative Patient-Reported Outcomes by Responder Subgroups (N = 2195) a
Values are expressed as n (%) or mean ± SD. unless otherwise indicated. Bold indicates statistically significant, p-value <.05. Dashes indicate not applicable. ASES, American Shoulder and Elbow Surgeons; CAT, Computer Adaptive Testing; QuickDASH, shortened version of the Disabilities of the Arm, Shoulder and Hand; PASS, Patient Acceptable Symptom State; PROM, patient-reported outcome measure; PROMIS, Patient-Reported Outcomes Measurement Information System; SANE, Single Assessment Numeric Evaluation; SST, Simple Shoulder Test; VAS, visual analog scale.
Postoperative missingness ranges from 12% to 58% for the partial responder subgroup (see Appendix Table A1, available online, for PROM-specific response rates in the pre- and postoperative periods). Because we defined partial responder as completing ≥1 PROM, the total postoperative subgroup sample sizes do not add to the total subgroup number.
Based on 1-way analysis of variance or chi-square test comparing all 3 subgroups.
Results of multinomial logistic regression with all significant demographic and clinical setting variables are presented in Table 3, including comparisons between nonresponders and complete responders and between partial responders and complete responders. The Akaike information criterion of the intercept-only model was 2458.05 and of the final model was 2360.92, which corresponded to a significant improvement of fit (P < .001). The model also did not violate assumptions on a test for goodness of fit (P = .579). When complete responders were compared with nonresponders, older age, more recent year of surgery, White race, workers’ compensation insurance, and ambulatory (vs hospital) setting were associated with responding at the 1-year follow-up. When complete responders were compared with partial responders, only more recent year of surgery and ambulatory surgery setting were associated with completing the 1-year follow-up. No individual-level factors were associated with the comparison of partial versus complete responders.
Multinomial Logistic Regression Comparing Complete Responders With Nonresponders and Partial Responders a
Bold indicates statistically significant, p-value <.05. Dashes indicate not applicable.
Results of our stepwise logistic regression predicting any response (complete and partial) from nonresponders are presented in Table 4. Six covariates were entered into the model, and 2 covariate (language and location) were excluded during forward conditional stepwise selection. The final model did not violate assumptions on a Hosmer-Lemeshow test for goodness of fit (P = .429); however, age and year of surgery had a significant inverse correlation (ρ = −0.123; P < .001), which may explain why there was a weak, negative β coefficient for age in this model (ρ = −0.019; P < .001; hazard ratio = 0.98). We observed the strongest magnitude of the β coefficient (ρ = 0.669; P < .001) for insurance type, suggesting workers’ compensation insurance to be the strongest predictor (almost 2 times the chance of complete responding compared with partial responding [hazard ratio = 1.95]) of responding to the 1-year follow-up PROMs compared with any other demographic or health care system variable. Post hoc analyses were conducted to compare postoperative PROM scores between insurance types, and scores were lower overall for patients with workers’ compensation compared with patients with all other insurance types combined (see Appendix Table A1, available online).
Results of Stepwise Logistic Regression Predicting Response (Complete or Partial) at 1 Year a
Bold indicates statistically significant, p-value is <.05. Dash indicates not applicable.
Discussion
The routine collection of PROMs is an important strategy to increase patient satisfaction, assess the effect of treatment, and improve quality of care. 8 However, completion of the entire battery of PROMs can be challenging for many patients. 3 Our large, retrospective study among patients who had undergone RCR showed the highest rate of nonresponders (55%), followed by complete responders (34%), and the lowest rate of partial responders (11%) for PROMs at the 1-year postoperative period. Our findings are consistent with the response rates found in the orthopaedics and shoulder surgery literature. In a study of >200,000 patients, Bernstein et al 3 reported an overall completion rate of 40%. Among studies of patients who underwent RCR, various studies have reported response rates ranging from 30% to 57% at a mean of 6 months to 4.5 years.21,24,38 However, none of these studies distinguished between complete and partial responder subgroups.
Our study showed demographic and health care system differences between responders and nonresponders; however, we did not find clinically significant clinical differences in preoperative PROMs (nonresponders vs complete plus partial or vs complete or partial). When we compared responders and nonresponders, predictors of responding to the 1-year follow-up included workers’ compensation insurance, patient race, year of surgery, and age. Complete responders were more likely to be older, have a later year of surgery, receive treatment in an ambulatory setting, and receive workers’ compensation compared with the nonresponder subgroup. Partial responders were more likely to have later year of surgery and to receive care in an ambulatory setting. Possible explanations for our findings are described below.
Age
Our finding that older age was associated with higher response rates is consistent with previous reports. ‡ Contrary to our finding, some studies found older age to be associated with worse response rates.3,10,15,16,26,34,38 When this occurs, it could distort (i.e., lower) postoperative PROM scores in a population of patients who have undergone RCR due to worse overall outcomes for this population. For example, older patients have higher retear rates.1,4,23,35
Later Year of Surgery
We found a positive trend in responding to PROMs over the 5-year study period. Higher response rates with more recent surgery are likely due to improved practices from our OBERD operations team over time. In 2017, the department underwent restructuring and management changes to drive improvements, such as (1) internal staff training to improve the knowledge base of clinical staff in regard to OBERD, (2) new patient paperwork within the OBERD system, (3) closer communication and engagement of the clinical department and the research department that allowed for clearer goals and key performance indicators to be established, and (4) the implementation of monthly patient compliance updates sent to providers to track patient engagement. Bernstein et al 3 also found a trend in improved collection rates year over year, which was attributed to the implementation of a variety of nonpunitive efforts, including hiring dedicated staff and increasing targeted advertising.
Ambulatory Setting
Patients being seen in an ambulatory setting were more likely to complete 1-year postoperative PROMs for both complete and partial responders. Possible explanations are that hospital-based patients may have more complex medical conditions (i.e., greater comorbidities) and that patients in an ambulatory setting are generally healthier and may have fewer medical problems to distract them from filling out forms.
Workers’ Compensation
The strongest predictor of responding to the 1-year postoperative PROMs was receiving workers’ compensation, wherein these patients were twice as likely to respond to PROMs compared with patients with other insurance types. Patients who receive workers’ compensation may be more likely to respond because (1) they are required to complete more paperwork and may be therefore accustomed to doing this, (2) they may think that not filling out the forms could disqualify them from getting compensation payments, and (3) they often have to visit their provider at a year after surgery to be examined as part of their case and would be prompted at the visit to fill out PROM forms at that time. Patients with workers’ compensation who undergo RCR are well-documented as having inferior outcomes.9,11,12,41 Post hoc analyses of available postoperative PROMs show significantly lower scores for patients with workers’ compensation compared with patients with all other insurance types combined (Appendix Table A1, available online). The overrepresentation of these patients could skew results by distorting population averages.
Strengths and Limitations
PROMs are critical for patient care improvement initiatives and population health strategies. This study is novel in that it focused on a single surgical procedure (RCR); included both preoperative and postoperative PROMs (where available); and assessed demographic, clinical, and health care system factors. To our knowledge, this is the largest study focusing solely on a population undergoing RCR to also include sociodemographic factors, patient-reported outcomes, and health care system variables.
This study has limitations. First, our patient population consisted of a low proportion of patients who identify as racial and ethnic minority individuals, which limits both the ability to generalize its findings to other populations and the ability to detect race- and ethnicity-based differences in likelihood of PROM response, should they indeed exist. Our patient population had a slightly higher socioeconomic status compared with the surrounding county (2020 census county mean income was $59,464), 40 which could introduce selection and generalizability biases into our findings. Second, we were not able to distinguish in our Current Procedural Terminology coding between primary and revision RCR. There is the potential that revision RCR could confound results because patients undergoing revision arthroscopic RCR may differ in terms of functional, clinical, and structural outcomes compared with patients who have undergone primary arthroscopic RCR. 35 Third, due to the relatively high level of missingness, we potentially missed variation in PROM scores by responder groups.
Conclusion
Clinical Implications
To understand the true results of RCR surgery, it is important to compare PROM responders and nonresponders to identify systemic factors that could affect results. This study suggests that our 1-year postoperative PROM scores for RCR may not accurately reflect the true results of our clinical population. Future quality improvement research could benefit from examining ways to improve response rates, including modality of survey administered, incentivizing form completion, and minimizing the quantity of measures administered. Enhanced operational efforts are needed to better target populations less likely to respond to PROMs and to adjust quality improvement data analytics to reflect potential response biases.
Supplemental Material
sj-pdf-1-ajs-10.1177_03635465231209441 – Supplemental material for Predictors of Postoperative Patient-Reported Outcome Measure Response Rates Among Patients With Rotator Cuff Repair
Supplemental material, sj-pdf-1-ajs-10.1177_03635465231209441 for Predictors of Postoperative Patient-Reported Outcome Measure Response Rates Among Patients With Rotator Cuff Repair by Nomi S. Weiss-Laxer, Sonja Pavlesen, Alfonso Arevalo, Joycelyn Jeffords, Mohammad N. Haider and Leslie J. Bisson in The American Journal of Sports Medicine
Footnotes
Acknowledgements
The authors acknowledge the contributions of Samuel Miller, who assisted with data merging, coding, cleaning, and interpretation.
Submitted March 27, 2023; accepted September 6, 2023.
One or more of the authors has declared the following potential conflict of interest or source of funding: L.J.B. has received hospitality payments from Arthrex. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
