Abstract
Background:
Prosthetic clinics in the United States must attain accreditation in order to receive reimbursement from Medicare. The accreditation process requires clinics to establish and implement performance management plans and quality improvement activities. This report describes the experience of seven prosthetic clinics in collecting patient-reported outcome data and using it to improve quality of services.
Objectives:
To describe the experience of prosthetic clinics implementing outcome monitoring and quality improvement activities as part of routine patient care.
Study design:
Qualitative, ethnographic design.
Methods:
Clinics incorporated the Orthotics Prosthetics Users’ Survey into routine care for patients aged 18 years and older who received a new lower limb prosthesis or socket. Orthotics Prosthetics Users’ Survey measures lower extremity functional status, quality of life, and satisfaction with device and services. Clinics selected Orthotics Prosthetics Users’ Survey-derived indicators on which to implement quality improvement action plans.
Results:
Seven clinics participated, but only three were able to sustain data collection. Two clinics initiated quality improvement activities focused on improving declining satisfaction or functional scores.
Conclusions:
Quality improvement activities based on patient-reported outcomes require a high degree of organizational commitment and support. External facilitation can support clinics’ quality improvement activities.
Clinical relevance
This project illustrates the challenges of sustaining quality improvement activities using patient-reported outcome data in prosthetic clinics.
Background
The Centers for Medicare and Medicaid Services (CMS) require prosthetic and orthotic clinics to attain accreditation in order to qualify for reimbursement. The American Board for Certification (ABC) in Orthotics, Prosthetics, and Pedorthics accredits clinics and determines the extent to which they meet quality care standards through performance management and improvement activities. 1 Some standards are specific, such as collecting data on patient satisfaction with products and services, while other quality standards are discretionary, allowing clinics to select measures they deem important. ABC directs clinic personnel to document actions taken to improve care when opportunities arise, but does not provide guidelines on how to implement quality improvement (QI) activities. ABC standards are similar to the International Organization for Standardization’s (ISO) standard 9001 that describes criteria for Quality Management Systems. 2
Clinics seeking accreditation may be limited in QI efforts by financial and resource limitations. Few clinics use electronic health records. They are reimbursed for device delivery by CMS and other insurers; however, they are not compensated for patient care activities, including the time taken to incorporate outcome data into routine clinical practice. 3 Productivity expectations may limit the time that prosthetists and orthotists can devote to implementing systematic data collection programs for QI activities, especially in small practices where clinicians fill multiple roles. In all, 60% of prosthetists sampled in a 2015 Practice Analysis survey 4 worked in a clinic with five or fewer clinicians, placing QI responsibility on a small group of people who may not have experience collecting, analyzing, and interpreting outcome data.
Prosthetic services must be tailored to patients’ functional status, personal preferences, functional goals, and disease states. Although a number of outcome measures exist for use with people with lower limb amputations,5–8 there is no consensus as to which instruments are optimal,5,6 leaving clinicians responsible for selecting measures suited to their patients’ varying needs. Until the recent introduction of the entry-level Masters degree, prosthetics and orthotics education in the United States provided few opportunities to acquire skills in use of outcome measures and QI procedures. Only 16% of prosthetists surveyed in the Practice Analysis had completed a graduate degree. 4 As per the National Commission on Orthotic and Prosthetic Education Core Curriculum for Prosthetists and Orthotists, graduates of the new entry-level Masters programs in prosthetics and orthotics must “assess the function and reliability of the device using scientifically validated outcome measures.”
Even clinicians well versed in research and measurement may find the outcomes and QI literature difficult to access and implement. Access to libraries and contemporary journal subscriptions is often limited to those who work in universities and large health systems although this may change as open access to journal content increases. Only 5% of prosthetists surveyed in the Practice Analysis were employed in university-based practices, while 69% worked in multi-facility organizations or hospital-based settings, and 22% in a private, single-facility practice. 4
Patient-reported outcome (PRO) measures are a cost-effective method of monitoring patient outcomes. PRO measures may be used systematically to quantify the effectiveness of a treatment plan, provide insights into the quality of clinical interventions, and allow clinicians to implement QI initiatives to address identified issues. To appreciate the experience of individuals receiving prosthetic services, clinicians need to know how patients function with their devices over time, how devices affect quality of life, and how they perceive the quality of the device and services received.
Given these resources and knowledge limitations, prosthetic facilities could benefit from external facilitation in the use of validated PRO measures to support QI activities for the patients they serve. Thus, the objective of this project was to describe the experiences of seven prosthetic clinics provided with external facilitation in collecting and sustaining PRO data collection as part of routine patient care and implementing QI activities. We used a qualitative, ethnographic approach.
Methods
By way of overview, university-based investigators provided external facilitation of QI activities to a convenience sample of prosthetic clinics. We planned to recruit five clinics through professional association newsletters, conference presentations, and professional networks. Clinic staff were responsible for all aspects of PRO data collection; they mailed or faxed scannable forms to the university-based investigators for data management and compilation of comparative reports. The university’s Institutional Review Board (IRB) approved the study, and participating clinicians provided verbal consent to participate. The IRB approved a waiver of consent for patients, as completing outcome measures was deemed to pose no greater than minimal risk and the PRO data received by the investigators were de-identified, containing no protected health information.
Patients
In exchange for external QI facilitation, clinics agreed to collect PRO data at admission, after device delivery, and at 2-month follow-up as part of routine practice from English-speaking patients with lower limb amputation aged 18 years or older. Eligibility criteria focused on patients who were receiving a new prosthesis or socket replacement; patients who received socket adjustments were ineligible. We planned to provide reports to clinics after they each collected data from 100 patients.
Instrument
The Orthotics Prosthetics Users’ Survey (OPUS) is a PRO measure tailored to prosthetics and orthotics patients; it measures lower limb functional status, quality of life, and satisfaction with device and services. 9 It was developed using contemporary psychometric methods and has been evaluated in the United States,6,10 Central America, 11 and Europe.12,13 OPUS is designed to collect data from children and adults who use orthoses or prostheses. 9 It is one of a limited number of measures designed specifically to assess patient satisfaction with prosthetic and orthotic services and devices7,10 as required by ABC.
Procedures
As part of external facilitation, we provided clinicians and administrative staff with an orientation to QI concepts and the project’s procedures. Each clinic determined procedures suited to their particular situation, including how to identify eligible patients, track patient appointments, and submit de-identified data to research staff.
Clinic staff asked patients to complete the OPUS at three time points, while prosthetists completed a one-page document to report Medicare’s Functional Classification Level, 14 also known as K-levels, and the patient’s demographic characteristics. Patients completed OPUS forms during their appointment or at home, depending on the procedures at each clinic. Clinicians instructed patients to mail forms to the clinic using postage-paid envelopes or return completed forms at subsequent appointments.
QI facilitation
The research team provided external facilitation of QI activities to each clinic. As part of the project launch, we discussed clinicians’ experience collecting PRO data and accreditation surveyor feedback. We provided an orientation to QI and the role we would serve. We encouraged clinicians to view an online QI educational program tailored to prosthetists titled “Quality Improvement: What Is It and How Do We Use It?” (https://northwesternuniversity.adobeconnect.com/_a826375917/p4ehdr8ob59/). The presentation provided instruction on implementing a QI project. We maintained weekly contact with clinics to monitor data collection and provide technical support. We planned an extended meeting when a clinic submitted data from 100 patients, after which we planned to share aggregate OPUS results plotted across time by level of amputation (above knee vs. below knee) and primary etiology (trauma vs. vascular) for each site and anonymously across all sites.
During QI consultation, we helped clinic leaders identify areas of concern that were considered amenable to change, feasible to address, and measurable over time. Once clinicians defined a quality measure to address issues they deemed important, we facilitated planning of a QI project by completing a root cause analysis 15 and developing an action plan, taking into consideration patient, clinician, and organization resources. Root cause analysis examines why a process fails or is suboptimal and develops a plan to prevent it from recurring.
We planned to conclude external facilitation approximately 1 year later when we would then share a report based on the aggregate data collected through the course of the project. Clinic leaders would report on their progress in implementing QI programs and their experience incorporating OPUS into routine clinic operations.
Results
Facilities
Seven prosthetic clinics from five states participated in the study. We recruited clinics over 2 years and exceeded the goal of five clinics, as the early-starting clinics struggled with data collection. Using classifications from the United States Bureau of the Census, 16 six clinics were located in urbanized areas, defined as an area populated by 50,000 or more people, and one was located in an urban cluster, an area with 2500–50,000 people. 17 Table 1 shows that site ownership, clinic setting, staffing levels, clinical experience, and education varied across participating clinics.
Facility characteristics.
FTE: full-time equivalent employee; PTE: part-time equivalent employee; urbanized area: 50,000 or more people; urban cluster: 2500–50,000 people; CP: certified prosthetist.
Facilities recruited late in study period.
Patients
No clinic met the initial goal of collecting complete data from 100 patients. Overall, clinicians collected data from 250 patients, resulting in 148 complete data sets. Patients completed 78% of the intake forms, 55% of the device delivery forms, and 49% of the follow-up forms; clinicians completed 98% of the demographic forms. Table 2 shows patient demographic characteristics for the four clinics that were able to collect demographic data on at least 10 patients.
Patient demographic characteristics by facility.
“Complete data sets” are those that contained the clinician and intake forms along with either the delivery or follow-up form.
QI activities
We experienced a slower-than-expected accrual of data across all sites. Hence, we modified our original goal to 10 complete data sets which included three PRO forms and the clinician form for each patient as a time to provide feedback of aggregate data. However, the small number of complete data sets collected by the end of the second year of the study led us to modify the goal again and include records that contained the clinician and patient PROs at intake along with either the delivery or follow-up forms. Three clinics met this revised goal, with the percentage of complete data sets ranging from a low of 52% at Facility D to a high of 75% at Facility B. During meetings with each clinic’s staff, we provided an overview of preliminary data to illustrate the feedback they would receive once more data were collected at each site. We used a root cause analysis approach to address data collection challenges. Ultimately, none of the other four clinics met data collection goals and did not participate in QI consultations or projects.
We planned another consultation when a clinic submitted 20 complete data sets: two clinics met this goal and a third submitted 14 complete data sets. Two facilities initiated QI projects, while the third clinic withdrew because of staff workload.
We encouraged the two clinics that implemented a QI project to define quality indicators that were relevant to their situation. With external facilitation, both defined (1) declining functional status, (2) declining service satisfaction, and (3) less than “satisfied” ratings on any item as quality indicators. We helped the clinics develop a semi-structured interview to guide discussion with patients about issues related to satisfaction or function. We prospectively alerted clinics when patients returned follow-up OPUS forms that revealed declining function or service satisfaction. Of the 39 follow-up forms received during the monitoring period, 17 patients had declining function or service satisfaction scores. Clinic leaders reviewed patient-level data to determine whether a follow-up interview was warranted. Clinic leaders reviewed the percentage of patients who declined in function or quality of life or who had ratings less than “agree” for satisfaction of service items to determine what QI activities were warranted. We provided benchmark data to the three clinics. We also produced a final report for the clinic that collected data but did not participate in a QI project.
Clinician feedback
Qualitative feedback from clinicians suggested that they understood the importance of monitoring patient progress and viewed participation in the project as an opportunity to grow professionally, make positive changes within their practice, and provide patients a voice about the quality of their services. They found the combination of function, quality of life, and satisfaction with the device and services to be unique and valuable information. One clinical manager noted that her department chose to participate in order “to have richer data to show whether the services we provide are actually improving patients’ function and participation.” Another clinician concurred, stating she was “pleased to document successes and give patients a chance to specify where we can improve.” Clinicians wanted to know how their site performed in comparison to other participating clinics. They believed they better understood the value of documenting outcomes as a means of justifying services.
Some clinicians were surprised by the time needed to collect data, while others were aware that response rates from patient surveys are typically low. “I think it is difficult to get any patient (or customer) to return surveys. The hard part is getting the patient’s attention that this survey is worth filling out.” Maintaining clinician enthusiasm was also an issue, as one clinician pointed out that “keeping a patient survey a priority has been difficult over a long period of time.” Clinicians cited time constraints and competing priorities as major barriers in data collection:
When I have a patient here for an appointment, I am focused on fitting the device and providing all of the education that goes along with it. I focus on getting all of our financial documents in order for the patient, and in trying to remember all of the steps required to fit and deliver a device, I am often not thinking about surveys.
Clinicians reported that patients had varied experiences completing study instruments. One clinician noted, “Patients respond well when we explain that this survey helps us to know whether our devices are really helping our patients.” One clinician speculated that data collection prompted patients to reflect and evaluate their prosthesis in relation to their functional level and quality of life, which is different from being asked only about satisfaction with services. Some had a different experience, reporting that patients expressed negative opinions of the time required to complete the survey, voiced concerns about revealing personal information, and expressed confusion over being asked to answer similar questions repeatedly. Clinicians reported having to spend time explaining the importance of data collection and making an ongoing effort to engage patients in providing feedback. Patient burden was increased somewhat by the collection of additional questions along with the OPUS modules that we used to co-calibrate with OPUS data.
Clinicians offered several suggestions to facilitate interpretation of the PRO data. They requested that we calculate minimal clinically important differences so that they could evaluate the magnitude of functional status, quality of life, and satisfaction changes. They thought it was important to document clinician opinions on changes in functional level. Several were concerned about the number of forms and suggested eliminating items to reduce patient burden. They thought that risk-adjusted outcomes would be helpful to account for variations across clinics in terms of patients’ age, amputation level, and etiology. A large comparative data set would provide a useful management tool by allowing clinicians to benchmark their performance, both within and across clinics. They also recommended several changes in report formats to enhance ease of data interpretation, including graphical displays and presentation of data from peer facilities.
Discussion
Implementing routine data collection to monitor patient outcomes proved to be a challenging endeavor for all clinics regardless of setting and external facilitation by the research team. Where modest success was achieved, it required exceptional planning, persistence, and tenacity on the part of management, clinical, and administrative staff. Four of the seven clinics were unable to sustain data collection. Barriers to participation included clinical and administrative staff turnover and prolonged job vacancies, few patients meeting inclusion criteria, competing clinical and administrative priorities, local coverage determination audits, closing of a clinical site, and difficulties tracking patients. More broadly, several factors may limit the adoption of evidence-based and QI practices in prosthetics including limited information regarding outcome measure sensitivity, limited time learning evidence-based practice in graduate programs, and a business model that emphasizes delivery of a device rather than service as a clinical care provider.
One clinic started data collection on 10 patients, allowing us to include its data in the demographic analysis, but was unable to submit follow-up surveys. One clinic ceased data collection before the first consultation meeting, citing limited administrative resources and competing clinical priorities. Two others began participating late in the project and did not collect sufficient data for a consultation meeting due to lower-than-expected patient volume and limited time to collect data.
Identifying eligible patients and integrating PRO collection into routine clinic operations demanded keen organizational planning and vigilant monitoring of appointments. Clinicians stressed the importance of having a designated person to assume ownership of data collection. The time devoted specifically to coordination varied from half an hour to 5 h per week, reflecting patient volume. Coordination involved screening new patients for eligibility, monitoring enrolled patients’ return appointments, and contacting patients who did not return forms. These tasks were more challenging at clinics that used paper-based scheduling and medical records: “Our office is mainly paper-based. Organization has been surprisingly difficult to maintain,” remarked one clinician. Flagging return appointments required vigilance, as the time from intake to delivery varied by many months at some clinics. Two of the clinics spent considerable time developing procedures that allowed staff to identify and track patients.
While clinicians agreed that PRO measures might play a vital role in improving organizational performance, several voiced concern that patient expectations may not be commensurate with the type of device ordered by the physician, deemed appropriate by the clinician, or covered by insurance. Patients may be dissatisfied with a low technology device that adequately meets their needs if they believe that a high technology device reflects a higher standard of care. Clinicians expressed concern that they could not address some satisfaction issues over which they had limited control, for example, device performance limitations due to patient factors, psychological issues, comorbid conditions, pain tolerance, insurer limitations, and expectations of cosmetic quality.
During the course of this project, the Office of the Inspector General of the US Department of Health and Human Services initiated compliance audits of the CMS’ local coverage determinations for lower limb prostheses.18–23 Compliance audits required many staff hours to retrieve documentation and justify claims. Two participating providers were unable to sustain data collection, and one cited audit response requirements as the reason for lower-than-expected patient recruitment.
Low numbers of PROs submitted during the last year of the project hampered completion of QI projects. Staff members used their judgment to decide whether decreases in function or quality of life, or low satisfaction scores warranted follow-up interviews. For example, low satisfaction with the cost of a device was deemed out of the control of the clinic; thus, the clinic decided not to pursue this issue. Clinicians also identified time constraints and the need to prioritize clinical and administrative work over QI as a barrier to making reminder calls for OPUS forms. However, clinicians were sometimes surprised at which patients reported declining scores and thought follow-up contact provided a good opportunity to address problems before they escalated.
Despite these challenges, some clinics persevered with data collection, employing a variety of strategies. Several clinics capitalized on strong patient relationships, making a concerted effort to engage patients in the assessment process, emphasizing the value of patients’ input, and expressing appreciation for their cooperation. Clinics were successful when they designated an administrative staff member as the point person and allowed time to organize and coordinate data collection, including tracking patient appointments, making follow-up phone calls, and mailing replacement forms.
A larger clinic with multiple prosthetists restructured weekly staff meetings to preview upcoming appointments to identify eligible or returning patients. The clinic used participation metrics in staff members’ annual performance appraisals to emphasize the importance of QI and promote data collection. As a method to encourage higher levels of form completion, one facility scheduled monthly drawings in which the clinician with the highest number of completed forms won a gift card. A clinic that enrolled late in the project integrated data collection into its electronic health record. Clinicians received automatic alerts when a new patient met eligibility criteria and active patients were requested to return so that the clinician could ask patients to complete PRO measures promptly. Clinics that scheduled follow-up appointments shortly after prosthesis delivery submitted a larger number of completed delivery and follow-up forms than those with long or inconsistent follow-up intervals.
Readers should note several study limitations. We recruited a limited number of self-selected clinics from the Midwestern United States. The extent to which we reached saturation of implementation issues is unknown without input from clinics in other regions. Clinics affiliated with national networks did not participate. It may be that larger organizations with electronic health records and a commitment to QI experience different challenges. We focused only on patients receiving new or substantially modified lower limb prostheses; clinics serving patients with different prosthetic–orthotic issues may encounter different challenges.
Conclusion
Outcome monitoring and QI initiatives require a high level of leadership support, organizational commitment, and sophisticated planning to sustain PRO data collection and QI activities as part of routine clinic operations. External facilitation is critical in helping clinics initiate data collection and QI planning, but insufficient in isolation, for maintaining them once past the initiation phase. Multiple opportunities exist to develop greater sophistication in quality management in prosthetic–orthotic care.
Footnotes
Acknowledgements
The authors are indebted to the staff members and patients at the participating facilities.
Author contribution
All authors contributed equally in the preparation of this manuscript. Copies and scoring instructions of the Orthotics Prosthetics Users’ Survey are freely available.
Declaration of conflicting interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: We certify that no party having a direct interest in the results of the research supporting this article has or will confer a benefit on us or on any organization with which we are associated and, if applicable, we certify that all financial and material support for this research and work are clearly identified.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was supported by the National Institute on Disability, Independent Living, and Rehabilitation Research through the Rehabilitation Engineering Research Center on Prosthetics and Orthotics (H133E080009) awarded to Northwestern University (PIs: Steven Gard and Stefania Fatone).