Abstract
Background
Previous research documents suboptimal preoperative or postoperative care for patients undergoing surgery. However, few existing quality measures directly address the fundamental element of surgical care: intra-operative care processes. This study sought to develop quality measures for intraoperative, preoperative, and postoperative care for carpal tunnel surgery, a common operation in the USA.
Methods
We applied a variation of the well-established RAND/UCLA Appropriateness Method. Adherence to measures developed using this method has been associated with improved patient outcomes in several studies. Hand surgeons and quality measurement experts developed draft measures using guidelines and literature. Subsequently, in a two-round modified-Delphi process, a multidisciplinary panel of 11 national experts in carpal tunnel syndrome (including six surgeons) reviewed structured summaries of the evidence and rated the measures for validity (association with improved patient outcomes) and feasibility (ability to be assessed using medical records).
Results
Of 25 draft measures, panelists judged 22 (88%) to be valid and feasible. Nine intraoperative measures addressed the location and extent of surgical dissection, release after wrist trauma, endoscopic release, and four procedures sometimes performed during carpal tunnel surgery. Eleven measures covered preoperative and postoperative evaluation and management.
Conclusions
We have developed several measures that experts, including surgeons, believe to reflect the quality of care processes occurring during carpal tunnel surgery and be assessable using medical records. Although quality measures like these cannot assess a surgeon's skill in handling the instruments, they can assess many important aspects of intraoperative care. Intraoperative measures should be developed for other procedures.
Payers, policymakers, and surgeons are paying increasing attention to the quality of the surgical care provided in the USA. Better surgical care increases the likelihood that patients will experience favorable outcomes and be free of complications. From a societal standpoint, this may yield economic benefits by reducing disability and avoiding the medical care costs associated with complications. Consequently, efforts to measure and improve the quality of surgical care are becoming more widespread. The Surgical Care Improvement Project now sets national standards for process measures such as prophylactic antibiotics, prophylaxis against deep venous thrombosis, use of preoperative beta-blockers, and prevention of ventilator-associated pneumonia [12]. The high cost attributed to complications has convinced a number of insurance companies to subsidize surgeons' efforts to monitor and improve quality [5]. Similarly, Medicare has experimented with pay-for-performance incentives for physicians [12]. In the future, these national efforts to evaluate surgical quality of care are likely to expand.
To measure the quality of surgical care, specific measures are needed. Process-oriented measures identify care processes that patients should or should not receive under specified circumstances. The purpose of such measures is to make existing standards of care explicit and measurable. Generally, quality measures delineate basic standards of care, in contrast to guidelines, which describe both minimum standards and best practices [26].
Most process measures pertaining to surgical procedures address the appropriateness of surgery (i.e., whether the surgery should be performed or not), preoperative care, or postoperative care. Very few existing measures address the fundamental element of surgical care: care during an operation. Yet good intraoperative care is critical to achieving favorable surgical outcomes. Some important aspects of intraoperative care are challenging to measure, such as the surgeon's dexterity in using the instruments or handling tissues. Nevertheless, other important aspects of intraoperative care can be assessed by reviewing operative reports and related records.
Our objective in this study was to develop a set of quality measures that can be used to assess, via medical record review, key aspects of intraoperative care as well as preoperative and postoperative care provided to patients undergoing carpal tunnel surgery. We chose to focus on carpal tunnel release surgery for two reasons. First, carpal tunnel syndrome causes substantial disability among working-age adults, which means that improving patient outcomes is likely to benefit both the affected individuals, through reduced symptoms, and their employers, through reduced absenteeism, improved productivity at work, and reduced healthcare expenditures [6]. Second, studies suggest that the outcomes of carpal tunnel surgery are not always optimal, with unsatisfactory results occurring in up to 10–15% of patients [1,8,13,14]. Some variability in outcomes appears attributable to patient characteristics [1,11]. However, some poor outcomes may be due to patients receiving less than optimal surgical care.
Quality measures should be based on high-quality studies whenever possible. However, randomized controlled trials do not exist for most healthcare processes, particularly many aspects of surgical care including intraoperative processes [2,25]. We, therefore, developed these measures using a variation of the RAND/UCLA Appropriateness Method. A particular strength of this method is its ability to consider available evidence and overcome important gaps by rigorously synthesizing the experience of expert clinicians [7]. Syntheses of clinical expertise are a valid and important form of evidence, as demonstrated by the fact that better adherence to measures developed using this method have been associated with improved patient outcomes in multiple studies. For example, judgments about the appropriateness of carotid endarterectomy were consistent with the findings of a subsequent randomized trial [9,10,22]. For arthroplasty of the knee and hip, judgments about the appropriateness of surgery were associated with improved quality of life [20].
Materials and Methods
Quality measure development follows a three-step process: (1) creating draft measures by integrating guidelines and literature, (2) refining and selecting final measures, and (3) testing the measures against a data source. Here, we report on the first two steps and the results of early pilot testing. This was one aspect of an effort to develop quality measures for carpal tunnel syndrome; other measures have been reported separately [16,17,21].
Developing Draft Measures
Developing draft measures was an iterative process involving collaboration among two hand surgeons, one general surgeon with experience developing measures of surgical quality, and two internists with expertise in quality measurement. First, we conducted a general literature search on carpal tunnel syndrome, searched MEDLINE, National Guidelines Clearinghouse, specialty society websites, and other sources of publicly available guidelines, and accessed personal reference collections. Team hand surgeons reviewed the guidelines and literature, chose key intraoperative and perioperative care processes that are likely to affect patient outcomes or that are widely recommended, then wrote draft measures.
Directed MEDLINE searches were conducted to identify evidence pertinent to the draft measures (search terms: surgery for carpal tunnel syndrome OR median neuropathy, additional MeSH terms used: surgical procedures, operative). For 845 citations identified, hand surgeons sequentially reviewed titles, abstracts, and articles to assess relevance to each draft measure. Draft measures were then refined, added, and deleted.
Next, the hand surgeons summarized, for each draft measure, the evidence supporting the relationship between the care process and patient outcomes, emphasizing the highest quality evidence identified. Given most evidence was not high quality, they used a simplified classification scheme: level 1, randomized controlled trial; 2, observational study; and 3, case series and expert opinion. Where level 1 evidence was not available, the summary described a chain of evidence or clinical rationale, including recommendations from recent guidelines.
Refining and Selecting Measures
Methods for refining and selecting quality measures were derived from the RAND/UCLA Appropriateness Method, a multidisciplinary, two-round, modified-Delphi process that enables researchers to obtain a quantitative assessment that reflects the judgment of a group of experts [7]. This well-established method (described below) has been used to develop quality measures for surgical care and determine the appropriateness of many surgical procedures. The method has reproducibility consistent with that of well-accepted diagnostic tests like screening mammography—i.e., separate panels examining the same topic have produced similar recommendations (kappas 0.51–0.83) [15,23]. Further, the measures developed using this method have been shown to have content, construct, and predictive validity, as evidenced by the association between adherence to the measures and improved patient outcomes in multiple studies. Additional details about this method have been published previously [7,22].
For this study, we selected a panel with 11 members: four hand surgeons, two orthopedists, an occupational medicine physician, a neurologist, a physiatrist, a family physician, and a physical therapist. Including a variety of clinical backgrounds is essential to the RAND/UCLA panel process because it increases the range of issues that the panel considers and discusses. National specialty societies recommended leaders in each specialty, and then we selected panelists representing a variety of geographic locations, expertise, and practice settings. Several surgeons were involved in the guideline development effort by the American Academy of Orthopedic Surgery (AAOS) [3,4].
The first round of ratings involved having panelists rate the measures at home. Panelists received the evidence summaries, draft measures, ballots, and instructions. During the second round, panelists met in person and research team members moderated discussions of each draft measure, the evidence, and first-round ratings. This panel method allows different attitudes to be expressed and contend with one another in order to allow true agreement or disagreement to emerge. This is in contrast to a consensus–panel method, which typically forces the group to reach agreement. Panelists suggested modifications to definitions of key terms and measures; these were adopted when a majority voted to do so. After all opinions had been voiced for a measure, all panelists marked private, equally weighted ballots.
For both rounds, panelists rated validity, feasibility, and importance on 9-point scales (9=highest). Validity meant: (1) adequate scientific evidence or professional consensus exists to support a link between the performance of care specified by the measure and improved clinical outcomes; and (2) based on the panelists' professional experience, health professionals with significantly higher rates of adherence to a measure would be considered higher-quality providers. Panelists were also instructed that, for a measure to be valid, both the care itself and the documentation of that care in the medical record must be important reflections of quality. Following standard procedures for this panel method, we interpreted validity ratings as follows: valid=median of 7–9 without disagreement; not valid=median of 1–3 without disagreement; uncertain validity=median of 4–6, or any median with disagreement. Disagreement was defined as three or more panelists rating in the 1–3 range and three or more in the 7–9 range [7].
We also included feasibility and importance to enable future users to prioritize the measures. Feasibility meant the potential ability to evaluate adherence to the measure using medical records as the only source of information on the care provided. Measures were considered infeasible if the median rating was below 4. Importance meant the magnitude of the potential effect on patient outcomes; there was no minimum threshold for importance.
Pilot Testing
After identifying measures meeting the validity and feasibility criteria, RAND staff developed a detailed tool for scoring them and pilot tested the measures and tool in a large workers' compensation provider organization (Kaiser Permanente Northern California Regional Occupational Health) and in a large workers' compensation insurance company (the California State Compensation Insurance Fund) [18]. Six nurses and one physical therapist, who routinely perform claims reviews within each organization, underwent a 2-day training in the use of the tool and scored several practice cases. Finally, they reviewed records for 28 patients who had been diagnosed with carpal tunnel syndrome (CTS) or conditions often confused with CTS. During the training and pilot testing, these abstractors provided feedback on the tool. The pilot test activities were approved by each of the institutional human subjects' protection committees; informed consent was not required.
Results
Panel Evaluation of Measures
We developed 24 draft measures and the panel made changes to all but two of them, including splitting one measure into two. Two intraoperative measures did not meet validity criteria. Panelists deleted one postoperative measure because it addressed a rare situation. The 22 remaining measures (88%) met the validity and feasibility criteria. For the intraoperative and perioperative measures, respectively, Tables 1 and 2 list the measures themselves. Tables 3 and 4 list median ratings for validity, feasibility, importance, and the highest level of supporting evidence.
Intraoperative care measures meeting validity and feasibility criteria
Preoperative and postoperative care measures meeting validity and feasibility criteria
Panelists' ratings and evidence level for intraoperative care measures
Ratings ≥7 indicated panelists thought the measure was valid
Ratings ≥4 indicated panelists thought the measure was potentially feasible
Level of evidence: 1 randomized controlled trial, 2 observational data, 3 case series or expert consensus
Panelists' ratings and evidence level for preoperative and postoperative quality measures
Ratings ≥7 indicated panelists thought the measure was valid
Ratings ≥4 indicated panelists thought the measure was potentially feasible
Level of evidence: 1 randomized controlled trial, 2 observational data, 3 case series or expert consensus
N/A not applicable
The
Pilot Testing
Of 28 patients in the pilot study, 14 had CTS and underwent carpal tunnel surgery for the first time during the study period; one additional patient had undergone surgery in the past.
After reviewing the operative reports for the intraoperative measures, the abstractors reported that several measures were challenging to score because the reports used unusual and highly variable terms to describe the operations. To investigate these concerns, a general internal medicine physician performed spot implicit reviews of operative reports, observed the substantial variability in terminology, and concluded that individuals with specific training or experience writing or reviewing these types of operative reports would be required to score the intraoperative measures. The seven abstractors reported no feasibility concerns for the pre- and postoperative care measures.
Discussion
We have developed nine measures that can be used to assess the quality of the intraoperative care provided to patients undergoing carpal tunnel surgery, as well as 11 measures pertaining to pre- and postoperative care. A multidisciplinary panel of national experts in carpal tunnel syndrome, including several hand surgeons and orthopedists, rated these measures as valid reflections of quality and feasible for use with medical record review.
The preoperative measures are generally intended to ensure that surgeons are able to make an accurate diagnosis and adequately assess the potential risks and benefits of carpal tunnel surgery. Because an incorrect diagnosis is a common reason that patients do not improve after carpal tunnel surgery, these preoperative measures are likely to reduce unnecessary and inappropriate operations [24]. In contrast, the intraoperative measures are designed to reduce the risks of serious surgical complications and persistent CTS symptoms. Damage to the median nerve (or its palmar cutaneous branch) is a particularly severe and disabling complication that several measures (#1, 3, 4, and 6–9) are designed to prevent. The remaining intraoperative measures (#2 and 5) are intended to reduce the chance that symptoms will not improve after surgery, also a fairly common occurrence [24]. Several of the postoperative measures assess whether providers identify and respond to these types of problems appropriately.
We know of very few other existing quality measures that focus on the quality of intraoperative care processes, regardless of the type of procedure. Assessing surgical appropriateness or outcomes is far more common. For example, a 2006 systematic review of quality measures for colon cancer identified several that assess the appropriateness of surgery, the type of procedure chosen, recurrence rates, complication rates, and mortality rates. None of the measures addressed specific intraoperative care processes [19].
The reason for this dearth in process measures for intraoperative care is unclear but it could relate to concerns about feasibility. Certainly, there are many critical aspects of the art of surgery that cannot be assessed by reviewing medical records, such as a surgeon's dexterity or meticulousness in achieving hemostasis. Nevertheless, operations are comprised of individual steps that surgeons can choose to perform or forgo, the purpose of operative reports is to document the steps taken, and the steps taken or foregone influence patient outcomes. For example, the number of lymph nodes harvested during the resection of colon cancer has been shown to influence the likelihood of recurrence [19]. The intraoperative measures, we developed comprise discrete individual steps in the performance of carpal tunnel surgery. Each of these steps is, in the opinions of our national experts, likely to influence specific patient outcomes, for reasons outlined above and explained in detail in the
The ultimate test of quality measures' feasibility involves applying them to medical records, and here we must acknowledge that our work to date is incomplete. Pilot testing the intraoperative measures enabled us to determine that nurses and general internists can have difficulty scoring them because some operative reports describe the procedures using unusual or variable terms. Our panelists, who included several surgeons, were generally very confident that the measures would be feasible, as evidenced by the fact that median feasibility scores were 7 or higher for seven of the nine intraoperative measures. Future work will need to confirm the feasibility of these measures by having surgeons who operate on the hand, or perhaps specially trained nurses, score them. If the intraoperative measures must be scored by surgeons rather than nurses, this will increase the cost of assessing quality of care. However, operative reports are concise documents that are relatively easy to obtain and review.
We are also planning future work to confirm the validity of the measures we have developed. For quality measures, the ultimate test of validity entails assessing whether better adherence is associated with better patient outcomes. However, most quality measures in wide use today have yet to be tested in this manner. We have developed a project that would compare adherence to these measures with patients' clinical outcomes as well as assess the relationship between quality, outcomes, and the costs of care and disability due to CTS.
Once the feasibility and validity of these measures has been confirmed, there are several ways that they could be used. Individual surgeons can use these measures to evaluate the quality of the care they provide. In some fields, board recertification is now contingent upon providers engaging in such activities. Practices with multiple surgeons can evaluate quality for the practice and, if warranted, develop an infrastructure that supports improvement. Such organizational efforts are particularly likely to be effective because they leverage the contributions of many individuals, and enable systems to be established that make adherence simpler. Payers might consider using these measures as a basis for referring patients to higher-quality providers, or as a basis for offering higher-quality providers greater remuneration. Because it is easier for payers to obtain and review operative reports than the entire medical record, the payers may find the intraoperative measures more feasible than the pre- and postoperative ones, which will require access to clinic notes. Finally, researchers can identify factors associated with better quality.
For the purposes of improving care, quality measures are most useful when they address basic standards of care that have not already been widely implemented. We suspect that baseline rates of adherence will be higher for some of these measures than for others. For example, because Medicare and other payers require histories and physical examinations to include specific elements, we suspect that the stipulations of measure #11 will usually be met. In contrast, we expect that the incision location and release of the transverse carpal ligament (measures #3 and 4) may not be documented consistently. Future research will be needed to confirm or refute these hypotheses.
These measures also have other limitations. As noted, not all important aspects of care for patients undergoing carpal tunnel surgery are amenable to direct measurement. Further, unique clinical circumstances can warrant exceptions to a measure. Justifiable exceptions are not problematic so long as they are rare and randomly distributed among populations of patients. The literature examining these practices is rather limited, and most of the measures are based on expert consensus. In contrast to some fields, such as cardiology, both musculoskeletal disorders and surgical care suffer for a lack of large, high-quality randomized controlled trials. Trials are not possible for all important care processes, however. For example, it would be unethical to randomly assign an incision location that can damage to the palmar cutaneous branch of the median nerve. The panel method we used offers a rigorous and ethically acceptable approach to determining the right care in such clinical situations.
In conclusion, this project has developed a set of measures that can be used to evaluate the quality of the care provided to populations of patients undergoing carpal tunnel surgery. The several measures focusing on intraoperative care processes represent an important advance in the science of measuring quality of care. These measures will be useful in efforts to improve quality of care for patients with carpal tunnel syndrome, whether initiated by providers, medical groups, payers, or policymakers. Intraoperative measures should be developed for other common operations.
Footnotes
Acknowledgments
The assistance of Christine Baker, MS, Executive Director of the California Commission on Health and Safety and Workers' Compensation, made this project possible. RAND staff members Rebecca Shaw, Belle Griffin, Liisa Hiatt, and Krasimir Karamfilov provided assistance with the research. We are grateful to the following individuals for their assistance: D. Lachlan Taylor, JD (Workers' Compensation Judge, California Commission on Health & Safety and Workers' Compensation), Doug Benner, MD (Medical Director, Regional Occupational Health, Kaiser Permanente Northern California), Gideon Letz, MD (Medical Director, California State Compensation Fund), Bernyce Peplowski, MD (past Medical Director, Zenith Insurance), Kathryn Mueller MD, MPH (Director for the Colorado Division of Workers' Compensation), and Jane Dereberry, MD (past Chairman, Concentra Occupational Health Research Institute, Austin, TX).
This project was jointly supported by the California Commission on Health and Safety and Workers ‘Compensation, a state-sponsored joint labor-management body charged with overseeing the health and safety and workers ‘compensation systems in California and recommending administrative or legislative modifications to improve their operation, and by Zenith Insurance, a workers’ compensation insurance company based in Woodland Hills, California. The RAND/UCLA team was fully responsible for the design and execution of this study, and the reporting of these results; the funders played no role in these activities.
Appendix
Intraoperative Care Measures Meeting Validity and Feasibility Criteria