Abstract
Introduction
With a reported incidence of up to 624 per 100 000 patients per year, pediatric hand fractures are a common presentation to emergency departments. 1 In Canada, these fractures are commonly referred to pediatric plastic surgeons (herein referred to as “hand surgeons”), yet most heal well without surgical intervention.2–4 This phenomenon has been observed outside of Canada as well.5–10 A recent scoping review showed that pediatric hand fracture referral patterns varied widely; objective measures and justification for referral criteria were also identified as a major knowledge gap. 11 In addition, a retrospective cohort study of over 1100 pediatric hand fractures presenting to this Canadian tertiary care center demonstrated that fewer than 10% required any surgical intervention. 2
In response to these findings, we developed a clinical prediction rule for pediatric hand fracture triage, the Calgary Kids’ Hand Rule (CKHR) (Figure 1). Clinical prediction rules are decision-making aides generated from clinical data that can be used to predict the presence or absence of a condition, determine a prognosis, and guide management. 12 The CKHR was derived from a retrospective cohort of pediatric hand fractures referred to our center, using multivariable regression analysis. 13 Derivation of the CKHR identified 6 significant clinical or radiologic predictor variables. During its derivation, the CKHR performed well, with 96% sensitivity and 45% specificity. Internal validation with bootstrapping also demonstrated good performance of the model (C-statistic = 0.88, calibration curve P = .94). 13 These predictors were incorporated into a user-friendly checklist with tick-boxes for each predictor. If any of the tick-boxes are marked, the fracture is predicted to be complex. If none are marked, the fracture is predicted to be simple. The premise of the CKHR is that accurate prediction of simple fractures could safely bypass the surgeon assessment and go directly to a nonsurgeon follow-up provider. For example, in our local context, this provider would be an occupational hand therapist.
The version of the Calgary Kids’ Hand Rule (CKHR) embedded within existing fax referral forms for this prospective evaluation.
The aim of this study was to prospectively validate the performance of the CKHR in a real-world setting and identify areas for improvement. We also looked at interrater reliability between referring physicians and surgeons and assessed user satisfaction with the CKHR through a survey.
Methods
Study Design and Data Collection
This prospective cohort study was a quality improvement initiative, and the local Research Ethics Board granted approval under the ARECCI guidelines (A pRoject Ethics Community Consensus Initiative). The study population included pediatric patients (< 18 years) referred to our hand surgery clinic with completed CKHR forms over an 18-month period (April 2019 through September 2020). Patients with hand injuries that did not involve fractures were excluded. Hand fractures were defined as fractures distal to the wrist, involving either the metacarpals or phalanges. Referring physicians include emergency physicians at our children's hospital site, emergency physicians at adult hospital sites, physicians at urgent care centers in the community and surrounding areas, and family physicians in their offices. The latter referral source (family physicians) did not have access to the CKHR and was not included in this project. Currently at our site, referring physicians (emergency physicians and urgent care physicians) will contact the on-call hand surgeon directly for cases that they believe need to be treated on an emergency basis. For urgent referrals (to be seen in the next 72 h) a referral form is faxed to the clinic which is triaged each morning by a hand surgeon. The CKHR was embedded into the existing fax referral form used by referring physicians (Figure 1). These physicians were asked to complete the CKHR form for each fracture they assessed, noting the presence or absence of the 6 predictors based on clinical examination (open fracture, malrotation) or on standard hand X-rays with 3 views (angulation, displacement, condylar involvement, dislocation/subluxation). For patients with multiple fractures, referring physicians were asked to complete a single form for all the fractures and to mark the presence of any predictors on any of the fractures. To ensure that the forms were being completed, referring physicians were also asked to tick a box indicating a simple fracture if none of the predictors were observed. It was emphasized that the CKHR was to be applied prior to any treatment. The CKHR and its purpose were reviewed in advance with referring physicians through their educational rounds and through newsletters. Posters explaining the use of the form were placed in each referral center (Figure 2). Fractures with one or more of these predictors marked on the form were predicted to be complex. All others were predicted to be simple. Referral practices and patient care were not altered based on predictions drawn from the CKHR. In particular, the referring physicians were free to attempt closed reductions as they saw fit. The data collected from these referrals represented predicted outcomes.
Poster provided to referring physicians with examples and descriptions of complex and simple fractures.
After referral, hand surgeons saw the patients in the clinic, independently completed the CKHR (blinded to the assessment completed by the referring physician), and provided standard care. Each patient's electronic medical record was reviewed 3 months after care began to determine the observed outcome: specifically, whether a patient received closed reduction, surgical intervention, or close follow-up (more than 3 plastic surgery clinic visits).
Primary Outcome—CKHR Performance
The primary study outcome was the predictive accuracy of the CKHR, determined by comparing predicted outcomes (based on the forms completed by referring physicians) to observed outcomes (clinical outcomes taken from chart review at 3 months). Predictive accuracy was quantified in terms of sensitivity (Sn), specificity (Sp), and concordance statistic (C-statistic). Sensitivity is defined as the number of accurately predicted complex fractures (true positives) divided by the total number of complex fractures (true positives + false negatives). Conversely, specificity is determined by the number of accurately predicted simple fractures (true negatives) divided by the total number of simple fractures (true negatives + false positives). The CKHR has been designed with a single-predictor cutoff to maximize sensitivity and thus minimize the number of missed complex fractures (false negatives). Naturally, this results in an over-prediction of complex fractures and consequent reduction of specificity. In this way, the CKHR prioritizes patient outcomes over efficiency gains. The C-statistic simultaneously considers the sensitivity and specificity of a clinical prediction rule, quantifying the relationship between the true positive (accurately predicted complex fractures) and the false positive rate (simple fractures predicted as complex). A C-statistic of 1.0 indicates a 100% true positive rate and a 0% false-positive rate. In contrast, a C-statistic of 0.5 indicates equal true positive and false positive rates (no better than random chance). Overall, an ideal clinical prediction rule possesses high sensitivity, high specificity, and a C-statistic close to 1.0.
Secondary Outcome—Interrater Reliability
Agreement between referring physicians and surgeons was examined as a secondary outcome. Interrater reliability between the 2 groups was calculated as percent agreement and kappa coefficient for each predictor and for the overall prediction of simple or complex.
To assess user satisfaction with the CKHR and to identify areas for improvement, an online survey was created for emergency physicians who used it. Results of this survey were reported as descriptive statistics (ie, proportions) and as qualitative themes. All statistical analyses were conducted using STATA (Version 17, StataCorp).
Results
Patient Population
Clinibase data revealed that there were approximately 1351 referrals to the pediatric plastic surgery clinic between April 2019 and September 2020 that included search terms related to hand injuries (hand, fracture, ligament, dislocation). Of these referrals, there were 475 patients with CKHR referral forms. After removing 46 patients with nonfracture hand injuries and another 64 with incomplete CKHR checklists (no boxes ticked), a total of 365 patients with pediatric hand fractures were available for analysis (Figure 3). There were 15 patients included with more than one fracture (only one form was completed per patient). The median patient age was 12 years with a range of 2 to 17 years. Most patients were male (n = 243, 67%).
Flow diagram representing the inclusion and exclusion of fractures for analysis in this study.
Predicted Outcomes
In the majority of completed referral forms, referring physicians reported zero predictors and thus the fractures were classified as simple (n = 201, 55%). The remaining fractures were predicted as complex (n = 164, 45%) as they were reported with at least 1 of the 6 predictors; of these, the majority presented with a single predictor (n = 127, 77%). A smaller number of complex fractures presented with 2 predictors (n = 33, 20%), and only 4 fractures were reported as having 3 or more predictors. Angulation was the most frequently observed predictor (n = 90, 25% of total fractures), followed by displacement (n = 36, 10%), malrotation (n = 23, 6%), open fracture (n = 20, 5%), dislocation (n = 18, 5%), and condylar involvement (n = 18, 5%).
Observed Outcomes
Observed outcomes, as determined by chart review, identified 260 simple fractures (71%) and 105 complex fractures (29%). Of the 105 complex fractures, the majority required closed reduction (n = 95, 90%) with a smaller number requiring surgery (n = 16, 15%) and only a few (n = 6, 6%) having more than 3 plastic surgery clinic visits. The total number of complex outcomes exceeded the number of complex fractures because some fractures had more than one complex outcome. The 16 fractures that required surgical intervention represented only 4% of all fractures.
Primary Outcome—CKHR Performance
There was an over-prediction of complex fractures by referring physicians using the CKHR, with 164 predicted versus 105 observed (Table 1). Sensitivity was 84%, with 88 of 105 complex fractures being accurately predicted. Conversely, 17 false negatives were observed (complex fractures that were incorrectly predicted to be simple). Specificity was measured to be 71%, accurately predicting 184 of 260 simple fractures. The CKHR also demonstrated good discrimination (C-statistic = 0.77).
Comparison of Predicted and Observed Outcomes. Predicted Outcomes were Determined by Referring physicians Completing the CKHR Forms. Observed Outcomes were Determined by Reviewing Patient Charts 3-Months Postinjury.
False negatives were reviewed in detail (Table 2). Of the 17 false negatives, nearly all of these (n = 15) were treated by the referring physician with a closed reduction in the emergency department. For 10 of the false negatives, it appears that the CKHR checklist may have been inappropriately applied to the postreduction X-rays. This was inferred in cases where predictors were observable on prereduction but not postreduction X-rays. Other reasons for false negatives included: failing to mark a predictor at all (n = 5) and failing to mark a predictor despite noting its presence elsewhere on the referral form (n = 2). In 4 of 5 cases where the referring physician failed to mark any predictor, the surgeon indicated at least one predictor as being present. Hand surgeons identified at least one predictor for 12 of the false negatives.
Review of False Negative Patients: Fractures Predicted as Simple That Were Deemed Complex by Outcomes.
Abbreviations: PIP, proximal interphalangeal; CRIF, closed reduction with internal fixation; ORIF, open reduction with internal fixation.
Secondary Outcome—Interrater Reliability
Hand surgeons also completed the CKHR form for each fracture, acting as a standard reference for the selections made by the referring physicians. Angulation was the least agreed-upon predictor (agreement = 84.1%), followed by displacement (88.3%). Agreement on the remaining predictors was high (range: 95.1-96.3%). Referring physicians reported a higher incidence of condylar involvement and malrotation. Conversely, surgeons reported a higher incidence of displacement and open fractures. Despite varying levels of agreement on the individual risk factors of complex fractures, overall agreement when predicting complex fractures was good (78.1%) (Table 3).
Interrater Reliability of Predictors as Assessed by Referring Physicians and Hand Surgeons.
Secondary Outcome—User Satisfaction and Survey Results
An online survey was conducted to collect feedback from emergency physicians regarding the CKHR, with a response rate of 43% (26 respondents out of 60 staff members surveyed). All respondents agreed the CKHR was easy to use, citing the clarity and simplicity (n = 8), tick-boxes (n = 6), and accompanying poster (n = 4) as strengths. Almost half of the respondents (n = 11) reported a level of uncertainty when marking the presence of certain risk factors, with condylar involvement (n = 4), open fracture (n = 3), and angulation (n = 3) being the least clear. The respondents also suggested various improvements to the referral form, including the addition of detailed descriptions for each predictor (eg, specifying the amount of angulation or displacement required) along with accompanying visual aids.
Discussion
Clinical prediction rules are decision-making aides based on statistical modeling that aim to streamline patient care while maintaining good outcomes. 12 They require careful design and calibration. If the sensitivity of the CKHR is too low, its predictions will underestimate the complexity of some hand fractures which could negatively affect patient outcomes. Conversely, if the CKHR is too sensitive (ie, lacks specificity), it will overestimate the number of complex hand fractures and negate its usefulness as a screening tool. It has been argued that prediction rules for children should be more assistive and less directive when it comes to clinical decision making. 14 This may be true for hand fracture referrals where clinical decision making must be considered alongside patient and caregiver values and preferences. The results of this prospective validation demonstrate the CKHR performed well and has the potential to simplify the care pathway for those patients that don't necessarily need to see a surgeon. However, we also identified the importance of continued education to ensure appropriate application by end users.
Over the past 10 years, as surgical wait times in Canada have come under scrutiny, it has become apparent that wait times to see pediatric surgeons for many conditions have increased dramatically. Furthermore, when looking at wait times for surgery, plastic surgery was among the top 3 subspecialties where surgeries were completed beyond target time windows. 15 The wait time to see a pediatric plastic surgeon at our hospital for nonurgent conditions is greater than 12 months. This situation has been further exacerbated by the COVID-19 pandemic. Elective referrals are increasingly displaced by urgent referrals, including a large volume of pediatric hand fractures. Based on data obtained from our clinical scheduling system, new hand fracture patients comprise over 25% of all new patient visits. It has thus become apparent to our group (and indeed others across the country) that pediatric hand fractures present a unique, well-defined, high volume referral condition where strategies to refine triage processes could improve overall access for all pediatric plastic surgery patients. Such strategies not only reduce the burden on the health care system but also lessen inconvenience to these children and their parents.16,17
In this study, 71% of patients (260/365) had uncomplicated fractures which required immobilization alone and did not require a closed reduction, surgical intervention, or more than three visits to a hand surgeon. Currently at our center all hand fractures, regardless of complexity, are referred to and seen by hand surgeons. Patients with simple fractures typically have 1 or 2 visits with a surgeon and are sent to an occupational hand therapist for a splint and active range of motion exercises. Recovery without incident is expected in 4 to 6 weeks. If these fractures were not referred to a surgeon and went directly to the hand therapist instead, more time would be available for the surgeon to see other types of problems requiring surgical expertise. Matching providers to patient needs is a key strategy for providing efficient health care, and the notion of directing simple hand fractures to nonsurgeons is not a novel practice. A team in the United Kingdom has already demonstrated the success of a hand therapist-led pathway for closed hand trauma. 18 If the prediction rule were in effect, the predicted simple fractures from this trial (55%) would not have been referred to a hand surgeon. This would have potentially allowed surgeons to see an additional 402 patients (201 simple fractures × 2 visits/patient).
Of course, appropriate redirection of patients relies on the accuracy of the prediction rule. This study demonstrates that the CKHR is good at predicting complex fractures (sensitivity = 84%, specificity = 71%). As such, the CKHR has the potential to optimize patient flow to appropriate care providers and make better use of health care resources. The sensitivity of the CKHR observed in this study is lower than in the retrospective derivation (96%). 13 This finding was not unexpected, as a real-world application of a clinical prediction rule introduces new sources of error that were absent in the derivation phase. 19
While survey results indicated that the CKHR was easy to use, this contradicts the finding that most of the false negatives were attributed to incorrect use of the CKHR form. Given that 15 out of the 17 false-negative cases underwent a closed reduction by the referring physician, we plan to further refine the CKHR referral form by including a tick box to indicate whether a closed reduction was performed, thus ensuring that these cases are referred to a hand surgeon. This addition would have eliminated all but 2 false negatives and the CKHR would have achieved near-perfect sensitivity (98%). Our analysis of the false negatives, combined with the information obtained from the interrater reliability assessment and the survey data, suggests that further and ongoing education of referring physicians on the use of the CKHR could greatly improve sensitivity in the clinical setting. Specifically, further clarification regarding angulation and displacement criteria would be beneficial. Education of the referring physicians would likely involve presentations and posters with example photographs and clear descriptions of the appropriate application of the tool. Additionally, there will be a need to educate the follow-up care providers to recognize false negatives or fractures that should otherwise be seen by a surgeon. We will need to ensure there is a continued close relationship and easy access to the hand surgeon to review any patient of concern identified by the follow-up providers.
Limitations
While this prospective study serves as temporal validation of the CKHR, it was carried out in the same center as that used for our original model derivation phase. As well, we did not capture all hand fracture referrals in the study period, as not all referring centers used the CKHR referral form. Further, an additional 64 incomplete referral forms were excluded from the analysis. The follow-up period of 3 months may also have been too short to capture the full extent of patient outcomes, particularly the number of plastic surgery visits. The study took place during the COVID-19 pandemic which may have played a role in decreasing the total number and variety of hand injuries that occurred during the study period.
Future Direction
External validation studies are presently underway at 2 other pediatric hospitals in Canada to assess the generalizability of the CKHR to other populations. Variation in patient populations, practice patterns, and clinical context that is observed within our external valuation sites will be informative. If we observe large changes in predictive performance, we will use this information to identify areas where the CKHR can be improved. In parallel, we will continue to prospectively evaluate the effects of false negatives generated by the CKHR to truly understand the implications of balancing sensitivity and specificity. In addition, we currently have a qualitative project underway looking at the barriers and facilitators of implementing a new care pathway incorporating the CKHR. This work will help us to understand areas where we will need to develop strategies to help make implementation a success and addressing the management concerns of simple fractures.
In conclusion, the Calgary Kids’ Hand Rule is an accurate clinical prediction rule, when used as intended. The clinical effectiveness of the CKHR depends upon wide awareness and informed application of the rule by end users. The preservation of strong performance metrics between the original derivation study, and this prospective study shows the CKHR is robust. With the potential to safely reduce the number of simple fractures referred to hand surgeons, the CKHR lays the groundwork for more efficient care in the future.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by a Canadian Society of Plastic Surgeons grant and an Alberta Health Services Surgery Strategic Clinical Network seed grant.
Presentation of Work
This work was presented at the 18th Annual Canadian Association of Health Services and Policy Research Meeting held virtually in May 2021.