Abstract
Study Design
Global Survey
Objective
To determine the accuracy, interobserver reliability, and intraobserver reproducibility of the AO Spine Upper Cervical Injury Classification System based on surgeons’ AO Spine region of practice (Africa, Asia, Central/South America, Europe, Middle East, and North America).
Methods
A total of 275 AO Spine members assessed 25 upper cervical spine injuries and classified them according to the AO Spine Upper Cervical Injury Classification System. Reliability, reproducibility, and accuracy scores were obtained over two assessments administered at three-week intervals. Kappa coefficients (ƙ) determined the interobserver reliability and intraobserver reproducibility.
Results
On both assessments, participants from Europe and North America had the highest classification accuracy, while participants from Africa and Central/South America had the lowest accuracy (P < .0001). Participants from Africa (assessment 1 (AS1):ƙ = .487; AS2:0.491), Central/South America (AS1:ƙ = .513; AS2:0.511), and the Middle East (AS1:0.591; AS2: .599) achieved moderate reliability, while participants from North America (AS1:ƙ = .673; AS2:0.648) and Europe (AS1:ƙ = .682; AS2:0.681) achieved substantial reliability. Asian participants obtained substantial reliability on AS1 (ƙ = .632), but moderate reliability on AS2 (ƙ = .566). Although there was a large effect size, the low number of participants in certain regions did not provide adequate certainty that AO regions affected the likelihood of participants having excellent reproducibility (P = .342).
Conclusions
The AO Spine Upper Cervical Injury Classification System can be applied with high accuracy, interobserver reliability, and intraobserver reproducibility. However, lower classification accuracy and reliability were found in regions of Africa and Central/South America, especially for severe atlas injuries (IIB and IIC) and atypical hangman’s type fractures (IIIB injuries).
Introduction
Classification schema are increasingly shared on global platforms; thus, international reliability and reproducibility analysis should be undertaken to see if limitations or modifications of the classification are necessary prior to widespread implementation. The strategy for validation attempts is well outlined by Audigé. 1 In short, a panel of experts evaluate the classification to determine its potential suitability for widespread use. Subsequently, international participants evaluate the classification to determine its generalizability. If both rounds of analysis demonstrate high levels of reliability, injury severity scores and injury management algorithms can be proposed as has been done for the AO Spine Thoracolumbar Injury Classification System.2,3
Similar to the AO Spine Sacral Injury Classification System, the AO Spine Upper Cervical Injury Classification System is reliant on computed tomography (CT) scans to classify upper cervical spine injuries. 4 This was a primary goal of AO Spine given the well-documented global inequalities in advanced imaging sites, especially the lack of access to magnetic resonance imaging (MRI) in regions like Western Africa and Central/South America.5,6 Therefore, while MRIs may have additional prognostic value when compared to CT scans for cervical spine trauma, injury classifications based on CT scans may allow for greater global application of the injury classification. 7
Unlike previous upper cervical spine injury classifications, which have focused on isolated areas of the upper cervical spine, the AO Spine Upper Cervical Injury Classification System unifies all upper cervical spine segments from the occipital condyles to the C2-3 joint.8-17 Although each upper cervical motion segment has unique properties based on its anatomical function, they share certain inherent biomechanics properties under physiologic and posttraumatic conditions. 18 Thus, injury classification principles may be sharable across upper cervical spine segments. Given the lack of an international validation for this newly proposed classification, the purpose of our study was to determine the intraobserver reproducibility and interobserver reliability of the AO Spine Upper Cervical Injury Classification System based on an international audience.
Methods
AO Spine Upper Cervical Injury Classification
This classification was designed to provide a simplistic easy-to-use schema. The injury is first classified based on the anatomical location of the injury, as designated by roman numerals (i.e. I. Craniocervical junction and occipital condyles; II. Atlas and C1-C2 joint; III. C2 ring, dens and C2-C3 joint). Injury types are then classified based on their level of stability and are assigned alphabetical grades. Type A — Purely bony injuries without any ligamentous injury or vertebral body translation; Type B — Tension band failures or ligamentous disruption/avulsion injuries without vertebral translation in any plane; Type C — Vertebral body translation in any plane consistent with subluxation or dislocation. Consistent with previous AO Spine classifications, neurologic injuries and injury modifiers are assigned, but they were not evaluated during this validation (Figure 1). Pictorial demonstration of the AO Spine Upper Cervical Injury Classification. The classification is based on injury location (I, II, or III) and injury type (A, B, C). Permission to use this figure was granted by the AO Foundation©, AO Spine, Switzerland.
Reproducibility and Reliability Evaluation
A total of 275 AO Spine members responded favorably to participating in an international validation of the AO Spine Upper Cervical Injury Classification System. Prior to participation, each study participant watched a video in English (https://www.youtube.com/watch?v=KyUYfa_JMb4), which described the classification and informed participants how to correctly classify a variety of upper cervical spine injuries. Each study participant was given the opportunity to ask questions to the instructor (one of the original creators of the classification schema) before participating in a sample validation of three upper cervical spine injuries. Another opportunity to ask the instructor questions was allotted to the study participants after the sample validation, but prior to participating in the reliability and reproducibility analysis.
The official reliability and reproducibility analysis of the AO Spine Upper Cervical Injury Classification System was conducted via a live, online webinar format as previously described. 19 A total of 25 CT videos of upper cervical spine injuries were evaluated with the webinar conducted in English. Key images of the injury, which were concurrently provided during the CT viewing, consisted of axial, sagittal, and coronal CT scans. Each video was played once at a rate of 2 frames/second. An online REDCap survey was then used to capture the study participants’ classification answers. The second assessment, where cases were re-randomized and re-assessed, took place three weeks after the first assessment.
Gold Standard Committee
The original creators of the classification system (all members of the AO Spine Knowledge Forum Trauma) provided the official classification of each injury film. Each injury film reached unanimous agreement within the gold standard committee before the study participants evaluated the injury.
Statistics
Percent agreement between study participants and the gold-standard committee was tabulated for anatomic location (I, II, or III), injury type (A, B, or C), and the combination of anatomic location and injury type. Intergroup comparisons based on AO region (Africa, Asia, Central/South America, Europe, Middle East, and North America) were performed.
Kappa coefficients (ƙ) were calculated based on the agreement between different study participants (interobserver reliability) and the same participant after a three-week interval (intraobserver reproducibility). Interobserver reliability and intraobserver reproducibility were calculated for anatomical injury location, injury type, and overall classification. All of the reported kappa values utilized Fleiss’ Kappa coefficient, which allows for missed ratings and comparisons between more than two validation members. 20 Interpretation of the reliability and reproducibility were based on the Landis and Koch convention, which categorized Kappa values as “slight” (<.2), “fair” (.2 - .4), “moderate” (.41 - .60), “substantial” (.61 - .8), and “excellent” (.81-1.0). 21 Pearson’s chi square test or Fisher’s exact test was utilized to screen for potentially relevant associations.
Results
Breakdown of AO Spine Members Who Participated in the AO Spine Upper Cervical Injury Classification including their AO Region, Subspecialty, and Surgical Experience.
Classification Accuracy
Proportion and Percent of Correctly Classified Injuries Based On AO Spine region.
Abbreviations: Proportion (P) – total number of correct responses / total number of injury films evaluated.
*Indicates statistical significance with P<.05.
When stratifying by injury location alone and by injury type alone, significant regional variability was noted for both location (P < .0001) and injury type (P < .0001) on assessment one and two. In general, accurate classifications were more likely for injury location than injury type on both assessments with Type B injuries being the least likely to be correctly classified (Supplemental Appendix B).
Interobserver Reliability
Interobserver Reliability of the AO Spine Upper Cervical Injury Classification System Based on AO Spine Region.
Identification of the injury location was excellent on assessment one for Asia (ƙ = .859), Europe (ƙ = .902), the Middle East (ƙ = .805), and North America (ƙ = .880), whereas it was substantial for Africa (ƙ = .793) and Central/South America (ƙ = .724). The results were similar for assessment two, but Asia fell from excellent reliability (ƙ = .859) to substantial (ƙ = .753). When evaluating injury type, Africa (ƙ = .413), Asia (ƙ = .580), Central/South America (ƙ = .470), and the Middle East (ƙ = .546) had moderate reliability, while North America (ƙ = .651) and Europe (ƙ = .642) had substantial reliability. No group changed their overall Landis and Koch rating between the first and second assessment. Type B injuries had the lowest reliability on both assessment one and two with reliability ranging from slight to moderate depending on the AO region (Supplemental Appendix C).
Intragroup Reproducibility
When evaluating intraobserver reproducibility for the overall classification, members from Africa (ƙ = .64 ± .19) achieved similar scores compared to Asia (ƙ = .68 ± .23), Central/South America (ƙ = .61 ± .25), Europe (ƙ = .74 ± .14), the Middle East (ƙ = .72 ± .21), and North America (ƙ = .75 ± .15). When evaluating by anatomical location, Africa also obtained similar reproducibility (ƙ = .83 ± .16) as Asia (ƙ = .86 ± .25), Central/South America (ƙ = .80 ± .27), Europe (ƙ = .92 ± .11), the Middle East (ƙ = .82 ± .24), and North America (ƙ = .92 ± .11). When evaluating reproducibility by injury type, similar trends were observed with Africa obtaining a reproducibility of (ƙ = .59 ± .23), which was comparable to Asia (ƙ = .63 ± .28), Central/South America (ƙ = .58 ± .23), Europe (ƙ = .71 ± .16), the Middle East (ƙ = .75 ± .20), and North America (ƙ = .70 ± .18).
Intraobserver reproducibility of the AO Spine Upper Cervical Injury Classification System based on the AO Region of the World.
Discussion
In order for an injury classification to receive widespread adoption, the classification should be comprehensible by an international group of surgeons with varying baseline demographics. To formally study whether the AO Spine Upper Cervical Injury Classification System met this threshold, we evaluated the impact of the surgeons’ AO region and its effect on the accuracy, interobserver reliability, and intraobserver reproducibility of the classification. In general, the classification demonstrated high accuracy, interobserver reliability, and intraobserver reproducibility, although there were some notable differences in the accuracy and reliability scores based on AO region. In particular, surgeons from Africa and Central/South America had lower accuracy and reliability scores, especially when evaluating moderate to severe injuries of the atlas (IIB and IIC) and atypical hangman’s type fractures (IIIB injuries).
One important distinction that differentiates this classification from previous AO Spine classifications, especially the AO Spine Thoracolumbar Injury Classification, is its reliance on CT scans instead of magnetic resonance imaging (MRI). 22 Previous literature has demonstrated inequalities in the access to MRI, which are especially prevalent in low-income countries. While limited access to CT scans also exist, they are available at a greater rate than MRIs. 23 The designers of the classification understand that access to CT scans can be problematic in lower income countries; however, CT imaging for high-energy trauma, especially spinal trauma has become standard of care. 24 Technological advances do appear to be on the horizon (digitalization of imaging), which may help equalize the current inequality gaps surrounding access to imaging, but this is unlikely to occur in the next decade. 22 The lower rates of advanced imaging (both CT and MRI) in Central/Latin America and Africa may result in classification accuracy rates lower than the remaining AO regions of the world.5,6 Although speculative, we hypothesize these AO regions may be more reliant on plain radiographs to identify spine trauma. Thus, even though CT scans are more ubiquitous than MRIs, they may still be underutilized in some regions of the world due to accessibility issues. In hindsight, a survey question asking each participant whether radiographs, CT, MRI, or a combination of imaging modalities was commonly used in their practice when identifying upper cervical spine injuries would have been highly instructive and valuable to identify potential gaps in imaging accessibility.
Hopefully, global access to imaging improves and CT and MRI scans become more prevalent in lower income countries, which may allow physicians to become familiar with classifying upper cervical spine injuries based on CT and MRI scans. In this event, incorporation of MRI images with CT scans for suspected upper cervical spine injuries is advantageous since MRI identifies injuries in 12% of obtunded patients with normal CT scans, albeit only 6% of patients have treatment plan alterations when an MRI is available in addition to a CT scan. 7 It is worth noting that although the validation was performed without MRI images, the ability to supplement CT scans with MRI images is already built into the classification system. In fact, the addition of MRI would likely improve the classification reliability of type B injuries, which can be identified by ligamentous disruption (MRI only) or tension band failure (MRI and/or CT scan).
One notable area for potentially improving the accuracy and reliability of this classification is through global education. In this regard, this study has provided a platform for bringing awareness to the challenges in identifying potential mechanisms and modes of failure of moderate to severe atlas injuries (type IIB and IIC) and atypical hangman equivalent injuries (IIIB). However, accurate classification and identification of these injuries is not a new problem given the relative rarity of these injury types. The Gehweiler classification was recently evaluated and had a similar moderate overall interobserver relaibility (k = .50). 24 While the AO Spine Upper Cervical Injury Classification System had similar interobserver reliability, it may be simpler to use, while also unifying the entire upper cervical spine into a singular classification system. The complexity of atypical hangman fracture variants is also well described with the mode of vertebral body fracture dependent on the injury mechanism. 8 Given the number of fracture variants present in IIIB injuries, an exhaustive validation of the different fracture patters is not possible, nor is it likely necessary, since these injury types are predominantly tension band failures with questionable stability. The crux of the AO Spine Upper Cervical Injury Classification System is built on identifying whether an injury is stable or not. 25 Purely bony injuries (type A) can commonly be treated with nonsurgical management (unless the injury is a modifier – for example, a dens fracture at the watershed line), injuries with questionable stability (tension band failures or ligament disruptions/avulsions) may be treated either nonsurgical or operative, while subluxation or dislocation of a vertebrae is treated with reduction and operative stabilization or Halo-immobilization. 26 Widespread acceptance of this hierarchical injury progression may aid future validation attempts across AO global regions.
As previously discussed, classification validations should be performed in a stepwise fashion. 1 The relatively high accuracy and interobserver reliability of this classification, which was performed on an international level, allows the classification to progress to injury severity score establishment. This process is dependent on the hierarchical progression of injury types (A, B, and C).2,26 Once injury severity scores are established, a treatment algorithm may be proposed and validated through a modified Delphi approach. This process will allow surgeons to grade the severity of an upper cervical spine injury and then determine appropriate management of the injury (operative or conservative management). 3
There were multiple limitations present throughout this study, which require further discussion. First, multiple different surgeon demographics may have affected the accuracy and reliability of a surgeons’ classification including subspecialty, practice location, work setting, trauma center involvement, and familiarity with reading CT scans. These demographic factors may overlap and provide some study confounding, which could not be completely accounted for. Second, study participants were comprised of AO Spine members who may understand the principles of AO classifications better than non-AO Spine members. Third, only injury scans uploaded to the de-identified AO Spine database were evaluated. Thus, no type IB injuries were examined and the reliability and accuracy of classifying these injuries is unknown.
Conclusion
The AO Spine Upper Cervical Injury Classification System found that, in general, the classification can be applied with high accuracy, interobserver reliability, and intraobserver reproducibility. However, differences in classification accuracy and reliability are present based on the surgeons AO region. Surgeons from Africa and Central/South America have lower accuracy and reliability classification scores on the overall classification, but the differences in the reliability and accuracy were most notable when classifying moderate to severe atlas injuries (IIB and IIC) and atypical hangman’s type fractures (IIIB injuries). Additional education on properly classifying these rare and severe injury types is warranted to improve classification accuracy and reliability.
Supplemental Material
Supplemental Material - Global Validation of the AO Spine Upper Cervical Injury Classification: Geographic Region Affects Reliability and Reproducibility
Supplemental Material for Global Validation of the AO Spine Upper Cervical Injury Classification: Geographic Region Affects Reliability and Reproducibility by Mark J. Lambrechts, Gregory D. Schroeder, Brian A. Karamian, Jose A. Canseco, Richard Bransford, Cumhur Oner, Lorin M. Benneker, Frank Kandziora, Rajasekaran Shanmuganathan, Rishi Kanna MD, Andrei F. Joaquim, Jens R Chapman, Emiliano Vialle, Mohammad El-Sharkawi, Marcel Dvorak, Klaus Schnake, Christopher K. Kepler, Alexander R. Vaccaro, and AO Spine Upper Cervical Injury Classification International Members in Global Spine Journal
Footnotes
Acknowledgments
The authors of the manuscript would like to thank Olesja Hazenbiller for her assistance in developing the methodology and providing support during the validation. We would also like to thank Hans Bauer, senior biostatistician at Staburo GmbH for his assistance with the statistical analysis.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was organized and funded by AO Spine through the AO Spine Knowledge Forum Trauma, a focused group of international Trauma experts. AO Spine is a clinical division of the AO Foundation, which is an independent medically guided not-for-profit organization. Study support was provided directly through AO Network Clinical Research.
ORCID iDs
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
