AO Spine-DGOU Osteoporotic Fracture Classification System: Internal Validation by the AO Spine Knowledge Forum Trauma

Abstract

Study Design

Cross-sectional survey.

Objectives

Injury classifications are important tools to identify fracture patterns, guide treatment-decisions and aid to identify optimal treatment plans. The AO Spine-DGOU Osteoporotic Fracture (OF) classification system was developed, and the aim of this study was to assess the reliability of this new classification system.

Methods

23 Members of the AO Spine Knowledge Forum Trauma participated in the validation process. Participants were asked to rate 33 cases according to the OF classification at 2 time points, 4 weeks apart (assessment 1 and 2). The kappa statistic (κ) was calculated to assess inter-observer reliability and intra-rater reproducibility. The gold master key for each case was determined by approval of at least 5 out of 7 members of the DGOU.

Results

A total of 1386 ratings (21 raters) were performed. The overall inter-rater agreement was moderate with a combined kappa statistic for the OF classification of 0.496 in assessment 1 and 0.482 in assessment 2. The combined percentage of correct ratings (compared to gold-standard) in assessment 1 was 71.4% and 67.4% in assessment 2. The average intra-rater reproducibility was substantial (κ = 0.74, median 0.76, range 0.55 to 1.00, SD 0.13) for the assessed fracture types.

Conclusions

The assessed overall inter-rater reliability was moderate and substantial in some instances. The average intra-rater reproducibility is substantial. It seems that appropriate training of the classification system can enhance inter- and intra-rater reliability.

Background

Osteoporosis represents a major public health concern with the elderly population steadily rising worldwide. ¹ Vertebral fractures are the most common pathological fractures as a result of low bone mass and bone fragility due to osteoporosis^2,3 leading to morbidity and potential mortality. ⁴ The management of osteoporotic vertebral fractures (OVFs) can be challenging due to patients’ age, existing comorbidities, reduced physiological reserves and often multipharmacy. ⁵ Osteoporotic changes in the elderly can either be a contributing factor in traumatic (low-energy) vertebral fractures or can be the cause of those fractures. ⁴ Injury classifications are important tools for patient care as well as for communication amongst caregivers. Classification systems, ultimately, should guide physicians in decision-making for optimal treatment of the patient. Therefore, classification systems should be simple, reproducible and should accentuate important fracture characteristics which are relevant for patient care. Commonly used and internationally accepted classifications for (thoracolumbar) vertebral fractures were not specifically developed for patients with osteoporotic changes. This differentiation is important since OVFs behave and are managed differently compared to traumatic (high-energy) vertebral fractures. ⁶

Recently, the Spine Section of the German Society for Orthopaedics and Trauma (DGOU), who have formed the working group “Osteoporotic Fractures”, have proposed a new classification for osteoporotic thoraco-lumbar spine fractures. ⁴

The final AO Spine-DGOU Osteoporotic Fracture (OF) classification system was developed after several consensus meetings with members the AO Spine Knowledge Forum Trauma. Blattert et al proposed treatment recommendations based on the OF classification: OF 1 and 2 should be treated conservatively, whereas OF 3 can either be managed conservatively or surgically. OF 4 and OF 5 are unstable fractures which should be treated surgically. ⁷ A large multi-center study from Germany, assessing performed treatment strategies of OVFs, showed a high accordance with the OF-based treatment recommendations. ⁶ Whereas treatment recommendations were assessed and validated with existing clinical data, data on the reliability of the AO Spine-DGOU OF classification is lacking.

Thus, the aim of the present study was to assess the reliability of the AO Spine-DGOU OF classification amongst members of the AO Spine Knowledge Forum Trauma.

Material and Methods

Development of Classification System

The methodology behind the creation of the AO Spine-DGOU classification has been described in detail in the publication of the Spine Section of the German Society for Orthopaedics and Trauma (DGOU). ⁴

The AO Spine-DGOU Osteoporotic Fracture (OF) Classification

This classification consists of 5 groups: OF 1, no deformation, only bone edema assessed with MRI, OF 2, deformation of 1 endplate with no or minor involvement of posterior wall, OF 3, deformation of 1 endplate with substantial involvement of posterior wall, OF 4, deformation of both endplates with or without involvement of posterior wall, and OF 5, fracture with anterior or posterior tension band failure. (Supplement 1)

Validation Process

Members of the AO Spine Knowledge Forum Trauma, an international group of spinal surgeons dedicated to the study of spinal trauma from different regions of the world, participated in the validation process. Participants were asked to rate cases according to the OF classification at 2 time points, 4 weeks apart (assessment 1 and 2), to avoid recall bias. Participants were not particularly familiar with the classification but an introduction to the classification with example cases was given and participating surgeons were able to use the classification poster for the assessments and were asked to provide any feedback they may have to improve the classification and validation process. All of the selected cases included osteoporotic thoracolumbar fractures with different morphological subtypes. A total of 33 cases were selected, and 7 DGOU members developed the gold-standard master key for each case. In cases of disagreement, consensus was reached. All cases were reviewed independently by 23 surgeons from the AO Spine Knowledge Forum Trauma representing different regions of the world. To support the surgeons in rating the cases, key-images showing the fracture, and short video sequences of computed tomography and/or MRI were provided. Cases were randomized in both assessments.

Statistical Analysis

The kappa statistic (κ) was calculated to assess the reliability of the classification system among different observers (inter-rater agreement) and the reproducibility for the same observer on separate occasions (intra-rater reproducibility). ⁸ The Fleiss kappa statistic measures the agreement of multiple raters who rate multiple subjects, with the rating based on multiple categories. The coefficients were interpreted utilizing the Landis and Koch grading system, which defines κ ≤ 0.2 as slight reliability (agreement/reproducibility), 0.2 < κ ≤ 0.4 as fair reliability, 0.4 < κ ≤ 0.6 as moderate reliability, 0.6 < κ ≤ 0.8 as substantial reliability, and κ > 0.8 as excellent reliability. ⁹ Inter-rater and intra-rater agreement was calculated for the different classification groups (OF 1-5) and overall, for the classification itself. Furthermore, agreement with the gold-standard (accuracy) was calculated overall and for each OF-subtype.

Development of the OF Classification System

The classification consists of 5 different subtypes (OF 1-5) ⁴ (Figure 1).

Figure 1.

Development of the OF classification system. OF 1: No vertebral deformation (vertebral body edema in MRI-STIR only). X-rays do not show vertebral deformation. OF 2: Deformation of 1 endplate without or with only minor posterior wall involvement (<1/5). OF 3: Deformation of 1 endplate with distinct posterior wall involvement (>1/5). OF 4: Deformation of both endplates with or without posterior wall involvement. This type can lead to the loss of integrity of the vertebral frame structure (complete burst), or vertebral body collapse, or pincer-type fracture. OF 5: Injuries with anterior or posterior tension band failure. These injuries have signs of distraction, rotation, or translation.

Results

Out of 23 raters, 2 were excluded from final analysis due to non-compliance in the assessment process. A total of 1386 (693 twice) ratings were performed. Of the 33 presented cases, 1 case was classified (gold-standard) as OF 1 (3%), ten as OF 2 (30.3%), 5 as OF 3 (15.1%), eleven as OF 4 (33.3%) and 6 as OF 5 (18.2%).

Inter-Rater Agreement

The overall inter-rater agreement was moderate with a combined kappa statistic for the OF classification of 0.496 in assessment 1 and 0.482 in assessment 2, respectively. The lowest level of agreement was seen in OF 1 (κ = 0.293, 0.181 in assessment 2), whereas the highest level of agreement was observed in OF 4 (κ = 0.625, 0.683 respectively). (Table 1) In both assessments, the inter-rater levels of agreement for OF 2, 4 and 5 were higher than the overall level of agreement.

Table 1.

Inter-rater Agreement of the OF Classification.

Classification	Assessment 1 (κ)	Assessment 2 (κ)
Total	0.496	0.482
OF 1	0.293	0.181
OF 2	0.576	0.579
OF 3	0.469	0.363
OF 4	0.625	0.683
OF 5	0.518	0.603

Agreement with Gold-Standard

The combined percentage of correct ratings (compared to gold-standard) in assessment 1 was 71.4% and 67.4% in assessment 2. (Table 2) In both assessments, the lowest level of agreement with the gold standard was seen in OF 1 and OF 5 fractures (61.9% and 38.1%, 59.5% and 67.5% respectively). Half of the raters assessed the OF 1 (1 case in each assessment) falsely as an OF 2. One of the 6 as OF 5 classified cases were rated wrongly by most of the raters (59.5%). The overall mean agreement with the gold-standard among raters in this study was 75.04% (median 75.76%, range 63.64% to 84.85%, SD 6.28) in assessment 1 and 75.90% (median 78.79%, range 57.58% to 84.85%, SD 8.65).

Table 2.

Agreement With Gold-Standard.

Classification	Assessment 1 (%)	Assessment 2 (%)
Combined	71.4	67.4
OF 1	61.9	38.1
OF 2	76.2	79.0
OF 3	76.2	67.6
OF 4	83.1	84.8
OF 5	59.5	67.5

Intra-Rater Agreement

The average intra-rater reproducibility was substantial (κ = 0.74, median 0.76, range 0.55 to 1.00, SD 0.13) for the assessed fracture types. Three of the reviewers showed a moderate level of agreement (0.4 < κ ≤ 0.6), whereas twelve raters had a substantial level of agreement (0.6 < κ ≤ 0.8) and 6 of the reviewers were assessed with an excellent level of agreement (κ > 0.8).

Discussion

The prevalence of OVFs varies between approximately 19% in women and 20% in men amongst the elderly population. ¹⁰ Due to a rising number of elderly people worldwide, the incidence rates of these fractures can be assumed to rise. OVFs represent a major public health concern due to increased morbidity and mortality, especially in patients with an age of 75 years and above. ¹¹ Accurate diagnosis and treatment of these fractures is crucial to avoid prolonged and chronic episodes of pain, to reduce morbidity and mortality and to prevent kyphosis and further collapse of the fractured vertebra which can result in neurological deficits. However, to date, there is no consensus guideline on the treatment of OVFs and conservative treatment is considered the primary treatment option. ¹² Existing classification systems on thoracolumbar vertebral fractures such as the AO Spine thoracolumbar spine injury classification system or the TLICS are widely accepted but are unsuitable for OVFs since these systems are not appreciating the distinct features of vertebral fractures with underlying osteoporotic changes.^13,14 Recently, a new classification specifically addressing OVFs was introduced by the working group on “Osteoporotic Fractures” of the DGOU and was then further developed in cooperation with AO Spine (AO Spine-DGOU Osteoporotic Fracture (OF) classification system) which grades OVFs into 5 groups and gives treatment recommendations (surgical vs conservative) for each classification group. ⁴ Thus, the aim of this study was to assess the inter-rater as well as intra-rater reliability amongst AO Spine Knowledge Forum Trauma members.

We assessed a moderate combined overall inter-rater agreement in both assessments (κ = 0.496 in assessment 1 and κ = 0.482 in assessment 2) which is in keeping with 2 previous studies assessing the inter-rater agreement of this classification amongst 6 surgeons in each study.^15,16 Furthermore, we found that our findings are comparable to the previous international validation study of the AO Spine thoracolumbar injury classification system (κ = 0.56). ¹⁷ Due to global osteoporotic changes of the vertebrae on the assessed imaging data, rating of the fractures might be more difficult (eg, degenerative endplate changes vs traumatic endplate changes) and therefore, the assessed inter-rater agreement might be falsely low. Interestingly, the inter-rater agreement declined between the first and the second assessment. OF 1 showed fair agreement (κ = 0.293) in assessment 1 and slight agreement (κ = 0.181) which represents the most inferior inter-rater agreement amongst the 5 classification groups. This relative low agreement rate may be also explained by the low prevalence of this morphology in the case sample as well as the difficulty to distinguish degenerative endplate irregularities from traumatic changes. OF 4 and OF 5 are recommended for surgical treatment and showed a substantial inter-rater agreement in assessment 2, which highlights the uniformity of rating and possibly treatment decisions. ⁷ The assessed moderate overall inter-rater agreement could be further due to lack of training of the included raters, which were all experienced raters with good training in well-established classifications who did not undergo training of the AO Spine-DGOU Osteoporotic Fracture (OF) classification system prior this assessment, which highlights the need for appropriate training of classification systems. This should be assessed further after appropriate training sessions of the respected classification system. We observed a combined percentage of correct ratings, compared to gold-standard, of 71.4% in assessment 1 and 67.4% in assessment 2. Like the inter-rater agreement, OF 1 was rated incorrectly the most often in both assessments and was rated worse in assessment 2 (61.9% and 38.1%). Half of the raters rated OF 1 as OF 2. This could be due to over-interpretation of MRI imaging ie, interpreting physiological endplate deformities with bone marrow edema as impression fracture. Therefore, assessment of both, CT and MRI are necessary to avoid an over-interpretation and to increase inter-rater and agreement with the gold-standard, which has been also shown in a previous assessment of 6 surgeons.^16,18 According to the newly introduced OF-score, both fracture types, in the worst clinical scenario, could be treated surgically. ⁷ However, a multi-center study showed that OF 1 are almost always and OF 2 are commonly treated conservatively. ⁶ A possible second solution to increase the intra-rater agreement as well as the agreement with the gold-standard would therefore be to fuse OF 1 and OF 2 into 1 entity and make surgical decision-making solely dependent on the modifiers according to the OF-score.

We assessed a substantial overall intra-rater reproducibility of 0.74 which is comparable to the AO Spine thoracolumbar spine injury classification system (κ = 0.68). This finding is also comparable to the assessment from Germany, which found a substantial intra-observer agreement when raters were provided with radiograph and MRI (κ = 0.64). This study, in addition, showed that the intra-observer reliability can be increased by providing CT-scans as an additional imaging modality, which highlights the importance of complete imaging.^16,17 The majority of the assessed raters showed a substantial or excellent reproducibility.

This study has certain limitations. Firstly, there might be a varying degree of familiarity with the OF-classification amongst the raters which results in a classification knowledge-inequality and could result in a potential bias. Furthermore, the sole use of key-images (in some cases) or vice versa might have an impact on the rater`s response. It is likely that some of the reviewers would change their response while assessing a complete imaging file rather than solely key images. Another potential limitation is the inequal frequency of different fracture subtypes and hence, the data on rare subtypes (eg, OF 1) should be interpreted with caution. Additionality, some of the presented images showed multiple fractures which might have caused confusion regarding the fracture to rate, decreasing the reliability rate. Furthermore, the members of the AO Spine Knowledge Forum Trauma might have varying degrees of knowledge on interpreting different imaging modalities and familiarity with the presented classification since not all members were involved in the development process, which could result in an inhomogeneous group of raters and could have a negative effect on the inter-rater agreement. Standardized training on imaging and of the OF-classification could limit these potential biases.

Conclusion

Like previously described AO Spine thoracolumbar fracture classifications as well as small inter- and intra-rater reliability assessments, the results from the present study showed moderate to substantial interrater reliability and substantial inter-rater reproducibility amongst a group of members of the AO Spine Knowledge Forum Trauma. As a next step, an international validation amongst spine surgeons with a higher number of raters representing different backgrounds, work settings, and regions should be performed to assess the AO Spine-DGOU OF classification reliability globally. It seems that appropriate training of the respected classification system is necessary to increase inter- and intra-rater reliability.

Supplemental Material

Supplemental Material - AO Spine-DGOU Osteoporotic Fracture (OF) Classification System: Internal Validation by the AO Spine Knowledge Forum Trauma

Supplemental Material for AO Spine-DGOU Osteoporotic Fracture (OF) Classification System: Internal Validation by the AO Spine Knowledge Forum Trauma by Julian Scherer, Andrei Joaquim, Alex Vaccaro, Rishi Kanna, Mohammad El-Sharkawi, Masahiko Takahata, Mohamed M. Aly, Gaston Camino-Willhuber, Ulrich Spiegl, Cumhur Oner, Jose A. Canseco, Ratko Yurac, Lorin Michael Benneker, Eugen Cezar Popescu, Richard Bransford, Chhabra Harvinder Singh, Frank Kandziora, Marko H. Neva, and Klaus John Schnake in Global Spine Journal