Abstract
Study Design
Cross-sectional study.
Objectives
Imaging classification of adolescent idiopathic scoliosis (AIS) is directly related to the surgical strategy, but the artificial classification is complex and depends on doctors’ experience. This study investigated deep learning-based automated classification methods (DL group) for AIS and validated the consistency of machine classification and manual classification (M group).
Methods
A total of 506 cases (81 males and 425 females) and 1812 AIS full spine images in the anteroposterior (AP), lateral (LAT), left bending (LB) and right bending (RB) positions were retrospectively used for training. The mean age was 13.6 ± 1.8. The mean maximum Cobb angle was 46.8 ± 12.0. U-Net semantic segmentation neural network technology and deep learning methods were used to automatically segment and establish the alignment relationship between multiple views of the spine, and to extract spinal features such as the Cobb angle. The type of each test case was automatically calculated according to Lenke’s rule. An additional 107 cases of adolescent idiopathic scoliosis imaging were prospectively used for testing. The consistency of the DL group and M group was compared.
Results
Automatic vertebral body segmentation and recognition, multi-view alignment of the spine and automatic Cobb angle measurement were implemented. Compare to the M group, the consistency of the DL group was significantly higher in 3 aspects: type of lateral convexity (0.989 vs 0.566), lumbar curvature modifier (0.932 vs 0.738), and sagittal plane modifier (0.987 vs 0.522).
Conclusions
Deep learning enables automated Cobb angle measurement and automated Lenke classification of idiopathic scoliosis whole spine radiographs with higher consistency than manual measurement classification.
Keywords
Introduction
Adolescent idiopathic scoliosis (AIS) is a three-dimensional spinal deformity of unknown etiology with a prevalence of 0.5%–6.8%.1–3 The deformity can lead to cosmetic deformity, psychological and social dysfunction, lower marriage and childbearing rates, higher suicide rates, and in severe cases, thoracic compression, reduced lung capacity, and even disability, seriously affecting the health of adolescents.1,4 Currently, the clinical assessment of scoliosis is mainly based on full spinal radiographs in the anteroposterior and lateral. The angle between the plumb lines of the endplates of the two most curved vertebrae at the ends of the scoliosis is called the Cobb angle. In general, AIS with a scoliosis Cobb angle number of less than 25° is mainly followed up by observation. Scoliosis with a Cobb angle of 25-40° can be prevented from getting worse by wearing an orthopedic brace. If conservative treatment is ineffective or if the degree of scoliosis is greater than 40°, surgery is required. 5 Accurate measurement of AIS is the key to determining the treatment plan, and in 2001 Lenke et al 6 proposed a new typology of AIS. A total of 6 types were classified, with corrections based on lumbar and sagittal thoracic curvatures. This typology takes full account of the characteristics of three-dimensional deformities. It has been widely used throughout the world over the past 20 years, and has become the gold standard for the typing of idiopathic scoliosis. The clinician usually performs a spinal classification based on Lenke’s classification rules. Cobb angle is measured using a ruler, marker, and other tools, and the Lenke classification is based on these measurements, which guide the surgical plan. However, there is inter- and intra-observer variability in the measurements. Therefore, doctors urgently need a set of automatic and accurate measurement and typing methods to replace manual operation.
In recent years, deep learning artificial intelligence has made great strides in imaging technology, and there have been some preliminary results in the diagnosis of some spinal conditions.7,8 However, there are still fewer studies on automated classification of adolescent idiopathic scoliosis. Computer-assisted measurement and classification has been reported in some cases, but most of them are still semi-automated. Current deep learning methods can only automatically measure the Cobb angle,9,10 or only identify the shape 11 or severity 12 of the curve, and cannot perform fully automatic Lenke classification.
This study focuses on the realization of the task of vertebral segmentation in multiple views of the spine and automatic Cobb angle measurement and Lenke auto-classification based on spine segmentation images in the framework of deep learning. The results are compared with the consistency of manual measurement and classification.
Materials and Methods
We used a cross-sectional study. The imaging data used in this study were obtained at the Department of Orthopaedics, Peking Union Medical College Hospital, from January 2013 to December 2020. All patients signed a medical research consent form for case imaging data and the ethics committee of our hospital approved the study. The study was approved by the Ethics Committee of Peking Union Medical College Hospital (No.S-K1857).
Inclusion and Exclusion Criteria
All cases that underwent model training and validation met the following criteria: (1) diagnosis of adolescent idiopathic scoliosis; (2) age of 10-20 years; (3) no history of spinal surgery; (4) Cobb angle <90°; (5) images clearly showing from C7 to the pelvis. Other scoliosis diagnoses were excluded. All cases had complete full spine images of anteroposterior position (AP), lateral position (LAT), left bending (LB) and right bending (RB).
Deep Learning-Based Automated Classification Methods
In this study, a semantic segmentation trial framework for AIS Lenke classification was developed. The above trial framework is divided into four trial steps: multi-view semantic segmentation, spine feature computation, automatic Lenke classification, and model performance evaluation (Figure 1). Framework of semantic segmentation trials for AIS Lenke stages. AP: anteroposterior position; LAT: lateral position; LB: left bending; RB: right bending.
The multi-view semantic segmentation process includes collecting AIS case samples, cleaning the sample data to remove outliers, constructing training and validation sets, and training the model to converge to the best performance. Data enhancement techniques such as rotation, left-right flip, ColorJit, histogram equalization are applied to the original training images. The dice similarity coefficient is used as an index to evaluate the semantic segmentation performance. The dice similarity coefficient ranges from 0 to 1. The closer the dice coefficient is to 1, the better the segmentation performance is. After the model training process, the semantic segmentation images of the four views of AP, LAT, LB and RB are output.
The spine feature calculation process takes the four view images as model input and outputs the mask image of the vertebral part after neural network calculation. Due to the noise in the input X-ray image data, scale transformation, image contrast adjustment, and region random sampling are applied to the original X-ray images, and the one with the best segmentation effect is taken as the model inference result. The evaluation criteria for the segmentation effect include the number of vertebrae that meet the requirements, no adhesion of the segmented vertebrae, and the minimum vertebral area. Automatically extract the vertebral aspect ratio data in AP images and RB and LB, calculate the Euclidean distance of the corresponding vertebral positions between multiple views of the spine, and select the one with the smallest Euclidean distance as the alignment object. For each vertebra, calculate the contour pixel position to solve for the minimum external polygon. In the measurement and fractionation step, the Cobb angle of AIS scoliosis, the central sacral vertical line (CSVL), the stabilizing vertebra (SV), the neutral vertebra (NV), the end vertebra (EV), and the lateral Cobb angle of the T5-T12, and other spinal feature data were calculated.
The Cobb angle of the AP image was calculated by finding the maximum angle between the upper endplate and the lower endplate edge line of the AP image, which was taken as the maximum Cobb angle of the spine. The second and third ranked Cobb angles were found in the region of the spine other than the maximum Cobb angle. Each Cobb angle value, starting position and left/right convexity characteristics were automatically recorded.
LAT image Cobb angle calculation method: the centre of mass was sampled at the upper, middle and lower part of the thoracic curve segments from T5 to T12, and the angle of the corresponding arc of the thoracic curve segment was calculated according to the equation of the circle where the thoracic curve was located by fitting the 3 sample points, and the angle of the arc of the corresponding arc of the thoracic curve segment was taken as the thoracic curve posterior convexity angle value. From the AP image, the starting position, the position of the apex vertebra, the direction of the apex vertebra, and the angle value, the Cobb angle of the corresponding position in the LB/RB image was searched and calculated, in which the vertebral body alignment of the LB/RB image with the AP image was applied by the method described in the image alignment process of the present experiment.
Lenke Automatic Classification
Scoliosis type (1-6) was determined according to the Lenke classification rules and Cobb angle measurements described above. Determination of the lumbar curve modifier was based on the position of the median sacral line in relation to the lumbar curve apex vertebrae. Determination of the thoracic curve modifier was based on the T5-T12 posterior convexity angle (Figure 2). Three repeated measurements were performed randomly for computerized classification. A case of automatic machine identification.
Manual Measurement and Classification
The classification performed by 3 spine surgeons using the traditional manual measurement classification method. They used manual protractors to measure angles on printed imaging films and classified them according to Lenke’s classification rules. The surgeons who performed the measurements were completely to the deep learning-based automated classification methods.
Statistical Analysis
Fleiss’ Kappa (Fleiss, 1971) was used to measure the agreement between multiple raters (e.g. more than two), while Cohen ‘s Kappa (Cohen, 1960) was used to measure the agreement between two raters. Kappa coefficients were calculated using SPSS software (SPSS Inc, Chicago, IL, USA). The Landis and Koch’s criteria for κ values were adopted:0-0.20 indicated slight agreement, 0.21-0.40 fair agreement, 0.41-0.60 moderate agreement, 0.61-0.80 substantial agreement, and 0.81-1.00 almost perfect agreement.
Results
506 cases (81 male patients and 425 female patients) and 1812 AIS full spine images in anteroposterior (AP), lateral (LAT), left bending (LB) and right bending (RB) positions were used for training. The mean age was 13.6 ± 1.8 years old and the mean maximum Cobb angle was 46.8 ± 12.0. Data from the training process of the U-Net semantic segmentation network model are shown in Figure 3. The DICE metrics achieved by the trained model based on the four view datasets are shown in Table 1. A total of 107 cases of AIS were included in the testing process. The segmentation time was approximately 30 seconds for the machine and 3 minutes for the manual operation. Curve of multi-view model training Loss and DICE change. DICE Metrics for Multi-View Models. AP: anteroposterior position; LAT: lateral position; LB: left bending; RB: right bending.
Consistency of Artificial Typing.
Consistency of Machine Typing.
Consistency of Manual Plus Machine Confluence Typing.
Discussion
The accurate measurement and classification of adolescent idiopathic scoliosis is a fundamental step in the preoperative design of the surgical plan. Traditionally, measurements are made by surgeons on X-ray films, which are susceptible to significant error. Jefferies 13 et al suggested that the 5° or so difference in scoliosis measurements is usually an error in measurement rather than a genuine difference in the degree of scoliosis itself. In 1994, Prujis 14 et al demonstrated that the measurement error of Cobb’s angle was approximately 3.2° and that this discrepancy difference should be considered in the diagnosis and management of scoliosis. Furthermore, errors in Cobb angle measurements can result in differences in subsequent classification. In 2001 Lenke, the proponent of the Lenke classification, reported on the reliability of the classification based on radiographs with a given angle. Lenke classification demonstrated good to excellent intra-observer reliability (κ = 0.74) and inter-observer reliability (κ = 0.893) when the size of the Cobb angle was already given. In contrast, Richards 15 et al found that intra-observer reliability (κ = 0.50) and inter-observer reliability (κ = 0.60) for Lenke typing were only fair when the Cobb angle needed to be measured from scratch. The Lenke classification is a valuable guide for surgical planning for AIS patients. However, errors in classification can result from inaccurate Cobb angle measurements and may ultimately affect the accuracy of preoperative planning and postoperative outcomes. It is therefore imperative that an objective, accurate, and rapid alternative method for eliminating errors in AIS measurements is developed without delay.
In recent years, image semantic segmentation has been used for medical image analysis.16,17 Image semantic segmentation is a computer vision method for predicting the classification of each pixel category in an image, which is one of the research areas of artificial intelligence. The majority of contemporary AI image segmentation frameworks are based on deep neural networks (DNNs) and deep learning techniques. Image semantic segmentation has emerged as a mainstream technology for medical image analysis due to its superior performance in terms of segmentation accuracy and precision when compared to traditional segmentation methods.18,19 The segmented images in this study were obtained from AIS radiographs, which are characterized by a limited number of samples, indistinct image boundaries, complex gradients, and a simple semantics and relatively fixed structure of the target object. The U-Net semantic segmentation framework was proposed by Ronneberger in 2015, 20 and is characterized by a small network size, a small training data requirement, and a symmetric upsampling and downsampling structure, which is adapted to the scenarios of high-resolution segmentation and low-resolution recognition of medical images. The U-Net semantic segmentation model is a variant type of the Convolutional Neural Network (CNN). The U-Net model has the characteristics of simple structure, easy training, adaptability to different image sizes, and strong generalization ability. Since its release in 2015, it has been proven to be a powerful semantic segmentation model in numerous medical imaging studies and experiments.21,22 Therefore, in this study, the U-Net semantic segmentation framework is used as a tool for AIS image segmentation.
Once the image segmentation has been completed, the Cobb angle can be measured. The process of automatic Cobb angle measurement can be conceptualized as a boundary recognition problem, whereby the machine is able to automatically recognize the contour of the spine and perform angle calculations based on the contour.
The team of Sardjono et al. 9 Proposed a model based on particles with points, and used 3 types of function fitting: segmented linear fitting, spline fitting, and polynomial fitting to extract the boundary contour and compute the Cobb angle. Comparative experiments were conducted with the manually measured Cobb degree, yielding more satisfactory results. Allen et al. 23 Proposed an activity model-based measurement algorithm to train the model for the purpose of angle measurement. Zhang 24 identified the boundary of the spine block by integrating the ideas of centre extraction with those of automatic boundary detection, using the principle of Hough transform, and then performed angle measurement based on the identified spine block.
The use of computer assistance led to an improvement in the interobserver κ values and consistency were improved in all 3 components of the Lenke classification, with the greatest impact observed in the six types classification. For the six types, lumbar curve modifier, and thoracic curve modifier, the average κ values were improved from 0.65 to 0.81, from 0.75 to 0.81, and from 0.83 to 0.92, respectively. However, the measurements still required manual determination of the anatomical landmarks of the vertebrae in scoliosis. 25 In this study, we used the utilize U-Net semantic segmentation neural network technique with a deep learning method, and the Dice coefficients were above 90%. Lenke’s typology is a three-dimensional typology, thus the assessment of lateral kyphosis is also of great importance. Previous literature has been primarily focused on the measurement of full spine orthostatic X-ray Cobb angle, with only a few reports on the identification of full spine lateralization. 26 This is due to the fact that full spine lateral X-ray is extremely difficult to interpret due to the occlusion of ribs and scapulae. In this study, we employed the methodology of centre-of-mass sampling at the superior, middle and inferior points of the thoracic curvature segments of T5-T12, respectively, and subsequently fitted the equation of the circle to the thoracic curvature. The 3 sampling points were located according to the aforementioned method, and the angle of the corresponding arc of the thoracic curvature segments was calculated. The angle of this arc of the circle was then taken as the value of the thoracic curvature posterior convexity angle, which greatly improves the accuracy rate.
The accuracy of identification varies according to the type of Lenke, due to the quantity of data. In this study, the highest classification consistency of the M Group was for type 5 with lumbar curve modifier C and thoracic curve modifier -, while lowest consistency was for types 2 and 3 with lumbar curve modifier B and thoracic curve modifier +, indicating that the M Group is still highly influenced by measurement error. The DL Group exhibited high consistency in all typing except for type 4. In the lumbar curve modifier, all of the participants demonstrated high consistency, with the exception of lumbar curve modifier B, which was slightly lower. Type 4 was mainly dependent on the quantity, given the limited number of cases. So the consistency of DL group was mainly dependent on the quantity of training data. It is anticipated that the accuracy will be further enhanced in the future by increasing the quantity of training data.
This study is limited by the following factors: firstly, patients with Cobb angles greater than 90° were excluded from the study, as the majority of the sample angles of adolescent idiopathic scoliosis were less than 90°. Secondly, and severe scoliosis with heavier vertebral rotation was more difficult to recognize. Due to the limited sample size, there is a lack of consistency in some scoliosis subtypes, such as Type 4. Should the sample size be expanded in future, it would be beneficial to enhance the accuracy and consistency of the method.
Conclusions
The implementation of automatic measurement typing of AIS is possible through the utilisation of MV-Unet-based multi-view neural networks, which offer a higher degree of consistency than traditional manual measurement typing.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is supported by. This study was supported by the Fundamental Research Funds for the Central Universities (No.3332021013), National High Level Hospital Clinical Research Funding (2022-PUMCH-A-120) and the CAMS Innovation Fund for Medical Sciences (CIFMS) (Grant No. 2022-I2M-C&T-A-005).
