Abstract
Study Design
Preliminary evaluation of new tool.
Objective
To ascertain whether the newly developed content-based image retrieval (CBIR) software can be used successfully to retrieve images of similar cases of adolescent idiopathic scoliosis (AIS) from a database to help plan treatment without adhering to a classification scheme.
Methods
Sixty-two operated cases of AIS were entered into the newly developed CBIR database. Five new cases of different curve patterns were used as query images. The images were fed into the CBIR database that retrieved similar images from the existing cases. These were analyzed by a senior surgeon for conformity to the query image.
Results
Within the limits of variability set for the query system, all the resultant images conformed to the query image. One case had no similar match in the series. The other four retrieved several images that were matching with the query. No matching case was left out in the series. The postoperative images were then analyzed to check for surgical strategies. Broad guidelines for treatment could be derived from the results. More precise query settings, inclusion of bending films, and a larger database will enhance accurate retrieval and better decision making.
Conclusion
The CBIR system is an effective tool for accurate documentation and retrieval of scoliosis images. Broad guidelines for surgical strategies can be made from the postoperative images of the existing cases without adhering to any classification scheme.
Introduction
Modern medical practice is moving toward clinical decision support techniques such as case-based reasoning and evidence-based medicine, 1 , 2 which has made it imperative to retrieve images with absolute precision to establish a clear diagnosis for planning therapeutic strategies. Traditional Picture Archiving and Communication System (PACS) tools use textural descriptions like keywords, file names, patient ID numbers, and so on to retrieve and communicate pictorial data. Needless to say, the precision of the system depends on the accuracy of the keyword fed in. Surgeons typically use different terms to describe the same lesion, usually because of lack of uniform lexicon for most disorders. In the context of adolescent idiopathic scoliosis (AIS), a classification system such as the Lenke system might be used to retrieve images of similar curves. 3 For example, one might query images of Lenke type 1B+ from the archive of scoliosis cases. This technique mandates a clear understanding of the classification by the person who has documented the image and the person who is retrieving it and a very high degree of reproducibility for the classification scheme itself. Unfortunately published literature is a bit divided on the interobserver reliability of most AIS classification schema. 4 , 5 , 6 , 7 , 8 , 9 The cause of such variability might be due to the known inter- and intraobserver variability of the given system, experience and training of the surgeon, differing outcome expectations by different surgeons, different interpretation of criteria proposed in literature, and so on. 10 , 11 It has been documented that for the field anatomic regions, up to 16% of errors have been reported during image retrieval alone. 12 This, then, is the relevance of the content-based image retrieval (CBIR) capability in the PACS to provide image indexing and retrieval on the basis of visual content rather than textural description. Such a tool might be ideally suited to AIS where most of the disease parameters are morphologic in nature, such as Cobb angle, curve location, shoulder level, trunk shift, and so on. 13 Currently significant variability exists in the manual extraction of features in scoliosis X-rays in terms of the selection of strategic vertebrae, measurement of Cobb angle, and assignment of curve type within a classification scheme, not to mention assigning surgical strategies. 10 , 11 These ambiguities may be ironed out to a considerable extent by using a CBIR-assisted PACS.
The typical manner in which diagnosis and therapy in clinical medicine is practiced involves recall of similar clinical situations from one's own experience, taught information, or literature search. The purpose of classification in AIS is largely to call upon the collective expertise to treat a given case based on stratification—in the context of AIS, based on morphologic features like curve location, magnitude, flexibility, shoulder symmetry, and so on. It is fascinating to postulate that when a new patient's image is fed into the data archive, a bundle of similar cases based on some sort of categorization is automatically extracted, and based on stored previous experience, the optimum management strategy is rolled out. The aim of the current study is to test the hypothesis of CBIR as opposed to keyword-based retrieval and to determine whether the software can help plan the surgical strategy for the given case.
Materials and Methods
The CBIR Software
The CBIR documentation system was originally created to eliminate the errors of data entry, pictorial feature measurement, and retrieval in scoliosis images. The database is particularly suited to spinal deformity because unlike other spinal disorders in this context most measured parameters are morphologic in nature (such as Cobb angle, flexibility of curves, trunk shift, shoulder level, etc.) with few “categorical” features needing to be measured (such as pain, paralysis, etc.). The original software was designed and tested by the authors in 2006 and found very reproducible in terms of feature documentation and extraction. 14 From 2006 to 2008, 62 operated cases of AIS with complete documentation and images were entered into the database. The images were indexed as follows. The standing anteroposterior (AP) X-ray of the full length of spine was digitized to obtain a DICOM (Digital Imaging and Communications in Medicine) image. Landmarks were acquired on the image with a mouse pointing device starting with the D12 vertebra, iliac crests, center of the S1 vertebra, and the four corners of all the vertebrae. The time taken for this digitization was a mean of 3 minutes and could be done without the help of a senior spine surgeon (in the current instance, a resident surgeon). The preoperative system has been validated to have an accuracy of 1.81 degrees in the measurement of Cobb angle with a standard deviation of 2.46 degrees described previously by the authors.14,15 Ten measured and derived features are computed based on a rule-based algorithm: curve type (if a classification is incorporated), number of curves and their direction, Cobb angle of each curve, apical vertebrae of each curve, end vertebrae of each curve (upper and lower), apical vertebral translation of each curve, spinal balance (measured as distance between center sacral vertical line and C7 plumb line), pelvic inclination, L4 tilt, and T1 tilt (Fig. 1). Supine side bending images to both sides were similarly indexed to measure the Cobb angle and apical and end vertebrae on bending. By adding side bending films, the precision of the image recall is considerably enhanced (Fig. 2). Lateral X-rays were also indexed to measure the kyphosis/lordosis angle alone but were not used for the current experimental setup because too many variables may reduce the number of accurate retrievals from a relatively small database. Postoperative standing AP and lateral X-rays were similarly documented. The indexing system for postoperative X-rays is unlike the preoperative images due to the overlap of the implant obscuring several of the landmark points in the image. This essentially uses end plate tilt angles not measured from the corner points of the vertebrae and is therefore useful only in extracting a limited number of features like the postoperative Cobb angle, spinal balance, and the numbers and levels of instrumented vertebrae. (This latter upgrade of the CBIR software has not been validated and is awaiting publication.) Similarly a system for mathematically deriving the apical vertebral rotation from the same indexed images has also been devised, which is under validation for internal consistency.
Image indexing and retrieval method. Please note indexed image and parameters generated automatically.
(A) Query standing anteroposterior X-ray, which yielded 11 cases (D). The right side bending image of the same patient (B) was used to filter these 11 cases, and it returned 7 images (E). The left side bending films (C) were used to further filter the results, yielding 3 results (F). The 3 cases finally short-listed (F) can be presumed to be an accurate likeness of the query case (A, B, and C) in terms of standing and side bending films.
Retrieval Settings
The current version of the software allows adjustment of the precision settings for retrieval based on most parameters measured. For this experimental setting, the sequence of importance was set as curve location, curve magnitude, and further parameters based on Euclidean distance of corresponding features between the images. It is possible to analyze the search result by prioritizing the features to be used for comparison, by assigning different weights to individual features. For example, the apical vertebrae might be designated with 0 variation, 1 disk level on either side (+1 disk to −1 disk), 1 vertebral level on either side, 1 vertebra + 1 disk on either side, and so on. Similarly for Cobb angle, measures retrieval might be with 0 degrees of variation, 5 degrees on either side (meaning −5 to +5 of the measured value), or 10 degrees on either side (implying −10 to +10 degrees), and so on. For this experimental setup, retrieval was set at 1 vertebral level on either side for the apical and end vertebrae and up to 40 degrees of Cobb angle measurement on either side. In practice, this means that when a query image is indexed and supplied to the database, the apical vertebra would be located automatically and all images with apex at the same level or one vertebra on either side be will get precedence over the retrieved sorted images. The automated Cobb angle measure would retrieve all images with similar Cobb angles and those from −40 to +40 degrees of the query image. Several other parameters that might be measured with this tool (like apical vertebral translation, pelvic inclination, inflexion vertebrae on side bending images, etc.) were all set at a wider margin of variability to allow more cases to be retrieved (obviously with less precision) because the database is presently not large enough to produce an accurate replica of every query image.
Test Methodology
As a preliminary test only single-observer testing was performed. A senior consultant studied each image and acted as the ground truth for the study. Five new cases of AIS were picked out of a deformity clinic and their AP, lateral standing, and supine side bending images digitized by an independent observer (orthopedic resident). The AP image alone was used as a query image in this experiment. The retrieved images in each of the five sets were reviewed by the senior consultant to compare with ground truth and the entire database was scanned to ascertain that none were overlooked. As reported in our previous study, 14 all curves falling into the same type could be picked out with 100% accuracy. The postoperative images of these resultant cases were then called for and the results compared by the senior author. Statistical data for the whole pool can also be generated to show several features like number of screws, levels of instrumentation, degree of correction achieved, and balance achieved depending on the data input at the time of image indexing. The optimal instrumentation patterns for each curve type may be derived from a large data set of similar cases. Multiple tests to assess intraobserver variation and multiple participants to check on interobserver variability were not performed in this experimental setting.
Results
Of the five image recalls tested, one yielded no results. There were no cervicothoracic curve patterns in the CBIR database. There were 22 images for the thoracic and lumbar double curve, 11 for single thoracic curve, 3 for double thoracic, and 8 for the thoracolumbar curve patterns. All the resultant images matched to the query image within the set limits of variability. There were no unretrieved images in the data pool for each query. In the illustrated example (Figs. 3 and 4), there were 8 cases corresponding to the thoracolumbar location of the apex. The Cobb angle varied from 42 to 82 with a mean of 65.38 in these cases. The operative results query is illustrated in Figs. 5 and 6, and 5 levels were instrumented in 4 cases and 6 levels in 4. The mean number of screws used was 8.7 but was 5.4 when the one posterior case was eliminated. The upper instrumented vertebra was D10 in 5 of 8 cases and D11 in 3. The lower instrumented vertebra was L3 in 7 of 8 and L2 in 1 of 8 patients. Based on the statistics available in the pool, in the queried patient it may be appropriate to select an anterior approach, instrumenting from D10 to L3 (Fig. 7). However, this statistical tool also has to be used with caution at this stage because the current version does not tell us the approach used for surgery, and posterior surgeries generally tend to use more screws and rods, thereby skewing the statistical data. Moreover, at this moment all recommendations of the CBIR software have to be counterchecked by a peer before surgical implementation until the software has been validated independently.
Image query of an experimental case of adolescent idiopathic scoliosis.
Results of retrieval of cases similar to the query image anteroposterior standing X-rays.
Postoperative images of the retrieved cases: anteroposterior view.
Postoperative images of the retrieved cases: lateral view.
Surgical strategy for the query image in Fig. 3 based on the data from the 8 previous similar cases.
Some of the limitations of using such liberal query settings are obvious in the given example (Fig. 5). Though the curve apex matches to one vertebral level in the images picked out, the Cobb angle varies considerably, which in many instances may change the instrumentation pattern. The magnitude and flexibility of the secondary curve was not set in the query, thereby resulting in some curves having structural secondary curves (image 8 in Fig. 5). Significant trunk shift (as in image 5 in Fig. 5) might also be one of the features that alters surgical strategy (though not so in this example). Therefore, an intelligent selection and weighting of features of the query image results in more accurate result set matching to the query image, which also implies a more precise surgical strategy being available.
Discussion
Attempts at classifying AIS really started with Shultess in 1905; his greatest contribution was to segregate curves based on their location along the vertebral column. Nothing much happened to this strategy over the next 50 years, though Ponsetti and Freidman and James built upon this concept of aggregating cases based on curve location.16 Harrington in 1972 proposed adding severity to location, 29 and King and Moe et al devised their schema to suit the emerging surgical strategies with Harrington instrumentation. 30 Coonrad et al (1998) studied 2,000 cases to formulate their scheme but it was not substantially different in curve identification from what was devised 90 years before. 31 The next major breakthrough happened in 1998 when Lenke and coworkers published their classification incorporating the sagittal plane in addition to the coronal plane. 17 Though already well known for several decades, the three-dimensional component of AIS was first used to classify the deformity by Poncet et al in 2001. 18 Nonetheless, they did not qualify how this might affect the management strategy of each curve type. The major contribution to this knowledge base from Asia came as the Peking Union Medical College (PUMC) classification published in 2005 yet again considering only the curve location on the coronal plane and inherent flexibility of the compensatory curves. 19
The key to understanding the future of classifications is to appreciate why we need them in the first place: classifications help to group, record, and retrieve cases that appear similar and have similar outcomes if managed similarly. This simplistic definition in reality encompasses several domains: aggregation and stratification, documentation, retrieval, communication, and prognostication but perhaps the most important, assigning therapy and outcome monitoring. Surgical strategy is often based on the morphologic classification though many authors prefer to plan their surgery without aligning to any schema. 20 , 21 , 22 Arlet advocates that an ideal system must have the following qualities16:
All curves patterns covered
Easily memorizable
Reliable and easily reproduced
Therapy orientated
Outcome related
Possess scope for expansion
In an attempt to qualify for the first of these conditions, the PUMC system expanded from the curve types of King to 36 curve types and by adding 1 more dimension (the sagittal plane) the Lenke system grew to 42 curve subtypes. Assuming that as our understanding of these deformities improves, in future more parameters like vertebral rotation, trunk shift, and spinal balance may all feature in the classification schema, the possible permutations and combinations would run into several hundreds, and it is extremely unlikely that anyone would be able to memorize any of the systems. What then is the future of AIS classifications?
The world is moving into digital archiving, retrieval, and communication of high-resolution images for practically all applications. The accuracy, validity, and role of several such systems have been studied by many authors. 23 , 24 , 25 , 26 , 27 , 28 When confronted with a new patient, if the surgeon is able to recall a strategy based on a classification or retrieve a series of similar cases from an archive of cases, he might use that experience to assign therapy and prognosis to the patient. The CBIR system has been demonstrated to successfully achieve this objective. Within the limitations of the experimental setting, it has been shown to effectively retrieve cases that correspond to query image and provide solutions for treatment without considering any schema of classification.
The present study has several limitations. It is a single-center study by one operating surgeon. The data entry was done by one trained person, and the results of retrieval reviewed and compared by one person. Due to the limited numbers in the data bank, the recall settings were set with liberal margins, which naturally brought up several “outliers” on query. Obviously the larger the data bank, the more exact matches might be obtained, and therefore the more successful the system would be. Although it is clearly established that AIS is a three-dimensional deformity in which the lateral view X-ray plays an important role, the present experimental setup does not use the lateral view for image recall (though they were also indexed to measure the kyphosis/lordosis angles) because too many variables may reduce the number of accurate retrievals from a relatively small database, but these may be added on as required in future studies. Moreover, one of the problems we often face with lateral X-rays is the poor visibility of several of the strategic vertebrae like T2, T5, and sometimes T10 and L2 as well. The accuracy of the recall mechanism can be enhanced by additionally short-listing the cases by apical and end vertebrae, inflexion vertebrae, and curve flexibility of primary and secondary curves on the supine bending films; curve dimension, apical vertebral translation of the main and secondary curves, and so on should also help ensure an exact match to the query image may be obtained. The surgical strategies brought up by the statistical tool need to be independently validated for optimal safety because the current version of the program does not tell us the surgical approach (posterior surgeries generally tend to use more screws and rods) thereby skewing the statistical data. It is presently recommended that all results of the CBIR software be verified by the operating surgeon before undertaking the operation until multicenter validation of the accuracy of the software is established. Another suggestion is that multiple surgeons feeding their cases as in the Scolisoft or Spine Tango registry would significantly enhance the potential of this system.
A concern has often been expressed that the development of powerful tools like the present one might effectively remove the “art” of preoperative planning from the “science” of scoliosis surgery. The authors do not believe that to be the case. There are several nonmorphologic features to be considered in taking up scoliosis cases for surgery; for example, age, skeletal maturity, other miscellaneous considerations (one child in the series had a diaphragmatic hernia repair in infancy that precluded anterior surgery) are but some of these. At this point of evolution, we would still recommend to use the tool to supplement independent planning based on a case-by-case evaluation of the patient.
Conclusion
The CBIR system has been found to be a user-friendly and reproducible system for documentation and retrieval of AIS cases in the experimental setting. It can also be shown to be of significant value as a substitute to memorizing complex classification systems. It appears to have value in accommodating expanding features of spinal curves in all three dimensions. In the current experimental setting, it appears to have significant potential in surgical planning of deformity cases. Obviously the database needs multicenter evaluation for consistency and reproducibility and also to enhance the value of the therapeutic strategy selected.
Disclosures
None