How Reliable is the Assessment of Fusion Status Following ACDF Using Dynamic Flexion-Extension Radiographs?

Abstract

Study Design

Reliability study.

Objectives

The radiographic diagnosis of non-union is not standardized. Prior authors have suggested using a cutoff of <1 mm interspinous process motion (ISPM) on flexion-extension radiographs, but the ability of practicing surgeons to make these measurements reliably is not clear.

Methods

29 practicing spine surgeons measured ISPM on 19 levels of ACDF from 9 patients. Surgeons relied on these measurements to report on fusion status. Inter-observer correlation co-efficients (ICC), standard error (SEM) and the minimum detectable difference (MD) of these measurements were calculated. We screened for clerical errors by checking measurements more than one standard deviation from the group mean.

Results

The ICC for ISPM was .76 (.64; .88) with a SEM of 1 mm and a MD of 2.76 mm. Agreement on fusion status was moderate, with an ICC of .6 (.44; .76). After screening for and removing clerical errors, the ICC improved to .82 (.71; .91), SEM improved to .83 mm, and MD improved to 2.29 mm. Six reviewers had an ICC >.9. The ICC from these high performing reviewers was .94 (.9; .97), SEM was .45 mm, and MD was 1.26 mm.

Conclusions

The MD of 2.29 mm in our study group was not precise enough to support a cutoff of <1 mm ISPM as the sole measurement technique in screening for non-union after ACDF, and there was only moderate agreement amongst surgeons on fusion status based on dynamic radiographs. More stringent techniques are necessary to avoid mis-diagnosing non-union in clinical studies. Future studies should consider auditing measurements to identify clerical errors.

Introduction

Fusion rates after anterior cervical discectomy and fusion (ACDF) have historically been reported to be very high for one- and two-level cases, with rates ranging up to 90-100% in some series. ¹ However, the methods employed to identify non-unions are quite variable across studies, and have included assessing for the presence of bridging bone across the operative levels on static images or CT scans, inter-spinous process motion (ISPM) on dynamic flexion-extension radiographs, and change in cobb angle of the operative segment on dynamic flexion-extension radiographs.^2-4 Given the variability in assessment methods, it is difficult to compare fusion rates and outcomes across studies. For example, one study listed a 100% fusion rate based on the presence of bridging bone on CT scans, but also reported that two of those patients had screw back-out at the operative levels, and the authors did not have any dynamic image assessment. ⁵ In contrast, studies that included an assessment of motion on dynamic flexion-extension radiographs have reported 12 month fusion rates as low as 64-65%,^6,7 and multiple authors have concluded that using a CT scan alone tends to under-estimate non-union incidence.^2-4,6-8 Furthermore, fusion assessment with CT is particularly challenging in cases with metallic interbody devices or with non-bone graft materials such as ceramics given the scatter created on the image. Such significant discrepancies raise concern that the reportedly excellent fusion rates reported following ACDF surgeries may be artificially inflated due to questionable radiographic methodologies.

Thus, there is need for a standardization in the radiographic assessment of fusion status. Several recent reviews have each concluded that the preferred initial screening method for non-union should be to assess dynamic ISPM, with the cutoff for suspected non-union being more than 1 millimeter (mm) of motion.^2-4 This includes reviews from the research committee of the Cervical Spine Research Society, ³ as well as the Joint Section report from the American Association of Neurological Surgeons and the Congress of Neurological Surgeons. ⁹

However, the ability of radiographic reviewers to measure ISPM reliably and reproducibly with a 1 mm accuracy is not clear. The majority of studies have utilized a small number of reviewers (typically 1-3), and correlation coefficients for the agreement between reviewers is infrequently published.^1,3,9 The authors of these studies have typically been senior level members of the major cervical societies, and it is not clear if the skill of the expert reviewers in these papers is reproducible amongst the overall population of spinal surgeons. Furthermore, from a statistical standpoint, correlation co-efficients represent an estimate of the variance in the measurement. A high correlation implies the variance amongst reviewers is low, but does not support a specific numeric measurement cutoff. Some studies have used receiver operating characteristics to support a cutoff number for ISPM, but it is not clear if the observed variance in measurements actually supports the level of accuracy necessary for that cutoff to be statistically valid. Thus, the purpose of this study was to determine the minimum detectable distance that could be reliably measured by practicing surgeons during assessment of ISPM on flexion-extension radiographs, and to determine if the previously suggested measurement cutoff of <1 mm ISPM is statistically reproduceable amongst practicing surgeons.

Methods

Study Design

Institutional Review Board approval was obtained for this study (STUDY00000102). This retrospective review of preexisting records was deemed to provide no more than minimal risk to subjects and a waiver of patient consent was granted. AO Spine Knowledge Forum Degenerative (KF Degen) consists of 54 surgeons and researchers from over 15 countries around the world. Since 2019, cases from this group have been entered into a prospective clinical registry, which includes radiographic images. We queried this database to identify cases of ACDF performed with an interbody device and an anterior plate performed for a degenerative indication that had 12-month post-operative flexion-extension images available. Cases were excluded if they included circumferential instrumentation, if there were no posterior spinous processes available to measure, or if they were performed for a non-degenerative indication (trauma, tumor, infection, deformity). Images were also excluded if the spinous processes were not visible due to patient body habitus or due to poor radiographic technique. Lastly, the assessment of motion requires that the patient has put in a good effort in both the extension and flexion images. Prior authors have suggested a cutoff of >4 mm of ISPM at the level adjacent to the fusion to assess adequacy of the motion effort, and thus we excluded images if they did not meet this threshold. ¹⁰ Each radiograph was reviewed by a committee of the three senior authors (CM, TY, DR) to verify that it met the inclusion criteria. Images from 9 patients with 19 levels of surgery were included. We elicited participation from amongst KF Degen members, and 29 surgeons agreed to participate. Each participant is a board certified and fellowship trained spinal surgeon who regularly performs anterior cervical fusions.

Radiological Assessment and Statistical Analysis

Each participant was asked to view a 2-minute training video which discussed the rationale for the study and also demonstrated proper measurement technique (Supplementary Appendix Video 1). Specifically, each reviewer was instructed to zoom in to maximize the measured level within the field of view on the screen, with a minimum magnification of at least 150%. The contrast on the image was then adjusted to maximize the visible difference between the bony landmarks and the surrounding soft-tissues. Surgeons were next asked to identify an obviously identifiable landmark around the tip of the spinous process, with the landmark being visible on both flexion and extension views simultaneously (Figure 1). ¹⁰ Measurement of ISPM is affected by the distance from the axis of rotation that the measurement is taken. Measurements taken at the very tip of the spinous process are therefore likely to have larger ISPM values than measurements taken near the facet or in the mid aspect of the spinous process. For this reason, reviewers were specifically asked to identify bony landmarks near the tip of the spinous process for measurement and this was also noted in the instructional video. The measurement in millimeters between the spinous process landmarks was recorded on both the flexion and extension image. The ISPM was then determined by subtracting the distance measured on the extension image from the distance measured at the same level, and using the same landmarks, on the flexion image. Each participant was required to attest that they had viewed the video and had also practiced the measurement technique on at least 10 of their own cases prior to participating in the radiographic study.

Figure 1.

Flexion (left) and Extension (right) images showing an example of using at least 150% magnification and adjusting the image contrast to identify bony landmarks (marked by arrow) for ISPM measurement.

Each case was de-identified of patient information and assigned a number. Each reviewer was then sent DICOM files of the images and was instructed to load the images onto their local PACS system for measurement. The reviewers also measured each image a second time, at least two weeks after their first measurement, to allow calculation of intra-observer correlation. Inter-observer correlation coefficients (ICC), Intra-observer correlation coefficients, Standard Error of Measurement (SEM), and the Minimum Detectable Difference (MD) were calculated off these measurements. Correlation Coefficient estimates and their 95% confidence intervals (CI) were calculated using a 2-way random-effect repeated-measures analysis of variance (ANOVA). Values of ICC less than .5, between .5 and .75, .75 and .9, and greater than .90 indicate poor, moderate, good, and excellent reliability, respectively. ¹¹ The SEM was derived from the standard deviation (SD) of all motion values and the ICC, ie, SEM = SD. The MD was then finally derived, based on the SEM using the following formula: MD = SEM*1.96*. ¹² Statistical analyses were performed using the software SAS version 9.4 (SAS Institute Inc., Cary, NC, USA). Numeric variables are presented as mean ± standard deviation (SD). Categorical variables are summarized using counts (n) and percentages (%).

Lastly, we asked reviewers to report on the fusion status of the operative level. Reviewers were instructed to use their expert clinical opinion and to rely on a combination of their ISPM measurements as well as other factors such as the absence of bridging and remodeled bone around the interbody, presence of screw or hardware loosening, and radiolucent lines around implants or at the interbody-endplate interface. The reviewer could report that the level was obviously fused, an obvious non-union, or indeterminate. We then calculated ICC for this assessment.

Clerical and Data Processing Errors

Clerical and data processing errors in research studies are common, occurring in up to 28% of data-entries. ¹³ Examples of these types of errors in radiographic measurement studies include forgetting to type a decimal thus leading the measurement to be off by an order of magnitude and transcribing a measurement from the wrong level. To identify these errors, surgeons were asked to go back and review their measurements for any level in which their measurement differed from the group median by one absolute deviation or more. If a clerical error in one of the above categories was identified, then the old measurement was discarded and a new measurement was taken. For transparency, both the initial calculations and then the updated calculations after removal of the clerical errors are both presented here.

Results

In the first round of measurement the inter-observer ICC was .76, the SEM was 1 mm and the MD was 2.76 mm (Table 1, Figure 2). Intra-observer variability was .71 (.56, .82). Agreement on fusion status was moderate with an inter-observer ICC of .6 (.44; .76). Surgeons were asked to identify gross clerical errors in their first round of measurement. A total of 181 measurements were found to be more than 1 absolute deviation from the group mean and were thus re-evaluated by the original surgeon. 62 of the measurements were changed by >1 mm, 95 further cases were changed 0-1 mm, and for 22 cases the surgeon did not find any error upon reviewing their measurement and thus the number did not change. After revising those errors, the inter-observer ICC improved to .82, the SEM improved to .83 mm and the MD improved to 2.29 mm (Table 1, Figure 3). We performed a sub-analysis of the six highest performing reviewers who had an ICC above .9, and found that the SEM improved to .45 mm and the MD improved to 1.26 mm (Table 1, Figure 4).

Table 1.

Statistical Outcomes.

	Interrater CC	SEM (mm)	MD (mm)
All reviewers initial result	.76 (.64; .88)	1.00	2.76
All reviewers after removal of clerical errors	.82 (.71; .91)	.83	2.29
Six expert reviewers	.94 (.89; .97)	.45	1.26

CC, Correlation coefficient; SEM, Standard error of measurement; MD, Minimum detectable difference; mm, millimeter.

Figure 2.

Measurement of ISPM (mm) from all reviewers in the first round of measurement.

Figure 3.

Measurement of ISPM (mm) from all reviewers after the removal of clerical errors.

Figure 4.

Measurement of ISPM (mm) from the six expert reviewers.

Discussion

While multiple prior reviews have supported the importance of dynamic imaging in assessing fusion status after ACDF,^2,3,6,7 the specific motion cutoffs have varied substantially across studies. A review from 2018 found 11 prior articles that had published cutoffs for non-union based on ISPM, ranging from 0-3 mm. ⁴ If fusion is complete both anteriorly in the disc space and also posteriorly through the facets, then motion at that segment should be 0 mm. Based on this assumption, at least six prior studies suggested a cutoff of 0 mm of ISPM for determining non-union. ⁴ However, requiring strict adherence to this cutoff would mean any motion of .1 mm or more would be considered a non-union, and it seems unlikely that surgeons can reproducibly achieve that measurement precision. Furthermore, some motion may still be observed due to plastic deformation of the fusion mass when the disc space is fused, but the facets remain open, which is common following anterior fusions. Thus, the majority of prior authors have advocated for a cutoff of 1-2 mm, but the validity of these cutoffs is not well established.^1-3,7,10,14

The gold standard for assessment of fusion status is intra-operative exploration, but few prior studies have compared radiographic techniques against this standard in large numbers.^8,14,15 Song et al reported on 125 patients who had undergone operative exploration with confirmation of fusion status. ¹⁰ Radiographs from that series were measured by two independent reviewers, and a receiver operator curve analysis found that the ideal cutoff for assessment of non-union should be an ISPM of .9 mm, with a sensitivity and specificity similar to that from CT scans. ¹⁰ The ICC of the two measurers was .825 (95% CI: .782 to .862). Based in part on these results, research committees from both the Cervical Spine Research Society, ³ as well as the Joint Section report from the American Association of Neurological Surgeons and the Congress of Neurological Surgeons, ⁹ have concluded that assessment of ISPM on dynamic radiographs should be the initial screening study for non-union, with a suggested cutoff of 1 mm.

However, the studies cited by the research committee reviews typically report only CC values for the observer’s measurements, and do not report on the minimum detectable difference (MD). Receiver operating characteristics of ISPM measurements have been used to support a specific cutoff of <1 mm, ¹⁰ but the ability of surgeons to accurately measure ISPM <1 mm is not clear. Thus, in this study, we aimed to determine the MD of ISPM measurements amongst practicing spine surgeons. Calculation of the MD represents the measurement distance at which it can be statistically certain that two measurements are different from each other, and takes into account both the precision of the number and also the reproducibility of the number across the multiple reviewers. Measurements less than the MD may be different simply due to measurement technique, and may not represent a true difference.

The ICC’s reported here (.76-.82) are high and similar to those reported by other studies, ¹⁰ which indicates that the ISPM for each level can be reliably distinguished. However, the MD was 2.29 mm, which indicates that the precision of the measured values was not high enough to support a statistical cutoff of <1 mm. In other words, an average radiographic reviewer may not have enough precision to reliably and consistently identify measurement differences <1 mm. Furthermore, the ICC agreement on fusion status was only .6, which implies moderate variability. This indicates that the surgeons in this study did not have consistently reliable assessments of fusion status based on dynamic radiographs alone, and implies that prior studies which have relied solely on plain radiographs for fusion assessment may not be accurately classifying the non-unions.

There could be several reasons for this. First, surgeons participating in a research study in which they have to take a large number of measurements may become fatigued over the course of taking many measurements, leading to a decrease in their precision. In contrast, in a clinical setting, the surgeon only needs to measure one patient’s radiographs at a time, and that measurement is likely to have immediate clinical relevance. Therefore, surgeons are likely to take more time or take greater care, and that may result in better precision than we observed in this research setting. Second, it is clear that the quality of measurements varied across reviewers. We performed a sub-analysis of the 6 highest performing reviewers who had ICC >.90 and found a MD of 1.26 mm, which is significantly more precise than the overall group MD of 2.29 mm. We asked these reviewers about their technique, and they report taking significant time and care in the measurements, in some cases up to 10 minutes per measurement, as opposed to 1 minute or less for most reviewers. This implies that the technique is quite reliable in the hands of experts who take great care in making the measurement. Thus, although the group average ISPM MD of 2.29 was above the common clinical thresh-hold of 1 mm, we believe our results do support the use of ISPM as a radiographic technique for expert reviewers. Bringing the image into maximum magnification (at least 150% or greater) and using the contrast tool to identify clear and reproduceable bony landmarks were common strategies used by the best performing reviewers to increase the precision of their technique (Figure 1).

Lastly, in the initial round of statistical analysis we identified several instances of clerical errors, including examples of transcribing a measurement from the wrong level, forgetting a decimal, or typing a wrong number by mistake. We believe that this supports our hypothesis that fatigue and imprecision played a significant role in our results; few surgeons would make the mistake of calling an interspinous process motion of 2 mm as being 2 centimeters (cm) in a clinical setting, since 2 cm would be grossly abnormal. A prior systematic review reported that data-processing errors of this kind are common, occurring in up to 28% of data entries (up to 2784 errors per 10 000 data-fields). ¹³ Each reviewer in this study sent us their measurements in a spread-sheet, which meant that we could not directly view how they took the measurement. We identified some of these errors by asking each reviewer to go back and check their measurement whenever it was more than one absolute deviation greater than the group mean. However, this method is imprecise, and may have missed some clerical errors that were less than one deviation from the mean. Given the prevalence of these data processing errors both in our study, as well as the reportedly high rate from prior review, ¹³ we suggest that future studies have some method of auditing the measurement itself. Specifically, a screenshot of the measurement could be saved and sent along with the spreadsheet so that if questions arise the image can be directly audited.

In contrast to prior studies that have utilized 1-3 radiographic reviewers, the primary strength of this study is the recruitment of 29 practicing spine surgeons from around the world, and its use of statistical techniques to calculate the minimum detectable difference in surgeon measurements. The reviewers utilized the imaging software at their home institution to make the measurements, which is meant to reproduce a real-world clinical scenario for how the measurements are regularly taken.

In summary, radiographic assessment of ISPM by human reviewers can be accurate in the hands of expert reviewers who take great care in the measurements, and approaches the precision needed to be statistically certain when using a motion cutoff of <1 mm. However, the average reviewer in this study was less precise with an MD of 2.29 mm, and agreement on fusion status was only moderate. Thus, an average reviewer may not be able to rely on dynamic radiographs alone for assessment of fusion status, and may benefit from additional imaging such as CT scans, or AI based computer aids. Prior studies that rely solely on dynamic radiographs may have inaccurately classified some cases of non-unions. Clerical errors can occur in data transcription, and future studies would benefit from incentivizing precise reading of radiographs as well as employing a method that allows auditing of the measurements for accuracy.