Abstract
The decision to intervene for internal carotid stenosis often depends on the degree of stenosis seen on duplex ultrasound (US). The aim of this study is to compare the diagnostic accuracy of two criteria: modified University of Washington (UW) and 2003 Carotid Consensus Panel (CCP). All patients undergoing US in an accredited (IAC) vascular laboratory from January 2010 to June 2015 were reviewed (n=18,772 US exams). Patients receiving a neck computed tomography angiography (CTA) within 6 months of the US were included in the study (n=254). The degree of stenosis was determined by UW/CCP criteria and confirmed on CTA images using North American Symptomatic Carotid Endarterectomy Trial (NASCET)/European Carotid Surgery Trial (ECST) schema. Kappa analysis with 95% confidence intervals (CIs) were utilized to determine duplex–CTA agreement. A total of 417 carotid arteries from 221 patients were assessed in this study. The modified UW criteria accurately classified 266 (63.9%, kappa = 0.321, 95% CI 0.255 to 0.386) cases according to NASCET-derived measurements. The sensitivity, specificity, and accuracy at ≥ 60% stenosis were 65.7%, 81.3%, and 81.9%. The CCP criteria resulted in 296 (70.9%) accurate diagnoses (kappa = 0.359, 95% CI 0.280 to 0.437). At ≥ 70% stenosis, the sensitivity, specificity and accuracy were 38.8%, 91.6%, and 87.1% for NASCET. Comparison of the duplex results to ECST-derived CTA measurements revealed a similar trend (UW 53.1%, κ = 0.301 vs CCP 62.1%, κ = 0.315). The CCP criteria demonstrate a higher concordance rate with measurements taken from CTAs. The CCP criteria may be more sensitive in classifying clinically significant degrees of stenosis without a loss in diagnostic accuracy.
Keywords
Introduction
Carotid duplex ultrasonography is largely used as a first line diagnostic tool for carotid stenosis in symptomatic and asymptomatic patients. The decision to offer surgical intervention is often dependent on duplex velocity in addition to patient presentation. Computer tomographic arteriography (CTA) has been found to be highly accurate for diagnosing the degree of stenosis with overall sensitivity and specificity values of 97% and 99%, respectively, 1 with the added benefit over carotid angiography of being non-invasive. Although CTA is very accurate for calculating the degree of stenosis, the advantages of duplex are the low cost, avoidance of contrast and radiation, and the ability to assess flow dynamics of the vessels.
There are several duplex criteria used to determine the degree of stenosis. At our institution, we transitioned to the 2003 Carotid Consensus Panel criteria (CCP). 2 The previous criteria used to interpret duplex velocities was a modification of the University of Washington (UW) thresholds. 3 The modification was based on work by Neale et al. 4 and had been internally validated in our vascular laboratory. The difference between the two criteria is most pronounced in stenosis >40–50%. The UW has four stratifications excluding occluded vessels while the CCP has three (Table 1). The objective of our study was to determine the diagnostic accuracy of the two different duplex criteria sets by comparing estimated values to CTA measurements of carotid stenosis.
Outlined criteria for the modified University of Washington (UW) system versus the 2003 Carotid Consensus Panel (CCP) criteria duplex system.
Primary criteria.
CCA, common carotid artery; EDV, end diastolic velocity; ICA, internal carotid artery; PSV, peak systolic velocity.
Methods
Study population
A retrospective review via current procedure terminology (CPT) query was conducted of all patients in our hospital system who received both a carotid ultrasonography (US) duplex study and a neck CTA within 6 months of the US between January 2010 and July 2015. The US duplex studies were performed in an Intersocietal Accreditation Commission (IAC) vascular laboratory. The indication for duplex included suspected symptomatic or asymptomatic cerebrovascular disease. The vessel of interest was the internal carotid artery (ICA). The vessel was excluded from the study data set if it had an occlusion or if there was an occlusion noted in the contralateral ICA, since this can elevate peak systolic velocities measured in the non-occluded artery. Exclusion parameters also included the presence of pseudoaneurysms, trauma, dissections, carotid body tumors, or previous endovascular carotid intervention. A total of 254 consecutive patients were captured by the query. Excluded patients were 13 with poor CTA quality, 16 with no duplex data, and 14 with carotid occlusion, resulting in 221 patients and a total of 417 carotids included in this study. IRB approval was obtained prior to initiation of the study, and patient informed consent was not required.
Computer tomographic arteriography
CTA scans were performed according to our institution’s protocol using 0.8–2-mm slices. Three-dimensional (3D) reconstructions were made using Sectra’s (Sweden) centerline vessel analysis tool from PACS imaging software. It reconstructed CTA image slides by mapping out the contrast opacity of the vessel and displaying the cross-sectional area and diameter at a given position (Figure 1). Degree of stenosis was calculated using the North American Symptomatic Carotid Endarterectomy Trial (NASCET) method 5 and the European Carotid Surgery Trial (ECST) method. 6 For NASCET, all diameter measurements were from the narrowest portion of the vascular lumen and were compared to the normal lumen superior to the lesion. For ECST, the stenosis was compared to the total diameter of the ICA. Each CTA was looked at by two members of the research lab trained in vascular imaging. Each worked independently and was blinded to the other’s readings. The averages of each value were calculated. If there was a severe discrepancy between the two measurements (i.e., an ICA was placed in two different stenosis categories by each reader), a third reader would measure blindly. Inter-observer agreement between readers was approximated by Spearman’s correlation coefficient of reader 1 versus reader 2. The data was compiled and any discordant readings not cleared by the third reader (usually measurements that were near the thresholds for higher grade stenosis) were adjudicated by an experienced vascular radiologist or surgeon.

CTA measurements of the stenosis were made using centerline data. Panels ‘a’ and ‘d’ demonstrate a coronal cross section of the common and internal carotid artery. The diameters at multiple axial cross sections are measured (b) to determine the maximum level of stenosis in the internal carotid artery. A 3D reconstruction (c) is generated by the centerline software as a visual aid. (Note: figure is in color online.)
Carotid duplex ultrasound
All carotid duplex US were performed by registered vascular technologists and reviewed by vascular surgeons and vascular medicine specialists with extensive US experience including Registered Physician in Vascular Interpretation (RPVI) or Registered Vascular Technologist (RVT) certification. Carotid duplex US was performed with an iU22 Ultrasound System (Philips, The Netherlands). Doppler spectral velocity analysis was performed using as small a sample volume as possible and a Doppler angle of insonation correction of < 60 degrees. Imaging was acquired utilizing a multi-frequency linear transducer with pulse wave Doppler. Standard protocol is to perform duplex while the patient is supine with the head elevated and rotated towards the contralateral side. The proximal to distal common carotid, external carotid, segments of the internal carotid (proximal, middle and distal) and vertebral arteries are examined. Data was uploaded and saved into our database.
Stenosis grading
For each patient, the degree of stenosis was categorized to both UW and CCP criteria based on peak systolic velocities. These were then compared to CTA stenoses that were determined using the vessel analysis. Kappa statistics including 95% confidence intervals (CIs ) were used to analyze agreement between the duplex and CTA findings (NASCET/ECST). The level of agreement was divided into the following: a value less than 0.20 demonstrated poor agreement; 0.21–0.40 demonstrated fair agreement; 0.41–0.60 demonstrated moderate agreement; 0.61–0.80 demonstrated good agreement; and 0.81–1.0 represented very good agreement.
Results
A total of 417 carotid arteries from 221 patients were assessed. Four comparisons were made:
stenosis category based on the UW duplex criteria versus CTA measurements derived from the NASCET method (UW-NASCET group)
stenosis category in the CCP duplex criteria versus CTA measurements derived from the NASCET method (CCP-NASCET)
stenosis category in the UW duplex criteria versus CTA measurements derived from the ECST method (UW-ECST)
stenosis category in the CCP duplex criteria versus CTA measurements derived from the ECST method (CCP-ECST).
Inter-observer agreement by Spearman’s correlation of the two CTA readers was 82% and 84% for NASCET and ECST, respectively (Figure 2A and B).

CTA measurements for each reader were plotted and Spearman’s correlation of the readers was calculated to assess inter-observer agreement. (a) NASCET inter-observer agreement was 82%. (b) ECST inter-observer agreement was 84%. CTA, computer tomographic arteriography; ECST, European Carotid Surgery Trial; NASCET, North American Symptomatic Carotid Endarterectomy Trial.
Concordance rates (Table 2) between UW criteria and NASCET-derived CTA stenosis measurements were as follows: 266 carotids (63.9%, n = 416) were concordant, 97 carotids (23.3%) had been placed in duplex categories that overestimated the degree of stenosis measured by CTA, and 53 carotids (12.7%) were underestimated based on duplex. In the comparison between CCP duplex thresholds and NASCET-derived measurements, there was a higher concordance rate between the designated duplex classification and CTA measurements. A total of 296 carotids (70.9%) were placed into a duplex category that also correlated with the degree of stenosis obtained from CTA measurements, while 76 carotids (18.2%) were overestimated and 45 carotids (10.7%) were underestimated by duplex (Table 2). Agreement was deemed to be ‘fair’ by kappa value 0.321 (SE-κ 0.034, 95% CI 0.26–0.39) for the UW-NASCET group and 0.359 (SE-κ 0.040, 95% CI 0.28–0.44) for the CCP-NASCET group.
Rate of concordance between stenosis based on duplex versus CTA measurements.
CCP, 2003 Carotid Consensus Panel; CTA, computer tomographic arteriography; ECST, European Carotid Surgery Trial; NASCET, North American Symptomatic Carotid Endarterectomy Trial; UW, University of Washington.
When the UW duplex classification was compared to the degree of stenosis obtained from ECST-derived CTA measurements, 221 carotids (53.1%, n = 417) were concordant, 60 carotids (14.4%) were overestimated, and 134 (32.2%) were underestimated measurements with a kappa value of 0.301 (SE-κ 0.032, 95% CI 0.237–0.364). In the CCP-ECST group, 259 carotids (62.1%) were concordant, 44 carotids (10.5%) were overestimated, and 115 carotids (27.5%) were underestimated measurements with a kappa value of 0.315 (SE-κ 0.036, 95% CI 0.24–0.49) (Table 2). Figure 3 graphically demonstrates the spread of CT measurements within each duplex category.

The spread of CTA stenosis measurements was plotted for each duplex category. The red line represents the average measurement and standard error of mean.
Figure 4 depicts correlation by linear regression of each stenosis designation by duplex criteria to its corresponding CTA measurement. The R2 for the UW-NASCET group versus UW-ECST was 0.393 and 0.327, respectively. The regression R2 for CCP-NASCET was 0.345, while the R2 for CCP-ECST was 0.280.

Linear regression was performed for each duplex stenosis designation to its corresponding CTA measurement.
Sensitivity and specificity were calculated for each group and receiver operating characteristic (ROC) curves were generated at points of overlap within the duplex criteria. In the UW group, ≥ 40%, ≥ 60%, and 80% stenoses were examined (Table 3A, Figures 5 and 6). Of note, the sensitivity, specificity, and accuracy at ≥ 60% stenosis were 65.7%, 81.3%, and 81.9% for NASCET and 57.7%, 91.4%, and 80.4% for ECST. At ≥ 80% stenosis, the sensitivity, specificity, and accuracy were 70.0%, 91.8%, and 91.3% for NASCET and 34.2%, 92.6%, and 87.7% for ECST. In the CCP group, ≥ 50% and ≥ 70% stenosis were examined (Table 3B, Figure 7). The sensitivity, specificity, and accuracy at ≥ 50% stenosis using CCP criteria were 73.3%, 82.3%, and 80.1% for NASCET and 57.5%, 87.7%, and 74.8% for ECST. At ≥ 70% stenosis using CCP criteria the sensitivity, specificity and accuracy were 38.8%, 91.6%, and 87.1% for NASCET and 33.3%, 95.3%, and 81.5% for ECST.
Stenosis detection using UW criteria: sensitivity and specificity.

Receiver operator curve (ROC) and the associated table for determining 60% or greater stenosis with UW duplex criteria when compared against NASCET-derived CTA measurements.

Receiver operator curve (ROC) and the associated table for determining 80% or greater stenosis with UW duplex criteria when compared against NASCET-derived CTA measurements.
Stenosis detection using CCP criteria: sensitivity and specificity.
CCP, 2003 Carotid Consensus Panel; ECST, European Carotid Surgery Trial; NASCET, North American Symptomatic Carotid Endarterectomy Trial; NPV, negative predictive value; PPV, positive predictive value; UW, modified University of Washington criteria.

Receiver operator curve (ROC) and the associated table for determining 70% or greater stenosis with CCP duplex criteria when compared against NASCET-derived CTA measurements.
CCP utilizes the PSV as the primary threshold, but also includes ratios and EDV as secondary criteria. Comparisons of sensitivity, specificity, and accuracy were also calculated utilizing PSV, PSV and ICA/CCA ratio, and all three cut-off criteria for the CCP > 70% stenosis (PSV, ICA/CAA ratio, and EDV) category using both NASCET and ECST derived measurements (Table 4). Sensitivity was highest utilizing NASCET CTA measurements when PSV was the sole cut-off value used (38.8%). Of note, the sensitivity decreased with the addition of the ICA/CCA ratio (22.9%) and both the ratio and EDV (11.4%). Specificity was higher regardless of cut-off threshold, when ECST measurements were used. Similar to sensitivity, accuracy was highest in the PSV only – NASCET group (87.1%) – and trended downward with the addition of the ICA/CCA ratio and EDV.
Stenosis detection using CCP criteria: sensitivity and specificity by >70% cut-off criteria.
CCP, 2003 Carotid Consensus Panel; ECST, European Carotid Surgery Trial; EDV, end diastolic velocity; ICA/CCA ratio, internal carotid artery/common carotid artery; NASCET, North American Symptomatic Carotid Endarterectomy Trial; NPV, negative predictive value; PPV, positive predictive value; PSV, peak systolic velocity.
Discussion
Previously published randomized control trials have shown the benefit of intervention for carotid stenosis in symptomatic and asymptomatic patients.5–7 These trials have mainly tried to delineate the benefit of carotid endarterectomy in symptomatic patients with severe stenosis (70–99%). They have also shown some benefit in symptomatic patients with moderate stenosis (50–69%). The improvement in intensive medical therapy has made these decisions harder with older data. For patients with asymptomatic moderate carotid stenosis, surveillance is an important component of a patient’s care. In one retrospective study by Olin et al., the frequency of patients with 60–79% internal carotid stenosis progressing to a higher category stenosis (80–99% stenosis or occlusion) was 5% at 1 year, 11% at 2 years and 20% at 3 years, and they were more likely to present with symptoms. 8 Many laboratories across the country have used varying duplex criteria to identify patients, symptomatic or asymptomatic, that require intervention.9,10 Some surgeons and interventionists rely solely on duplex scanning, although there has been a move towards duplex in addition to CTA for procedural planning.1,11–15
The impetus for this study stemmed from an inconsistency in reporting from different imaging departments. As our hospital system acquired new hospitals, it was noted that different criteria were used, with the UW predominantly used in vascular laboratories and CCP in radiology departments. Migrating to one set of duplex criteria would lead to less confusion among patients, non-vascular providers, and staff by avoiding disparate information when patients received scans from varying departments. However, questions remained if one set of criteria was better than the other, prompting this study. Importantly, parsing out the differences between the two sets of criteria can help prevent unnecessary consultations, imaging, and interventions that drive up cost to the patient and to the healthcare system, as shown in a previous study. 9
Our results demonstrated a higher concordance rate between the CCP classification of stenosis with CT measurements for both NASCET- and ECST-derived stenosis measurements, compared to the UW. When the UW duplex guidelines were used, there were more instances when the carotid was placed in a stenosis category that was higher than what was actually measured on the CTA (UW-NASCET 23.3%, UW-ECST 14.4%), more so than with the CCP criteria (CCP-NASCET 18.2%, CCP-ECST 10.5%). Kappa analysis demonstrated greatest agreement between the CCP duplex classification and NASCET-derived CTA measurements (0.359 (SE-κ 0.040, 95% CI 0.28–0.44)) than the UW criteria (0.321 (SE-κ 0.034, 95% CI 0.26–0.39)). A similar trend was observed when comparing kappa values for CCP and UW duplex criteria to ECST.
Overall, CTA measurements using NASCET measurements have better agreement with either of the duplex criteria than ECST. In addition, there was less incidence of a carotid stenosis being overestimated under the NASCET method, based on peak systolic velocities using either of the duplex criteria. This was not surprising, as both duplex criteria were formulated and verified using NASCET stenosis measurements from angiography. Furthermore, compared to NASCET, the ECST method often yields a higher calculated stenosis for a given carotid artery.
As was the case with CTA concordance, the CCP duplex classification system was more specific and accurate in classifying clinically significant degrees of stenosis, especially for those with moderate stenosis (UW specificity and accuracy at ≥ 60% stenosis: NASCET 81.3% and 81.9%, ECST 91.4% and 80.4% vs CCP specificity and accuracy at ≥ 70% stenosis: NASCET 91.6% and 87.1%, ECST 95.3% and 81.5%). Interestingly, the specificity was higher when compared against ECST regardless of which set of US criteria were used. In a study by AbuRahma, where the CCP duplex criteria were validated against angiography, they reported similar findings of a sensitivity of 99%, specificity of 86%, and an overall accuracy of 95% for detecting ≥ 70% stenosis using a PSV of ≥ 230 cm/s. 16
When comparing the sensitivity, specificity, and accuracy within the CCP ≥ 70% stenosis category, we found that specificity did not vary by measurements using solely PSV, PSV and the ICA/CCA ratio, and all three components (PSV, ICA/CCA ratio, and EDV). However, there were downward trends in both sensitivity and accuracy with each additional criterion added to PSV due to increased false negative duplex findings (Table 4). This observation suggests that PSV may be a more reliable predictor of degree of stenosis.
Limitations
There are several limitations of this study. First, interpretation of the duplex ultrasound was based mostly on peak systolic velocities. Other factors such as end diastolic velocities and ICA/CCA ratios were not analyzed. This may have contributed to the lower sensitivity of duplex imaging found in this study than what was previously reported in other literature. 2 Second, there is also inherent inter-observer variability in calculating stenosis degree on CTA images. Sources of variation introduced by the reader can include differences in windowing to observe calcification, in defining the centerline to obtain the axial cuts across the vessel, and in the reader’s interpretation of what is the narrowest lumen in a morphologically irregular calcification. Even their interpretation of a normal lumen in a vessel with post-stenotic narrowing, for example, can cause wide differences in their estimates of stenosis. Although all individual observers were taught to measure stenosis by a vascular radiologist and the inter-observer agreement was high, observers were not formally trained radiologists. Both these factors may have contributed to a lower kappa value than expected. Another possibility is that both the UW and CCP duplex criteria had been derived and verified against measurements from angiographies and not CTAs. There are also technical considerations to be made, including variations in how the ultrasound was performed. While our duplexes were performed in an accredited vascular lab where examinations were performed according to a standardized protocol in terms of positioning and Doppler angle, there is inherent variability introduced when different sonographers are used.
Conclusion
The importance of this study is not only to unify to one consistent and more accurate system of measurement for our institution, but also to devise more appropriate surveillance and a pathway to intervention if necessary. Not all patients that have carotid disease determined on duplex US need undergo CT angiography to further classify stenosis. Surgeons and interventionists should have a consistent and accurate assessment specification of carotid stenosis using low cost imaging to guide their need for intervention on patients with carotid disease.
Footnotes
Acknowledgements
These data were presented in part at the 2016 Annual Meeting of the Midwestern Vascular Surgery Society, Columbus, OH, 8–10 September 2016.
The authors are grateful to the Department of Radiology at University Hospitals and the members of the Vascular Laboratory at University Hospitals.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
