Reliability and accuracy of individual Alberta Stroke Program Early CT Score regions using a medical and a smartphone reading system in a telestroke network

Abstract

Introduction

The aim of this study was to assess individual regions of the Alberta Stroke Program Early CT Score in noncontrast head computed tomography interpretations using a smartphone in a telestroke network, by comparison to a medical monitor.

Methods

The review board of our institution approved this retrospective study. A factorial design with 188 patients, four radiologists and two reading systems was used. Accuracy and reliability were evaluated.

Results

Very good interobserver agreements were observed on the total Alberta Stroke Program Early CT Score for both the medical and smartphone reading systems, with intraclass correlation coefficients of 0.91 and 0.84 respectively. Interobserver agreements were moderate to very good for the medical reading system (all intraclass correlation coefficients >0.74), whereas they were fair to very good for the smartphone (intraclass correlation coefficients ranged from 0.31–0.81). All intraobserver agreements were good (intraclass correlation coefficient >0.64), except for internal capsule (0.48) and M2 (0.55) regions. The areas under the receiver-operating curve ranged from 0.69–0.89 for the medical system, while for the smartphone ranged from 0.44–0.86. No statistical differences were observed between medical and smartphone reading systems for each region (all p > 0.05).

Discussion

If radiologists are better trained in the evaluation of the lesions in the insula, the internal capsule and the M2 regions, the total and the dichotomised Alberta Stroke Program Early CT Score will be more precise. Hence, ruling out contraindications to thrombolysis administration will be improved, allowing assessment of head computed tomography in a telestroke network using a smartphone to be a common practice.

Keywords

Accuracy agreement Alberta Stroke Program Early CT Score displays reliability receiver-operating characteristic smartphone stroke telehealth telestroke

Introduction

In emergency telestroke networks, noncontrast head computed tomography (NCCT) is still the first imaging examination to detect haemorrhagic and ischaemic strokes in patients with acute neurological deficits.^1,2 The Alberta Stroke Program Early CT Score (ASPECTS)³ is commonly used to evaluate the size and location of ischaemic lesions in the middle cerebral artery (MCA) territory. The ASPECTS measures the extension of an ischaemic insult according to hypodensities in 10 specific brain regions. Scores <6 indicate that a stroke is greater than two-thirds of the MCA territory, and has been considered a major contraindication for the use of intravenous (IV) thrombolytic therapy with recombinant tissue plasminogen activator (rtPA).^1,2,4,5

In order to increase the availability of neuroradiologists, and to improve reading times, mobile solutions based on computer tablets, laptops and smartphones have been recently evaluated.^6–8 Literature has been published analysing dichotomised ASPECTS scores but not individual ASPECTS regions using mobile solutions. One study has focused on differences in individual ASPECTS regions between head NCCT scans and other diagnostic modalities, such as head CT angiography (CTA) and CT perfusion (CTP).⁹ To the best of our knowledge, there is no study comparing the potential impact of using a mobile solution such as a smartphone to score different ASPECTS regions on head NCCTs. We aimed to study the accuracy and reliability of a smartphone in the evaluation of individual ASPECTS regions in patients with acute ischaemic stroke.

Methods

The Institutional Review Board (IRB) of our hospital approved this retrospective study and the requirement for informed consent was waived. A factorial design with 188 patients, four radiologists, and three reading systems was used. Accuracy and reliability were evaluated. A complete description on how to perform dichotomised ASPECTS evaluation can be found elsewhere.⁶ Herein, we present a full description on individual ASPECTS regions and its corresponding functional anatomy.

The ASPECTS score is based on a segmental assessment of 10 defined regions of the MCA as follows: C: caudate head, I: insular ribbon, IC: internal capsule, L: lentiform nucleus, M1: anterior MCA cortex, M2: MCA cortex lateral to insular ribbon, M3: posterior MCA cortex, M4: anterior MCA territories, M5: lateral MCA territories, and M6: posterior MCA territories (Figure 1). For each of these regions, the observer reports the presence of hypoattenuations every time that a subacute, chronic ischaemic or a haemorrhagic lesion has been excluded. A normal ASPECTS score is 10, which indicates the total absence of hypodensities. However, hyperacute infarcts can also present with normal ASPECTS scores in a small percentage of patients. The final ASPECTS evaluation is calculated by subtracting one point for each region with early infarct signs from the maximum score of 10.

Figure 1.

Individual regions for evaluation of the Alberta Stroke Program Early CT Score (ASPECTS). C: caudate head; I: insular ribbon; IC: internal capsule; L: lentiform nucleus; M1: anterior MCA cortex; M2: MCA cortex lateral to insular ribbon; M3: posterior MCA cortex; M4: anterior MCA territories; M5: lateral MCA territories; M6: posterior MCA territories; MCA: middle cerebral artery.

The sample included 188 randomly selected NCCT examinations of patients who presented to our hospital emergency room between 2013–2018 with acute stroke symptoms, and for whom the stroke code was activated. The gold standard was defined by three experienced neuroradiologists in our hospital (two-thirds of the available neuroradiologists in agreement).

Image analysis was performed by four neuroradiologists (three with more than 10 years of experience and one with four years of experience in neuroradiology), using a medical workstation and a smartphone, for a total of 1504 interpretations. The routine reading system for CT interpretations at our hospital is a medical workstation with the Digital Imaging and Communication in Medicine (DICOM)-compliant viewer software Agfa IMPAX 6.5 (AGFA HealthCare, Mortsel, Belgium). Images were displayed using an E-2620 BARCO monitor (BARCO N.V., Kortrijk, Belgium), which is a two-megapixel LCD grayscale medical display with a dot pitch of 0.249 mm, a spatial resolution of 1600 × 1200 pixels, and maximum luminance of 700 cd/m². This reading system, hereafter referred to as Medical-IMPAX, was used as the reference device in this study. The smartphone selected was the Samsung Galaxy S8 Plus (Samsung Electronics, South Korea) smartphone, with a display of 146.5 mm (5.8 inches), 570 pixels per inch, a spatial resolution of 1440 × 2960 pixels, and a maximum luminance of 1000 cd/m² was selected. The viewer software used on this smartphone was the Agfa XERO Viewer 3.0 (Agfa HealthCare, Mortsel, Belgium). This reading system is hereafter referred to as Smartphone-XERO. Both reading systems were used in an interpretation room with controlled illumination, having ambient light of approximately 20 lux, according to the American Association of Physicists in Medicine (AAPM) TG18 recommendations.¹⁰

Interpretations were performed over several sessions in which the reading system used was selected randomly, as well as the patients order for each session.

The reliability evaluation was performed by using intraobserver and interobserver reliability measurements such as the intraclass correlation coefficient (ICC) as absolute agreement, based on a mean-rating (four raters) on a two-way mixed-effects model. ICC estimates and their 95% confidence intervals were calculated using the IBM SPSS Statistics 19 software (IBM Corp., Armonk, New York, USA). ICC values less than 0.5 are indicative of poor reliability, values between 0.5–0.75 indicate moderate reliability, and values between 0.75–0.9 indicate good reliability. Values greater than 0.90 indicate excellent reliability.¹¹

In addition, receiver-operating characteristic (ROC) curves were performed to evaluate diagnostic accuracy. The area under the curve (AUC) for ROC curves and trapezoidal nonparametric ROC curves was plotted¹² by averaging the curves of each radiologist using the threshold method¹³ with STATA 13 software (Stata Corp., College Station, Texas, USA).

Results

Of the 188 patients, 90 (47.87%) were male. The patients’ ages ranged from 30–97 years, with a mean of 71.3 years. According to the gold standard, the distribution of cases, of interest for the ASPECTS evaluation, were as follows: 25 haemorrhagic lesions, 109 with any ischaemic lesion in the MCA territory, and 68 with acute ischaemic lesions involving the MCA territory.

Table 1 shows the interobserver agreement among four observers by individual ASPECTS regions and reading systems. Table 2 shows the intraobserver agreement between medical and smartphone reading systems by individual ASPECTS regions. The AUCs for medical and smartphone reading systems specifying individual ASPECTS regions are shown in Table 3. No statistical differences were observed between medical and smartphone reading systems for each ASPECTS region (all p > 0.05). The ROC curves are shown in Figures 2 and 3. For each region, the shapes of the ROC curves were very similar for all reading systems with exception of the insular ribbon and M2. Although moderate intraobserver agreement was found for the internal capsule, adequate and similar AUC values were documented for medical and smartphone reading systems (0.72 and 0.73, respectively) as well as identical ROC curves for the aforementioned ASPECTS regions (Figure 2).

Table 1.

Interobserver agreement among four observers by individual Alberta Stroke Program Early CT Score (ASPECTS) regions and reading systems.

Region	Reading system	ICC	95% CI	Agreement
Total ASPECTS	Medical	0.91	0.81–0.96	Very good
	Smartphone	0.84	0.68–0.93	Very good
By region
C: caudate head	Medical	0.89	0.77–0.95	Very good
	Smartphone	0.81	0.63–0.92	Very good
IC: internal capsule	Medical	0.74	0.48–0.89	Good
	Smartphone	0.31	–0.33–0.7	Fair
L: lentiform nucleus	Medical	0.85	0.7–0.94	Very good
	Smartphone	0.84	0.68–0.93	Very good
I: insular ribbon	Medical	0.57	0.14–0.82	Moderate
	Smartphone	0.46	–0.11–0.77	Moderate
M1: anterior MCA cortex	Medical	0.78	0.56–0.9	Good
	Smartphone	0.64	0.26–0.85	Good
M2: MCA cortex lateral to insular ribbon	Medical	0.75	0.5–0.89	Good
	Smartphone	0.40	–0.15–0.74	Moderate
M3: posterior MCA cortex	Medical	0.68	0.35–0.86	Good
	Smartphone	0.70	0.4–0.87	Good
M4: anterior MCA territories	Medical	0.64	0.29–0.84	Good
	Smartphone	0.74	0.49–0.89	Good
M5: lateral MCA territories	Medical	0.72	0.45–0.88	Good
	Smartphone	0.71	0.42–0.87	Good
M6: posterior MCA territories	Medical	0.80	0.6–0.91	Good
	Smartphone	0.70	0.42–0.87	Good

CI: confidence interval; ICC: Intraclass correlation coefficient; MCA: middle cerebral artery.

Table 2.

Intraobserver agreement between medical and smartphone systems by individual Alberta Stroke Program Early CT Score (ASPECTS) regions.

Region	ICC	95% CI	Agreement
Total ASPECTS	0.85	0.76–0.9	Very good
By region
C: caudate head	0.74	0.58–0.83	Good
IC: internal capsule	0.48	0.18–0.67	Moderate
L: lentiform nucleus	0.71	0.54–0.81	Good
I: insular ribbon	0.67	0.47–0.79	Good
M1: anterior MCA cortex	0.67	0.47–0.79	Good
M2: MCA cortex lateral to insular ribbon	0.55	0.29–0.71	Moderate
M3: posterior MCA cortex	0.64	0.42–0.77	Good
M4: anterior MCA territories	0.67	0.48–0.79	Good
M5: lateral MCA territories	0.73	0.58–0.83	Good
M6: posterior MCA territories	0.66	0.45–0.78	Good

CI: confidence interval; ICC: Intraclass correlation coefficient; MCA: middle cerebral artery.

Table 3.

Area under the curve (AUC) for the receiver-operating characteristic (ROC) by individual Alberta Stroke Program Early CT Score (ASPECTS) regions and reading systems.

Region	Reading system	AUC	SE	95% CI	Chi-square	p-Value
C: caudate head	Medical	0.89	0.042	0.81–0.97	1.10	0.295
	Smartphone	0.83	0.050	0.73–0.92
I: insular ribbon	Medical	0.69	0.146	0.41–0.98	2.96	0.085
	Smartphone	0.44	0.019	0.41–0.48
IC: internal capsule	Medical	0.72	0.145	0.44–1	0.00	0.973
	Smartphone	0.73	0.145	0.45–1
L: lentiform nucleus	Medical	0.87	0.039	0.8–0.95	0.03	0.854
	Smartphone	0.86	0.040	0.78–0.94
M1: anterior MCA cortex	Medical	0.82	0.045	0.73–0.91	1.80	0.180
	Smartphone	0.74	0.052	0.64–0.84
M2: MCA cortex lateral to insular ribbon	Medical	0.80	0.052	0.7–0.91	2.44	0.119
	Smartphone	0.68	0.069	0.54–0.81
M3: posterior MCA cortex	Medical	0.72	0.053	0.61–0.82	0.77	0.379
	Smartphone	0.67	0.057	0.55–0.78
M4: anterior MCA territories	Medical	0.72	0.049	0.62–0.81	0.47	0.493
	Smartphone	0.68	0.048	0.59–0.78
M5: lateral MCA territories	Medical	0.69	0.059	0.57–0.8	3.16	0.075
	Smartphone	0.78	0.047	0.69–0.88
M6: posterior MCA territories	Medical	0.72	0.053	0.62–0.82	0.00	0.944
	Smartphone	0.72	0.052	0.62–0.82

CI: confidence interval; ICC: Intraclass correlation coefficient; MCA: middle cerebral artery.

Figure 2.

Receiver-operating characteristic (ROC) curves for regions. C: caudate head; I: insular ribbon; IC: internal capsule; L: lentiform nucleus.

Figure 3.

Receiver-operating characteristic (ROC) curves for regions. M1: anterior MCA cortex; M2: MCA cortex lateral without insular ribbon; M3: posterior MCA cortex; M4: anterior MCA territories; M5: lateral MCA territories; M6: posterior MCA territories; MCA: middle cerebral artery.

Discussion

Differences in the ROC curves were documented for the insular ribbon and M2 regions with AUCs of 0.44 and 0.68, respectively, for the smartphone reading system. However, these differences were not found for the remaining eight ASPECTS territories.

In contrast, interrater reliability, for the internal capsule and M2 was compromised, but not for the insular ribbon. All the remaining regions performed well in terms of interrater reliability. Therefore, readers should take special care when evaluating lesions involving the insula, internal capsule and M2 territories every time that a smartphone reading system is utilised. The internal capsule is predominantly a white matter structure and is hypodense, when compared to the adjacent grey matter nuclei. This is a particularly challenging territory for the detection of ischaemic lesions even after judicious adjustment of the contrast window. Isolated infarcts of the internal capsule are rare given that its major blood supply depends anterior choroidal artery, and is present, mostly with internal carotid artery occlusions. Abnormal findings in this area are usually reported after a proximal occlusion and extensive ischaemic insults. For the M1–M6 territories, the limits between these regions are not easily defined, as they are contiguous in the axial and the cranio-caudal planes.

Our analysis found very good total ASPECTS interrater reliability. However, in a previous analysis dichotomised ASPECTS scores showed conflicting results.⁶ ASPECTS < 6 is a well-known contraindication to IV rtPA administration. Hence, when using a smartphone, lesions involving the internal capsule, insula and M2 regions should be carefully evaluated every time that treatment decisions are made based on dichotomised ASPECTS scores.

In our previous evaluations, the intraobserver agreement between medical and smartphone reading systems was good (kappa = 0.66),¹⁴ and the AUC values were 0.76 and 0.74 respectively, with no significant differences (p = 0.11). However, equivalence between the two reading systems was achieved at 7.9%.⁶ These results may constitute an additional risk when ruling out contraindications for IV rtPA administration, while using a smartphone device. Hence, improvement in the detection of lesions in the three aforementioned territories will end up in more accurate total and dichotomised ASPECTS scores. Total and by-region ASPECTS interobserver agreements in our study, were higher than those obtained in a previous study by Finlayson et al.,⁹ while using the medical reading system, with the exception of the insula. In the same study, other imaging techniques were incorporated, using only a medical reading system and not a mobile solution.

Other studies have evaluated ASPECTS reliability and accuracy, when using a tablet computer.^7,8,15–17 A systematic review by Caffery et al.¹⁸ concluded that the diagnostic accuracy of radiological interpretation was not compromised by using a tablet computer to interpret CT, but this conclusion is only applicable to the Apple iPad. In the studies reviewed, the detection of haemorrhage performs well, however, only the study of McLaughlin et al.¹⁹ evaluated ASPECTS, measuring only the concordance of total ASPECTS with a gold standard. There was no evaluation of intraobserver agreements nor interobserver concordance; in addition, neuroradiologists were not evaluated individually, as they performed interpretations in consensus. Therefore, there are no previous studies indicating which are the regions difficult to interpret when using a tablet or a smartphone. The study of McLaughlin et al.¹⁹ showed that junior and senior residents had low agreement with the reference standard when the ASPECTS was scored between 4–7, indicating the relevance of the training in the interpretation and evaluation of ASPECTS, as these scores are in the limit of the thrombolysis decision (ASPECTS < 6). In our hospital, interpretation of head NCCTs is performed only by experienced neuroradiologists; therefore, the use of mobile devices allows for contacting them every time that this is required.

Preliminary studies using smartphones in the evaluation of early signs of infarction in NCCT brain scan were performed by Mitchell et al.²⁰ and Demaerschalk et al.,²¹ however, these studies never evaluated ASPECTS reliability or accuracy. To the best of our knowledge, this is the first study that compares a medical workstation and a smartphone reading system for the interpretation of head CT images from patients with acute stroke symptoms, and the first of its kind considering individual ASPECTS territories.

This study has several limitations. Our random sample had few cases of acute ischaemic lesions in the MCA territory, which were interpreted by all neuroradiologists (19/188). Cases that were considered to evaluate ASPECTS scores are those with a report of high confidence in the presence of ischaemic lesion in the MCA territory (109/188), and those with acute lesions (68/109), according to our gold standard. However, for the ROC and agreement evaluations, in repeated measures designs, missing data are not permitted. Hence, if any observer reported a case with no ischaemic lesions or acute lesions, the ASPECTS score was not evaluated for this specific reading and the patient was excluded for the final calculation. The cases that were reported as ischaemic lesions in the MCA territory by the four readers numbered 49, including only 19 interpreted as acute stroke. The limitation relies on the fact that we could not find statistical differences in the AUC comparisons. Statistically, significant results are required in this context in order to avoid type II errors. Equivalence evaluations are strongly recommended,²² however this requires higher sample sizes.

Detection of haemorrhagic lesions, hyperdense MCA, ischaemic lesions in the MCA territory, acute ischaemic lesion in the MCA territory, and ASPECTS <6, in head NCCTs using a smartphone were previously evaluated by our group. High levels of reliability and accuracy were found,^6,14 except for ASPECTS < 6. In the current article, we focused on the evaluation of individual ASPECTS regions to detect those specific territories that significantly increase interrater reliability influencing the overall and dichotomised ASPECTS scores. Hence, ruling out contraindications to the administration of IV thrombolytics will likely improve after specific regions such as the insula, internal capsule and M2 territories are more carefully reviewed by experienced readers while using handy mobile solutions in the context of a telestroke network.

Footnotes

Acknowledgements

The authors wish to thank the radiologists who carried out the interpretations as well as their institutions and the National Department of Science, Technology and Innovation of Colombia for funding this study (Grant 1204-744-55680).

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: National Department of Science, Technology and Innovation (COLCIENCIAS), Grant 1204-744-55680.

References

Jauch

Saver

Adams

Jr , et al. Guidelines for the early management of patients with acute ischemic stroke: A guideline for healthcare professionals from the American Heart Association/American Stroke Association. Stroke 2013; 44: 870–947.

Demchuk

Hill

Barber

, et al. Importance of early ischemic computed tomography changes using ASPECTS in NINDS rtPA Stroke Study. Stroke 2005; 36: 2110–2115.

Pexman

JHW

Barber

Hill

, et al. Use of the Alberta Stroke Program Early CT Score (ASPECTS) for assessing CT scans in patients with acute stroke. Am J Neuroradiol 2001; 22: 1534–1542.

Tan

Wan-Yee

Paliwal

, et al. Good intracranial collaterals trump poor ASPECTS (Alberta Stroke Program Early CT Score) for intravenous thrombolysis in anterior circulation acute ischemic stroke. Stroke 2016; 47: 2292–2298.

Barber

Demchuk

Zhang

, et al. Validity and reliability of a quantitative computed tomography score in predicting outcome of hyperacute stroke before thrombolytic therapy. ASPECTS Study Group. Alberta Stroke Programme Early CT Score. Lancet 2000; 355: 1670–1674.

Salazar

Granja

Useche

, et al. Evaluation of the accuracy equivalence of head CT interpretations in acute stroke patients using a smartphone, a laptop or a medical workstation. JACR 2019; 16: 1561–1571

Salazar

Useche

Granja

, et al. Mobile device for thrombolysis decisions for telestroke. Colomb Med 2018; 49: 254–260.

Salazar

Useche

Granja

, et al. Ruling out brain CT contraindications prior to intravenous thrombolysis: Diagnostic equivalence between a primary interpretation workstation and a mobile tablet computer. Int J Telemed Appl 2017; 2017: 7.

Finlayson

John

Yeung

, et al. Interobserver agreement of ASPECT score distribution for noncontrast CT, CT angiography, and CT perfusion in acute stroke. Stroke 2013; 44: 234–236.

10.

Samei E, Badano A, Chakraborty D et al. Assessment of display performance for medical imaging systems: executive summary of AAPM TG18 report. Med Phys 2005; 32: 1205–1225

11.

Altman

DG.

Practical statistics for medical research. 1st Edition. London: Chapman and Hall/CRC, 1991.

12.

DeLong

Clarke-Pearson

DL.

Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach.

Biometrics 1988; 44: 837–845.

13.

Fawcett

An introduction to ROC analysis.

Pattern Recogn Lett 2006; 27: 861–874.

14.

Salazar

Granja

Useche

, et al. Remote head CT evaluation for acute stroke diagnosis using a smartphone: Reliability and diagnostic equivalence with a primary medical interpretation workstation. In: Eleventh international conference on ehealth, telemedicine, and social medicine (eTELEMED 2019) Athens, Greece, 24–28 February 2019, pp.129–134. Wilmington: IARA.

15.

McLaughlin

Moloney

O'Neill

, et al. CT of the head for acute stroke: Diagnostic performance of a tablet computer prior to intravenous thrombolysis. J Med Imaging Radiat Oncol 2017; 61: 334–338.

16.

Park

Choi

Lee

, et al. An assessment of the iPad 2 as a CT teleradiology tool using brain CT with subtle intracranial hemorrhage under conventional illumination. J Digit Imaging 2013; 26: 683–690.

17.

Yoshimura

Nihashi

Ikeda

, et al. Comparison of liquid crystal display monitors calibrated with gray-scale standard display function and with γ 2.2 and iPad: Observer performance in detection of cerebral infarction on brain CT. Am J Roentgenol 2013; 200: 1304–1309.

18.

Caffery

Armfield

Smith

AC.

Radiological interpretation of images displayed on tablet computers: A systematic review.

Br J Radiol 2015; 88: 20150191.

19.

McLaughlin

Moloney

O'Neill

, et al. CT of the head for acute stroke: Diagnostic performance of a tablet computer prior to intravenous thrombolysis. J Med Imaging Radiat Oncol 2017; 61: 334–338.

20.

Mitchell

Sharma

Modi

, et al. A smartphone client-server teleradiology system for primary diagnosis of acute stroke. J Med Internet Res 2011; 13: e31.

21.

Demaerschalk

Vargas

Channer

, et al. Smartphone teleradiology application is successfully incorporated into a telestroke network environment. Stroke 2012; 43: 3098–3101.

22.

Obuchowski

NA.

Testing for equivalence of diagnostic tests.

Am J Roentgenol 1997; 168: 13–17.