Abstract
Background:
Patients with lesions suspicious for skin cancer often present to primary care physicians (PCPs), who may have limited training in skin cancer diagnosis.
Objective:
To measure the impact of an adjunctive handheld device for PCPs that employs elastic scattering spectroscopy (ESS) on the diagnosis and management of skin cancer.
Methods:
Fifty-seven PCPs evaluated 50 clinical images of skin lesions (25 malignant and 25 benign), first without and then with knowledge of the handheld ESS device output, and in each case indicated if a lesion was likely to be benign or malignant.
Results:
The diagnostic sensitivity of the PCPs with and without the use of the ESS device was 88% (95% CI, 84%-92%) and 67% (95% CI, 62%-72%), respectively (P < .0001). In contrast, no significant difference was observed in the diagnostic specificity. The management sensitivity of the physicians with and without the use of the ESS device was 94% (95% CI, 91%-96%) and 81% (95% CI, 77%-85%), respectively (P = .0009). Similarly, no significant difference was observed in the management specificity.
Conclusion:
The use of the ESS device may have the potential to help improve skin cancer diagnosis and confidence in management decision-making in a primary care setting.
Keywords
Introduction
Skin malignancies are the most common type of cancer diagnosed in the United States (US), 1 yet the clinical diagnosis of this cancer remains a challenge for most primary care physicians (PCPs), to whom initial presentations usually occur. Incidence rates of skin cancer, including malignant melanoma (MM) and keratinocyte carcinoma (KC), have increased by 44% and 77%, respectively, in recent decades.1,2
Skin cancer is highly curable if detected early, with a 99% 5-year survival rate for MM when diagnosed at a localized stage. However, this figure drops to 66% for regional stage, and 27% for distant stage. 1 There is a dire need to improve the early diagnosis of skin cancer, in order to maximize survival rate and decrease morbidity as well as associated healthcare costs. 3
Patients are most likely to visit PCPs for the initial examination of suspicious skin lesions before being referred to a specialist. 4 The standard method used by PCPs to diagnose skin cancer is visual inspection, with or without the use of diagnostic aids such as the ABCDE rule, the Chaos and Clues method, the 2 Step Algorithm, or the Glasgow 7-point checklist.5 -8 However, the accuracy of this naked-eye examination is highly dependent on the PCP’s experience, due to the high similarity between early-stage malignant lesions and benign lesions. 9 Consequently, reviews of the literature have reported that PCPs generally have lower sensitivity of both diagnostic accuracy and biopsy/referral decision accuracy as compared to dermatologists. 10
Recently, optical technology-based methods have been developed that significantly improve the accuracy of non-invasive skin cancer diagnosis. These approaches include educational interventions, dermoscopy, reflectance confocal microscopy, optical coherence tomography, multiphoton excited fluorescence imaging, and others.11 -14 However, significant barriers to their use in primary care include the need of specialized training, high cost of implementation, and generally a lack of comprehensive validating evidence.11 -13 A long-standing challenge is to develop new technologies for the early diagnosis of skin cancer, which are non-invasive and accurate, easy to use, and cost-effective for general practice.
A recent comparative study evaluating the diagnostic performance of non-invasive techniques for melanoma diagnosis found that optical spectroscopy achieved the best performance in terms of specificity and sensitivity. 15 Elastic-scattering spectroscopy (ESS) is a specialized form of optical spectroscopy that is sensitive to the micro- and nano-scale structure of a diseased tissue and is therefore suitable for skin cancer assessment.16,17
Here, we investigate the potential for utilization of a spectral classification algorithm that employs ESS measurements in conjunction with the machine learning approach of convolutional neural networks, implemented in a portable ESS Device to help detect skin cancer. 17 The ESS device was developed to be used as an adjunctive tool for the evaluation of suspicious skin lesions in a primary care setting, not as a diagnostic tool. We compared the diagnosis and management performance (sensitivity, specificity, and AUC) in primary care physicians (PCP) with and without the use of the ESS device for detecting skin cancer. Ultimately, this study aims to determine whether the use of an ESS device can help PCPs improve their diagnostic accuracy of skin malignancies.
Methods
ESS and Artificial Intelligence System
The investigative device is a handheld device that employs elastic scattering spectroscopy (ESS) and convolutional neural networks (CNNs) to detect cancerous skin lesions. 17 Developed as an adjunctive tool to help PCPs in their diagnostic and management decision, the device classifies lesions as either malignant (“Investigate Further”) or benign (“Monitor”). Additionally, for “Investigate Further” classified lesions, a score from 1 to 10 is provided which corresponds to the amount of spectral similarity a lesion has to malignant lesions in studies, with 10 representing the highest amount.
The algorithm of the ESS device was trained and validated with over 20 000 spectral recordings from over 4500 skin lesions, including histologically-confirmed melanoma and KC; as well as unbiopsied benign lesions, identified by dermatologists.18 -22 None of the ESS spectral recordings used in the training process were employed in the testing set used in this study.
Test Lesions
A total of 50 cases of skin lesions were selected from the clinical validation study of the ESS device. 23 These cases included 25 malignant lesions and 13 benign lesions, which had been biopsied and assessed histologically; and 12 unbiopsied benign lesions, diagnosed by dermatologists. Characteristics of the test lesions are presented in Table 1. Of note, actinic keratoses were not treated as malignant for the purposes of this classification, and severely atypical nevi were considered to need further evaluation by a dermatologist. For each case, high resolution digital clinical images were used. In addition, the patient’s clinical information, including prior skin cancer history, risk factors, and physical examination results were included.
Characteristics of Test Lesions and Corresponding Device Classifications (N = 50).
Pigmentation of lesion was determined by an internal dermatologist reviewing the images of the lesion cases and assessing whether or not the lesion was or was not pigmented.
Determination of if a lesion was melanocytic was performed by an internal dermatologist classifying the different parent pathology classifications into buckets of melanocytic or non-melanocytic.
Basal cell carcinoma, squamous cell carcinoma, melanoma, and severely atypical melanocytic nevus lesions were considered malignant and all others benign. Severely dysplastic melanocytic nevi were considered malignant because they are often managed with re-excision by dermatologists.
50 lesions utilized in the study were procured from 44 patients.
Lesions were randomly selected to match the prevalence of lesions in the clinical study that was performed. The device performance for the lesions in this study was 96% sensitivity and 36% specificity, compared to the prior clinical study performance of 97% and 26% (37% for unbiopsied benign lesions and 22% for biopsied benign lesions). Thus, the randomly selected cases were similar in performance to the device performance. The malignant and benign subgroups were based on predefined proportions applicable to primary care and enrolled lesions matching these criteria were selected from the clinical study dataset to match the appropriate diagnoses. The order in which lesions were presented was also randomized.
Primary Care Physicians’ Evaluation
A total of 57 U.S. board-certified PCPs participated in this study. All PCPs provided written informed consent. Eligibility criteria and PCPs’ experience as well as demographic information are provided in Table 2, respectively. All PCPs watched a training video about how to use the ESS device and were required to pass a training quiz before moving forward with the study.
Descriptive Characteristics of the PCPs (N = 57).
The study was conducted in 2 phases; in each phase, PCPs evaluated all 50 cases independently and in random order. In the first phase, each PCP evaluated the skin lesion cases using 1 or more digital images and the patient’s clinical information for each case alone. PCPs were blinded to the classifier output from the device. For each case, PCPs completed a questionnaire about their diagnosis of the lesion (malignant or benign), their recommended management decision (biopsy/referral to a dermatologist or not), and their confidence level on their management decision (scale 0-3, where 0 = no confidence, 1 = slight confidence, 2 = moderate confidence, and 3 = high confidence).
In the second phase of the study, PCPs were asked to evaluate the same lesions from the first phase in a different randomized order; but this time, with the additional knowledge of the ESS device output, as part of the clinical information provided for each case. As in the first phase, for each case, all PCPs completed the questionnaire about their diagnosis, management decision, and level of confidence. At no point in the study were the physicians informed of the study’s distribution of benign and malignant lesions (ie, 25 of each).
Statistical Analysis
Each performance parameter, including sensitivity and specificity, was calculated using the total number of lesions evaluated (n = 1425; 25 lesions × 57 PCPs). Corresponding 95% confidence intervals (CIs) were calculated for both sensitivity and specificity using the Wilson method, as outlined in Saha et al. (2016), 24 to account for potential within-cluster correlation. To compare diagnostic sensitivity and specificity of the physicians with and without knowledge of the device output, we used the method of moments for clustered matched paired data, where a cluster was the set of lesion cases within each PCP combination. A 2-sided P-value of less than .05 was considered to indicate a statistically significant difference.
The same procedure was followed to determine the sensitivity and specificity of management decision, where the sensitivity was defined as the probability of deciding to further evaluate a lesion through a biopsy or referral, given that the lesion was malignant, and specificity was defined as the probability of deciding not to further evaluate a lesion given that the lesion was benign.
Inter-physician variability was assessed using logistic regression with Generalized Estimating Equations to account for within PCP correlation (assuming a compound symmetry correlation structure).
Shifts in the levels of confidence of the physicians in their management decision, with and without (baseline) knowledge of the device output, were compared using an unweighted Kappa statistic. Statistical associations between the level of confidence of the PCPs and their sensitivity and specificity were assessed using multivariate logistic regressions where the dependent variable is a correct diagnosis (yes/no). The model includes a random effect of the lesion and physician to account for correlations. The P-values come from the logistic regression and measure the association between each covariate and the outcome while adjusting for other variables in the model.
All statistics analyses were performed using SAS software (Version 9.4, SAS Institute Inc).
Results
The 50 lesions evaluated by PCPs were dispersed across the body, had a mixture of surfaces, textures, colors, and pigmentations; 26% were melanocytic, 76% were biopsied, and there were a range of final lesion etiologies as determined by histopathology; inherent device classification is also provided (Table 1).
Among 57 PCPs included, participating physicians represented a diversity of areas of practice, time practicing medicine, and practice types (Table 2). Further demographics, as well as range of patients, frequency of biopsies performed, referrals to dermatologist, and baseline level of confidence in skin lesion assessment can be found in Table 2.
The diagnostic sensitivity of the PCPs was 67% (958/1425; 95% CI, 62%-72%) independently, i.e. without the use of the ESS device, and 88% (1261/1425; 95% CI, 84%-92%) with the device (P < .0001). In contrast, the diagnostic specificity independently and with the use of the ESS device was 53% (761/1425; 95% CI, 49%-57%) and 40% (577/1425; 95% CI, 37%-44%), respectively (P = .0516; Table 3). Balanced accuracy (TP + TN/all cases) of PCP diagnostic performance was 0.588 and 0.645 independently without and with the use of the ESS device, respectively. Notably, the proportion of melanomas correctly diagnosed by PCPs included in the test lesion set were 82% (186/228) without use of the ESS device and 97% (221/228) with use. The number of false negatives for melanoma without the use of the device was 42 and with the use of the device was 7; for severely atypical nevi, false negatives without and with the use of the device were 39 and 3, respectively.
Diagnostic and Management Performance for Detection of Skin Cancer With and Without the Use of ESS Device.
Data in parentheses are the number of cases correctly identified as malignant (for sensitivity analysis) and benign (for specificity analysis) over the total number of lesions evaluated (25 lesions × 57 physicians = 1425). Bolded values represent statistical significance (p < .05).
The management sensitivity of the PCPs without and with the use of the ESS device was 81% (1160/1425; 95% CI, 77%-85%) and 94% (1352/1425; 95% CI, 91%-96%), respectively (P = .0009; Table 3). The management specificity of the PCPs was 36% (516/1425; 95% CI, 31%-42%) without the device and 31% (437/1425; 95% CI, 28%-34%) with the device (P = .3558; Table 3).
The interclass correlation, as an estimate of inter-physician variability in diagnostic decision, improved from .19 (slight agreement between PCPs) without the use of the ESS device to .50 (fair agreement between PCPs) with the use of the device. Similarly, the interclass correlation for management decision improved from .14 (slight agreement between PCPs) without the use of the device to .51 (fair agreement between PCPs) with the use of the device. The level of confidence of PCPs in their diagnosis and management decision also significantly increased by using the device (Table 4, P < .0001).
Shifts in Levels of Confidence of PCPs in Their Management Decision.
Kappa statistic: .1490; OR (95% CI): 4.210 (3.764-4.708); P < .0001.
The area under the curve for diagnosis of malignancy for PCPs for 4 confidence levels increased from 0.619 to 0.683 with addition of the device (P < .001; Figure 1).
Area under the curve for diagnostic sensitivity and specificity with and without the device with 4 levels of physician confidence (none, slight, moderate, and high). AUC with no device ROC was 0.6194 (0.0096 SE), AUC with device ROC was 0.6828 (0.0063 SE), which is a difference of 0.0634 (P < .001, 0.0085 SE, 95% CI 0.0466-0.0801).
Discussion
Our results support a beneficial role for the ESS device, which uses ESS and CNN, as an adjunctive tool for the evaluation of potentially malignant skin lesions in primary care settings. Use of the ESS device by PCPs significantly improved their diagnostic and management sensitivity and was associated with a decrease in specificity that approached significance for diagnosis and was not significant for management. There is a net improvement of PCP detection accuracy with the availability of device output. ESS use also significantly improved inherent validity as reported by area under the curve (AUC) for both diagnosis and referral.
While other commercial options are available, there is a great degree of variation among accuracy and little data to support their use by PCPs. A multispectral image analysis device (MELA Sciences, Irvington, NY) was an FDA-approved device that used a proprietary algorithm to aid with the decision to biopsy a suspicious skin lesion. Studies showed the device had high sensitivity, over 98%, but low specificity, around 10%. It was discontinued for sale or clinical use in 2017. Another adjuvant tool for melanoma detection, a device utilizing electrical impedance spectroscopy, reported over 96% sensitivity and 34% specificity. 25 This is the only currently FDA-approved device for melanoma detection but is restricted to use by dermatologists, given a lack of validation/approval in a primary care context and restrictions for usage on benign lesions such as seborrheic keratoses.26,27 Therefore, additional evidence for use of efficacious devices in clinical settings outside of dermatologic practice is both high yield for primary care, the front line of most suspicious lesion presentation, and in contribution to the high degree of trust needed for regulatory recognition and implementation of such devices. The present study contributes evidence for a diagnostically valid device that can improve detection in this context and meaningfully impact behavior.
This context is important when considering the increase in AUC that we report from 0.62 to 0.68. In this study, the majority of cases involved physician-biopsied lesions, which are notably more difficult for physicians to classify as benign or malignant compared to patient-concerned lesions often featured in image-based AUC studies. The receiver operating characteristic (ROC) curve in this scenario, which describes performance at different classification thresholds, was computed factoring in the variability in physician performance across 50 lesion cases. The rise in AUC implies that PCPs using the device would improve clinical utility and the quality of dermatologist referrals. This is particularly substantial for primary care, where skin cancer detection accuracy is typically lower than dermatology specialty care. Furthermore, given similar ROC sensitivity, PCP specificity increases (ie, when sensitivity is maintained at 94%, the specificity of the PCPs is 13.7% without device output, and 30.9% with device output).
Our results showed an increase in the levels of confidence of the PCPs in their management decision with the use of the ESS device; importantly, our results also indicated that higher confidence is associated with higher sensitivity and specificity. The inter-physician variability decreased with the use of the ESS device, which may indicate a decline in the subjectivity of the PCPs in their assessments. Of note, use of this device in prior clinical studies has shown minimal user-related variations. 16 Another strength of our study are the skin lesions used include a wide range of cases representative of those found in primary care practice. Additionally, due to an approach that is agnostic to lesion and skin type given its lack of surface-level feature analysis, ESS provides an inherent advantage as compared to other detection methods like Pigmented Lesion Assays (PLAs) and image-based CNN algorithms in its utility for a variety of lesions including both melanocytic and non-melanocytic lesions.28,29
There are a few limitations to this study. First, the study design did not allow for in-vivo lesion evaluations; therefore, though clinical history was provided, tactile evaluation of the lesions was not possible. Second, PCPs who participated in the study expressed interest in the ESS device and may have had a particular interest in skin cancer, resulting in a selection bias. Further, the ratio of benign:malignant lesions among the test pool was not reflective of primary clinical practice, where the vast majority of encountered lesions are benign. Future studies will include the analysis of the performance of the ESS device per lesion type.
In conclusion, the ESS device has the potential to help PCPs decide which skin lesions warrant further biopsy/referral for possible skin cancer. This technology improves sensitivity and inherent validity of skin cancer diagnosis and management. The use of the ESS device in a primary care setting is further supported by a reduction in the subjectivity of the PCPs regarding their evaluations and the limited training required for its use, as well as the increased diagnostic and management performance.
Footnotes
Abbreviations (Listed in Order of Appearance)
Elastic Scattering Spectroscopy (ESS)
Primary Care Physicians (PCPs)
Malignant Melanoma (MM)
Keratinocyte Carcinoma (KC)
Convolutional Neural Networks (CNN)
Pigmented Lesion Assay (PLA)
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Todd Thames and Thomaz de Campos Silva are advisors for DermaSensor, Inc. Laura Korb Ferris is a consultant for DermTech and an investigator for Castle Biosciences, SkinAnalytics, and DermTech.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was sponsored by DermaSensor, Inc.
