Sage Journals: Discover world-class research

Abstract

Background:

Elastic Scattering Spectroscopy (ESS), an optical tissue sampling technique, distinguishes between benign and malignant tissue in vivo without the need to perform a surgical biopsy. A handheld device that employs ESS enabled with an artificial intelligence algorithm was developed as an objective tool to aid primary care physicians (PCPs) in their management of lesions suspicious for skin cancer. The aim of this study was to assess and compare the diagnostic and management performance of PCPs with and without the use of the ESS device in detecting skin cancer.

Methods:

In this clinical utility study, 108 PCPs evaluated 100 skin lesion cases (50 aided with the device output and 50 unaided by the device). For each case, PCPs provided a diagnosis, management decision, and level of confidence in that decision initially without, and then subsequently with, device output. Sensitivity, specificity, AUC, and confidence in their assessment prior to, and then with, device output were compared.

Results:

With visual assessment assisted by device output, diagnostic sensitivity increased significantly from 71.1% to 81.7% (P = .0085) and referral sensitivity increased significantly from 82.0% to 91.4% (P = .0027) compared to visual assessment only. Device-aided diagnostic specificity decreased from 60.9% to 54.7% (P = .1896), and referral specificity decreased from 44.2% to 32.4% (P = .0256). Overall management performance (ie, AUC) also increased from 0.708 to 0.762, and increased from 0.567 to 0.682 for lesions which physicians reported low confidence in their unassisted management decision. Physicians reporting high confidence in their management assessment increased from 36.8% to 53.4%.

Conclusion:

Use of the ESS device output by PCPs significantly improved their diagnostic and management sensitivities as well as their overall management performance. The findings suggest the ESS device can improve PCP skin cancer detection and confidence in their skin lesion evaluation and management.

Keywords

melanoma squamous cell carcinoma basal cell carcinoma artificial intelligence machine learning skin cancer technology diagnostic tools elastic scattering spectroscopy

Introduction

Skin cancer is the most common malignancy in the U.S.¹ The most common skin cancers are basal cell carcinomas (BCC) and squamous cell carcinomas (SCC), which arise from keratinocytes in the epidermis and are collectively referred to as keratinocyte carcinomas (KCs); approximately 5.4 million are diagnosed in the U.S. each year.^2,3 Melanoma, the next most common type of skin cancer, has lower incidence but higher risk of mortality than KCs; in 2023, an estimated 186 680 new cases are expected to be diagnosed in the U.S.^1,2 In the U.S., approximately 7990 people die from melanoma and 8000 die from KCs each year.^1,4,5 Most skin cancers are treated surgically, and treatment of more advanced melanoma and KCs often result in significant surgical morbidity to sensitive areas.

Access to a dermatologist for diagnosis and management of cutaneous malignancies is challenging. Patients experience long waits for dermatology appointments, with 40% of Americans facing scarce dermatological services, and only 15% of high-risk Americans have ever been screened for skin cancer.^6
-8 This underscores the pivotal role of primary care physicians (PCPs) in promptly detecting cutaneous malignancies. However, differentiating malignant from benign skin lesions can be challenging for physicians, particularly non-dermatologists. The “ABCDE” (asymmetry, border irregularity, color changes, diameter, evolution) criteria for the evaluation of a melanocytic growth are helpful for detecting superficial spreading melanomas; however, these criteria result in variable performance and do not aid in detection of KCs.⁹

To address this need, various non-invasive approaches have been developed to aid clinicians, including imaging algorithms, electrical impedance spectroscopy, radiation-based photometric analysis, optical coherence tomography, and dermoscopy. For many of these technologies, researchers have developed algorithms using artificial intelligence that are applied to the lesion data to provide an automated cancer risk assessment.^10
-15 While promising, many face challenges to their widespread implementation, including narrow clinical scope (ie, only 1 type of skin cancer assessed), high cost, slow speed, and/or complicated output requiring extensive training.^14,15 Algorithm-based skin lesion imaging devices have shown potential to improve physician cancer detection, but further testing and improvements are needed, as commercially-available melanoma smartphone applications have a mean sensitivity of only 28%. Until recently, no skin cancer detection devices nor AI-based algorithms have been approved by the US Food and Drug Administration (FDA) for use by PCPs nor patients.^15,16

Elastic Scattering Spectroscopy (ESS), an optical tissue sampling technique, captures sub-cellular information to distinguish between benign and malignant tissue in vivo without the need to remove tissue.¹⁷ A handheld device employing ESS and an AI-powered algorithm was developed as an objective tool for PCPs to aid in their management of lesions suspicious for skin cancer. The device has shown promise in several clinical studies to date.^18
-22 The algorithm was developed to be highly sensitive in discerning malignant skin lesions (melanoma, BCC, and SCC) from benign skin lesions of concern to PCPs, the sensitivity across studies ranged from 90% to 96% and specificity ranged from 21% to 61%, depending on lesion selection criteria. This is the first technology to be cleared by the FDA to provide PCPs with an immediate, objective report of the skin cancer risk of a lesion; it is also the first technology to ever be cleared or approved for any user for all 3 common types of skin cancer.

In this study, PCPs actively practicing in the U.S. were presented with lesion cases (lesion images with patient and lesion clinical information) enrolled during the DERM-SUCCESS clinical validation study.²³ The purpose of the study was to evaluate the impact of the device results on PCPs decision-making regarding skin lesion evaluations. The DERM-SUCCESS study analyzed device performance on 1579 lesions from 1005 enrolled patients across 22 primary care study sites, and this companion study evaluated the device’s ability to impact and improve upon PCP skin cancer diagnostic and management performance compared to current practice without use of the device.

Methods

Study Overview

This multi-reader multi-case (MRMC) study was designed to assess and compare the sensitivity and specificity of PCPs in their clinical assessment of lesions suggestive of skin cancer with and without knowledge of the device result. This study was reviewed and approved by an ethical approval granting committee and an IRB-approved written informed consent was used and obtained for all study participants prior to participation in the DERM-SUCCESS clinical study.

Device Information

The ESS device (DermaSensor™ device, manufactured by DermaSensor Inc., Miami, FL, USA) has been described previously was cleared by the FDA for use by non-dermatology expert physicians in January 2024.^21,23 Briefly, this device emits light pulses dispersed over isolated areas of the lesion, with backscattered optical reflectance representing its architecture (eg, nuclear and chromatic characteristics). The ESS device was trained and validated with >20 000 spectral scans from >4500 lesions including histologically-confirmed melanoma, BCC, SCC, and benign lesions as well as unbiopsied benign lesions diagnosed by board-certified dermatologists from 6 clinical studies conducted in primary care and dermatology settings.^23
-27

In accordance with FDA requirements, the AI algorithm was locked prospectively; thus, algorithm development occurred with neither knowledge of the device data nor device results. The device classifies lesions as “Investigate Further” or “Monitor” based on similarity of their scanned properties to malignant or benign lesions. For “Investigate Further” lesions, a 1 to 10 score reflects the degree of spectral similarity to malignant lesions from prior studies. The likelihood of malignancy (positive predictive value) in the DERM-SUCCESS study ranged from 6% for a spectral score of “1” to 61% for a score of “10.”

Study Design

This study was conducted in 2 phases. In each phase, PCP study participants reviewed high-resolution digital images and clinical information for 50 skin lesion cases, which included 25 malignant cases, independently and in random order. These prospectively-collected lesion cases were a randomly selected subset of lesions from the DERM-SUCCESS clinical study and were all biopsied as suspicious for melanoma, BCC, and/or SCC by primary care physicians. All were subsequently diagnosed by 2 to 5 dermatopathologists depending on pathology type and discordance; thus, all lesions are representative of the pathologies of common primary care lesions suspicious for skin cancer. This selection process also ensured the device performance was reflective of that in the clinical study. Additionally, a standardized photograph protocol was implemented in the clinical study where captured images were reviewed by a panel of physicians (including both dermatologists and PCPs) for image quality and consistency prior to inclusion.

Physicians were recruited for participation via a survey platform (Sermo, Inc., United States) and were compensated for their time per the platform’s standards. In the first phase, each physician evaluated the 50 skin lesion cases based on their assessment alone, without the device output. In the second phase, following a break of at least 2 h, physicians evaluated the same lesions in a different randomized order, based on the same clinical information provided for each case but with the addition of the device output. For both phases and every lesion case, physicians completed a questionnaire about their diagnosis (malignant or benign), their recommended management decision (further evaluate or not), and their confidence level on their management decision (1-10 scale, where 1 = no confidence and 10 = high confidence). Physicians were blinded to the proportion of benign versus malignant lesions included in the study. A power calculation predicted 93% power with 100 readers for 25 malignant lesions, assuming a 10% increase in diagnostic sensitivity (a measurement of PCPs’ ability to detect malignant lesions).

One hundred and eighteen (118) board-certified PCPs completed at least 1 lesion case in the study, and 108 PCPs completed all 100 lesion case assessments and were included in effectiveness analysis. Physician referral sensitivity, a measurement of PCPs’ ability to appropriately refer malignant cases, was a primary endpoint to assess improvements in skin cancer management decisions. To ensure PCP performance with the device was better than random chance, a second co-primary endpoint was evaluated (whether sensitivity + specificity > 1).

Statistical Analyses

Co-primary endpoint 1 was referral sensitivity of the physicians using biopsy results as the reference standard, which was defined as the number of true positive cancers that were referred for further evaluation and tested at a 1-sided .025 level of significance. The second co-primary endpoint was referral sensitivity and specificity of the physicians with knowledge of device result using biopsy results as the reference standard, tested at a 1-sided .025 level of significance.

Additionally, physicians’ diagnostic sensitivity was analyzed with and without device result knowledge. The hypotheses compared the true diagnostic sensitivity for physicians with and without device result knowledge. The Area Under the Curve (AUC) of the Receiver Operating Curve (ROC) was calculated based on physicians’ referral decisions and confidence levels, using histopathology diagnosis as the reference. PCP referral and diagnostic specificity with and without device result knowledge were evaluated, including a confidence interval of the difference between the 2 specificities, calculated using the Obuchowski and Rockette ANOVA approach. All analyses were conducted using SAS Version 9.4 and using R.

Results

Of the 108 U.S. board certified physicians included, 48.1% (n = 52) were Internal Medicine physicians and 51.9% (n = 56) were Family Medicine physicians (Supplemental Table S1). There were 65.7% (n = 71) male and 34.3% (n = 37) female physicians. Years in practice varied with most physicians reporting 21+ years in practice (31.5%), followed by 1 to 5 years (19.4%,) 11 to 15 years (17.6%), 6 to 10 years (17.6%), and 16 to 20 years (13.9%). The majority of physicians reported practicing in various types of non-academic settings, with 13.9% (n = 15) practicing in an academic center. Among this group, 40.7% (n = 44) reported referring patients with skin lesions to a dermatologist “most of the time” and 49.1% (n = 53) reported referring “sometimes”; no physicians in the study reported doing so “never” or “always.” The majority, 56.5% (n = 61), rated themselves as having “intermediate” competency in skin lesion assessment.

Patient cases (n = 50) had a median age of 59 years, were 100% of White race, and the majority had Fitzpatrick skin type of 2 (36.0%, n = 18) or 3 (32.0%, n = 16; Supplemental Table S2). Half of lesions included were characterized as high risk (50%, n = 25), with the same clinical study proportions for basal cell carcinoma (40%, n = 10), squamous cell carcinoma (36.0%, n = 9), melanoma (16%, n = 4), and severely dysplastic nevi (8.0%, n = 2); half were low risk (50%, n = 25), which included benign melanocytic nevi (36.0%, n = 9), seborrheic keratosis (36.0%, n = 9), and benign other (28.%, n = 7; see footnote; Supplemental Table S3). Other characteristics of the patients and lesions are shown in Supplemental Tables S2 and S3, including risk factors such as new/changing lesions, ultraviolet light exposure, and family history. Patient characteristics in the lesion cases sample set were very similar to those of the DERM-SUCCESS clinical study.²³ In both studies, there were no clinically meaningful differences between the selected lesion cases and the clinical study cases for any lesion characteristics, including location, pigmentation, texture, or size.

In accordance with the study protocol, the prevalence of malignant lesions in this study was 50%, which was higher than the 14.2% prevalence (224 malignant lesions of 1579 total lesions) in the clinical study (Supplemental Table S3). The device sensitivity for the 25 malignant lesions in this study was 96.0% (compared to 95.5% in the clinical study), and the device specificity for the 25 benign lesions was 20.0% (compared to 20.7% in the clinical study). Within those 2 groups of 25 cases each, the lesion pathology breakdown was also similar to the clinical study pathology.

The 25 malignant and 25 benign lesions were assessed by each of the 108 PCPs, yielding 2700 lesion assessments of each type without the device result and 2700 of each type with the device result, totaling 5400 assessments for malignant lesions and 5400 for benign lesions. The management sensitivity of physicians with the device result was significantly higher than that without the device result (91.4% vs 82.0%, P = .0027); thus, the physicians’ false negative rate decreased by more than half, from 18.0% to 8.6% (Table 1). The diagnostic sensitivity for PCPs with the device result of 81.7% (95% CI: 72.4%-90.9%) was also significantly higher compared to their sensitivity without the device result of 71.1% (95% CI: 63.4%-78.8%; P = .0085; Table 1).

Table 1.

Management and Diagnostic Sensitivity With and Without the Device.^c

	Lesion assessments^a	Estimate (%) and 95% confidence interval^b		P-value^b
Outcome	Lesion assessments^a	Without device	With device	P-value^b
Management sensitivity (co-primary endpoint)	n = 5400	82.0% (76.4%-87.6%)	91.4% (85.7%-97.1%)	.0027
Diagnostic sensitivity	n = 5400	71.1% (63.4%-78.8%)	81.7% (72.4%-90.9%)	.0085

For sensitivity analyses, only positive (malignant) lesion cases are included; for specificity analyses, only negative (benign) lesion cases are included.

MRMC analysis of variance (ANOVA) method of Obuchowski and Rockette.²⁸

The authors recognize that there is debate about the definition of melanoma and high-risk melanocytic lesions. The FDA required that severely dysplastic nevi should be considered positive for referral to dermatology. Thus, the DERM-SUCCESS clinical study and this study grouped highly dysplastic nevi, atypical junctional melanocytic lesions, melanoma in situ, and invasive melanoma as all “high risk” lesions.

For benign lesions, the associated management specificity was 44.2% (95% CI: 36.0%-52.4%) without the device result and 32.4% (95% CI: 20.7%-44.1%) with the device result (P = .0256). Statistical significance testing for specificity was not planned but using the significance testing that was used for sensitivity (ie, a 1-sided .025 level of significance), the specificity decrease was not significant. PCP diagnostic specificity for benign lesions was not significantly different without the device output 60.9% (95% CI: 52.5%-69.3%) as compared to specificity with the device output 54.7% (95% CI: 42.3%-67.1%; P = .1896). PCPs incorrectly referred an additional 319 of 2700 (11.8%) benign lesion cases when aided by the device output, but correctly referred an additional 254 of 2700 (9.4%) malignant lesion cases when aided by the device output. For benign lesions that PCPs had referred unaided, 21.2% of these were not referred when aided by the device output.

The referral sensitivity of the PCPs with knowledge of the device result yielded a sensitivity of 91.4% (95% CI: 85.7%-97.1%) and a specificity of 32.4% (95% CI: 20.7%-44.1%). The PCPs with the device result were significantly better at detecting malignant lesions than expected by chance, with an odds ratio of 6.8 (95% CI: 4.7-9.8; P < .0001). The 382 device true positive results changed PCP reads from incorrectly managed to correctly managed malignancies. By comparison, 81 device true positive results and 49 device false negative results (130 in total) changed PCP reads from correctly managed to incorrectly managed malignancies (Figure 1). Thus, device use impact on skin cancer management has a net 2.9× ratio of increased detection (382 positively impacted vs 130 negatively impacted), supporting the device’s benefit/risk profile.

Figure 1.

Flow chart of PCP decision-making for lesion management and impact of device results.

High confidence in the physician management decision (defined as a rating of 8-10) was reported for 43.2% of malignant lesions without the device result and 63.3% of malignant lesions with the device result. For all lesion assessments combined (n = 5400), the proportion of assessments made with a high level of confidence in the physician management decision was 36.8% without the device result and 53.4% with the device result (Figure 2).

Figure 2.

Level of Confidence in management decision (n = 10 800).

PCP management sensitivity for malignant lesions with device output information was higher for low-confidence referral decisions (81.9% [95% CI: 70.4%-93.3%]) when compared to corresponding decisions without the device output (76.8% [95% CI: 67.2%-86.4%]). For high-confidence referral decisions, PCP management sensitivity for malignant lesions with device output was also higher (95.6% [95% CI: 91.7%-99.4%]) as compared to corresponding decisions without device output (88.7% [95% CI: 83.4%-94.0%]) as shown in Table 2. In addition, PCPs completed 58% more high confidence (ie, 8-10) referrals of malignant lesions with device output (n = 1635) than they completed without device output (n = 1034).

Table 2.

Management Sensitivity and Specificity According to Confidence Level^a (n = 10 800).

A: Low/High Confidence.^b

Level of confidence	#Malignant lesions	#Physician positive	Sensitivity (95% CI)^a	#Benign lesions	#Physician negative	Specificity (95% CI)^a
Without device			n = 2700			n = 2700
Low	630	468	74.3% (67.6%-81.0%)	829	331	39.9% (32.7%-47.1%)
High	2070	1746	84.3% (78.4%-90.3%)	1871	863	46.1% (36.5%-55.7%)
With device			n = 2700			n = 2700
Low	361	296	82.0% (75.7%-88.3%)	630	142	22.5% (15.6%-29.5%)
High	2339	2172	92.9% (87.6%-98.2%)	2070	733	35.4% (21.1%-49.7%)
B: Low/Medium/High Confidence.^c
Level of confidence	#Malignant lesions	#Physician positive	Sensitivity (95% CI)^a	#Benign lesions	#Physician negative	Specificity (95% CI)^a
Without device			n = 2700			n = 2700
Low	319	245	76.8% (67.2%-86.4%)	379	136	35.9% (26.4%-45.3%)
Medium	1215	935	77.0% (71.0%-82.9%)	1502	662	44.1% (36.2%-51.9%)
High	1166	1034	88.7% (83.4%-94.0%)	819	396	48.4% (36.6%-60.1%)
With device			n = 2700			n = 2700
Low	182	149	81.9% (70.4%-93.3%)	301	67	22.3% (13.0%-31.6%)
Medium	807	684	84.8% (77.3%-92.3%)	1229	379	30.8% (21.2%-40.5%)
High	1711	1635	95.6% (91.7%-99.4%)	1170	429	36.7% (21.1%-52.2%)

Obuchowski and Rockette method for 95% confidence interval.

Two-level classified as: low (1-5) and high (6-10).

Three-level classified as: low (1-4), medium (5-7), and high (8-10).

The PCPs’ observed AUC performance was 0.708 without device output and improved to 0.762 with device use (Figure 3a, Supplemental Figure S1A). The device standalone AUROC was 0.780 in the clinical study.¹⁸ As mentioned previously, the PCP management performance with use of the device output was 91.4% (95% CI: 85.7%-97.1%) for sensitivity (Figure 3b) and 32.4% (95% CI: 20.7%-44.1%) for specificity. Fixing the PCP sensitivity of 91.4% on the ROC curve without use of the device, the interpolated PCP specificity level was 23.2%. Therefore, for PCPs unaided by the device result, interpolated specificity would be approximately 23.2% instead of 32.4% to achieve the same level of skin cancer detection that was achieved with use of the device, a difference of 9.2%. Additional ROC and AUC analyses were performed for lesions in which PCPs had low confidence in their clinical assessment when unaided by the device output. For such low confidence lesions, the PCPs’ observed AUC was 0.567 without the device output and was 0.682 with the device output (Supplemental Figure S1B).

Figure 3.

(a) Physician management ROC curves unaided and aided by device and (b) physician sensitivity values unaided and aided by device.

The DERM-SUCCESS clinical validation study and this companion clinical utility study were performed prior to FDA clearance, which resulted in device labeling for patients 40 years and older. Subanalysis of our study for patients above age 40 years (22 malignant, 22 benign) found the management sensitivity of physicians with the device result (95.9%, 95% CI: 94.5%-97.2%) was significantly higher than it was without the device result (83.6%, 95% CI: 81.0%-86.1%). Management specificity also decreased with device output (32.6%, 95% CI: 29.5%-35.9%) as compared to without (41.5%, 95% CI: 37.8%-45.1%). Among this subgroup, the PCPs’ observed AUC performance improved to 0.800 (95% CI: 0.787%-0.812%) with device use from 0.704 (95% CI: 0.686%-0.722%) without device output. PCP diagnostic sensitivity in patients over 40 also increased when aided by the device (88.3%, 95% CI: 86.1%-90.4%) as compared to without it (73.3%, 95% CI: 70.2%-76.4%), and diagnostic specificity minimally decreased with device output (53.1%, 95% CI: 49.5%-56.7%) as compared to without it (58.1%, 95% CI: 54.8%-61.4%).

At the end of the study, physician participants were asked a series of 7 questions relating to device applicability in clinical practice. Only 1 of the 108 PCPs (<1%) reported no benefits from the device. Among the other PCPs, the most commonly-reported benefits of the ESS device were: “providing an immediate, objective result to inform your management of suspicious lesions” (82%), “detecting more skin cancer” (82%), and “providing you with greater confidence in your clinical assessments and management decisions” (81%). Also, 98% of PCPs agreed (81% strongly agreed and 17% agreed) that skin cancer is a disease deserving of better surveillance in the primary care setting. The PCPs’ most commonly-reported estimate of their own sensitivity for correctly managing malignant lesions was 71% to 80%.

Discussion

We show that the availability of the AI-enabled, non-invasive handheld ESS device output significantly increased the sensitivity of PCPs from 82.0%, when using only standard of care visual inspection and patient history, to 91.4% when they were also provided with the device result (P = .0027). Thus, availability of the device output halved the physicians’ false negative rate from 18.0% to 8.6%. When comparing to the physicians’ clinical performance in the DERM-SUCCESS clinical study,²³ the unaided, standard of care sensitivity of the physicians for both studies was similar, that is, 83.0% for the clinical study and 82.0% for this utility study. When comparing to the device’s performance in the clinical study, the PCPs’ aided sensitivity of 91.4% was lower but similar to that of the device algorithm’s standalone sensitivity of 96.0%.

The increases in sensitivity demonstrate the potential for a direct benefit to PCPs’ ability to correctly manage high risk, potentially cancerous lesions when aided by the device result. Increasing skin cancer detection by 9.4% (ie, 91.4% compared to 82.0%) is highly impactful, given that 5.6 million cutaneous malignancies are diagnosed in the U.S. each year. Notably, while consensus among dermatologists and dermatopathologists has yet to be reached,²⁹ any significant increase in detection must be weighed against concerns for the overdiagnosis of malignancy. For this study, 254 additional malignant lesions were correctly managed, while 319 additional benign lesions were incorrectly referred with use of the device output.

To our knowledge, this is the first prospective clinical utility study completed for FDA clearance for any kind of skin cancer detection device for PCPs. Our results demonstrate a significant improvement in both diagnostic and management sensitivity of physicians with device result availability, while maintaining acceptable levels of specificity for all patients and for those patients 40 years and above, suggesting that the integration of this AI-powered device into their practice is beneficial to PCP skin cancer evaluations. The significant 10.6% improvement in diagnostic sensitivity (15.0% for patients 40 years and above), coupled with an observed drop of 6.2% in specificity (5.0% for patients 40 years and above) that was not significant, further supports that use of the device may enable physicians to evaluate skin lesions more confidently and correctly. We also found increasing physician performance as physician confidence increases, both with and without device use. Furthermore, device use increased physician confidence in management decisions and, notably, physician sensitivity was consistently higher with device use than without device use across different levels of physician confidence.

The observed physician AUC of 0.708 without device use was lower than the AUC of 0.762 with device use, and both were lower than standalone device AUC of 0.780. Physician management performance with use of the device was 91.4% sensitivity and 32.4% specificity. This difference in AUC with versus without the use of the device result was even more pronounced for lesions in which PCP management confidence was low (0.567 vs 0.682, respectively). Further, physicians reported increased confidence in their assessments with the availability of the device result. Results demonstrated consistent benefits of device use in terms of improved sensitivity across a wide range of physician, patient, and lesion characteristics.

Generally, PCPs are aware that they have room for improvement in their detection of skin cancer, as their most commonly self-reported estimate of sensitivity for correctly managing malignant lesions was 71% to 80%. Collectively, responses by the physician study participants show their strong agreement on the device benefits in clinical practice and the need for improved skin cancer surveillance in primary care, which demonstrate (1) a strong unmet need for augmentation of PCPs’ clinical assessments and management decisions in this context, and (2) that use of the ESS device could help meet that need for nearly all physicians that participated in this study.

This study has a few limitations of note. Due to ethical considerations since the device was experimental during the clinical studies, the DERM-SUCCESS Part 1 clinical validation study exclusively evaluated device performance compared to PCPs assessment and did not provide a device result to impact PCP lesion management. Thus, this companion reader study evaluated the impact of the same device results on a randomized subset of the same PCP-selected lesions, but it was not the same PCPs performing in-person lesion evaluations. Therefore, although patient and lesion clinical information and lesion tactile information (eg, smooth or rough, elevated or flat) was provided, the individual clinician’s own tactile evaluation of the lesions was not possible; while this does not mimic true clinical practice, this is consistent with care provided via telemedicine. Also, recall bias could impact re-evaluation of lesions, and all lesions were from Caucasian patients with lighter skin types; relative performance with and without aid of the device output in darker skin types was not assessed. Inherent to the study design and endpoints, the ratio of benign to malignant lesions among the test pool was not reflective of primary clinical practice where the large majority of lesions that are encountered are benign, though this does not impact the reported performance results (ie, sensitivity, specificity, and AUC). However, the incorporation of KCs among the test pool is a strength that more accurately represents the distribution of malignant lesions encountered in primary care.

In conclusion, this study demonstrated that, when added to clinical information and lesion tactile information, PCP use of the AI-enabled ESS device was able to significantly improve skin cancer diagnosis and management sensitivity, as well as physician overall diagnostic accuracy (ie, AUC). The findings demonstrate that utilization of the ESS device output may improve physician performance in the management of suspicious lesions and referral of skin cancer to ensure timely diagnosis. Future investigations may provide further evidence of device performance and/or impact on physician diagnosis and management on varied atypical lesion and skin types in clinical practice settings with variable pre-test probability (patient-identified lesions, dermatology vs primary care).

Supplemental Material

sj-docx-1-jpc-10.1177_21501319251342106 – Supplemental material for DERM-SUCCESS FDA Pivotal Study: A Multi-Reader Multi-Case Evaluation of Primary Care Physicians’ Skin Cancer Detection Using AI-Enabled Elastic Scattering Spectroscopy

Supplemental material, sj-docx-1-jpc-10.1177_21501319251342106 for DERM-SUCCESS FDA Pivotal Study: A Multi-Reader Multi-Case Evaluation of Primary Care Physicians’ Skin Cancer Detection Using AI-Enabled Elastic Scattering Spectroscopy by Laura K. Ferris, Erik Jaklitsch, Elizabeth V. Seiverling, Thomas Agresta, Peggy Cyr, Laurie Caines, Na Wang and Daniel M. Siegel in Journal of Primary Care & Community Health

Supplemental Material

sj-docx-2-jpc-10.1177_21501319251342106 – Supplemental material for DERM-SUCCESS FDA Pivotal Study: A Multi-Reader Multi-Case Evaluation of Primary Care Physicians’ Skin Cancer Detection Using AI-Enabled Elastic Scattering Spectroscopy

Supplemental material, sj-docx-2-jpc-10.1177_21501319251342106 for DERM-SUCCESS FDA Pivotal Study: A Multi-Reader Multi-Case Evaluation of Primary Care Physicians’ Skin Cancer Detection Using AI-Enabled Elastic Scattering Spectroscopy by Laura K. Ferris, Erik Jaklitsch, Elizabeth V. Seiverling, Thomas Agresta, Peggy Cyr, Laurie Caines, Na Wang and Daniel M. Siegel in Journal of Primary Care & Community Health

Supplemental Material

sj-docx-3-jpc-10.1177_21501319251342106 – Supplemental material for DERM-SUCCESS FDA Pivotal Study: A Multi-Reader Multi-Case Evaluation of Primary Care Physicians’ Skin Cancer Detection Using AI-Enabled Elastic Scattering Spectroscopy

Supplemental material, sj-docx-3-jpc-10.1177_21501319251342106 for DERM-SUCCESS FDA Pivotal Study: A Multi-Reader Multi-Case Evaluation of Primary Care Physicians’ Skin Cancer Detection Using AI-Enabled Elastic Scattering Spectroscopy by Laura K. Ferris, Erik Jaklitsch, Elizabeth V. Seiverling, Thomas Agresta, Peggy Cyr, Laurie Caines, Na Wang and Daniel M. Siegel in Journal of Primary Care & Community Health

Footnotes

Acknowledgements

The study team would like to take this time to commend Joseph Massaro, PhD (deceased) for writing the initial study statistical plan and for his contributions to the study design. We would like to thank Kim Ann Dukes, PhD, Margaret Shea, Kanisha Mittal, Lindsey Furton, Kaitlin Hartlage, and Laurent Billot, PhD for their contributions to the statistical analyses.

ORCID iDs

Erik Jaklitsch

Elizabeth V. Seiverling

Daniel M. Siegel

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was sponsored by DermaSensor, Inc.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Supplemental Material

Supplemental material for this article is available online.

References

Siegel

Miller

Wagle

Jemal

Cancer statistics, 2023. CA Cancer J Clin. 2023;73(1):17-48. doi:10.3322/CAAC.21763

American Cancer Society. Basal and Squamous Cell Skin Cancer Statistics | American Cancer Society. Published October 31, 2023. https://www.cancer.org/cancer/types/basal-and-squamous-cell-skin-cancer/about/key-statistics.html

Rogers

Weinstock

Feldman

Coldiron

BM.

Incidence estimate of nonmelanoma skin cancer (Keratinocyte Carcinomas) in the US population, 2012. JAMA Dermatol. 2015;151(10):1081-1086. doi:10.1001/JAMADERMATOL.2015.1187

Karia

Han

Schmults

CD.

Cutaneous squamous cell carcinoma: estimated incidence of disease, nodal metastasis, and deaths from disease in the United States, 2012. J Am Acad Dermatol. 2013;68(6):957-966. doi:10.1016/J.JAAD.2012.11.037

Nehal

Bichakjian

CK.

Update on Keratinocyte Carcinomas. N Engl J Med. 2018;379(4):363-374. doi:10.1056/NEJMRA1708701/SUPPL_FILE/NEJMRA1708701_DISCLOSURES.PDF

Tsang

Resneck

JS.

Even patients with changing moles face long dermatology appointment wait-times: a study of simulated patient calls to dermatologists. J Am Acad Dermatol. 2006;55(1):54-58. doi:10.1016/J.JAAD.2006.04.001

Hester

Thomas

Cederna

, et al. Increasing access to specialized dermatology care: a retrospective study investigating clinical operation and impact of a university-affiliated free clinic. Dermatol Ther. 2021;11(1):105-115. doi:10.1007/S13555-020-00462-Z

Coups

Geller

Weinstock

Heckman

Manne

SL.

Prevalence and correlates of skin cancer screening among middle-aged and older white adults in the United States. Am J Med. 2010;123(5):439-445. doi:10.1016/J.AMJMED.2009.10.014

Goldsmith

SM.

A unifying approach to the clinical diagnosis of melanoma including “D” for “Dark” in the ABCDE criteria. Dermatol Pract Concept. 2014;4(4):75. doi:10.5826/DPC.0404A16

10.

Malvehy

Hauschild

Curiel-Lewandrowski

, et al. Clinical performance of the Nevisense system in cutaneous melanoma detection: an international, multicentre, prospective and blinded clinical trial on efficacy and safety. Br J Dermatol. 2014;171(5):1099-1107. doi:10.1111/BJD.13121

11.

Monheit

Cognetta

Ferris

, et al. The performance of MelaFind: a prospective multicenter study. Arch Dermatol. 2011;147(2):188-194. doi:10.1001/ARCHDERMATOL.2010.302

12.

Emery

Hunter

Hall

Watson

Moncrieff

Walter

FM.

Accuracy of SIAscopy for pigmented skin lesions encountered in primary care: development and validation of a new diagnostic algorithm. BMC Dermatol. 2010;10:9. doi:10.1186/1471-5945-10-9

13.

Ulrich

Von Braunmuehl

Kurzen

, et al. The sensitivity and specificity of optical coherence tomography for the assisted diagnosis of nonpigmented basal cell carcinoma: an observational study. Br J Dermatol. 2015;173(2):428-435. doi:10.1111/BJD.13853

14.

Higgins

Lee

Leffell

DJ.

Point of care cutaneous imaging technology in melanoma screening and mole mapping. F1000Prime Rep. 2014;6:34. doi:10.12703/P6-34

15.

Pyun

Min

Goo

, et al. Real-time, in vivo skin cancer triage by laser-induced plasma spectroscopy combined with a deep learning-based diagnostic algorithm. J Am Acad Dermatol. 2023;89(1):99-105. doi:10.1016/J.JAAD.2022.06.1166

16.

Sun

Kentley

Mehta

Dusza

Halpern

Rotemberg

Accuracy of commercially available smartphone applications for the detection of melanoma. Br J Dermatol. 2022;186(4):744. doi:10.1111/BJD.20903

17.

Bigio

Boyer

Mourant

. Theoretical and experimental investigations of elastic scattering spectroscopy as a potential diagnostic for tissue pathologies. Paper presented at: Advances in Optical Imaging and Photon Migration (1994), paper OPTTM265; 21–March 23, 1994; Florida; OPTTM.265. doi:10.1364/AOIPM.1994.OPTTM.265

18.

Manolakos

Patrick

Geisse

, et al. Use of an elastic-scattering spectroscopy and artificial intelligence device in the assessment of lesions suggestive of skin cancer: a comparative effectiveness study. JAAD Int. 2023;14:52-58. doi:10.1016/J.JDIN.2023.08.019

19.

Hartman

Trepanowski

Chang

, et al. Multicenter prospective blinded melanoma detection study with a handheld elastic scattering spectroscopy device. JAAD Int. 2024;15:24-31. doi:10.1016/j.jdin.2023.10.011

20.

Jaklitsch

Thames

de Campos Silva

Coll

Oliviero

Ferris

LK.

Clinical utility of an AI-powered, handheld elastic scattering spectroscopy device on the diagnosis and management of skin cancer by primary care physicians. J Prim Care Community Health. 2023;14:21501319231205979. doi:10.1177/21501319231205979

21.

Rodriguez-Diaz

Manolakos

Christman

, et al. Optical spectroscopy as a method for skin cancer risk assessment. Photochem Photobiol. 2019;95(6):1441-1445. doi:10.1111/PHP.13140

22.

Tepedino

Baltazar

Hanna

Bridges

Billot

Zeitouni

NC.

Elastic scattering spectroscopy on patient-selected lesions concerning for skin cancer. J Am Board Fam Med. 2024;37(3):427-435. doi:10.3122/JABFM.2023.230256R2

23.

Merry

Croghan

McCormick

Chatha

Leffell

Clinical performance of novel elastic scattering spectroscopy (ESS) in detection of skin cancer: a blinded, prospective, multi-center clinical trial [initial results]. Cutis. 2022;110(6 Suppl):31.

24.

Tepedino

Baltazar

Hucks

Chatha

Zeitouni

NC.

Use of elastic scattering spectroscopy on patient selected lesions that are concerning for skin cancer. Cutis. 2022;110(6 Suppl):35-36.

25.

Salmon

Bonning

Use of elastic-scattering spectroscopy and machine learning when assessing skin lesions suggestive of skin cancer. J Dermatol Phys Assist. 2021;15(4):64-65.

26.

Hartman

Tepedino

Fung

, et al. Validation of a handheld elastic-scattering spectroscopy device on lesions suggestive of melanoma. J Dermatol Physician Assist 2022;16(4):51.

27.

Manolakos

Rabinovitz

Geisse

Bonning

Rodriquez-Diaz

Bigio

I CA

. Clinical validation of a handheld elastic scattering spectroscopy artificial intelligence device. Paper presented at: Podium Presentation, American Academy of Dermatology Innovation Academy; July 20-24, 2022; Vancouver, CA.

28.

Obuchowski

Rockette

HE.

Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests an anova approach with dependent observations. Commun Stat Simul Comput. 1995;24(2):285-308. doi:10.1080/03610919508813243

29.

Esserman

Thompson

Reid

, et al. Addressing overdiagnosis and overtreatment in cancer: a prescription for change. Lancet Oncol. 2014;15(6):e234-e242. doi:10.1016/S1470-2045(13)70598-9

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.03 MB

0.36 MB