Abstract
The limited number of specialists and diseases’ long-tail distribution create challenges in diagnosing oral tumors. Health care facilities with sole practicing pathologists face difficulties when encountering the rare cases. Such specialists may lack prior exposure to uncommon presentations, needing external reference materials to formulate accurate diagnoses. An image search or content-based image retrieval (CBIR) system may help diagnose rare tumors by providing histologically similar reference images, thus reducing the pathologists’ workload. However, the effectiveness of CBIR systems in aiding pathologists’ diagnoses through interactive use has not been evaluated. We conducted a remote evaluation in a near-clinical environment using Luigi-Oral, an interactive patch-based CBIR system that uses deep learning to diagnose oral tumors. The database comprised 54,676 image patches at multiple magnifications from 603 cases across 85 oral tumor categories. We recruited 15 general pathologists and 13 oral pathologists with varied experience to evaluate 10 retrospective test cases from 2 institutions using this dedicated system. At top-1 and top-3 differential diagnoses, the overall diagnostic accuracy among the 2 groups was significantly higher with Luigi-Oral than without (12.05% and 21.61% increase, P = 0.002 and P < 0.001, respectively). Improvements were more evident for tumor cases in which the category was underrepresented in the database, benefiting novice and experienced pathologists. Misdiagnoses using Luigi-Oral could be due to inappropriate query input, poor retrieval performance in cases with a rare morphologic type, the difficulty of diagnosis without elaborate clinical information, or the system’s inability to retrieve accurate categories with convincing images. This study proves the clinical usability of an interactive CBIR system and highlights areas for improvement to ensure adequate assistance for pathologists, which potentially reduces pathologists’ workload and provides accessible specialist-level histopathology diagnosis.
Keywords
Introduction
Many specific oral tumor types meet the World Health Organization’s (WHO’s) rare cancer definition, with <6 incidence per 100,000 population per year (Hendra et al. 2020; Peraza et al. 2020; Bokhari and Greene 2025). Due to this inherent rarity, diverse category and histologic variations (Appendix Table 1; Appendix Fig. 1), and multifactor considerations, such as tumor location (Wilson et al. 2018), diagnosing oral tumors poses significant challenges, particularly in regions with limited specialists. Practice is further hindered by limited access to databases and evolving diagnostic criteria and testing methods, increasing pathologists’ workload and risking fatigue, delays, and potentially fatal outcomes (Fitzpatrick and Migliorati 2025). An artificial intelligence (AI)–based tool could aid diagnosis and reduce pathologists’ workloads (Retamero et al. 2024). Several tools have been proposed to aid oral pathology diagnosis (Giraldo-Roldan et al. 2023; Cai et al. 2024; Zayed et al. 2024). However, the support is suitable only for limited oral tumor categories.
One possible application for multiple tumor categories is an interactive image search system, technically termed content-based image retrieval (CBIR). In pathology, 2 types of CBIR systems are used: whole-slide image (WSI) based and patch based. WSI-based systems use entire slides as queries, providing contextual slide-level information for image retrieval, while patch-based systems allow selection of specific regions of interest. In morphologically variable areas such as the head and neck, patch-based systems offer advantages by focusing on pertinent areas and excluding irrelevant regions compared with end-to-end WSI-level CBIR approaches (Chen et al. 2022; Li et al. 2023; Shafique et al. 2024; Alfasly et al. 2025). Moreover, pathologists can capture query images using digital microscopes or smartphones with the patch-based system, eliminating the need for expensive WSI scanners, thereby enhancing accessibility to advanced diagnostic tools even in small hospitals and health care facilities in developing countries (Kiehl 2022). By democratizing access to advanced diagnostic capabilities, patch-based CBIR offers a cost-effective solution for improving health care worldwide.
Implementing patch-based CBIR systems involves human intervention for the appropriate selection of query images and review of the retrieved results. A pathologist’s assessment of high positivity for a category may not align with the highest calculated similarity. Therefore, clinical usefulness cannot be judged solely by retrieval accuracy with preselected query patches. Although studies have reported pathologists’ positive experiences with CBIR algorithms regarding histological similarity (Schaer et al. 2019; Chen et al. 2022; Hashimoto et al. 2023), practical usability, potential pitfalls, and areas for system improvement in preclinical settings remain underexplored. By presenting relevant image references, CBIR potentially enhances diagnosis workflow, mitigates fatigue due to excessive workload, and reduces misdiagnosis. Therefore, our study aimed to evaluate the CBIR system’s practical usability in diagnosis accuracy and time and determine factors affecting pathologists’ decisions when using the system.
Here, we introduce Luigi-Oral, a web-based CBIR system for oral tumor histopathology, covering >80% of oral tumor categories in the WHO classification (El-Naggar et al. 2017). This broad coverage includes most cases encountered in oral pathology, including rare types, such as odontogenic and salivary gland tumors (Mendenhall et al. 2011; Marin et al. 2021; Alsanie et al. 2022; Gazendam et al. 2023). Using Luigi-Oral allowed us to explore the CBIR system’s effectiveness in enhancing the diagnostic workflow and addressing the challenges associated with rare and diverse oral tumors.
Methods
Diagnosed histopathology slides of oral tumors were collected from Tokyo Medical and Dental University Hospital between January 2001 and September 2022 (approval No. D2019-087). The slides were scanned and tumor areas were annotated by a pathology resident and verified by board-certified oral pathologists. Square image patches were extracted from tumor areas, resulting in a database of 54,677 patches from 605 cases across 85 oral tumor categories. The Luigi-Oral model was trained using a noncontrastive self-supervised learning method, TiCo (Zhu et al. 2022), on a subset of the database images, employing ResNet-18 as the backbone. The trained model computed image representations were stored as a retrievable database. To retrieve similar images, query images were encoded using the same model, followed by a nearest-neighbor search based on cosine similarity to identify the top-k most similar database images. A web-based user interface (https://luigi-pathology.com) was developed for query image upload and result observation. The training process and retrieval performance of the Luigi-Oral feature extractor model are discussed in prior study (Herdiantoputri et al. 2024).
A remote multicase–multireader evaluation was conducted with 10 test cases from different categories, collected from Teikyo University Hospital’s 2018
Detailed methodology, including development, evaluation, statistical analysis, and code availability, can be found in the appendix. The methodology summary is presented in Figure 1.
Summary of the study method. The workflow consists of the training of feature encoder, database construction, query case collection and evaluation by pathologists. In the evaluation, each case was presented to the evaluators through an online whole-slide image (WSI) viewer and assigned to either without Luigi-Oral (without the assistance of Luigi-Oral) or with Luigi-Oral (Luigi-Oral usage is allowed) for evaluation. Optionally, the evaluators could use textbook references before rising up to 3 differential diagnoses. Image created with BioRender.
Results
Evaluators and Cases
From September 12, 2023, to January 26, 2024, 246 valid answers were collected, comprising 123 answers when Luigi-Oral was allowed and 123 cases when it was not. When Luigi-Oral usage was prohibited, the evaluators were assisted by Luigi-Oral for 9 (7.3%) cases by mistake. These answers were therefore excluded from the analyses. Finally, 123 cases with Luigi-Oral and 114 cases without Luigi-Oral were included in the analyses. Evaluator distribution on each case is presented in Appendix Table 3.
Diagnostic Accuracy
Analyzed using the intention-to-treat approach adapted for the diagnosis assistive tool, Luigi-Oral usage significantly improved evaluator accuracy for top-1 and top-3 differential diagnoses. For top-1 diagnoses, we observed a 12.05% increase in accuracy (P = 0.002), whereas for top-3 diagnoses, the improvement was 21.61% (P < 0.001) (Table 1). Accuracy distribution among different pathologist groups revealed specific patterns. For top-1 accuracy, significant differences were found between senior and junior general pathologists (P = 0.042) and also between senior oral pathologists and junior general pathologists (P = 0.017). On top-3 accuracy, we found no significant differences between junior general pathologists and other groups.
Diagnostic Accuracy with and without Luigi-Oral at Top-1 and Top-3 Differential Diagnoses and the Respective Odds Ratios. a
The Wald test was used to assess statistical differences in accuracy. Sample sizes were n = 123 (with Luigi-Oral) and n = 114 (without Luigi-Oral) independent patient samples for each variable. P values were 2 sided. Odds ratios and confidence intervals (CIs) were calculated using logistic regression with generalized estimating equations, adjusting for confounding factors, such as profession, experience level, and case numbers in the database.
P < 0.01; ***P < 0.001.
At the case level, Luigi-Oral improved evaluator accuracy in 8 of 10 cases (Table 2). Underrepresented categories in the database showed significant improvements (Appendix Table 1). The accuracy of the oncocytoma case increased by 33.34% (P = 0.041) for top-1 and by 33.33% (P = 0.014) for top-3 diagnosis. The accuracy of the chondroblastic osteosarcoma case increased even more substantially, with a 45.45% increase (P = 0.009) for top-1 and a 63.63% increase (P < 0.001) for top-3 diagnosis. Significant improvements were also observed in the secretory carcinoma case for top-3 diagnosis, with a 34.62% increase (P = 0.049), and in the adenomatoid odontogenic tumor case for top-3 diagnosis, with a 41.66% increase (P = 0.020). These categories may have been more rarely encountered by pathologists in clinical practice. However, categories represented by >10 cases in the database, likely more commonly encountered, were diagnosed with higher accuracy without Luigi-Oral.
Case-Level Diagnostic Accuracy with and without Luigi-Oral at Top-1 and Top-3 Differential Diagnoses and the Respective Odds Ratios. a
The Wald test was used to assess statistical differences in accuracy. Sample sizes were n = 123 (with Luigi-Oral) and n = 114 (without Luigi-Oral) independent patient samples for each variable. P values were 2 sided. Odds ratios and confidence intervals (CIs) were calculated using logistic regression with generalized estimating equations, without considering confounding factors.
P < 0.05; **P < 0.01; ***P < 0.001.
Luigi-Oral usage also significantly improved evaluator accuracy for top-1 and top-3 differential diagnoses in all cases on a per-protocol approach (Appendix Table 4). Significant improvements were observed across the same set of cases, except for the secretory carcinoma case at the top-3.
At the evaluators’ category level, evaluator accuracy of junior general pathologists was significantly higher with Luigi-Oral at top-3 differential diagnoses (27.64% increase; P < 0.001). The same was observed for the accuracy of junior oral pathologists at the top-3 differential diagnoses (33.75% increase; P < 0.001). The accuracy of senior oral pathologists was significantly higher with Luigi-Oral at the top-1 (21.79% increase; P = 0.001) and top-3 (22.67% increase; P = 0.007) differential diagnoses (Fig. 2A). Different categories of evaluators showed higher accuracy without Luigi-Oral on different categories of tumors, likely related to their clinical experiences (Fig. 2B).
Breakdown of the evaluators’ category-level accuracy. (
The frequency of the evaluators’ answer choices at the top-1 is plotted in Figure 2C and Appendix Figure 4A and B. Several cases were confused with other tumor categories even with Luigi-Oral (e.g., the lateral periodontal cyst case was mostly confused with glandular odontogenic cyst among other odontogenic cysts and tumors), and the fibrous dysplasia case was confused mostly with other fibro-osseous diseases. The secretory carcinoma case was confused with cystadenoma, salivary duct carcinoma, and metastasizing ameloblastoma, which may have a similar morphology. However, without Luigi-Oral, the secretory carcinoma case was confused with cemento-ossifying fibroma, which bears no morphological resemblance (Appendix Fig. 4C).
The composition of the retrieved image results across various cases is intriguing. Top-10 accuracy, majority-10, and top-10 %query rates for the fibrous dysplasia case were 44%, 30%, and 90%, respectively, which were still higher than the categories it was confused with (10%, 20%, and 50% for ossifying fibroma and 12%, 10%, and 80% for cemento-osseous dysplasia). Queries for the chondroblastic osteosarcoma case resulted in a 0% majority-10 due to a single case in the database but achieved a high top-10 %query rate of 69.23%. The diagnostic accuracy of top-1 moderately correlated with majority-10 (r = 0.332), top-10 retrieval accuracy (r = 0.437), and top-10 %query (r = 0.346). However, it correlated weakly with the number of total cases in the database (r = 0.125) (Fig. 3A–D).
Correlations between diagnostic accuracy, retrieval accuracy, and misdiagnosis. (
Evaluators consulted Luigi-Oral for 97 (78.9%) cases when allowed. The remaining answers were 14 by senior oral pathologists, 5 by junior oral pathologists, 5 by junior general pathologists, and 2 by senior general pathologists (Appendix Fig. 5A). Although usage was allowed, Luigi-Oral was not used in 35.90% of diagnoses by senior oral pathologists, 29.41% by junior oral pathologists, 15.15% by junior general pathologists, and 5.88% by senior general pathologists, suggesting senior oral pathologists rely less on Luigi-Oral for their diagnosis. Regardless, some senior oral pathologists still accurately diagnosed the odontogenic keratocyst case and the adenoid cystic carcinoma case (Appendix Fig. 5B). The comparison of the diagnosis accuracy using the per-protocol approach is shown in Appendix Table 4.
Misdiagnosis
As many as 31 answers consulted Luigi-Oral but did not include the accurate diagnosis within the 3 differential diagnoses (11 by junior general pathologists, 9 by senior general pathologists, 4 by junior oral pathologists, and 7 by senior oral pathologists). The following reasons potentially caused misdiagnoses: (1) unsuitable query image(s) uploaded, resulting in low retrieval performance; (2) low retrieval performance despite appropriate query image(s); (3) Luigi-Oral results did not convince the evaluators to decide on the accurate diagnosis, leading to mistrust; or (4) indeterminate reasons (Fig. 3E).
From 31 misdiagnoses when Luigi-Oral was used, we found 6 misdiagnoses in which query images were unsuitable (3 on different cases; Fig. 3F). Most unsuitable queries included nontissue areas and cropped at an exceedingly low magnification. We also found 2 misdiagnoses for which query images were of nonrepresentative tumor areas, on the odontogenic keratocyst case, and 10 misdiagnoses for which retrieval performance was poor. Potential mistrust occurred in the adenomatoid odontogenic tumor, fibrous dysplasia, and secretory carcinoma cases. For the other misdiagnoses (11/31), the reasons were unclear. In most instances, the category’s highest retrieval rank for the correct diagnosis was lower than the submitted top-1 differential diagnosis’ retrieval rank (8/11) (Appendix Table 5).
Diagnosis Time
Twenty-six answers were assisted by Luigi-Oral and the WHO book, 71 by Luigi-Oral only, 91 without any reference, and 49 by the WHO book only. The median answering time with Luigi-Oral only was lower (196 s) than using the WHO book (265.5 s). Considering only the correct diagnosis at the top-1 differential diagnoses, 15 answers were assisted by both, 39 by Luigi-Oral only, 47 without any reference, and 24 by the WHO book. The median time spent answering accurately was lower (187 s) with the assistance of Luigi-Oral than with the WHO book (227 s) (Fig. 3G). Evaluators uploaded an average of 2.19 images to Luigi-Oral per case. Experience level and familiarity with oral tumor cases may influence the diagnosis time. The junior general pathologist group consistently exhibited higher median times than senior general pathologists and junior oral pathologists across different methods. When Luigi-Oral was used, senior oral pathologists exhibited higher median times than other groups did (Fig. 3H).
Discussion
Many AI systems, including CBIR, are designed for collaborative use with health care professionals, making it crucial to assess their performance in real-world collaborative scenarios (Sakamoto et al. 2022; Wataya et al. 2022; Raciti et al. 2023; Singh et al. 2023; Yasaka et al. 2023). By using oral tumors in our evaluation, our study is the first to explore an interactive patch-based CBIR system as an assistant for diagnosing rare tumors, where CBIR is most applicable. It is more appropriate than traditional supervised classification when training samples are limited, as the latter risks overfitting. CBIR also facilitates critical human decision making through interpretable visual similarity matches. Because no similar AI tools exist for this application, the study evaluated diagnostic workflows with and without Luigi-Oral instead of comparing it to another system. By testing its use among pathologists with varying levels of expertise and experience, we provide insights into both the tool’s effectiveness and its potential impact on decision-making across diverse professional backgrounds.
Studying rare tumors poses challenges; therefore, we used a carefully curated set of representative cases. Although the sample size was limited owing to the rarity of these conditions, we selected cases from diverse parent categories to ensure a comprehensive evaluation. Recognizing that general pathologists may have limited exposure to such rare cases, we considered the lasting impressions these cases often leave. Therefore, instead of sequential diagnosis with a washout period, as is common in computer-aided diagnosis research (Steiner et al. 2020; Ba et al. 2022; Wu et al. 2024), we used random assignments for each pathologist.
We allowed pathologists full discretion in their system usage when it was made accessible. This simulates realistic clinical settings in which familiarity with certain cases may negate the need for assistance. Our approach aimed to understand how often the system would be used voluntarily, which cases would prompt its use, and how evaluators would interact with it in practice.
Our results revealed that the interactive CBIR system improved diagnostic accuracy within a limited time frame, especially in pinpointing differential diagnoses to morphologically similar categories. The system requires human intervention, such as selecting diagnostically relevant areas and making final diagnoses based on presented similar cases. Therefore, CBIR retrieval performance and pathologists’ knowledge and experience markedly influence diagnostic accuracy. Despite this complexity, a notable improvement in diagnostic accuracy was observed across pathologists with varying experience and specialties, including senior oral pathologists and junior general pathologists. Senior oral pathologists likely benefited from their nuanced histopathological knowledge aided by image references from Luigi-Oral in making final differential diagnosis decisions. Consistent with the findings that AI tools benefit less experienced medical professionals (Wataya et al. 2022; Yasaka et al. 2023), junior general pathologists in our study were more receptive to Luigi-Oral’s suggestions and used them to select their answers.
Despite the limited training time on the system, we observed a lower median time to diagnosis when using Luigi-Oral’s assistance compared with using the WHO book alone, even after excluding misdiagnosed cases. This indicates the potential of the CBIR system to become an efficient reference tool by serving as a preliminary search method before conducting a more thorough analysis using textbooks and expert consultations. In clinical practice, pathologists may initially use CBIR for preliminary diagnosis and subsequently confirm findings with the WHO book to increase certainty. The recent introduction of a web-based version of the WHO book suggests the potential for complete digitalization, such as linking CBIR results directly to relevant sections in digital reference books to improve diagnostic efficiency.
Misdiagnosis with the CBIR system may stem from mistrust of retrieved cases, inappropriate query inputs, the CBIR algorithm’s low retrieval accuracy, or indeterminate reasons. Based on these findings, several critical factors must be addressed while developing an enhanced CBIR system to maximize benefits and minimize misdiagnoses. Retrieved images alone may not suffice to instill pathologists with the confidence required for final diagnoses. In diagnostic pathology, pathologists frequently rely on comprehensive clinical, diagnostic, and radiologic data as well as histology (Van Der Laak et al. 2021). By enhancing this process, the CBIR system could incorporate additional information regarding retrieved cases, such as histologic features or tumor categories, functioning akin to a comprehensive digital atlas. Integrating links to references and recommending ancillary tests would further aid investigations, providing interpretable results and valuable diagnostic guidance. Suggestions and clinical guidance for follow-up tests are accessible for research purposes on the Luigi-Oral website following this evaluation.
To effectively employ the CBIR system, users should receive comprehensive training on its interface, behavior, and database characteristics. This knowledge is essential for effectively selecting regions of interest, interpreting results, and extracting meaningful information. As users gain proficiency with the system, they can operate it more efficiently, thereby enhancing overall diagnostic accuracy.
The CBIR model’s performance require rigorous testing to accurately capture complex histological features (Dehkharghanian et al. 2023; Tommasino et al. 2023; Shang et al. 2024). Although retrieval performance correlated with diagnostic accuracy, decisions were not solely based on the retrieved cases. Even with low top-10 accuracy, a single accurate image substantially improved diagnostic accuracy, as observed in the chondroblastic osteosarcoma case. Nonetheless, challenges persist for lateral periodontal cysts and secretory carcinomas, of which retrieval performance remains poor due to their similarity to other types or rare variants. This underscores the importance of a database with diverse morphological representations, rather than just numerous similar cases of the same tumor type.
The system encountered difficulties differentiating mineralized tissue components, likely because the feature extractor prioritized high-level image similarity over semantic similarity (Fig. 3I). Addressing these limitations requires continuous improvements of the feature extraction model (Hameed et al. 2021). State-of-the-art models trained on histopathological images, along with advanced vision–language foundation models for histology (Kang et al. 2023; Chen et al. 2024), show promise in enhancing retrieval accuracy and semantic understanding, potentially improving diagnostic precision and efficiency (Ignatov et al. 2024; Lu et al. 2024).
The limitation of this study is that it may be subject to evaluator bias regarding AI-based diagnosis systems. In addition, the small sample size and single-read design differ from standard multireader, multicase studies, potentially limiting generalizability. However, as a pioneering pilot study, our findings offer valuable insights, with statistically significant results in some cases indicating a strong effect warranting further investigation. Larger clinical trials, involving more cases and diverse tumor types across varied medical institutions and geographical regions, are necessary. While patch-based CBIR systems are advantageous where WSI scanners are unavailable or processing large image data remains impractical, this study used WSI screenshots for remote evaluation. User-uploaded smartphone images enable quick uploads but require robust models to handle lower quality and color variations. Therefore, future exploration of this input method for resource-limited settings is necessary.
Overall, this study proves that an interactive CBIR system is useful to improve the diagnosis performance of pathologists with varied backgrounds and experience levels. Some areas should be improved to ensure safety since this system is not intended to replace qualified pathologists. With model performance improvement, supporting information, and adequate user training, this system potentially reduces pathologists’ workload and helps provide accessible specialist-level histopathology diagnosis.
Author Contributions
R.R. Herdiantoputri, contributed to conception, design, data acquisition, analysis, and interpretation, drafted and critically revised the manuscript; D. Komura, S. Ishikawa, contributed to conception, design, data analysis, and interpretation, drafted and critically revised the manuscript; M. Ochi, contributed to conception, design, drafted and critically revised the manuscript; Y. Fukawa, M. Tsuchiya, Y. Kikuchi, T. Ushiku, contributed to data acquisition, critically revised the manuscript; K. Oba, Y. Matsuyama, contributed to data analysis, critically revised the manuscript; T. Ikeda, contributed to data acquisition, analysis, and interpretation, critically revised the manuscript. All authors gave final approval and agree to be accountable for all aspects of the work.
Supplemental Material
sj-docx-1-jdr-10.1177_00220345251329042 – Supplemental material for Preclinical Evaluation of an Interactive Image Search System of Oral Pathology
Supplemental material, sj-docx-1-jdr-10.1177_00220345251329042 for Preclinical Evaluation of an Interactive Image Search System of Oral Pathology by R.R. Herdiantoputri, D. Komura, M. Ochi, Y. Fukawa, K. Oba, M. Tsuchiya, Y. Kikuchi, Y. Matsuyama, T. Ushiku, T. Ikeda and S. Ishikawa in Journal of Dental Research
Footnotes
Acknowledgements
We acknowledge and thank our evaluators and colleagues who helped during the recruitment: Drs. or Profs. Ayataka Ishikawa, Felix Manirakiza, Agoeng Tjahajani, Pretty Trisfilha, Janti Sudiono, Tetsuya Kitamura, Athira KP, Katsutoshi Hirose, Shingo Sakashita, Yuko Kinowaki, Kaori Shima, Mana Ideyama, Naomi Yada, Yuki Kato, Shrija G., Ryohei Kuroda, Piyush Asnani, Yu Usami, and other contributors who chose to remain anonymous.
In memory of Dr. Kou Kayamori, whose contributions and dedication greatly advanced this research. He will be deeply missed
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was supported by the AMED Practical Research for Innovative Cancer Control under grant number JP 24ck0106873 and JP 24ck0106904 to S.I., JSPS KAKENHI Grant-in-Aid for Scientific Research (S) under grant number 2H04990 to S.I., and JSPS KAKENHI Grant-in-Aid for Scientific Research (B) under grant number 21H03836 to D.K.
Data Availability
The raw data used in this study are not publicly available to preserve participant privacy. The data generated and analyzed during this study are available from the corresponding author upon reasonable request.
A supplemental appendix to this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
