Performance of latest AI models,RAG,and MCP on lung cancer-related questions

Abstract

Background

Large language models (LLMs) have advanced rapidly. However, concerns remain regarding their reliability in clinical settings due to the inherent issues of hallucinations and inadequate referencing.

Materials and Methods

We evaluated six current LLMs: GPT-4.1 (GPT), o3, Gemini-2.5-Pro-Preview-0506 (Gemini), Grok-3 (Grok), Qwen3-235B-A22B (Qwen3), and Claude Sonnet 4 (Claude), as well as two technologies that extend LLM capabilities using external knowledge bases: retrieval-augmented generation (RAG) and Model Context Protocol (MCP). Each model was evaluated using 50 questions selected from a 132-question pool developed based on the Chinese Medical Association guideline for clinical diagnosis and treatment of lung cancer (2024 Edition). Three models—Qwen, GPT, and Grok—were further analyzed to assess performance changes with RAG and MCP integration. All responses were independently reviewed by two qualitative evaluators.

Results

Overall, o3 achieved the highest accuracy (50%), followed by GPT (48%) and Gemini (48%), then Grok (44%), Qwen (40%), and Claude (36%). However, implementing RAG (LLM-RAG) or MCP (LLM-MCP) significantly improved accuracy, with statistical differences observed between baseline LLMs and their RAG- or MCP-enhanced counterparts. Lexical richness and semantic noise both diminished, whereas the semantic clarity and accuracy of verbs, noun-verb combinations, and content words improved.

Conclusions

The six latest LLMs performed similarly on lung cancer-related questions. The integration of RAG or MCP significantly enhanced accuracy while simplifying sentence structure, focusing more on the main topics, and using more accurate vocabulary.

Keywords

large language model artificial intelligence lung cancer ChatGPT retrieval-augmented generation model context protocol

Introduction

Lung cancer is one of the most commonly diagnosed cancers and the leading cause of cancer-related deaths worldwide. About 80% of all cases are linked to smoking, while other risk factors include secondhand smoke, radon, coal combustion, occupational carcinogens, and air pollution.¹ According to the Cancer Statistics, 2025 report, there will be approximately 226,650 new cases of lung cancer in the USA in 2025.² Lung cancer causes nearly 2.5 times more deaths than second-ranking colorectal cancer. Moreover, prognosis is closely tied to the stage at diagnosis, with 5-year survival rates ranging from 98% for stage IA to under 5% for stage IV.^3,4 These statistics underscore the urgent need for new strategies to improve early detection and survival outcomes for lung cancer.⁵ Patients often encounter challenges in acquiring adequate medical information to gain a comprehensive understanding of their health conditions.

Recent advances in artificial intelligence (AI) offer promising tools to address this challenge. In healthcare, AI has demonstrated potential in enhancing diagnostic precision and cancer staging.^6–8 The rise of consumer-facing large language models (LLMs) has also allowed patients to access medical information through conversational platforms—a trend that continues to expand.⁹ Trained on massive datasets, LLMs can generate contextually relevant responses to prompts involving text, images, or multimedia.¹⁰ However, they often process data from mixed-quality sources, sometimes producing inaccurate or outdated responses.¹¹ This issue, known as “LLM hallucination,” can generate plausible but incorrect information, raising concerns about its reliability in clinical use.^12,13

To improve accuracy, technologies such as retrieval-augmented generation (RAG) and Model Context Protocol (MCP) have been developed. RAG strengthens LLMs by incorporating external knowledge bases (EKBs) through embedding models. This involves converting clinical guidelines into vector formats and integrating them into the model's retrieval process.^14,15 RAG has been successfully used in specialties like general surgery and urology.^16,17 MCP enables AI agents to select tools based on context and supports human-in-the-loop workflows, allowing users to inject data or authorize decisions.¹⁸ It facilitates EKB integration, improving response reliability, and has recently shown promise across multiple domains.^19,20 Despite their potential, RAG and MCP remain in early stages of adoption, and their effectiveness is not yet well established.^18,21 Prior studies assessing LLM accuracy often involved earlier models without current multimodal capabilities.^22,23 Additionally, few studies have evaluated LLM performance in non-English languages such as Chinese—the most widely spoken language globally.^24,25

To explore the potential of the latest AI models and technologies in addressing lung cancer-related questions in the Chinese context, we evaluated the performance of six leading LLMs in addressing questions on lung cancer screening, diagnosis, treatment, and follow-up, based on the Chinese Medical Association guideline for clinical diagnosis and treatment of lung cancer (2024 Edition), within a Chinese-language context.²⁶ We also examined how the application of RAG and MCP influenced model performance.

Materials and methods

Question and LLM preparation

The Chinese Medical Association guideline for clinical diagnosis and lung cancer treatment (2024 Edition) provides medical staff at all levels, including clinicians, imaging specialists, and laboratory technicians, with comprehensive and reasonable recommendations about lung cancer. From an initial pool of 132 questions derived from this guideline, we selected 50 questions based on their commonality and coverage to ensure a comprehensive representation of the guideline's pivotal content for in-depth analysis (Supplementary Material A). These clinically relevant questions cover four domains: screening (14, 28%), diagnosis (12, 24%), treatment (16, 32%), and follow-up (8, 16%). As of May 22, 2025, the following six LLMs, which belong to the leading publicly available LLMs, were evaluated: OpenAI's GPT-4.1 (GPT; released on April 14, 2025)²⁷; OpenAI's o3 (released on April 16, 2025)²⁸; Anthropic's Claude Sonnet 4 (Claude; released on May 22, 2025)²⁹; xAI's Grok-3 (Grok; released on February 19, 2025)³⁰; Google's Gemini-2.5-Pro-Preview-0506 (Gemini; released on May 6, 2025)³¹; Qwen's Qwen3-235B-A22B (Qwen; released on April 29, 2025),³² we obtained the results by using the above names through the official website APIs. LLMs with RAG (LLMs-RAG) and LLMs with MCP (LLM-MCP) were also examined, including Qwen-RAG, GPT-RAG, Grok-RAG, Qwen-MCP, GPT-MCP, and Grok-MCP.

RAG and MCP workflows

We used the application programming interfaces (APIs) of various LLMs in combination with the local RAG and MCP systems to complete the task (Figure 1). In the RAG system, we break down the textual guideline, setting the chunk size to 3000 characters and creating a 200-character overlap between adjacent chunks. We chose this length because when setting the 2000-character chunk size, we found that the absence of answer content led to a very low accuracy rate. Given that modern LLMs are capable of processing longer input sequences, we opted to increase the chunk length to better preserve contextual integrity. The text was then encoded into vectors using BAAI's bge-m3. Considering a user's input, the same encoding model was then used to convert the query into a vector. The similarity between the question embedding and the embedding of the document chunks in the vector store was then calculated. The most relevant document chunks were selected as the augmented context information for the current question. The provided question and associated documents were then merged into a new prompt. The LLM was then tasked with answering questions based on the provided information. In the MCP system, the “filesystem” was used as the MCP server for LLMs to obtain EKB. It is a Node.js server implementing MCP for file system operations, such as searching, reading, and editing files. The host is a user-facing AI application that orchestrates LLM interactions and enforces access controls. It uses an LLM to interpret user requests and initiate tool calls via an MCP client. The server provides access to external files. The client launches the server as a subprocess and exchanges JSON-RPC messages with it over a stdio transport.^33,34 Based on the given question, the LLM, referencing the tool definitions provided by the server, first generates a request to invoke the “list_directory” tool to retrieve the list of files. The client initiates the tool invocation request, and the server executes the operations and returns the results. After identifying the relevant file, the LLM similarly generates a call request for “read_file” to access the complete content. The LLM then generates the final answer based on the context and the file content retrieved by the server.

Figure 1.

(Left panel) directly asking questions to the LLM; (middle panel) asking questions to the LLM through the RAG system referring to EKB; and (right panel) and asking questions to the LLM through the MCP system with reference to EKB. The RAG system identifies the most relevant text chunks based on the input prompt and submits them together for further processing. While the MCP system orchestrates a closed-loop workflow where the LLM identifies the required data, the framework executes the retrieval, and a second inference pass synthesizes the results into a grounded final response.

Response generation and grading

An independent assessor meticulously entered the questions into the LLMs, maintaining the exact format, wording, and punctuation as presented in the source data during May 10–30, 2025. This approach ensured consistency across all LLMs. Following the input, the same assessor then gathered the responses produced by the LLMs. Each prompt was entered into a new conversation using a fresh chat window, and the previous conversations were cleared to avoid bias from the LLMs remembering past interactions. To minimize inaccuracies resulting from prompts, we used the same instruction: “As a thoracic surgeon in a top-tier hospital in China, you will answer patients’ questions regarding lung cancer with meticulousness and comprehensiveness.” When we access EKB through MCP or RAG, the instruction we followed was:

As a thoracic surgeon in a top-tier hospital in China, you will answer patients’ questions regarding lung cancer with meticulousness and comprehensiveness based on the Chinese Medical Association guideline for clinical diagnosis and lung cancer treatment. Do not add your own content.

To balance representativeness and efficiency, we selected 50 domain-stratified, commonly encountered clinical questions from 132, prior to viewing any model results. Two researchers independently evaluated the full dataset using a randomized, blinded evaluation protocol. They assessed accuracy based on NCCN guideline and Chinese Medical Association guideline, categorizing the responses as correct or incorrect while blinded to the model sources.³⁵ We employed a rigorous rubric that focused solely on accuracy, disregarding the influence of language style. Reviewers were instructed to identify verbose yet inaccurate responses. Only those answers that provide complete information in accordance with the reference guidelines were considered correct; those that did not provide the correct answer at all will be regarded as incorrect; otherwise, they were judged partially correct (Supplementary Material A). Due to the rigor of clinical issues, during our analysis process, we regarded partially correct results as incorrect. A Cohen's kappa value of 0.68 suggests substantial inter-rater agreement, beyond what would be expected by chance.³⁶ Any disputes at the review phase were resolved through consensus.

Readability assessment

Text readability was assessed using the Alpha Readability Chinese (ARC) analysis tool.³⁷ The evaluation covered multiple subcategories, including lexical and syntactic richness, noun and verb semantic accuracy, content word semantic accuracy, as well as semantic richness, clarity, and noise. The higher the score of lexical and syntactic richness, the more varied the vocabulary and sentence structure are, and for patients, the comprehensibility becomes worse. Higher semantic accuracy values for verbs, nouns, noun-verb combinations, and content words indicate more precise word usage, which in turn helps patients obtain information more accurately. Semantic richness refers to the abundance of the topic, semantic clarity refers to the concentration of the topic, and semantic noise represents the degree to which the text topic leans towards unimportant topics or deviates from important topics. The lower the values of semantic richness and semantic noise, and the higher the value of semantic clarity, the more tightly the content focuses on the main topic, thereby enhancing readability and clinical communication. These indicators are essential for measuring comprehensibility and gauging the difficulty of both human- and AI-generated responses. To establish a benchmark, 20 texts were selected from the 10th edition of Surgery published by People's Medical Publishing House. ARC evaluation of these texts provided a standard reference for undergraduate reading levels.

Statistical analysis

All data analyses were conducted using SPSS software (version 27.0.1; IBM, Armonk, NY, USA). Percentages of correct answers were calculated for each LLM, LLM-RAG, and LLM-MCP across the four sections. To test for significant differences in accuracy across models, chi-square tests were applied. For paired data, McNemar tests were used. When chi-square assumptions were violated (i.e., expected count <5 in >20% of cases or any expected count <1), comparisons between categorical variables were performed using two-sided generalized Fisher's exact tests. For continuous paired data (e.g., response length, reading ease), paired t-tests were applied when data were normally distributed; otherwise, the Wilcoxon signed-rank test was used. For continuous unpaired data, independent samples t-tests were performed when data were normally distributed and variances were homogeneous; otherwise, the Mann–Whitney U-test was used.To compare multiple groups, analysis of variance (ANOVA) or the Kruskal–Wallis test was conducted, including for readability scores derived from LLMs. For post-hoc multiple comparisons, either the Bonferroni correction or the Games–Howell method was employed.

Results

The latest LLMs displayed similar accuracy on lung cancer-related questions

Among the 50 questions, the accuracy rates of the six latest LLMs (as of May 30, 2025) did not differ significantly. Of these, o3 achieved the highest accuracy (50.0%), while Claude recorded the lowest (36.0%) (Figure 2). Across the four domains, the six base LLMs showed no statistically significant variation in accuracy. o3 achieved the highest accuracy (66.7%) in the diagnosis section. In the follow-up section specifically, the models showed the lowest baseline performance (12.5%).

Figure 2.

The accuracy of responses from LLMs and RAG/MCP variants in lung cancer screening, diagnosis, treatment, follow-up, and overall data.

Statistical differences observed between LLM and LLM-RAG, as well as between LLM and LLM-MCP

In aggregate, overall accuracy improved substantially after integrating RAG and MCP. Among the models enhanced with RAG or MCP, Grok-RAG, GPT-MCP, and Grok-MCP performed best, each reaching an accuracy of 96.0% (Table 1). GPT-RAG followed with 94%, while Qwen-MCP and Qwen-RAG ranked lowest among enhanced models at 84%. Grok-RAG and Grok-MCP also demonstrated the greatest improvement, with a 52% increase in accuracy over their base versions. Post-hoc analysis further showed no significant accuracy differences among the six enhanced variants when RAG or MCP was applied individually, a trend consistent across all four subcategories. However, statistical differences were observed between baseline LLMs and their RAG- or MCP-enhanced counterparts.

Table 1.

Comparison of LLMs and their RAG/MCP variants on 50 questions, n (%).

Models	Total	Outcomes		p
Models	Total	Correct	Wrong	p
Qwen	50	20 (40.0)	30 (60.0)
Qwen-RAG	50	42 (84.0)	8 (16.0)	<.001*
Qwen-MCP	50	42 (84.0)	8 (16.0)	<.001*
GPT	50	24 (48.0)	26 (52.0)
GPT-RAG	50	47 (94.0)	3 (6.0)	<.001*
GPT-MCP	50	48 (96.0)	2 (4.0)	<.001*
Grok	50	22 (44.0)	28 (56.0)
Grok-RAG	50	48 (96.0)	2 (4.0)	<.001*
Grok-MCP	50	48 (96.0)	2 (4.0)	<.001*

Note. Results are presented as frequency%. *P < .05 was deemed significant.

Across the four domains, after enhancement, Grok-RAG and GPT-MCP answered all eight follow-up questions correctly, reflecting the largest improvement in accuracy. Notably, both variants also answered all questions correctly in three of the four subcategories (Figure 2). Although most enhanced models exhibited statistically significant gains, some exceptions were noted. In the screening category, no significant differences were found between Qwen and Qwen-RAG, Qwen and Qwen-MCP, GPT and GPT-RAG, or Grok and Grok-RAG. Similarly, within the treatment section, comparisons between Qwen and its RAG and MCP variants and between GPT and GPT-MCP also failed to reach statistical significance.

There were statistically significant differences in response length

According to the Kruskal–Wallis test, the six LLMs differed significantly in word count and clustered into three gradients. Gemini produced the longest responses, followed by Qwen, Grok, and o3. GPT and Claude generated the shortest outputs. After applying RAG or MCP, most models showed a clear reduction in word count; however, the GPT group diverged from this trend, with GPT-MCP generating noticeably longer responses (Supplementary Material B).

Readability assessment

Our assessment of Chinese readability covers nine indicators across three dimensions. We compared the readability of GPT's responses with that of the undergraduate references. The results indicated that the readability of GPT was better than the undergraduate reading level in Chinese (Table 2). Other LLMs’ readability is at this level or even higher (Supplementary Material C). After applying these two techniques, the scores of Qwen's variants in terms of lexical richness and semantic noise both decreased (Tables 3 and 4). This indicates that the base model has more variable lexical and deviation from the main topic, thus has poorer readability. Second, from the perspective of semantic accuracy indicators, the semantic accuracy values of verbs, noun-verb combinations, and content words in the responses of Qwen-RAG and Qwen-MCP are significantly lower than those of Qwen. In other words, Qwen-RAG and Qwen-MCP use more difficult words, and the words they use are more semantically precise from a semantic perspective. From the perspectives of semantic clarity indicators, similar findings were also observed. Specifically, the semantic clarity value of Qwen-RAG's and Qwen-MCP's responses was higher than that of Qwen's. This indicates that the topic distribution probability of Qwen-RAG and Qwen-MCP's responses deviated more to the right from the normal distribution, and the topics were more concentrated, with clearer semantics. In terms of semantic noise, Qwen-RAG and Qwen-MCP scored lower than Qwen; this value indicates that Qwen may cover a wider range of topics or concepts, albeit at the cost of clarity. By contrast, GPT-MCP and GPT-RAG increased accuracy but produced longer outputs that were rated as more difficult to read across five indicators, although the changes were minor. Overall, the GPT series showed relatively stable variation. When comparing LLM-RAG with LLM-MCP, there is no difference between the two variants of Qwen, while GPT only shows differences in semantic noise and syntactic richness. However, Grok demonstrates differences across seven indicators (Supplementary Material D).

Table 2.

Comparison of the readability of GPT's answers and undergraduate Chinese standard reference.

Dimensions	The readability of Chinese texts	Answers of GPT	Undergraduate Chinese reading standard reference	p
Vocabulary	Lexical Richness^*,b	4.83 (4.66, 4.94)	5.99 (5.67, 6.06)	< .001***
Syntax	Syntactic Richness^*,b	2.16 (2.10, 2.18)	2.19 (2.18, 2.20)	< .001***
Semantics	Semantic Accuracy (n.)^**,a	3.18 ± 0.36	3.12 ± 0.28	.569
Semantics	Semantic Accuracy (v.)^**,a	7.54 ± 0.51	7.51 ± 0.34	.829
Semantics	Semantic Accuracy (n. & v.)^**,a	5.47 ± 0.60	5.27 ± 0.25	.054
Semantics	Semantic Accuracy (c)^**,a	5.92 ± 0.49	5.70 ± 0.27	.020***
Semantics	Semantic Richness (n.)^*,b	0.25 (0.23, 0.28)	0.28 (0.27, 0.30)	.003***
Semantics	Semantic Clarity (n.)^**,b	0.07 (0.05, 0.10)	0.04 (0.03, 0.07)	< .001***
Semantics	Semantic Noise (n.)^*,b	11.89 (6.83, 15.39)	47.67 (33.92, 109.83)	< .001***

*The larger the index value, the greater the text difficulty, and the harder it is to read.

**The larger the index value, the smaller the text difficulty, and the easier it is to read.

***P < .05 was deemed significant.

Independent samples t-test.

Mann–Whitney U-test.

Table 3.

Comparison of the Readability of Qwen’s Answers and Qwen-RAG’s Answers

Dimensions	The Readability of Chinese Texts	Answers of Qwen	Answers of Qwen-RAG	P
Vocabulary	Lexical Richness^{*, a}	5.15±0.15	4.68±0.29	P<0.001***
Syntax	Syntactic Richness ^{*, b}	2.13(2.10,2.16)	2.11(2.07,2.16)	P=0.191
Semantics	Semantic Accuracy (n.) ^{**, b}	3.05(2.80,3.24）	3.08(2.81,3.45)	P=0.119
Semantics	Semantic Accuracy (v.) ^{**, a}	7.50±0.62	7.28±0.75	P=0.038***
Semantics	Semantic Accuracy (n. & v.) ^{**, b}	5.45(5.14,5.63)	5.40(4.99,5.69)	P=0.717
Semantics	Semantic Accuracy (c) ^{**, b}	5.72(5.50,5.92)	5.64(5.26,5.99)	P=0.821
Semantics	Semantic Richness (n.) ^{*, a}	0.26±0.03	0.26±0.04	P=0.169
Semantics	Semantic Clarity (n.) ^{**, b}	0.62(0.04,0.09)	0.81(0.06,0.11)	P=0.004***
Semantics	Semantic Noise (n.) ^{*, b}	16.30(10.49,24.91)	11.74(7.66,18.66)	P=0.001***

*The larger the index value, the greater the text difficulty, and the harder it is to read.

**The larger the index value, the smaller the text difficulty, and the easier it is to read.

***P < 0.05 was deemed significant.

Paired t-test.

Wilcoxon signed-rank test.

Table 4.

Comparison of the Readability of Qwen’s Answers and Qwen-MCP’s Answers

Dimensions	The Readability of Chinese Texts	Answers of Qwen	Answers of Qwen-MCP	P
Vocabulary	Lexical Richness^{*, a}	5.15±0.15	4.58±0.33	P<0.001***
Syntax	Syntactic Richness ^{*, b}	2.13(2.09,2.16)	2.13(2.06,2.17)	P=0.257
Semantics	Semantic Accuracy (n.) ^{**, b}	3.05(2.80,3.24）	2.87(2.63,3.22)	P=0.466
Semantics	Semantic Accuracy (v.) ^{**, a}	7.50±0.62	7.06±0.86	P=0.003***
Semantics	Semantic Accuracy (n. & v.) ^{**, b}	5.45(5.14,5.63)	5.09(4.76,5.52)	P=0.017***
Semantics	Semantic Accuracy (c) ^{**, b}	5.72(5.50,5.92)	5.46(5.06,5.77)	P=0.014***
Semantics	Semantic Richness (n.) ^{*, a}	0.26±0.03	0.26±0.05	P=0.524
Semantics	Semantic Clarity (n.) ^{**, b}	0.62(0.04,0.09)	0.80(0.06,0.12)	P<0.001***
Semantics	Semantic Noise (n.) ^{*, b}	16.30(10.49,24.91)	10.34(6.41,16.17)	P<0.001***

The larger the index value, the greater the text difficulty, and the harder it is to read.

The larger the index value, the smaller the text difficulty, and the easier it is to read.

***

P < 0.05 was deemed significant.

Paired t-test.

Wilcoxon signed-rank test.

Discussion

Generative AI chatbots have emerged as a transformative technology and continue to evolve rapidly. Past studies have demonstrated the effectiveness of generative AI chatbots in delivering high-quality health information,^22,38 and we focused on the capabilities of the latest LLMs and the related technologies. This study extends previous work in several important ways. First, earlier research primarily evaluated models such as ChatGPT and Google Bard.^22,39 The present study includes a broader range of both leading proprietary and open-source LLMs. Second, we examined the impact of recent technologies, specifically, the integration of RAG and MCP, to enable LLMs to reference EKBs when answering lung cancer-related questions.

Our findings indicate that the six latest LLMs as of May 30, 2025, including the open-source Qwen, demonstrated similar accuracy across questions from the same domain. Global LLMs are predominantly trained on English corpora, which limits their vocabulary coverage for Chinese characters. This leads to inefficient tokenization, where complex medical terms are often fragmented into meaningless characters. Furthermore, the lack of pre-training on Chinese clinical texts restricts their ability to interpret cultural-specific medical terminology. This linguistic misalignment is likely a significant factor contributing to their lower baseline performance.^40,41 In the follow-up category, however, only 12.5% of responses were correct, which was different from the results of the previous study.⁴² Although the models’ training cutoffs spanned the publication dates of the referenced guidelines, no major performance differences emerged across clinical domains. This may be attributed to the paywalled nature of the EKB content, which likely prevented access during model pretraining. However, in the screening domain, the LLMs demonstrate a higher question-answering accuracy compared to other domains. The marginal benefit conferred by the EKB enhancement measures is relatively limited, potentially attributable to the widespread availability of information in this area, which has already been extensively incorporated into the models’ pre-training data.

As noted in the results, no significant differences were detected between the RAG- or MCP-enhanced counterparts. For consistency, we used Qwen in the non-thinking mode throughout the study, as applying its thinking mode could improve the outcomes.³² As an open-source LLM, Qwen offers the advantage of being deployable offline or locally, making it compatible with RAG and MCP for enhanced accuracy while ensuring data confidentiality. Both RAG and MCP enabled models to access EKBs and generate comparable outputs. However, MCP required complete file uploads, resulting in higher resource demands and latency. RAG necessitated careful preparation, including an embedding model, a retrieval system, and a well-processed EKB to deliver precise, contextually relevant responses.¹⁴ Therefore, the choice between the two should depend on the specific operational context.

In line with past studies,^17,20 nearly all LLM-RAG and LLM-MCP variants in our research significantly outperformed their baseline counterparts across the 50 questions. Among them, three models made only two errors each. For incorrect Chinese names of drugs such as “Durvalumab,” after referring to EKB, all were answered correctly. These findings underscore that, without referencing materials containing the correct answers, the models exhibited strong hallucinations. Notably, these hallucinations disappeared once standard, accurate content was provided. This observation highlights the importance of supplying high-quality reference material, which enables LLMs to access and apply the most current and trustworthy domain-specific information.^14,43 A deeper review of the incorrectly answered questions from these six and other variants revealed two main patterns: in most cases the results failed to present the full correct answer, and in one case the Grok-MCP response deviated from the expected direction; both problems stemmed from incomplete citations (Table 5). In addition, GPT always provides citation annotations. However, for Qwen and Grok, the citation markers sometimes are absent in the generated content, which can create the illusion that there is no reference to EKB. Future studies should also focus on optimizing ensemble approaches to enhance the retrieval precision and improve the citation annotation quality of RAG and MCP frameworks.

Table 5.

Sample questions and answers.

Which lung cancer tumor markers might show false increases in patients with renal failure?

Expected answer: Patients with renal failure may experience false increases in CYFRA 21-1 and ProGRP levels.

Qwen-RAG: In patients with renal failure, there may be a false increase in the tumor marker cytokeratin 19 fragment antigen (CYFRA 21-1) in the detection of non-small cell lung cancer (NSCLC). The reason for this false increase may be related to the decreased metabolic clearance rate caused by renal insufficiency. When interpreting the results, it is necessary to pay attention to excluding the influence of renal function status on the detection values [1]. It should be noted that the false increases of other NSCLC markers such as carcinoembryonic antigen (CEA) and squamous cell carcinoma antigen (SCCA) in renal failure have not been explicitly mentioned, but the interference of CYFRA 21-1 has been clearly indicated in the guidelines [1].

What treatment measures do patients with stage IA non-small cell lung cancer need to undergo after surgery?

Expected answer: For patients with stage IA non-small cell lung cancer, regular follow-up is recommended after complete surgical resection (R0 resection). For patients with positive surgical margins in stage IA after incomplete resection (R1 or R2 resection), they can choose to undergo another surgery or local radiotherapy (Class 2B evidence).

Grok-MCP: According to the “Guideline for Clinical Diagnosis and Lung Cancer Treatment” issued by the Chinese Medical Association (2024 Edition), for patients with stage I A (T1a/b/c N0) non-small cell lung cancer (NSCLC) who have undergone complete resection (R0 resection), the recommended postoperative management is routine follow-up, without adjuvant chemotherapy or radiotherapy. The guidelines specify that these patients should receive regular clinical monitoring to facilitate early detection of disease recurrence or progression. Follow-up evaluations typically include a comprehensive medical history review, physical examination, and chest CT imaging—contrast-enhanced scans may be utilized as clinically indicated. The recommended follow-up schedule consists of visits every 3 to 6 months during the first 3 years post-surgery, annually in the 4th and 5th years, and continued annual surveillance thereafter. Additionally, patients are strongly encouraged to discontinue smoking. xxbThis management approach is supported by high-level evidence-based recommendations (Class I evidence), reflecting the low risk of recurrence in stage IA NSCLC following complete surgical resection. Given that the potential benefits of adjuvant therapies do not outweigh their associated risks and adverse effects in this patient population, routine surveillance is considered the optimal strategy for timely identification of recurrence or new primary tumors, avoiding unnecessary pharmacological or radiation interventions.

In terms of readability, Gemini achieved the highest lexical richness and semantic noise score, whereas o3 displayed the weakest semantic clarity but excelled in verb accuracy, noun–verb combinations, and content word selection. Both LLM-RAG and LLM-MCP not only improved accuracy but also optimized responses. Their lower lexical and syntactic richness reflected a more focused linguistic style, with reduced semantic noise that enhanced coherence and clarity, thereby aiding comprehension. They also demonstrated higher verb accuracy, content word accuracy, and combined noun–verb semantic accuracy, producing responses that were more specific and contextually appropriate—critical for delivering accurate information, though potentially at the cost of readability. The loss of readability caused by providing accurate information is inevitable. Meanwhile, the concentration and clarity of the topic, as well as the simplicity of the vocabulary and sentence structure, have significantly contributed to enhancing patient readability of the content. However, both GPT-MCP and GPT-RAG have an overall trend of becoming more difficult to read. The average word counts also increased for both variants, with GPT-MCP demonstrating a statistically significant difference. This pattern does not appear to be associated with answer accuracy. For instance, in response to the question “What are the non-surgical treatment options for SCLC?,” GPT-MCP generated a wrong but longer answer. This phenomenon may be attributed to the inclusion of additional relevant information in the responses following the provision of a more comprehensive context, which is also what distinguishes GPT from the other two models. The comparison between LLM-RAG and LLM-MCP shows that distinct models produce varying outcomes. No differences were observed between the two variants of Qwen, whereas Grok exhibited discrepancies across nearly all metrics when these two techniques were applied. This suggests that Qwen is insensitive to technical modifications, while Grok displays a dependency on such changes. Prior research suggests AI can simplify text readability.²⁰ In this study, no specific optimization for readability was carried out. We are confident that performance and readability can be effectively enhanced through the use of prompt engineering, fine-tuning, or specialized algorithm optimization.^44–46 Whether AI can consistently balance accuracy with accessibility, ensuring both correct answers and broad comprehensibility, remains uncertain and warrants further investigation.

The data used by LLM during pre-training has undergone quality control, and false information and advertisements have been filtered out. However, there is still the problem of inaccurate responses, which is also known as hallucination. This study demonstrated that the application of RAG and MCP significantly enhanced the accuracy of LLM and provided more accurate information. Grok-RAG, GPT-MCP, and Grok-MCP achieved an accuracy of 96.0%. A notable advantage of LLMs equipped with EKB is their ability to provide easy-to-understand and accurate responses to common questions, enabling patients to access instant and reliable medical information. Although no significant ethical issues, such as racial discrimination, hate speech, or inciting harm, were observed in our experiments, enhanced governance remains imperative in clinical practice to avert potential misinterpretation. Since LLMs rely on internet connectivity for operation, the storage of personal sensitive information in memory and its potential exposure through prompts may pose significant risks to individual privacy. Local deployment offers a viable solution by enabling standalone usage without internet dependence, thereby enhancing data security. Open-source LLMs such as Qwen represent a favorable option in this context.

Nevertheless, our study has limitations. First, although the questions spanned various topics, their total number was relatively limited, and each question was posed only once. Despite this, statistically significant distinctions were identified among specific LLM variants. Second, unlike real-world clinical environments, we evaluated LLM accuracy using a set of common lung cancer-related questions. The researchers conducted a blinded evaluation with randomized response orders and strictly adhered to the scoring criteria, ensuring that judgments were unaffected by response length or stylistic variations. However, the questions raised by the patients are more complex and diverse, which might pose a challenge for LLM in extracting useful information and returning it. Despite full access to a comprehensive knowledge repository, the variants exhibit suboptimal exploitation, yielding only partial content synthesis and inadequate citation attribution, which represents a deficiency. To achieve competence in clinical practice, further advancements and empirical investigations in these areas will be essential in the future.

Conclusion

In conclusion, our preliminary findings revealed no statistically significant differences in accuracy among the six LLMs across four domains and 50 questions. However, integrating RAG or MCP markedly improved accuracy, uncovering meaningful distinctions across all questions. Grok-RAG, GPT-MCP, and Grok-MCP achieved accuracy rates as high as 96%. These systems simplified vocabulary and structure, sharpened thematic focus, and improved semantic coherence. Verb usage, including verb–noun collocations and content noun choices, also became more precise, yielding contextually appropriate and semantically specific sentences. The open-source model Qwen showed notable gains with RAG or MCP, though its accuracy remained relatively lower; its local deployability and open-source nature, however, provide important advantages for data privacy. Both RAG and MCP enhanced accuracy comparably and allowed dynamic updating of reference content, highlighting their potential as adaptive, high-precision tools for patient clinical education. Nonetheless, the uneven performance of EKB search in our experiments illustrates the instability of some AI applications in medicine. Thus, rigorous expert evaluation remains essential to ensure AI systems support, rather than burden, patients and medical staff.

Supplemental Material

sj-xlsx-1-dhj-10.1177_20552076261427503 - Supplemental material for Performance of latest AI models, RAG, and MCP on lung cancer-related questions

Supplemental material, sj-xlsx-1-dhj-10.1177_20552076261427503 for Performance of latest AI models, RAG, and MCP on lung cancer-related questions by Xinjie Zhao, Miaomiao Yang, Kang Tian, Hui Jiang, Deyu Guo, Yadong Wang and Jiajun Du in DIGITAL HEALTH

Supplemental Material

sj-docx-2-dhj-10.1177_20552076261427503 - Supplemental material for Performance of latest AI models, RAG, and MCP on lung cancer-related questions

Supplemental material, sj-docx-2-dhj-10.1177_20552076261427503 for Performance of latest AI models, RAG, and MCP on lung cancer-related questions by Xinjie Zhao, Miaomiao Yang, Kang Tian, Hui Jiang, Deyu Guo, Yadong Wang and Jiajun Du in DIGITAL HEALTH

Supplemental Material

sj-xlsx-3-dhj-10.1177_20552076261427503 - Supplemental material for Performance of latest AI models, RAG, and MCP on lung cancer-related questions

Supplemental material, sj-xlsx-3-dhj-10.1177_20552076261427503 for Performance of latest AI models, RAG, and MCP on lung cancer-related questions by Xinjie Zhao, Miaomiao Yang, Kang Tian, Hui Jiang, Deyu Guo, Yadong Wang and Jiajun Du in DIGITAL HEALTH

Supplemental Material

sj-docx-4-dhj-10.1177_20552076261427503 - Supplemental material for Performance of latest AI models, RAG, and MCP on lung cancer-related questions

Supplemental material, sj-docx-4-dhj-10.1177_20552076261427503 for Performance of latest AI models, RAG, and MCP on lung cancer-related questions by Xinjie Zhao, Miaomiao Yang, Kang Tian, Hui Jiang, Deyu Guo, Yadong Wang and Jiajun Du in DIGITAL HEALTH

Supplemental Material

sj-xlsx-5-dhj-10.1177_20552076261427503 - Supplemental material for Performance of latest AI models, RAG, and MCP on lung cancer-related questions

Supplemental material, sj-xlsx-5-dhj-10.1177_20552076261427503 for Performance of latest AI models, RAG, and MCP on lung cancer-related questions by Xinjie Zhao, Miaomiao Yang, Kang Tian, Hui Jiang, Deyu Guo, Yadong Wang and Jiajun Du in DIGITAL HEALTH

Footnotes

Acknowledgments

First and foremost, I would like to extend my sincere gratitude to all those who have supported me in the completion of this manuscript. I am especially indebted to my thesis advisors, Professor Jiajun Du and Professor Guanghui Wang, whose expertise, dedication, and guidance have been invaluable throughout this process. Despite their demanding schedules, they generously devoted their time, energy, and thoughtful insight to the supervision of my work, greatly contributing to the development and refinement of my ideas. I would also like to express my heartfelt appreciation to Dr Yadong Wang for his insightful suggestions on language and his guidance during the revision of this thesis.

ORCID iD

Jiajun Du

Ethical considerations

This study is neither a clinical trial nor a human trial; therefore, an IRB approval is not required. Therefore, the consent statement is not necessary.

Author contributions

Conceptualization (X.Z., M.Y.); data curation (X.Z., M.Y.); formal analysis (H.J., X.Z., M.Y.); investigation (X.Z., M.Y.); methodology (X.Z., M.Y.); resources (X.Z.); visualization (X.Z., K.T., M.Y.); writing‒original draft (X.Z., M.Y.); writing‒review and editing (Y.W., D.G., X.Z., M.Y.); supervision (J.D., Y.W.). All authors accepted the final draft of the manuscript.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data availability statement

All data supporting this article are available in the article and in Spplementary Material, and the original data are available from the corresponding author upon reasonable request.

Guarantor

JD.

Supplemental material

Supplemental material for this article is available online.

References

Wang

Zhang

Liu

, et al. Artificial intelligence in precision medicine for lung cancer: A bibliometric analysis. Digital Health 2025; 11. DOI: 10.1177/20552076241300229.

Siegel

Kratzer

Giaquinto

, et al. Cancer statistics, 2025. CA Cancer J Clin 2025; 75: 10–45.

Nooreldeen

Bach

Nooreldeen

, et al. Current and future development in lung cancer diagnosis. Int J Mol Sci 2021; 22: 8661.

Aokage

Suzuki

Saji

, et al. Segmentectomy for ground-glass-dominant lung cancer with a tumour diameter of 3cm or less including ground-glass opacity (JCOG1211): a multicentre, single-arm, confirmatory, phase 3 trial. Lancet Respir Med 2023: 540. DOI: 10.1016/S2213-2600(23)00041-3.

Roointan

Ahmad Mir

Ibrahim Wani

, et al. Early detection of lung cancer biomarkers through biosensor technology: a review. J Pharm Biomed Anal 2019; 164: 93–103.

Elkenawy

E-SM

Alhussan

Khafaga

, et al. Greylag goose optimization and multilayer perceptron for enhancing lung cancer classification. Sci Rep 2024; 14: 1.

Ebrahim

Rabie

El-Kenawy

E-SM

, et al. Optimization algorithms for deep learning prediction of liver cirrhosis: a survey. Metaheuristic Optim Rev 2025; 3: 01–11.

Gao

. Comparison of A.I. models in lung cancer staging and subtyping. J Thorac Oncol 2024; 19: 16.

Wang

Qian

Briggs

, et al. The use of chatbots in oncological care: a narrative review. Int J Gen Med 2023; 16: 1591–1602.

10.

Huang

Yan

, et al. From large language models to large multimodal models: a literature review. Appl Sci 2024; 14: 5068.

11.

Kadoya

Arai

Tanaka

, et al. Assessing knowledge about medical physics in language-generative AI with large language model: using the medical physicist exam. Radiol Phys Technol 2024; 17: 929–937.

12.

Keshavarz

Bagherieh

Nabipoorashrafi

, et al. ChatGPT in radiology: a systematic review of performance, pitfalls, and future perspectives. Diagn Interv Imaging 2024; 105: 251–265.

13.

Lee

Frieske

, et al. Survey of hallucination in natural language generation. ACM Comput Surv 2023; 55: 1–38.

14.

Gao

Xiong

Gao

, et al. Retrieval-augmented generation for large language models: A survey. 10.48550/arXiv.2312.1099 (2024, accessed 30 May 2025).

15.

Lewis

Perez

Piktus

, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. 10.48550/arXiv.2005.11401 (2021, accessed 30 May 2025).

16.

Sun

Owens

, et al. Development of a liver disease-specific large language model chat interface using retrieval-augmented generation. Hepatology (Baltimore, MD) 2024; 80: 1158–1168.

17.

Khene

Z-E

Bigot

Mathieu

, et al. Development of a personalized chat model based on the European association of urology oncology guidelines: harnessing the power of generative artificial intelligence in clinical practice. Europ Urol Oncol 2024; 7: 160–162.

18.

Hou

Zhao

Wang

, et al. Model context protocol (MCP): Landscape, security threats, and future research directions. 10.48550/arXiv.2503.23278 (2025, accessed 12 July 2025).

19.

da Silva

LMV

Köcher

Gehlhoff

. Beyond formal semantics for capabilities and skills: model context protocol in manufacturing. 10.48550/arXiv.2506.11180 (2025, accessed 12 July 2025).

20.

Ehtesham

Singh

Kumar

. Enhancing clinical decision support and EHR insights through LLMs and the model context protocol: an open-source MCP-FHIR framework. 10.48550/arXiv.2506.13800 (2025, accessed 12 July 2025) .

21.

Tozuka

Johno

Amakawa

, et al. Application of NotebookLM, a large language model with retrieval-augmented generation, for lung cancer staging. Jpn J Radiol 2024; 43. DOI: 10.1007/s11604-024-01705-1.

22.

Rahsepar

Tavakoli

Kim

GHJ

, et al. How AI Responds to Common Lung Cancer Questions: ChatGPT vs Google Bard. Radiology 2023; 307. DOI: 10.1148/radiol.230922.

23.

Iannantuono

Bracken-Clarke

Karzai

, et al. Comparison of large language models in answering immuno-oncology questions: a cross-sectional study. Oncologist 2024; 29: 407–414.

24.

Piao

Chen

, et al. Assessing the performance of large language models (LLMs) in answering medical questions regarding breast cancer in the Chinese context. Digital Health 2024; 10. DOI: 10.1177/20552076241284771.

25.

Zhang

Chu

, et al. Evaluation of large language models under different training background in Chinese medical examination: a comparative study. Frontiers Artif Intellig 2024; 7. DOI: 10.3389/frai.2024.1442975.

26.

中华医学会肿瘤学分会. 中华医学会肺癌临床诊疗指南（2024版）. 中华医学杂志 2024; 104. DOI: 10.3760/cma.j.cn112137-20240511-01092.

27.

OpenAI. Introducing GPT-4.1 in the API. https://openai.com/index/gpt-4-1/ (14 April 2025, accessed 30 May 2025).

28.

OpenAI. Introducing OpenAI o3 and o4-mini, https://openai.com/index/introducing-o3-and-o4-mini/ (16 April 2025, accessed 30 May 2025).

29.

Anthropic. Introducing Claude 4. https://www.anthropic.com/news/claude-4 (23 May 2025, accessed 30 May 2025).

30.

xAI. Grok 3 Beta—The Age of Reasoning Agents. https://x.ai/news/grok-3 (19 February 2025, accessed 30 May 2025).

31.

Comanici

Bieber

Schaekermann

, et al. Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. 2025. DOI: 10.48550/arXiv.2507.06261.

32.

Yang

, et al. Qwen3 technical report. 2025. DOI: 10.48550/arXiv.2505.09388.

33.

Errico

Ngiam

Sojan

. Securing the model context protocol (MCP): risks, controls, and governance. 10.48550/arXiv.2511.20920 (2025, accessed 28 January 2026) .

34.

Ray

. A survey on model context protocol: architecture, state-of-the-art, challenges and future directions. Authorea Preprints 2025. DOI: 10.36227/techrxiv.174495492.22752319/v1.

35.

Network

NCC.

NCCN clinical practice guidelines in oncology: non-small cell lung cancer (Version 4.2025). https://www.nccn.org/professionals/physician_gls/pdf/nscl.pdf (23 May 2025, accessed 30 May 2025).

36.

Cohen

. A coefficient of agreement for nominal scales. Educ Psychol Meas 1960; 20: 37–46.

37.

雷蕾. 韦瑶瑜. 刘康龙. Alphareadabilitychinese:汉语文本可读性工具开发与应用. 外语与外语教学 2024; (1): 83–93.

38.

Hanguang

Feizhong

Xingyue

, et al. A comprehensive survey of large language models and multimodal large language models in medicine. Inf Fusion 2025; 117: 102888.

39.

Musheyev

Pan

Gross

, et al. Readability and information quality in cancer information from a free vs paid Chatbot. JAMA network Open 2024; 7: e2422275.

40.

Cui

Yang

Yao

. Efficient and effective text encoding for Chinese LLaMA and alpaca. 10.48550/arXiv.2304.08177 (2024, accessed 28 January 2026).

41.

Wang

Chen

Song

, et al. CMB: a comprehensive medical benchmark in Chinese. 10.48550/arXiv.2308.08833 (2024, accessed 12 July 2025) .

42.

Pan

Musheyev

Bockelman

, et al. Assessment of artificial intelligence chatbot responses to top searched queries about cancer. JAMA Oncol 2023; 9: 1437.

43.

A survey on RAG meeting LLMs: towards retrieval-augmented large language models. In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2024, Vol. 24, pp.6491–6501. DOI: 10.1145/3637528.3671470.

44.

Karim

Khafaga

Eid

, et al. A novel bio-inspired optimization algorithm design for wind power engineering applications time-series forecasting. Biomimetics 2023; 8: 21.

45.

Khodadadi

Roghani

Caso

, et al. Data-driven PSO-CatBoost machine learning model to predict the compressive strength of CFRP- confined circular concrete specimens. Thin-Walled Struct 2024; 198: 111763.

46.

Elshewey

Alhussan

Khafaga

, et al. EEG-based optimization of eye state classification using modified-BER metaheuristic algorithm. Sci Rep 2024; 14: 1.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

0.08 MB

0.01 MB