Sage Journals: Discover world-class research

Abstract

Objective

To evaluate the performance of generative AI tools, specifically Ernie Bot and ChatGPT, in supporting online medical consultations in China, focusing on their accuracy, safety, and empathy, and to assess their potential role in addressing the supply-demand gap in the healthcare system.

Methods

We collected 233 trigeminal neuralgia consultations from a Chinese medical platform, including patient questions and doctor replies. Each question was input into ChatGPT-3.5 and Ernie Bot with role-specific prompts to generate large language models (LLMs) responses. Four blinded raters—two doctors and two patients—evaluated all responses using DISCERN and a modified PEMAT. Lexical, syntactic, and semantic analyses were conducted, with Spearman correlations assessing links between linguistic features and perceived quality.

Results

While doctors led in reliability, Ernie Bot scored highest overall, especially in empathy and clarity, likely due to stylistic choices rather than true understanding. Despite their fluency, LLMs remain prone to factual errors. Text analysis showed distinct linguistic patterns, with several features significantly correlated with perceived quality.

Conclusion

LLMs demonstrate strengths in perceived empathy and clarity but fall short in clinical accuracy and depth when addressing complex cases. Although they outperform doctors in communication-related aspects, their limitations in high-risk decision-making remain evident. As such, LLMs hold promise as adjunct tools for non-urgent consultations, but further refinement is needed to meet the standards of precise and personalized healthcare delivery.

Keywords

Healthcare consultation large language models response quality evaluation medical expertise patient experience text features

Introduction

Since the launch of China's healthcare reform in 2009, medical resources have expanded steadily..¹ However, this growth remains inadequate given the rising demand driven by an aging population and improved living standards, with persistent disparities in resource distribution.² By 2023, China had 4.782 million licensed (assistant) doctors and 5.637 million nurses, translating to 3.4 doctors per 1000 people—lower than many developed nations. Meanwhile, medical demand surged, with outpatient visits reaching 9.55 billion, hospital admissions at 302 million, and 243 million cross-regional treatments, underscoring the urgent need for optimized resource allocation.^3,4

Amidst these challenges, rapid internet development has driven the expansion of online healthcare services. By 2024, China had 1.099 billion internet users, with a penetration rate of 78.6%.⁵ Online consultations have gained popularity, especially among younger populations,⁶ and have helped alleviate the mismatch of medical resources by providing access to remote and underserved areas.^7,8 However, these platforms have not fundamentally resolved supply shortages. Online consultations rely on part-time doctors with limited availability, resulting in low service capacity and inconsistent quality—data shows an average of just 0.38 patients consulted per doctor per day, with an online-to-offline workload ratio of 1:14.⁹ Additionally, doctors often lack experience in virtual consultations, while communication barriers such as uncertain response times and limited real-time interaction further impact patient experience and service quality.¹⁰

The emergence of ChatGPT in 2022 marked a milestone in AI, particularly in large language models, offering a potential solution to the shortage of online healthcare consultations. Unlike traditional doctor-led telemedicine, LLMs can rapidly process large volumes of patient inquiries, providing preliminary diagnoses and health management recommendations, thereby improving efficiency.^11,12 LLMs have demonstrated early applications in medical imaging analysis and disease prediction, with some showing preliminary diagnostic capabilities.^13,14 For example, ChatGPT has passed the United States Medical Licensing Examination and achieved high accuracy in identifying certain conditions.¹⁵ ChatGPT has been shown to perform comparably to human doctors in cerebrovascular consultations, with higher perceived completeness and empathy but slightly lower clarity.¹⁶ In another study, GPT-4o generated more accurate, concise, and comprehensive responses than doctors in thyroid eye disease consultations.¹⁷ Google's medically trained model AMIE also outperformed clinicians in both accuracy and patient experience in simulated consultations.¹⁸

The rapid integration of LLMs in healthcare has brought unprecedented opportunities, yet also raises critical ethical and regulatory challenges. Despite their extensive medical knowledge, LLMs’ real-world clinical reliability remains unverified, as their diagnostic quality is constrained by training data, reasoning abilities, and adaptability to new medical knowledge, potentially leading to inaccuracies, particularly in complex or rare diseases.^19–21 Recent researches highlight critical and ethical risks including limited factual accuracy, unclear accountability, and inadequate protection of patient consent and data privacy.^22–25 LLMs may produce confident but potentially misleading content, which could exacerbate health disparities and undermine clinical trust, especially in low-resource or linguistically diverse settings. Furthermore, compared to human doctors, LLMs face challenges in patient communication, emotional support, and personalized care, and patient acceptance of AI-generated medical advice varies across demographics.

This study aims to compare and evaluate the quality of medical responses provided by doctors and large language models, exploring the potential applications of LLMs in online healthcare consultations. The research objectives are as follows: 1. Assess the efficacy and feasibility of responses through expert and patient evaluations. 2. Analyze the textual features of responses using text mining techniques. 3. Examine the impact of textual features on the perceived efficacy and feasibility of the responses.

To ensure experimental consistency and control task variability, we focus on trigeminal neuralgia, a representative symptom-driven condition characterized by sudden, severe facial pain. The diagnosis of trigeminal neuralgia heavily relies on patient-reported symptoms rather than imaging or laboratory data, making it suitable for evaluating language-based consultation systems.^26,27 Its structured diagnostic path and stable informational demands help reduce noise from task heterogeneity,²⁸ while its moderate sensitivity and clear boundaries enhance data collection and ethical feasibility in an online setting.²⁹ Prior studies also suggest that LLMs tend to perform more reliably in structured, language-dependent tasks than in complex, multimodal diagnostic scenarios.^30,31 Moreover, the majority of existing LLM evaluations are centered on English-language models, overlooking the performance and risks of localized models in non-English healthcare contexts. To address this gap, we compare ChatGPT with Ernie Bot-a mainstream Chinese-language LLM developed by Baidu—focusing on how linguistic and cultural embedding may shape perceived empathy and clarity in medical dialogue. This comparison is particularly relevant given the rapid expansion of localized AI tools and their growing use in digital health services in China and other multilingual regions.

Material and methods

Data sources

We selected xunyiwenyao, a prominent Chinese online medical platform, as the data source due to its comprehensive collection of patient-doctor interactions. This study focuses on trigeminal neuralgia, a condition affecting 1.6 to 3 per 1000 individuals,²⁷ characterized by intense facial pain. Given that its diagnosis primarily relies on symptom descriptions rather than medical tests, it is ideal for text-based consultations. Using bazhuayu (https://www.bazhuayu.com/), we extract critical data from April to May 2024, including patient symptoms, treatment history, inquiries, doctor analyses, and recommendations (Figure 1). Additionally, to ensure accurate simulation of LLM responses, we collect doctors’ titles, specialties, and affiliated departments.

Figure 1.

Example of online consultation for trigeminal neuralgia on a medical inquiry website.

Text generation with large language models

This study selected Ernie Bot and ChatGPT to generate large language model responses to patients’ questions. Ernie Bot is trained predominantly on Chinese-language web data, knowledge graphs, and search logs, with stylistic norms tailored to local user expectations. In contrast, ChatGPT is developed by OpenAI using primarily English corpora, often reflecting a more neutral, structured, and less affective tone. These distinctions allow us to explore whether model-specific language and interaction styles influence patient-perceived communication quality. For detailed technical background, please refer to the Supplemental Appendix A.

To improve the accuracy and applicability of LLMs in medical dialogues, we designed a role-playing prompt that includes attributes such as professional level and department affiliation.³² All experimental procedures involving LLMs were conducted following standardized workflows in accordance with TRIPOD-AI guidelines for AI evaluation transparency. Specifically, we employed ChatGPT (gpt-3.5-turbo) and Ernie Bot (ERNIE-3.5–8 K), for which structured prompts and model parameters were predefined to ensure reproducibility. The structured prompt design comprised three components: Role Setting, defining the model as a senior neurologist with expertise in trigeminal neuralgia and related interventions; Task Description, instructing the model to perform comprehensive case analyses, consider differential diagnoses, reference authoritative clinical guidelines, and provide evidence-based therapeutic and lifestyle recommendations; and Writing Requirements, ensuring that responses employed accurate medical terminology, were patient-friendly, scientifically rigorous, well-structured, and empathetic. Relevant model parameters, including temperature, top_p, and maximum token limits, were consistently applied across all experiments. The exact prompts, selected patient questions, corresponding responses from doctors, ChatGPT, and Ernie Bot, along with relevant model configuration parameters, are provided in Supplemental Appendix B.

During the data collection process, all questions were input individually into the gpt-3.5-turbo model and the ERNIE-3.5–8 K model to collect their responses. To eliminate potential learning biases or contextual influences, we adopted a new chat mode, ensuring that each question was answered independently. This process was conducted from April 2024 to May 2024, ensuring that no context from previous questions influenced subsequent responses, thereby minimizing systematic errors.

Evaluation of response effectiveness and feasibility

This cross-sectional observational study combining subjective assessments and objective text analysis was conducted in China from April to June 2024. Table 1 presents the subjective assessment of indicators. Two independent evaluation groups were designed to assess responses based on efficacy and feasibility. Efficacy was evaluated by two medical experts with over 10 years of clinical experience in tertiary medical centers, focusing on response reliability and quality. The DISCERN tool quantified efficacy, with questions 1–8 assessing answer reliability, 9–15 evaluating treatment quality, and question 16 providing an overall quality assessment.³³ Feasibility was assessed by two patients with clinical experience in treating trigeminal neuralgia, focusing on clarity, usefulness, empathy, and actionability. Evaluations were conducted using the Patient Education Materials Assessment Tool (PEMAT), which includes two subscales: PEMAT-U for understandability and PEMAT-A for the practicality of suggested actions.^34,35

Table 1.

Subjective assessment indicators. Specific scoring criteria for each indicator can be found in Supplemental Appendix C.

Evaluation panel	Dimensions	Indicator	Rating scale
Expert Evaluation	Efficacy	Reliability	8–40
Expert Evaluation	Efficacy	Quality	9–45
Patient Evaluation	Feasibility	Clarity	1–4
		Usefulness	1–4
		Comprehensiveness	1–4
		Understandability	1–10
		Actionability	1–3
		Empathy	1–2

To ensure the reliability and objectivity of subjective evaluations, we implemented a standardized multi-rater protocol incorporating structured training, pre-rating consistency testing, and rigorous bias control procedures. Briefly, all annotators were trained using validated instruments, followed by a consistency check to ensure inter-rater reliability. During the rating process, a triple-blind strategy was adopted to minimize potential source-related bias, including anonymization of response sources, randomized evaluation order, and independent rating assignment. For detailed training content, procedures, and quality assurance measures, please refer to Supplemental Appendix D and Supplemental Appendix E.

Analysis of response text features

We analyzed the textual features of the responses from three perspectives: words, sentences, and meaning. For word analysis, we utilized Python and the Jieba library to segment the text and gather frequency statistics, in order to ensure accurate recognition of specialized terminology, we extracted key terms related to “trigeminal neuralgia” from over 2000 academic papers in the CNKI database.

For sentence-level analysis, we examined the overall and average sentence lengths and conducted a differential analysis of high-frequency word pairs using a 2-gram model.

At the meaning level, we focused on evaluating the complexity, emotional tone, and semantic similarity of the responses. We measured text complexity using perplexity with a pre-trained BERT model.³⁶ Emotional polarity was assessed by analyzing the sentiment of individual words, while text and semantic similarities were quantified using the TF-IDF and Doc2Vec models.

Statistical analysis

Statistical analysis was performed using SPSS (IBM 26.0). The Intraclass Correlation Coefficient (ICC) was calculated to assess inter-rater reliability across all evaluated dimensions. We used a two-way random effects model with average measures and consistency type based on established psychometric standards. This model assumes raters and subjects are randomly sampled, focus on consistency in relative ranking regardless of absolute scores, and averages ratings from two raters per item. Calculations were performed in SPSS using a two-way ANOVA framework. For full methodological details and formulae, please refer to Supplemental Appendix F. Good consistency was indicated by an ICC value above 0.70. A one-way ANOVA tested whether differences between the three types of responses were significant. Spearman's correlation analysis.³⁷ explored the relationship between textual features and manual evaluation dimensions, quantifying the impact of text features on perceived characteristics with Spearman's Rho. The Spearman's Rho value, ranging from −1 to +1, indicates the degree of monotonic relationship between variables, with +1 as a perfect positive correlation, −1 as a perfect negative correlation, and values near 0 indicating no significant relationship.

Results

Differences in perceptual dimensions of responses from human doctors and large language models

Figure 2 presents the evaluation results of specialists and patients. In terms of ‘efficacy,’ doctors’ answers (mean 25.04, SD 3.05) were more reliable than those of ChatGPT (mean 23.40, SD 2.24) and Ernie Bot (mean 24.65, SD 2.74) (Figure 2(A)). However, Ernie Bot (mean 26.04, SD 3.05) had the best ‘Quality’ of answers (Doctor: mean 24.90, SD 2.23; ChatGPT: mean 24.45, SD 2.48) (Figure 2(B)). Although Ernie Bot outperformed ChatGPT, some repeated sentences in both responses may have come from the same information source. For feasibility, LLMs scored higher on ‘Clarity’ (Doctor: mean 3.37, SD 0.47; ChatGPT: mean 3.39, SD 0.58; Ernie Bot: mean 3.68, SD 0.44) (Figure 2(C)), ‘Usefulness’ (Doctor: mean 2.77, SD 0.61; ChatGPT: mean 3.43, SD 0.49; Ernie Bot: mean 3.82, SD 0.34) (Figure 2(D)), ‘Comprehensiveness’ (Doctor: mean 2.76, SD 0.63; ChatGPT: mean 3.36, SD 0.43; Ernie Bot: mean 3.80, SD 0.32) (Figure 2(E)), ‘Understandability’ (Doctor: mean 5.52, SD 0.63; ChatGPT: mean 6.83, SD 0.78; Ernie Bot: mean 7.42, SD 0.86) (Figure 2(F)), ‘Actionability’ (Doctor: mean 1.70, SD 0.59; ChatGPT: mean 2.76, SD 0.36; Ernie Bot: mean 2.85, SD 0.28) (Figure 2(G)), and ‘Empathy’ (Doctor: mean 0.11, SD 0.28; ChatGPT: mean 0.48, SD 0.50; Ernie Bot: mean 0.32, SD 0.47) (Figure 2(H)). Tukey Post Hoc Test showed significant pairwise differences (P < .001) across evaluation dimensions.

Figure 2.

(A) Reliability scores (scores range from 8 = worst to 40 = best) for the three groups answering the efficacy dimension. (B) Quality scores (scores range from 9 = worst to 45 = best) for the three groups answering the efficacy dimension. (C) Clarity scores for the three groups answering the feasibility dimension (scores range from 1 = worst to 4 = best). (D) Usefulness scores (scores range from 1 = worst to 4 = best) for the three groups answering the feasibility dimension. (E) Comprehensiveness scores (scores range from 1 = worst to 4 = best) for the three groups answering the feasibility dimension. (F) Understandability scores for the three groups answering the feasibility dimension (scores range from 1 = worst to 10 = best). (G) Actionability scores for the three groups answering the feasibility dimension (scores range from 1 = worst to 3 = best). (H) Empathy scores of the three groups answering the feasibility dimension (scores range from 1 = worst to 2 = best).

To better quantify the magnitude of observed differences between human and LLM-generated responses across evaluation dimensions, we calculated Cohen's d as a standardized measure of effect size (Table 2). Relative to doctor responses, Ernie Bot achieved large effect sizes in actionability (d = 2.48), comprehensiveness (d = 2.09), and usefulness (d = 2.13), as well as moderate to large effects in clarity and empathy (d = 1.24–0.55).

Table 2.

Pairwise Cohen's |d| effect sizes across perceptual dimensions.

	Doctor & ChatGPT Cohen's \|d\|	Doctor & Ernie bot Cohen's \|d\|	ChatGPT &Ernie bot Cohen's \|d\|
Reliability	0.80	1.23	0.50
Quality	0.90	1.38	0.57
Empathy	0.90	0.55	0.32
Usefulness	1.19	2.13	0.93
Comprehensiveness	1.12	2.09	1.15
Clarity	0.91	1.24	0.39
Actionability	2.17	2.48	0.27
Understandability	1.84	2.51	0.72

Based on the above data, we can preliminarily infer that LLMs—particularly Ernie Bot—demonstrated strong performance in terms of response quality and patient-perceived experience. Patients may have perceived Ernie Bot's responses as more helpful and emotionally supportive across specific dimensions. Notably, the differences between doctor- and LLM-generated responses yielded large effect sizes, suggesting statistically meaningful and perceptually salient differences. In contrast, the perceptual differences between ChatGPT and Ernie Bot were marginal. While effect size estimates help contextualize statistical significance, their practical and clinical implications should be interpreted with caution, particularly given the limited sample size and the inherently subjective nature of the evaluations.

Furthermore, systematic evaluation of the medical accuracy of consultation texts indicated that, although LLMs perform well in dimensions such as empathy, they remain susceptible to generating hallucinations. These hallucinations appear plausible but are factually incorrect, thereby posing potential clinical risks. Error analysis indicated that human doctors produced 2 incorrect answers (0.86%), Ernie Bot 6 incorrect answers (2.58%), and ChatGPT 17 incorrect answers (7.30%). In certain cases, LLMs misprioritized therapeutic options; for instance, electrical stimulator implantation, which is primarily a palliative intervention, was occasionally suggested as a first-line treatment. Table 3 presents the primary error types along with representative examples. The data indicate that human doctors rarely committed Explanation or Treatment Plan Errors, whereas LLMs (Ernie Bot and ChatGPT) frequently exhibited errors across all four categories (Table 3). Representative examples and a detailed classification of hallucination types are provided in Supplemental Appendix G, along with the potential clinical consequences if such responses were adopted by patients. The observed error types include diagnostic misjudgments, inaccurate etiological explanations, and inappropriate treatment recommendations, particularly in high-risk interventions.³⁸

Table 3.

Summary of primary error types in human and LLM responses.

Error type	Example summary	Typical source
Diagnostic Error	Misattributing occipital or postauricular pain to trigeminal neuralgia	Ernie Bot, ChatGPT
Treatment Plan Error	Recommending nonstandard or extreme procedures (e.g., parathyroid surgery, glioma resection, unnecessary nerve resection)	ChatGPT, Ernie Bot
Explanation Error	Incorrect rationale for symptoms or treatment (e.g., trigeminal neuralgia causes facial swelling; misattributing unrelated symptoms)	Doctor, ChatGPT, Ernie Bot
Misinformation Provided	Overstating surgical efficacy or omitting critical patient selection criteria	Ernie Bot

Textual feature differences between responses from human doctors and large language models

Figure 3 shows the results of the text analysis. Doctors’ responses had significantly higher frequencies of nouns (Doctor: mean 25.62, SD 0.35; ChatGPT: mean 23.59, SD 0.23; Ernie Bot: mean 23.86, SD 0.19), time words (Doctor: mean 0.71, SD 0.07; ChatGPT: mean 0.43, SD 0.04; Ernie Bot: mean 0.62, SD 0.04), verbs (Doctor: mean 20.96, SD 0.35; ChatGPT: mean 18.43, SD 0.17; Ernie Bot: mean 17.92, SD 0.15), and adjectives (Doctor: mean 3.57, SD 0.17; ChatGPT: mean 2.48, SD 1.09; Ernie Bot: mean 2.73, SD 0.07) compared to AI-generated responses (P < .001 for all). In contrast, doctors’ responses contained fewer conjunctions (Doctor: mean 4.39, SD 0.19; ChatGPT: mean 6.86, SD 0.11; Ernie Bot: mean 6.61, SD 0.10) and prepositions (Doctor: mean 2.06, SD 0.12; ChatGPT: mean 3.88, SD 0.07; Ernie Bot: mean 3.74, SD 0.06) (P < .001 for both). The higher noun frequency in doctors’ replies suggests they provide more specific, practical information, while the greater use of time words and verbs shows their focus on giving clear guidance, such as “follow up in three months” or “reduce sugar and salt.” LLM responses, with a higher proportion of function words, reflect their reliance on these words to maintain sentence structure and coherence.

Figure 3.

(A) Distribution of lexical shares for the three groups of responses. (B) Distribution of perplexity for the three sets of responses. (C) Sentiment polarity distribution of the three sets of responses. (D E) Semantic similarity and textual similarity. comp1: Mean similarity of pairwise comparisons of ChatGPT and doctor responses, comp2: Mean similarity of pairwise comparisons of Ernie Bot and doctor responses, comp3: Mean within-group similarity of doctor responses, comp4: Mean within-group similarity of ChatGPT responses, comp5: Mean within-group similarity of Ernie Bot responses, comp6: Mean of similarity between patient questions and doctor replies, comp7: Mean of similarity between patient questions and ChatGPT replies, comp8 :Mean of similarity between patient questions and Ernie Bot replies. Detailed explanations of all abbreviations and variables presented are available in Supplemental Appendix F.

In terms of meaning, ChatGPT and Ernie Bot's responses had lower PPL values and were concentrated in the low PPL range compared to doctors (Doctor: mean 3.17, SD 4.58; ChatGPT: mean 1.48, SD 0.18; Ernie Bot: mean 1.46, SD 0.27) (Figure 3(B)). Doctors’ responses, more flexible and varied, often include metaphors, similes, and idioms, resulting in higher PPL values and a long-tail distribution. Regarding sentiment, doctors’ responses had more dispersed emotional polarity, with patients perceiving a stronger emotional tendency (Doctor: mean 0.45, SD 0.36; ChatGPT: mean 0.35, SD 0.18; Ernie Bot: mean 0.37, SD 0.15) (Figure 3(C)). Statistically significant differences were found between doctors and the AI (P < .001), but no significant difference was found between ChatGPT and Ernie Bot (P = 0.335).

For similarity, doctors’ responses showed lower intra-group average semantic similarity (comp1: 0.61, comp2: 0.60, comp3: 0.58, comp4: 0.62, comp5: 0.61) (Figure 3(D)) and text similarity (comp1: 0.07, comp2: 0.09, comp3: 0.05, comp4: 0.07, comp5: 0.07) (Figure 3(E)). In contrast, ChatGPT and Ernie Bot's responses had higher average text similarity to doctors’ replies, reflecting that doctors use more varied language in their responses. While both LLMs showed language proficiency, they exhibited higher semantic similarity, indicating more content homogeneity. Regarding question-answer similarity, doctors’ responses differed more from the questions’ wording, while LLMs closely followed the question's wording (comp6: 0.06, comp7: 0.18, comp8: 0.21) (Figure 3(E)). However, doctors’ responses were more contextually relevant and focused on the core of the question, while LLM responses often diverged from the main topic (comp6: 0.80, comp7: 0.47, comp8: 0.54) (Figure 3(D)).

Impact of textual features on perception

Table 4 presents the results of the correlation analysis between text features and perceived response attributes in healthcare consultations. The appropriate use of prepositions, nouns, conjunctions, and verbs in text can enhance the quality and reliability of responses from doctors, as well as improve patients’ perceptions of empathy, clarity, usefulness, comprehensiveness, understandability, and actionability. In particular, the correlations between understandability and the use of nouns (Spearman ‘s Rho = 0.80, P < .001), verbs (Spearman ‘s Rho = 0.74, P < .001), and prepositions, (Spearman ‘s Rho = 0.78, P < .001) as well as comprehensiveness and the use of verbs (Spearman ‘s Rho = 0.73, P < .001) and nouns (Spearman ‘s Rho = 0.76, P < .001), are nearing high correlations. The use of nouns (Spearman ‘s Rho =0.51, P < .001) and verbs (Spearman ‘s Rho = 0.50, P < .001) also shows a moderate correlation with clarity (Spearman ‘s Rho > 0.50). At the sentence level, shorter average sentence lengths can improve the actionability of the text (Spearman ‘s Rho = 0.15, P < .001). At the semantic level, lower text perplexity can improve performance across all human evaluation metrics, especially in terms of the understandability (Spearman’s Rho = 0.60, P < .001) and actionability (Spearman’s Rho = 0.52, P < .001) of the response text, where the relationship is moderately correlated. Additionally, emotional support in the text can enhance the empathy felt by patients (Spearman’s Rho = 0.25, P < .001) and the relevance between the response text and the question can improve the usefulness (Spearman’s Rho = 0.40, P < .001) comprehensiveness (Spearman’s Rho = 0.35, P < .001), understandability (Spearman’s Rho = 0.40, P < .001), and actionability (Spearman’s Rho = 0.43, P < .001) of the response, and these metrics show moderate correlations.

Table 4.

Spearman correlations between text features and perceived response quality dimensions. Conj, Noun, Prep, and Verb indicate the proportion of conjunctions, nouns, prepositions, and verbs in the text, respectively.

	Reliability	Quality	Clarity	Usefulness	Comprehensiveness	Understandability	Actionability	Empathy
Conj	0.55(P < .001)	0.57(P < .001)	0.46(P < .001)	0.66(P < .001)	0.68(P < .001)	0.73(P < .001)	0.69(P < .001)	0.21(P < .001)
Noun	0.57(P < .001)	0.60(P < .001)	0.51(P < .001)	0.71(P < .001)	0.76(P < .001)	0.80(P < .001)	0.69(P < .001)	0.21(P < .001)
Prep	0.53(P < .001)	0.56(P < .001)	0.43(P < .001)	0.60(P < .001)	0.65(P < .001)	0.74(P < .001)	0.64(P < .001)	0.22(P < .001)
Verb	0.54(P < .001)	0.57(P < .001)	0.50(P < .001)	0.70(P < .001)	0.73(P < .001)	0.78(P < .001)	0.70(P < .001)	0.25(P < .001)
Length	−0.09(P = .04)	−0.05(P = .23)	−0.07(P = .17)	−0.08(P = .03)	−0.02(P = .15)	−0.08(P = .04)	−0.15(P < .001)	−0.19(P < .001)
PPL	−0.39(P < .001)	−0.41(P < .001)	−0.39(P < .001)	−0.46(P < .001)	−0.48(P < .001)	−0.59(P < .001)	−0.52(P < .001)	−0.27(P < .001)
Emotion	0.41(P < .001)	0.45(P < .001)	0.37(P < .001)	0.54(P < .001)	0.57(P < .001)	0.64(P < .001)	0.53(P < .001)	0.25(P < .001)
Relevance	0.31(P < .001)	0.31(P < .001)	0.28(P < .001)	0.40(P < .001)	0.35(P < .001)	0.40(P < .001)	0.43(P < .001)	0.15(P < .001)

Length refers to the average number of words in the text. PPL measures the text perplexity, indicating how uncertain or unpredictable the text is. Emotion represents the level of emotional support conveyed in the response. Relevance assesses the semantic similarity between the response and the question.

We emphasize that the findings should be interpreted as statistical associations rather than causal relationships. Spearman's rank correlation is a non-parametric method that captures monotonic associations but does not imply directionality or causality between variables.³⁹ The observed negative correlation between noun usage and perceived understandability does not necessarily indicate that reducing noun density would directly enhance comprehension. LLMs capable of producing clearer responses may adopt an overall simplified linguistic style, of which lower noun frequency is merely one co-occurring feature.⁴⁰ Moreover, multiple linguistic features—such as reduced use of medical jargon, enhanced emotional tone, or simplified sentence structure—often change simultaneously, making it difficult to isolate the independent contribution of any single variable. These feature clusters may reflect broader latent constructs, such as a “patient-friendly communication style”.^41,42 Therefore, we caution against overinterpreting individual correlations as definitive drivers of patient's perceived communication quality.

Discussion

LLMs enhance clarity and emotional support but fall short in medical depth

LLMs demonstrate notable advantages in clarity, language fluency, and emotional support when compared to human doctors, particularly in symptom-driven consultations such as those involving trigeminal neuralgia. Our findings revealed higher ratings for LLMs in empathy and clarity dimensions, consistent with prior studies showing that LLM-generated responses tend to be more readable and emotionally engaging for patients.⁴³

Effect size analyses further support these distinctions. Ernie Bot achieved large effect sizes relative to doctors in several user-centered dimensions. These results suggest that patients perceived its responses as more actionable and emotionally supportive, particularly in low-risk informational scenarios such as lifestyle guidance or postoperative care. Although these differences were statistically significant, their practical or clinical relevance warrants cautious interpretation. Perceived advantages in style and tone do not necessarily equate to diagnostic accuracy or medical appropriateness. Given the limited sample size and the subjective nature of evaluation metrics, further research is needed to determine whether LLMs can safely and effectively complement human expertise in more complex clinical contexts.

A detailed examination of the textual data indicated that these high empathy scores often stemmed from templated expressions—e.g., recurring phrases such as “I understand your discomfort”—which lacked adaptation to the patient's specific context or atypical concerns. This suggests that emotional resonance may arise more from stylistic fluency than from genuine contextual understanding.^23,44 Furthermore, these stylistic features may obscure substantive flaws. Despite their empathetic tone, LLMs occasionally produced responses containing factual inaccuracies or oversimplified clinical reasoning, such as inappropriate treatment suggestions or omission of relevant comorbidities. This observation is consistent with concerns about factual hallucinations in LLM-generated medical content,⁴⁵ where emotional fluency may conceal medically unreliable or even unsafe advice. These findings underscore the need to integrate fact-verification mechanisms into empathetic generation pipelines, ensuring that communication quality does not come at the cost of clinical reliability.²⁴

LLMs simplify medical text but lack professionalism

Our findings reveal that LLMs outperform human doctors in terms of linguistic clarity, structural coherence, and conciseness, consistent with prior research indicating that AI-generated responses are more readable and systematically organized.^46,47 This linguistic advantage is also supported by studies showing that LLMs tend to use shorter sentences, more functional words, and lower topic density, which facilitate patient comprehension and engagement.^48,49 Such simplification improves the emotional tone and overall user experience.⁵⁰ However, this stylistic strength may come at the cost of reduced professional specificity. Our analysis shows that LLMs frequently rely on generalized terms such as “certain medications” or “individual differences,” and exhibit highly patterned use of hedging and neutral phrasing. These generic expressions—while promoting consistency and reducing cognitive load—can weaken the delivery of risk disclosures and personalized medical reasoning.

While our findings suggest that language features such as reduced noun density or higher emotional expressiveness are positively associated with perceived clarity and empathy, this does not imply improved factual accuracy. In fact, recent work has shown that LLM-generated medical texts—although stylistically clearer—may contain more factual inaccuracies compared to those written by clinicians.⁵¹ Other studies have found that mainstream LLMs like ChatGPT and DeepSeek still struggle with diagnostic reasoning tasks, with accuracy rates under 60% on standardized clinical scenarios.⁵² However, targeted improvements, such as the use of retrieval-augmented generation, have shown promise in enhancing output correctness in breast cancer Q&A tasks.⁵³ These findings reinforce the need to interpret associations between linguistic features and perceived quality cautiously, as stylistic fluency may sometimes mask factual errors or limit clinical reliability.

Beyond linguistic features and user perceptions, the deployment of LLMs in medical contexts necessitates careful ethical and legal considerations. Prior work has documented the “hallucination” phenomenon in AI-generated medical advice, posing direct risks to patient safety if misinformation is accepted uncritically.²² Legal responsibility remains unclear as existing frameworks lack explicit regulations for AI-related clinical errors, complicating accountability.²⁵ Transparency and patient autonomy are further challenged by opaque AI decision-making processes and insufficient disclosure of AI involvement in care.⁵⁴ Additionally, AI systems must be monitored to prevent perpetuating health disparities caused by biased training data, a concern highlighted in recent mental health AI ethics analyses.^23,24 Incorporating these dimensions underscores the imperative for multidisciplinary governance frameworks that balance innovation with safety, fairness, and patient rights.

Reasons for higher ratings of Ernie Bot

Several factors may explain the higher subjective scores received by Ernie Bot, especially in dimensions such as empathy, clarity, and actionability. First, its fine-tuning on native Chinese clinical language likely improved lexical familiarity, syntactic fluency, and cultural congruence, thereby enhancing perceived trustworthiness.^55,56 Second, Ernie Bot frequently employed emotionally supportive phrasing (e.g., “I understand your concern,” “It's best to consult a doctor promptly”), which has been associated with increased perceived empathy and user satisfaction in prior studies.^57,58

In contrast, ChatGPT's outputs often included cautious disclaimers (e.g., “I cannot provide medical advice”), reflecting its global deployment and risk-averse design. While important for safety, this approach may reduce perceived usefulness in low-risk scenarios. Prior literature also highlights this tradeoff between communicative warmth and legal safeguards in medical AI applications.^24,59 These findings underscore the importance of balancing linguistic relatability and clinical appropriateness in deploying LLMs for patient-facing tasks.

Limitations and future directions

While our study offers novel insights into the performance of LLMs in structured, symptom-driven diagnostic tasks, several limitations should be acknowledged. First, we focused exclusively on trigeminal neuralgia, a disease with well-defined diagnostic pathways and high linguistic homogeneity, which facilitated internal validity and fair comparison between human and AI-generated responses. However, this narrow disease context may limit the generalizability of our findings to other clinical scenarios with more complex, heterogeneous presentations.^30,60 In particular, the unique communication patterns and emotional demands associated with psychological, oncological, or pediatric conditions remain unexplored. Additionally, although we employed a rigorous rating framework involving both medical professionals and patient representatives, the limited number of raters may constrain the generalizability of subjective assessments, even though interrater reliability was shown to be high.

Furthermore, this study was limited to single-turn, text-based interactions, which fall short of capturing the complexity of real-world, multi-turn clinical dialogues. Unlike the structured patient queries used in our dataset, actual online consultations typically involve iterative clarification, contextual reasoning, and dynamically evolving emotional needs. As such, the ability of LLMs to maintain coherence, resolve ambiguity, and deliver context-sensitive responses requires further empirical investigation. In addition, our correlation-based textual analysis does not account for potential confounding variables and cannot establish causal relationships between specific linguistic features and perceived quality.

Future research should therefore extend the evaluation of LLMs along multiple dimensions: (1) covering a broader spectrum of diseases with varying linguistic and clinical complexity; (2) simulating real-time, multi-turn interactions that better approximate actual doctor—patient communication; and (3) incorporating multimodal inputs such as clinical images and structured health records to assess model performance in more realistic and information-rich contexts. Methodologically, future work could leverage causal inference frameworks,⁶¹ mediation models, or experimental manipulations such as language-style interventions⁶² to explore whether specific linguistic cues causally influence user ratings.

Finally, we emphasize the need for interdisciplinary efforts to address ethical and regulatory challenges associated with medical LLM deployment—particularly those concerning hallucination risks,^22,54 transparency and user awareness,⁴⁵ and responsibility attribution in the event of medical harm.⁶³ These considerations are essential for ensuring that future LLM applications in healthcare are not only effective but also safe, accountable, and aligned with broader principles of medical ethics.

Conclusion

This study reinforces that while LLMs demonstrate notable advantages in clarity, emotional support, and accessibility, especially in delivering responses that patients perceive as helpful and understandable, they remain limited in handling complex clinical cases. Our findings align with prior research indicating that LLMs may generate text that is emotionally supportive yet potentially flawed in factual accuracy, particularly in domains requiring in-depth medical reasoning and personalized decision-making. The structured, empathetic language used by LLMs can enhance user trust but may also mask critical deficiencies in medical knowledge, echoing concerns about hallucination risks in AI-generated medical content. Thus, we advocate a cautious and principled approach to the deployment of LLMs in clinical contexts. LLMs may be safely applied to low-risk, non-urgent, and well-defined tasks, such as providing general health information, patient education, follow-up reminders, or summarizing non-critical patient-reported symptoms, where potential inaccuracies pose minimal risk. For moderate-risk tasks, including preliminary interpretation of patient symptoms, drafting differential diagnoses, summarizing consultation notes, or offering explanations of routine test results, LLM outputs should always be reviewed and validated by qualified healthcare professionals. High-stakes applications, including initial patient triage, medication prescribing, surgical planning, or critical diagnostic decisions, should not rely on LLMs without expert supervision because even minor errors could directly compromise patient safety. By adhering to this tiered framework, LLMs can be effectively leveraged to enhance patient communication and support clinical workflows while minimizing the risks associated with hallucinations or misinformation.

To advance the safe and effective integration of LLMs in healthcare, we recommend three key research and implementation directions. First, future studies should assess LLMs’ performance in multi-turn, context-rich dialogues that simulate realistic clinical encounters, as current evaluations often rely on isolated, single-turn response. Second, subjective quality scores should be complemented by real-world outcome measures to better capture clinical utility. Third, broader cross-lingual and multi-disease validation is essential, given potential cultural-linguistic biases and the varying demands of different clinical tasks. On the practical front, implementation should incorporate expert-in-the-loop validation systems to mitigate hallucination risks and ensure accountability. Additionally, robust legal and ethical frameworks must be established to clarify the responsibilities, transparency obligations, and informed consent requirements associated with LLM usage in clinical contexts. Only through such multi-layered safeguards can we ensure that LLMs contribute meaningfully and safely to the future of digital health.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076251388140 - Supplemental material for Comparing large language models and human doctors in symptom-driven online medical consultations: A case study on trigeminal neuralgia

Supplemental material, sj-docx-1-dhj-10.1177_20552076251388140 for Comparing large language models and human doctors in symptom-driven online medical consultations: A case study on trigeminal neuralgia by Liantan Duan, Zhong Yao, Xiaoyu Li, Yifei Wu and Dongfang Sheng in DIGITAL HEALTH

Supplemental Material

sj-xlsx-2-dhj-10.1177_20552076251388140 - Supplemental material for Comparing large language models and human doctors in symptom-driven online medical consultations: A case study on trigeminal neuralgia

Supplemental material, sj-xlsx-2-dhj-10.1177_20552076251388140 for Comparing large language models and human doctors in symptom-driven online medical consultations: A case study on trigeminal neuralgia by Liantan Duan, Zhong Yao, Xiaoyu Li, Yifei Wu and Dongfang Sheng in DIGITAL HEALTH

Footnotes

Acknowledgements

The authors gratefully acknowledge the invaluable contributions of all volunteers who participated in this study. This research was supported by the National Social Science Fund of China (Grant No. 23BTQ086), whose financial assistance is sincerely appreciated.

Ethics approval and consent

As this study did not involve real patient clinical interventions or the collection of private data, and all content was based on publicly available platform information or subjective evaluation tasks conducted by volunteers, formal approval from an institutional ethics board was not required. However, we adhered to fundamental principles of medical ethics in organizing the study, and all raters provided informed consent.

ORCID iDs

Liantan Duan

Zhong Yao

Xiaoyu Li

Yifei Wu

Dongfang Sheng

Consent for publication

All authors have read and approved the final manuscript. Dongfang Sheng acts as the guarantor of the study and accepts full responsibility for the integrity of the work.

Author contribution(s)

Liantan Duan contributed to conceptualization, visualization, writing the original draft, and editing and revising the final manuscript. Zhong Yao was responsible for conceptualization, investigation, data curation, and writing the original draft. Xiaoyu Li and Yifei Wu were responsible for data curation and formal analysis. Dongfang Sheng contributed to reviewing and editing the final manuscript, funding acquisition, and conceptualization. All authors read and approved the final manuscript.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: National Social Science Fund of China under Grant No. 23BTQ086.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Availability of data and materials

The original data and materials used in this study are available from the corresponding author upon reasonable request.

AI usage declaration

AI tools were used only for English language polishing; all research design, analysis, and scientific content were solely handled by the authors.

Supplemental material

Supplemental material for this article is available online.

References

Cao

Bai

Cao

, et al. Comparing regional distribution equity among doctors in China before and after the 2009 medical reform policy: a data analysis from 2002 to 2017. Int J Environ Res Public Health 2020; 17: 1520.

Zhu

Hsieh

Mao

. Spatio-temporal variations of licensed doctor distribution in China: measuring and mapping disparities. BMC Health Serv Res 2020; 20: 159.

http://www.gov.cn/lianbo/bumen/202408/content_6971248.htm

https://www.nhsa.gov.cn/art/2024/7/25/art_7_13340.html

https://www3.cnnic.cn/n4/2025/0117/c88-11229.html

https://www.iimedia.cn/c1061/102861.html

Qian

Lei

, et al. Experiences and challenges of emerging online health services combating COVID-19 in China: retrospective, cross-sectional study of internet hospitals. JMIR Med Inform 2022; 10: e37042.

Gajarawala

Pelkowski

. Telehealth benefits and barriers. J Nurse Pract 2020; 17: 218–221.

Yan

Song

. Provision of paid web-based medical consultation in China: cross-sectional analysis of data from a medical consultation website. J Med Internet Res 2019; 21: e12126.

10.

Mishra

. When patients connect with physicians on Facebook: physician perspectives on benefits, challenges, and strategies for managing interaction. Health Technol (Berl) 2018; 9: 505–515.

11.

Bohr

Memarzadeh

. The rise of artificial intelligence in healthcare applications. Artif Intell Healthc 2020; 1: 25–60.

12.

Wang

Wen

Dou

, et al. Performance and exploration of ChatGPT in medical examination, records and education in Chinese: pave the way for medical AI. Int J Med Inf 2023; 177: 105173–3.

13.

Placido

Yuan

Hjaltelin

, et al. A deep learning algorithm to predict risk of pancreatic cancer from disease trajectories. Nat Med 2023; 29: 1113–1122.

14.

Mehandru

Miao

Almaraz

, et al. Evaluating large language models as agents in the clinic. npj Dig Med 2024; 7: 1–3.

15.

Kung

Cheatham

Medenilla

, , et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. Dagan A, editor. PLOS Digital Health 2023; 2: e0000198.

16.

Roy

Atallah

Piper

, et al. Comparison of quality, empathy and readability of physician responses versus chatbot responses to common cerebrovascular neurosurgical questions on a social Media platform. Clin Neurol Neurosurg 2025; 255: 108986–6.

17.

Wang

Yang

Zeng

, et al. Evaluating the performance of ChatGPT in patient consultation and image-based preliminary diagnosis in thyroid eye disease. Front Med (Lausanne) 2025; 12: 1546706.

18.

Schaekermann

Palepu

, et al. Towards conversational diagnostic artificial intelligence. Nature 2025; 642: 442–450.

19.

Pressman

Borna

Gomez-Cabello

, et al. AI And ethics: a systematic review of the ethical considerations of large language model use in surgery research. Healthcare 2024; 12: 825.

20.

Otokiti

Ozoude

Williams

, et al. The need to prioritize model-updating processes in clinical artificial intelligence (AI) models: protocol for a scoping review. JMIR Res Protoc 2023; 12: e37685–5.

21.

Ferdush

Begum

Hossain

. ChatGPT and clinical decision support: scope, application, and limitations. Ann Biomed Eng 2023; 52: 1119–1124.

22.

Morley

Machado

CCV

Burr

, et al. The ethics of AI in health care: a mapping review. SSRN Electr J 2020; 260: 113–172.

23.

Longoni

Bonezzi

Morewedge

. Resistance to medical artificial intelligence. J Consum Res 2019; 46: 629–650.

24.

Meadi

Sillekens

Metselaar

, et al. Exploring the ethical challenges of conversational AI in mental health care: scoping review. JMIR Ment Health 2025; 12: e60432–2.

25.

Gerke

Minssen

Cohen

. Ethical and legal challenges of artificial intelligence-driven healthcare. Artif Intell Healthc 2020; 1: 295–336.

26.

Maarbjerg

Di Stefano

Bendtsen

, et al. Trigeminal neuralgia – diagnosis and treatment. Cephalalgia 2017; 37: 648–657.

27.

Ashina

Robertson

Srikiatkhachorn

, et al. Trigeminal neuralgia. Nat Rev Dis Primers 2024; 10: 1–22.

28.

Werner

Harden

Lawton

. Pathways to a diagnosis of trigeminal neuralgia: a qualitative study of patients’ experiences. BMC Primary Care 2025; 26: 65.

29.

Battistelli

Izzo

Ercole

, et al. The role of artificial intelligence in the management of trigeminal neuralgia. Front Surg 2023; 10: 1310414.

30.

Singhal

Azizi

, et al. Large language models encode clinical knowledge. Nature 2023; 620: 172–180.

31.

Nori

King

McKinney

, et al. Capabilities of GPT-4 on Medical Challenge Problems. 2023 Mar 20;

32.

Xue

Zhang

Gan

, et al. Quality and dependability of ChatGPT and DingXiangYuan forums for remote orthopedic consultations: comparative analysis. J Med Internet Res 2024; 26: e50882–2.

33.

Dağcı

Çam

Dost

. Reliability and quality of the nursing care planning texts generated by ChatGPT. Nurse Educ 2024; 10: 109–114.

34.

Mistry

Chesley

Chin

, et al. Advancing health equity-agency for healthcare research and quality research and action agenda. Health Serv Res 2023; 58: 275–280.

35.

Yılmaz

IBE

Doğan

. Talking technology: exploring chatbots as a tool for cataract patient education. Clin Exp Optom 2024; 108: 56–64.

36.

Miaschi

Brunato

Dell’Orletta

, et al. What Makes My Model Perplexed? A Linguistic Investigation on Neural Language Models Perplexity. 2021.

37.

Spearman

. The proof and measurement of association between two things. Int J Epidemiol 2010; 39: 1137–1150.

38.

Zhang

Cui

, et al. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv.org. 2023.

39.

Cohen

West

, et al. Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences. London: Routledge, 2013.

40.

Bender

Koller

. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020;Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

41.

Pennebaker

Mehl

Niederhoffer

. Psychological aspects of natural language use: our words, our selves. Annu Rev Psychol 2003; 54: 547–577.

42.

Schwartz

Eichstaedt

Kern

, et al. Personality, gender, and age in the language of social Media: the open-vocabulary approach. Preis T, editor. PLoS One 2013; 25;8:e73791.

43.

Zhang

Jin

, et al. Physician versus large language model chatbot responses to web-based questions from autistic patients in Chinese: cross-sectional comparative analysis. J Med Internet Res 2024; 26: e54706–6.

44.

Jin

Chandra

Verma

, et al. Better to ask in English: cross-lingual evaluation of large language models for healthcare queries. In: Proceedings of the ACM Web Conference 2022, 2024 May 8, pp.2627–2638.

45.

Agarwal

Jin

Chandra

, et al. MedHalu: Hallucinations in Responses to Healthcare Queries by Large

46.

Language Models. arXiv (Cornell University). 2024.

47.

Gilson

Safranek

Huang

, et al. How Does ChatGPT Perform on the Medical Licensing Exams? The Implications of Large Language Models for Medical Education and Knowledge Assessment. MedRxiv 2022 Dec 26.

48.

Zweck

, et al. Doctor versus AI: patient and physician evaluation of large language model responses to rheumatology patient questions, a cross sectional study. Arthritis Rheumatol 2024; 76: 479–484.

49.

Hanish

Cherian

Baumann

, et al. Reducing the use of Complex words and reducing sentence length to. Arthroscopy, Sports Med Rehabil 2023; 5: e1–e9.

50.

Romoff

Brunette

Peterson

, et al. The role of large language models in improving the readability of orthopaedic spine patient educational material. J Orthop Surg Res 2025; 20: 531.

51.

Wang

Yin

, et al. Emotional intelligence of large language models. J Pacific Rim Psychol 2023; 1: 17.

52.

Williams

CYK

Subramanian

Ali

, et al. Physician- and Large Language Model-Generated Hospital Discharge Summaries: a Blinded, Comparative Quality and Safety Study. medRxiv. Cold Spring Harbor Laboratory, 2024 Sep 30.

53.

Tordjman

Liu

Yuce

, et al. Comparative benchmarking of the DeepSeek large language model on medical tasks and clinical reasoning. Nat Med 2025; 31: 2550–2555.

54.

Yarkoni

. Personality in 100,000 words: a large-scale analysis of personality and word use among bloggers. J Res Pers 2010; 44: 363–373.

55.

Hong

Zhang

, et al. Evaluation of the integration of retrieval-augmented generation in large language model for breast cancer nursing care responses. Sci Rep 2024 Dec 28; 14: 30794.

56.

Wang

Liu

, , et al. MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback. arXiv (Cornell University). 2023 Jan 1.

57.

Lekadir

Frangi

Porras

, et al. FUTURE-AI: international consensus guideline for trustworthy and deployable artificial intelligence in healthcare. Br Med J 2025; 388: e081554.

58.

Sorin

Brin

Barash

, et al. Large language models and empathy: systematic review (preprint). J Med Internet Res 2024; 26: e52597–7.

59.

Lee

Suh

Zhan

, et al. Large Language Models Produce Responses Perceived to be Empathic. arXiv (Cornell University). 2024 Mar 26.

60.

Cabitza

Rasoini

Gensini

. Unintended consequences of machine learning in medicine. JAMA 20178; 318: 517.

61.

Zakrzewska

Akram

. Neurosurgical interventions for the treatment of classical trigeminal neuralgia. Cochrane Database Syst Rev 2011; 9: CD007312.

62.

Pearl

. Causal inference in statistics: an overview. Stat Surv 2009; 3: 96–146.

63.

Price

Cohen

. Privacy in the age of medical big data. Nat Med 2019; 25: 37–43.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.11 MB

0.93 MB