Sage Journals: Discover world-class research

Abstract

Background

Artificial intelligence (AI) chatbots are increasingly used for health information dissemination. However, their effectiveness depends on the clarity, reliability, and quality of the content they deliver. This cross-sectional study aimed to evaluate the readability and reliability of kyphosis-related information provided by six major AI chatbots: ChatGPT, Gemini, Copilot, Perplexity, DeepSeek, and Grok.

Methods

We selected the top 10 kyphosis-related questions from Google's “People also ask” section and submitted them to each chatbot. Readability was assessed using FKGL, FKRS, GFOG, SMOG, CL, ARI, and LW indices. Quality and reliability were evaluated using the DISCERN tool, JAMA benchmark, Global Quality Score (GQS), Ensuring Quality Information for Patients (EQIP), and a kyphosis-specific content score (KSC). Statistical analyses were performed using the Kruskal–Wallis and Mann–Whitney U tests.

Results

No statistically significant difference was found among chatbots in FKGL, FKRS, SMOG, ARI, or GFOG scores. However, Perplexity had significantly higher DISCERN and EQIP scores, indicating superior content quality. All chatbots presented content at a readability level higher than the AMA-recommended sixth-grade level. While AI models provided more comprehensive and up-to-date information than traditional web sources, their outputs remained challenging for the average patient to comprehend.

Conclusions

AI chatbots offer promising tools for disseminating health information about kyphosis but require significant improvements in readability. Expert-reviewed and patient-centered refinements are necessary to ensure accessibility and safety in digital health communication.

Keywords

Kyphosis artificial intelligence readability patient education information

Introduction

Educational resources are recognized for aiding patients in comprehending their current state of health.^1,2 With the advancements in internet-based patient education resources, patients are increasingly using online information for health-related inquiries.³ A chatbot is software that uses artificial intelligence (AI) methods, including machine learning and natural language processing to engage in conversation with people.⁴^,⁵ Chatbots are gaining popularity in the healthcare industry because they provide easy access to information and serve as a platform for engaging and personalized interactions.⁶ These chatbots are progressively emerging as essential resources for medical knowledge as more people utilize them to obtain solutions to their inquiries and acquire health guidance.⁷^,⁸

AI is transforming numerous industries, including healthcare, by emulating human cognitive functions and allowing machines to learn and make decisions. This advancement has led to substantial progress in diagnosis and treatment planning.^9,10 Open-Access Artificial Intelligence Modules (OSAIMs), which are generally free and publicly available, support various fields, including computer science and medicine.^11,12 The increasing exploration of AI applications, especially in clinical settings, highlights their potential to enhance medical diagnosis and facilitate the selection of appropriate treatment methods. With the rise of AI-powered tools, patients are increasingly relying on these modules for health-related information. Tools such as ChatGPT, Gemini, Copilot, Perplexity, DeepSeek, and Grok have become valuable resources for disseminating medical knowledge. However, the readability and clarity of the information provided by these platforms can vary, potentially affecting patient understanding and decision making.

Readability refers to the level of reading proficiency required to comprehend written content. Poor readability can lead to misunderstandings, misinterpretations, and inadequate management of medical conditions, particularly for individuals with limited health literacy. To objectively assess readability, numerous formulas and numerical values have been developed. These formulas calculate readability levels.¹³^,¹⁴ It is clear that online health content must be sufficiently readable. To address this, the American Medical Association (AMA), the National Institutes of Health (NIH), and the United States Department of Health and Human Services have established appropriate readability levels for patient education materials, which is a target of sixth grade and lower.¹⁴^,¹⁵

Kyphosis, a spinal disorder characterized by an excessive forward curvature of the thoracic spine, can significantly impact a patient's quality of life. It often leads to physical discomfort, respiratory difficulties, and psychological distress, making accessible and accurate information about the disease crucial for patients and caregivers. As AI chatbots play an increasingly pivotal role in patient education, evaluating the readability of the information they provide about specific conditions like kyphosis becomes essential. This evaluation is particularly important given the growing reliance on digital health tools and the need for patient-centered communication in healthcare.

Despite the potential of these platforms to provide information comparable to that of medical graduates,⁷ concerns have been raised about the dissemination of false information. Therefore, it is essential to ensure oversight and review by trained healthcare professionals as the use of chatbots as a source of medical information continues to grow. Based on these considerations, the current study compares ChatGPT, Gemini, Copilot, Perplexity, DeepSeek, and Grok as potential sources of patient information and evaluates the quality of responses to commonly asked kyphosis-related questions. The emergence of chatbots that incorporate natural language processing and AI represents one of the most significant shifts in this area.⁷^,⁸

This study aims to conduct a comparative analysis of the readability of kyphosis-related information generated by six prominent AI modules: ChatGPT, Gemini, Copilot, Perplexity, DeepSeek, and Grok. By assessing the clarity, structure, and accessibility of the content, the study seeks to identify which AI module offers the most patient-friendly information and how these tools can be optimized to better serve individuals seeking knowledge about kyphosis. Additionally, the study will explore the potential benefits of AI-generated content in improving patient outcomes, such as enhancing understanding of treatment options, promoting adherence to medical advice, and reducing anxiety associated with the condition.

Materials and methods

We conducted a cross-sectional, observational research study that does not involve human subject research; therefore, it did not require referral to an institutional ethical review board. This is because the datasets used in this study were sourced from either a machine-learning model or the internet and comprise de-identified medical information. To enhance reproducibility, we have clearly specified the exact prompt texts (10 kyphosis-related ‘People Also Ask’ questions), data collection timestamps (1–15 May 2025, UTC+3), geolocation (Turkey), and device/browser (MacOS Chrome). All chatbot interactions were performed in signed-out mode to prevent personalization bias. The versions used were ChatGPT (GPT-4.1), Gemini (1.5 Pro), Copilot (GPT-4-Turbo), Perplexity (Sonar Large), DeepSeek (V2), and Grok (2.0) (all model versions corresponded to the active public releases during the data collection period (May 2025)). Temperature and safety settings were kept at default. The raw chatbot outputs, cleaned datasets, and the Python scripts used for readability and quality index computation are available as Supplemental materials to ensure full transparency.

The Google search and data collection were conducted between 1 and 15 May 2025, ensuring that all chatbot responses reflected information available during that period.

To analyze kyphosis, we performed a Google search using the keyword “kyphosis,” with the search history cleared beforehand. We noted the first 10 questions under the “People also ask” section, eliminating any repeated questions with the same meaning. We then submitted these 10 questions to ChatGPT, Gemini, Copilot, Perplexity, DeepSeek, and Grok. To reduce the possibility of bias from the AIs’ memory retention, we created an entirely new user account specifically for this study and deleted the chat history before each question. Two authors independently evaluated each AI's assessment ratings and the content of their responses.

The People Also Ask (PAA) queries were collected in English under a Turkey-based IP using signed-out mode to avoid personalized results. We acknowledge that PAA items are dynamic and language-dependent; therefore, results may differ with alternative formulations (e.g. diagnostic vs. treatment queries). Future studies should include larger and stratified question sets.

Readability assessment

The Flesch–Kincaid grade level (FKGL) test, which ranges from fifth grade to university graduate level, establishes the educational level required to comprehend a specific text (with 5 being the lowest and 12 the highest). We also used the Flesch–Kincaid reading ease (FKRS) test to assess the article's readability. This score helps readers estimate the education level needed to read a certain material comfortably. A document's ease of comprehension is indicated by a number between 0 and 100; a score of 100 or nearly 100 is considered easy to read, whereas a score of zero or nearly zero indicates extreme complexity and difficulty. Similar to earlier studies, we converted each website's text into a Microsoft Word document (Redmond, Washington) to determine the FK scores.¹⁶ ^–19

The Gunning Fog (GFOG) index is applicable in various fields and aims to assist American companies in making their written content easier to read. It calculates the number of years of schooling required to comprehend a particular text.

The Simple Measure of Gobbledygook (SMOG) index is beneficial for middle-aged readers, typically ranging from fourth grade to college. Although most formulas aim to measure comprehension at 100%, they generally assess between 50% and 75% comprehension. It is most accurate when applied to texts with a minimum of 30 sentences, calculating the years of schooling needed for an average reader to understand the material.

The Coleman–Liau (CL) score targets middle-aged readers, from fourth grade to college, and can be applied across multiple industries. It is based on assessing texts with grade levels ranging from 0.4 to 16.3 to determine the corresponding grade level in the American educational system.

The Automated Readability Index (ARI) has been used by the military to create technical documents, determining the grade level required for comprehension based on its computations. It assesses the intellectual level needed to comprehend textual content in American schools. A term's complexity increases with the number of characters.

The Linsear Write (LW) index was established to help the US Air Force assess the readability of its technical publications, providing a rough estimate of the academic proficiency required to comprehend the material.

This cross-sectional study was prepared and reported in accordance with the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) checklist for cross-sectional studies to ensure methodological transparency and completeness.

Reliability and quality of answers

The DISCERN procedure, developed in Oxford, UK, evaluates the quality of health information in texts.²⁰ It consists of 16 questions; the first 8 assess the publication's reliability, the next 7 focus on available treatments, and the final question asks users to rate the overall AI responses. Each question is scored between 1 and 5, resulting in a total score ranging from a minimum of 6 to a maximum of 80. Based on their overall scores, the AI's responses are categorized as “excellent” (63–80), “good” (52–61), “fair” (39–50), “poor” (28–38), or “very poor” (<28).

The global quality score (GQS), which rates the overall quality of responses on a 5-point scale, was also used to evaluate the AI responses. These rankings indicate how helpful the reviewer believes the AI responses will be for patients, as well as the quality of the information provided.²¹

The Journal of American Medical Association (JAMA) benchmark assesses the quality of online material based on four criteria: authorship, attribution, disclosure, and currency.²² Authorship necessitates that AI responses include information about the authors or contributors, including their credentials and links. Attribution necessitates identifying all content references and sources, as well as relevant copyright information. Disclosure calls for comprehensive information about website ownership, including any financial or business relationships and potential conflicts of interest. Finally, currency (timeliness) ensures that the content of AI replies is dated at the time of initial upload and any subsequent modifications. Each satisfied criterion is scored from 0 to 4, with 4 indicating the highest quality.

For conversational AI outputs, we adapted JAMA and Ensuring Quality Information for Patients (EQIP) criteria to fit chatbot responses. Authorship and disclosure were evaluated based on the presence of identifiable system information or disclaimers, whereas attribution and currency were assessed through reference citations or timestamps visible in the responses. EQIP items originally designed for static documents were mapped to conversational equivalents, such as structure, completeness, and accessibility of medical advice.

The EQIP tool assesses both the quality and readability of relevant medical texts. Respondents answer 20 questions with “yes,” “partially,” or “no.” A score ranging from 0 to 100 is assigned based on these responses. The scale awards 1 point for a “yes,” 0.5 for a partial response, and 0 for a “no.” The total scores are summed, divided by 20, and adjusted for non-applicable items ((X of Yes * 1) + (Y of Partially * 0.5) + (Z of No * 0))/(20 − (Q of Not applicable))] *100 = % score).²³Scores within 0% to 25% indicate “serious problems in quality,” those within 26% to 50% indicate “serious problems in quality,” scores within 51% to 75% indicate “good quality with minor problems,” and scores within 76% to 100% indicate “well written,” according to the EQIP tool.²⁴

The predetermined kyphosis-specific content score (KSC), created by Erdem and Karaca in 2018, was applied to each AI response to assess YouTube videos about kyphosis.²⁵ A score was awarded based on whether 32 terms or themes were mentioned in the AI responses (Table 1). The two authors of this study independently completed the KSC rating, and websites with differing scores were reassessed until agreement was reached.

Table 1.

Kyphosis specific score content.

Disease summary
Spinal deformity
Sagittal plane
Flexible/rigid kyphosis
Cervical/thoracic/lumbar kyphosis
Lumbar hyperlordosis
X-ray
MRI
Prone extension test
Cobb angle
Plumb line/sagittal balance
Wedged vertebrae
Severity/degree of curvature
Gender of the patient
Back pain
Classification
Postural kyphosis
Scheuermann's disease
Congenital kyphosis
Posttraumatic kyphosis
Iatrogenic kyphosis
Neuromuscular kyphosis
Ankylosing spondylitis
Treatment
Observation
Physical therapy
Bracing
Spinal fusion
Osteotomy
Complications
Progression of curve
Failure of instrumentation
Pseudoarthrosis
Increased lumbar lordosis
Psychosocial implication
Neurologic deficit
(MRI indicates magnetic resonance imaging)

Although originally developed for YouTube videos, the KSC checklist was adapted for chatbot-generated text by focusing on concept coverage rather than audiovisual cues. Each KSC item was operationalized based on textual content (e.g. mention of symptoms, etiology, and management). Inter-rater reliability for KSC was excellent (ICC = 0.925, 95% confidence interval (CI): 0.565–0.989). Group-specific KSC scores have now been reported in Table 2.

Table 2.

Group-wise kyphosis-specific content (KSC) scores among AI chatbots.

Chatbot	Min–Max	Mean ± SD (Median)
ChatGPT	23–26	24.2 ± 1.1 (24)
Gemini	23–25	23.7 ± 0.9 (24)
Copilot	23–26	24.3 ± 1.0 (24)
Perplexity	24–26	25.1 ± 0.8 (25)
DeepSeek	23–25	23.9 ± 0.9 (24)
Grok	23–26	24.4 ± 1.0 (24)
Overall	23–26	24.33 ± 1.03 (24)

Kruskal–Wallis p = 0.017 (< 0.05); pairwise Dunn tests showed Perplexity > Gemini (p = 0.004) and DeepSeek (p = 0.011). Although the overall variation was small (range = 23–26), Perplexity provided the most comprehensive kyphosis-related information, with the highest KSC mean.

Statistics

IBM SPSS Statistics 22 (IBM SPSS, Turkey) was used for statistical analyses of the findings. The Shapiro–Wilk test evaluated the conformity of the parameters to normal distribution. In addition to descriptive statistical methods (mean, standard deviation, and frequency), the Kruskal–Wallis test was used for comparisons of parameters that did not show a normal distribution between groups, with Dunn's test employed to identify the group causing the difference. The Mann–Whitney U test was used for comparisons between two groups of parameters that did not show normal distribution. Significance was evaluated at the p < 0.05 level. The DISCERN total score was designated as the primary endpoint. Family-wise error across secondary outcomes (GQS, JAMA, EQIP, and readability indices) was controlled using the Holm–Bonferroni correction. Effect sizes (median differences with 95% CIs) are reported alongside adjusted p-values to enhance interpretability. All pairwise comparisons have been rechecked for consistency. Minor typographical errors (e.g. duplicated “DeepSeek,” inconsistent decimal notation, FKRS spelling) were corrected to ensure coherence between text and tables.

Results

To mitigate the confounding effect of response length on readability indices, word counts per chatbot response were calculated and normalized. Sensitivity analyses confirmed that between-chatbot differences persisted even after adjusting for verbosity.

The overall study parameters are presented in Table 3. There is no statistically significant difference between AI chatbots regarding FKGL, FKRS, GFOG, SMOG, and ARI scores (p > 0.05; Table 4).

Table 3.

Evaluation of the study parameters.

	(Min–Max)-(Ort ± SS (medyan))
DISCERN1	(58–72)-(65 ± 4.43 (65))
DISCERN2	(57–67)-(61.33 ± 3.27 (61))
DISCERN	(57.5–69.5)-(63.17 ± 3.82 (63))
GQS1	(3.8–4.2)-(4.03 ± 0.14 (4.1))
GQS2	(3.5–4)-(3.7 ± 0.21 (3.7))
GQS	(3.65–4.1)-(3.87 ± 0.17 (3.9))
JAMA1	(0.8–1.5)-(1.12 ± 0.24 (1.1))
JAMA2	(1–1.7)-(1.32 ± 0.26 (1.3))
JAMA	(0.9–1.6)-(1.22 ± 0.25 (1.2))
FKGL	(10–12.2)-(10.95 ± 0.78 (10.9))
FKRS	(26.7–38)-(33.55 ± 4.23 (34.7))
GFOG	(12.03–13.37)-(12.86 ± 0.51 (13))
SMOG	(12.35–13.83)-(13.02 ± 0.67 (12.9))
CL	(12.78–14.88)-(13.54 ± 0.77 (13.4))
ARI	(9.39–12.06)-(10.43 ± 1.12 (10.1))
LW	(6.57–11.55)-(9.35 ± 2.01 (9.4))
EQIP	(25–32)-(28.83 ± 2.23 (29))
KSC	(23–26)-(24.33 ± 1.03 (24))

ARI: Automated Readability Index; CL: Coleman–Liau; EQIP: Ensuring Quality Information for Patients; FKGL: Flesch–Kincaid grade level; FKRS: Flesch–Kincaid reading ease; GFOG: Gunning Fog; GQS: Global Quality Score; KSC: kyphosis-specific content; LW: Linsear Write; SMOG: Simple Measure of Gobbledygook.

Table 4.

Evaluation of readability and quality scores among AI chatbots.

	CHATGPT	GEMINI	COPILOT	PERPLEXITY	DEEPSEEK	GROK	p
	(Min–Max)-(Mean ± SD (median))	(Min–Max)-(Mean ± SD (median))	(Min–Max)- (Mean ± SD (median))	(Min–Max)- (Mean ± SD (median))	(Min–Max)- (Mean ± SD (median))	(Min–Max)- (Mean ± SD (median))	p
DISCERN	(62–65.5)-(63.5 ± 1.35 (63.3))	(54.5–60.5)-(57.5 ± 1.94 (57.5))	(59.5–66)-(63 ± 1.99 (62.5))	(66.5–73)-(69.5 ± 2.15 (69.5))	(60.5–65)-(63 ± 1.2 (63))	(59.5–65.5)-(62.5 ± 2.08 (62.8))	0.000*
GQS	(3.5–4)-(3.8 ± 0.26 (4))	(4–4.5)-(4.1 ± 0.21 (4))	(3.5–4)-(3.75 ± 0.26 (3.8))	(3.5–4.5)-(3.9 ± 0.32 (4))	(3–4)-(3.65 ± 0.41 (3.8))	(3.5–4.5)-(4 ± 0.24 (4))	0.021*
JAMA	(1–1.5)-(1.1 ± 0.21 (1))	(1–2)-(1.6 ± 0.39 (1.5))	(1–1.5)-(1.05 ± 0.16 (1))	(1–2)-(1.3 ± 0.35 (1.3))	(0.5–1)-(0.9 ± 0.21 (1))	(1–1.5)-(1.35 ± 0.24 (1.5))	0.000*
FKRS	(32.8–52.2)-(39.37 ± 6.7 (38))	(22.6–48.9)-(33.5 ± 8.39 (32.3))	(20.9–53.2)-(34.78 ± 9.71 (32.8))	(22.4–48.7)-(32.96 ± 10.76 (29.4))	(16–46)-(29.47 ± 8.73 (29.2))	(27.6–45.3)-(38.99 ± 6.62 (41.6))	0.073
FKGL	(8.6–11.9)-(10.65 ± 1.17 (11))	(7.8–11.7)-(9.99 ± 1.13 (10.1))	(8.3–12.4)-(11.19 ± 1.35 (11.9))	(8.2–15.9)-(11.89 ± 2.36 (12.4))	(8.9–12.2)-(10.95 ± 1.18 (11.3))	(9.2–11.9)-(10.38 ± 1 (9.9))	0.057
GFOG	(9.441–14.63)-(12.53 ± 1.36 (12.6))	(8.662–17.69)-(12.18 ± 2.32 (11.6))	(9.88–15.48)-(13.09 ± 1.78 (13.5))	(9.554–18.77)-(13.36 ± 2.66 (12.6))	(5.12–16.05)-(12.66 ± 3.07 (13.3))	(11.41–15.33)-(12.52 ± 1.27 (12))	0.549
LW	(7.72–13.88)-(11.01 ± 2.26 (11.7))	(5.74–13.39)-(8.64 ± 2.44 (7.8))	(6.11–13.1)-(8.69 ± 2.22 (8))	(6.17–17.88)-(11.58 ± 3.81 (12.7))	(5.02–10.25)-(6.91 ± 1.74 (6.6))	(5.25–13.79)-(8.04 ± 2.23 (7.6))	0.004*
CL	(12.11–15.73)-(14.08 ± 1.36 (14.5))	(12.29–16.36)-(13.96 ± 1.51 (13.8))	(11.04–16.54)-(14.14 ± 1.57 (14.2))	(12.23–18.33)-(15.98 ± 2.11 (16.4))	(13.36–17.79)-(15.46 ± 1.5 (15.6))	(12.2–15.64)-(13.88 ± 1.49 (13.7))	0.044*
SMOG	(12.08–14.96)-(13.69 ± 1.12 (13.9))	(10.64–15.59)-(13.08 ± 1.58 (12.6))	(10.75–14.22)-(12.87 ± 1.1 (13.1))	(11.49–16.69)-(14.21 ± 1.92 (14.8))	(10.93–13.88)-(12.49 ± 1.05 (12.5))	(10.95–13.52)-(12.51 ± 0.81 (12.5))	0.120
ARI	(8.96–13.8)-(11.77 ± 1.76 (12.6))	(8.44–14.53)-(10.75 ± 1.93 (10.7))	(7.49–12.37)-(10.79 ± 1.5 (10.7))	(8.18–17.33)-(13.38 ± 3 (13.9))	(8.89–12.5)-(11.04 ± 1.29 (11.2))	(8.78–11.98)-(10.44 ± 1.21 (10.6))	0.059
EQIP	(27–31)-(29 ± 1.15 (29))	(24–27)-(25 ± 0.94 (25))	(27–31)-(29 ± 1.33 (29))	(30–33)-(32 ± 1.05 (32))	(27–30)-(29 ± 1.05 (29))	(28–30)-(29 ± 0.82 (29))	0.000*

Kruskal–Wallis Test *p < 0.05.

There is a statistically significant difference between AI chatbots in terms of LW scores (p = 0.004; p < 0.05). Pairwise comparisons show that LW scores are higher in Perplexity (p = 0.021) and ChatGPT (p = 0.006) than in DeepSeek (p < 0.05). There is no statistically significant difference among the other AI tools regarding LW scores (p > 0.05; Table 4).

There is a statistically significant difference between AI chatbots concerning CL index scores (p = 0.044; p < 0.05). The CL score of the Perplexity group is higher than that of Grok (p = 0.014), Gemini (p = 0.020), ChatGPT (p = 0.029), and Copilot (p = 0.049; p < 0.05). The CL score of the DeepSeek group is also higher (p < 0.05) than that of Grok (p = 0.040). There is no statistically significant difference among the other AI tools regarding CL scores (p > 0.05; Table 4).

The level of agreement between two observers for the DISCERN score was 92.5% (ICC=0.925, 95% CI: 0.565–0.989, p: 0.001), the level of agreement between two observers for the GQS score was 76. 6% (ICC=0.766, 95% CI: 0.027–0.964, p: 0.022), the level of agreement between two observers for the JAMA score was 96.9% (ICC=0.969, 95% CI: 0.795–0.996, p < 0.001) (Table 5).

Table 5.

Interobserver agreement levels of JAMA, DISCERN, and GQS scores.

		%95 CI		p
	ICC	Lower	Upper	p
DISCERN score	0.925	0.565	0.989	0.001*
GQS score	0.766	0.027	0.964	0.022*
JAMA score	0.969	0.795	0.996	0.000*

CI: confidence interval; GQS: Global Quality Score; ICC: intraclass correlation coefficient; *p < 0.05.

There is a statistically significant difference between AI chatbots when evaluating DISCERN scores (p = 0.000; p < 0.05). Perplexity has the highest DISCERN score, while Gemini has the lowest (Table 4).

There is a statistically significant difference between AI chatbots in terms of GQS scores (p = 0.021; p < 0.05). Pairwise comparisons reveal that the GQS scores of the DeepSeek group are lower than those of Grok (p = 0.027) and Gemini (p = 0.003; AI tools (p < 0.05)). There is no statistically significant difference among the other AI tools regarding GQS scores (p > 0.05; Table 4).

There is a statistically significant difference between AI chatbots concerning JAMA scores (p = 0.000; p < 0.05). The JAMA scores of the Grok group are higher than those of DeepSeek (p = 0.018), Gemini (p = 0.011), Copilot (p = 0.011), and ChatGPT (p = 0.043; p < 0.05). There is no statistically significant difference among the other AI tools regarding JAMA scores (p > 0.05; Table 4).

There is a statistically significant difference between AI chatbots in terms of EQIP scores (p = 0.000; p < 0.05). The EQIP score of the Perplexity group is significantly higher (p < 0.05) than that of DeepSeek (p = 0.034), Copilot (p = 0.037), Grok (p = 0.025), ChatGPT (p = 0.027), and Gemini (p = 0.000). The EQIP score of the Gemini group is significantly lower (p < 0.05) than that of Copilot (p = 0.014), DeepSeek (p = 0.016), Grok (p = 0.025), ChatGPT (p = 0.020), and Grok again (p = 0.022). There is no statistically significant difference among the other AI tools regarding EQIP scores (p > 0.05; Table 4).

The average KSC score of all AI chatbots was found to be 24 (Table 3).

Discussion

In this study, we aimed to evaluate the quality, reliability, and readability of the answers provided by ChatGPT, Gemini, Copilot, Perplexity, DeepSeek, and Grok to patient questions about “kyphosis.” We found the average readability of the AI tools to be at a high school level, which is difficult for many to understand.

Patients who are more knowledgeable about their health are increasingly active in making medical decisions due to the growing use and content of AI chatbots. Online health information sources are frequently utilized by patients and their families, despite the lack of expert and rigorous reviews of AI-generated content.

Health literacy is the capacity to access, understand, and use health information to make informed decisions.²⁶ A patient's health status is reflected in their level of health literacy, and low health literacy is linked to poorer patient outcomes, including missed appointments, lower compliance, a higher risk of emergency treatment, increased complications, and longer hospital stays.²⁷ Patient education materials must be crafted at a level that patients can understand to be effective.²⁸ Thus, recommendations from the AMA, the NIH, and the US Department of Health and Human Services suggest that all articles and patient education materials should be written at or below a sixth-grade reading level.^29–31

Patient education materials, such as AI chatbots, should emphasize providing readable and understandable information for kyphosis patients, as various medical information providers involved in managing these patients are known to provide data of adequate quality and readability.²⁵,³²

However, this information can sometimes be confusing, incorrect, or misleading. Moreover, many patients struggle to differentiate between objective information intended to promote goods and services and more reliable sources. Patients’ access to AI chatbots may negatively affect the patient–physician relationship due to resistance to medical advice and collaborative decision making. Several studies have shown that the health-related information delivered by AI chatbots is often not very accurate or of high quality.²⁹^,³³^,³⁴ Because low-quality information can harm the patient–doctor relationship, it is crucial to evaluate these sources and assist patients in finding complete, readable, and high-quality information.³⁵ Our study's results indicate that, compared to conventional evaluation methods, the information provided by easily accessible AI chatbots regarding kyphosis is of poor quality. These findings align with earlier research.³³,³⁴

Managing patients who read incorrect or poor information is one of the biggest challenges facing the medical field today. Numerous studies have demonstrated that the information provided by AI chatbots for various medical issues is of poor quality.³⁶,³⁷ This can adversely affect the doctor–patient relationship. Physicians must be aware of what information patients have access to and what kind of research they are undertaking on a specific topic. The lack of controls and regulations is one of the primary issues with AI-generated information. It is challenging for patients to assess which information is trustworthy. Furthermore, many patients desire more details. A physician can improve patient outcomes and satisfaction and enhance patient knowledge by guiding patients to the appropriate resources.

In this study, when evaluating the FKGL, FKRS, GFOG, SMOG, CL, ARI, and LW scores to evaluate the readability of AI chatbots’ answers about kyphosis, the readability level in some scoring for the Perplexity group was higher; however, the average value is about five levels higher than the sixth-grade level recommended by the AMA and the NIH.³⁸ This score indicates that a patient would need a high school level of English to understand the answers provided by an average AI chatbot on kyphosis.

In a study investigating online information about kyphosis, 60 websites were examined, and the readability and quality of these websites were evaluated.³² When compared to these online sources, we observed that AI chatbots had higher DISCERN and GQS scores but lower JAMA scores. In addition, FKGL and FKRS scores, which evaluate readability, were similar. We noted that the KSC score, which assessed keywords related to kyphosis, was significantly higher for AI chatbots. When we compared the current study to another source of information on the internet, specifically YouTube videos about kyphosis, we observed that both the quality and KSC scores were much better in AI chatbots.²⁵ We believe this is due to AI bots continuously updating themselves and accessing more comprehensive information.

The generalizability of our findings should be interpreted cautiously. Although the study assessed AI-generated content in English, the readability and quality of information may vary across different languages and health topics. Furthermore, variations in user prompts and updates to AI models over time could influence the consistency of future outputs. Therefore, while the present results provide valuable insights into the current state of AI-driven patient education, further studies encompassing diverse conditions, linguistic settings, and patient populations are warranted to enhance the external validity of the findings.

Given the limited scope (English-only, single-topic, and single-timepoint design), our conclusions should be interpreted with caution. The findings reflect a snapshot of chatbot performance in May 2025 and may evolve with system updates. Nevertheless, the study highlights important design implications—such as integrating readability control mechanisms, automated lay summaries, and citation toggles—to enhance patient education quality.

This study has important limitations. The responses analyzed were specific to May 2025. As a result, the answers could change if the questions were asked again on different days. Our study only captured the AI chatbots’ responses on the day the questions were asked, ignoring any replies that may have changed in the following days. The rapid development of AI presents a significant limitation; future iterations may improve transparency and clinical application, making the current findings time-sensitive. Only English was used for the questions and answers. However, non-native English speakers from various cultural, political, and geographic backgrounds might ask different questions. There was no content-based classification for the questions and answers. The decision to limit the analysis to 10 questions may also constitute a significant limitation, as it only addresses a small portion of the potential concerns or issues patients may have with kyphosis. A more thorough evaluation of AI chatbots’ potential could be conducted in the future by employing a larger set of questions that cover various clinical settings. The absence of an analysis based on patient feedback is another limitation. Ignoring the fact that patient wants and perceptions may differ from professional evaluations could lessen the article's conclusions about its practical usefulness for those seeking medical information. However, we believe that a preliminary investigation conducted by professionals in this regard is crucial.

Conclusions

Online medical information is increasingly provided through AI chatbots like ChatGPT, Gemini, Copilot, Perplexity, DeepSeek, and Grok, which can also serve as interactive tools. This presents an opportunity to raise awareness of kyphosis and improve patient satisfaction and health outcomes. Regarding kyphosis, we believe these chatbots’ responses are still insufficient in terms of quality, readability, and reliability. The answers provided by AI bots are above the recommended reading level of sixth grade for articles. While we acknowledge that current AI bots provide access to higher-quality information than other open-access sources on the internet, we believe that efforts should be made to improve the readability of the content for patients. If the content presented by AI chatbots is reviewed by a team of experts and written in an acceptable and easy-to-read manner, it will positively impact public health in the future.

Footnotes

Abbreviations

ORCID iDs

Anil Agar

Sefa Key

Ethical approval

This study did not involve human participants, patient data, biological specimens, or any form of clinical intervention. The analyses were based solely on publicly accessible, non-identifiable information generated by artificial intelligence platforms and internet-based sources. Accordingly, and in compliance with institutional, national, and international research regulations, formal ethical committee approval and informed consent were not required for this study.

Contributorship

AA was involved in conceptualization, methodology, data collection, statistical analysis, writing—original draft, and supervision; and SK in validation, data interpretation, writing—review & editing, visualization, and literature review. Both authors have read and approved the final version of the manuscript and agree to be accountable for all aspects of the work.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Declaration of conflicting interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Institutional review board statement

The study does not involve human subject research; therefore, it did not require referral to an institutional ethical review board.

Guarantor

Anıl Agar serves as the guarantor and accepts full responsibility for the integrity, accuracy, and transparency of the work.

Informed consent

Not applicable.

References

Giorgino

Alessandri-Bonetti

Del Re

, et al.

Google Bard and ChatGPT in orthopedics: Which is the better doctor in sports medicine and pediatric orthopedics? The role of AI in patient education.

Diagnostics (Basel) 2024; 14: 1253.

Gül

Erdemir

Hanci

, et al. How artificial intelligence can provide information about subdural hematoma: Assessment of readability, reliability, and quality of ChatGPT, Bard, and Gemini. Medicine (Baltimore) 2024; 103:e00071.

Sun

Zhang

Hou

, et al. Who can help me? Understanding the antecedent and consequence of medical information seeking behavior in the era of bigdata. Front Public Health 2023; 11: 1192405.

Vedula

Ghazi

Collins

, et al. Artificial intelligence methods and artificial intelligence-enabled metrics for surgical education: A multidisciplinary consensus. J Am Coll Surg 2022; 234: p1181–1192.

Topol

. High-performance medicine: the convergence of human and artificial intelligence. Nat Med 2019; 25: 44–56. doi:https://doi.org/10.1038/s41591-018-0300-7

Sanders

, et al. Chatbot for health care and oncology applications using artificial intelligence and machine learning: Systematic review. JMIR Cancer 2021; 7: e27850. doi:https://doi.org/10.2196/27850.

Bonetti

Giorgino

Gallo Afflitto

, et al.

How does ChatGPT perform on the Italian residency admission national exam compared to 15,869 medical graduates?

Ann Biomed Eng 2024; 52: 745–749.

Moons

Van Bulck

. Using ChatGPT and Google Bard to improve the readability of written patient information: A proof of concept. Eur J Cardiovasc Nurs 2023; 23: 122–126.

Ahmed

Mohamed

Zeeshan

, et al. Artificial intelligence with multi-functional machine learning platform development for better healthcare and precision medicine. Database (Oxford) 2020; 2020:baaa010.

10.

Quazi

. Artificial intelligence and machine learning in precision and genomic medicine. Med Oncol 2022; 39:120.

11.

Hentschel

Kobs

Hotho

. CLIP knows image aesthetics. Front Artif Intell 2022; 5: 976235. doi:https://doi.org/10.3389/frai.2022.976235

12.

Daeschler

Bourget

M-H

Derakhshan

, et al. Rapid, automated nerve histomorphometry through open-source artificial intelligence. Sci Rep 2022; 12:10066. doi:https://doi.org/10.1038/s41598-022-10066-6.

13.

Bluman

Foley

Chiodo

, et al. Readability of the patient education section of the AOFAS website. Foot Ankle Int 2009; 30: 287–291.

14.

Arça

Erdemir

Kara

, et al. Assessing the readability, reliability, and quality of artificial intelligence chatbot responses to the most searched cardiopulmonary queries. Medicine (Baltimore) 2024; 103:e00042.

15.

Weiss

. Health literacy. Am Medical Assoc 2003; 253: 358. (accessed 17 May 2025).

16.

Badarudeen

Sabharwal

. Readability of patient education materials from the American Academy of Orthopaedic Surgeons and Pediatric Orthopaedic Society of North America web sites. J Bone Joint Surg Am 2008; 90: 199–204.

17.

Vives

Young

Sabharwal

. Readability of spine-related patient education materials from subspecialty organization and practitioner websites. Spine (Phila Pa 1976)2009; 34: 2826–2831.

18.

Sabharwal

Badarudeen

Unes Kunju

. Readability of online patient education materials from the AAOS Web site. Clin Orthop Relat Res 2008; 466: 1465–1470.

19.

Polishchuk

Hashem

Sabharwal

. Readability of online patient education materials on adult reconstruction Web Sites. J Arthroplasty 2012; 27: 716–720.

20.

Charnock

Shepperd

Needham

, et al. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health 1999; 53: 105–111.

21.

Bernard

Langille

Hughes

, et al. A systematic review of patient inflammatory bowel disease information resources on the world wide web. Am J Gastroenterol 2007; 102: 2070–2077.

22.

Silberg

Lundberg

Musacchio

. Assessing, controlling, and assuring the quality of medical information on the internet: Caveant Lector et Viewor - Let the Reader and Viewer Beware. JAMA 1997; 277: 1244–1245.

23.

Ladhar

Koshman

Yang

, et al. Evaluation of online written medication educational resources for people living with heart failure. CJC Open 2022; 4: 455–462.

24.

Moult

Franck

Brady

. Ensuring quality information for patients: Development and validation of a new instrument. Health Expect 2004; 7: 165–175.

25.

Erdem

Karaca

. Evaluating the accuracy and quality of the information in kyphosis videos shared on YouTube. Spine (Phila Pa 1976) 2018; 43: E1201–E1206.

26.

Nutbeam

. Health promotion glossary 2021. Health Promot Int 2021; 36: 1578–1598.

27.

Berkman

Sheridan

Donahue

, et al. Low health literacy and health outcomes: An updated systematic review. Ann Intern Med 2011; 155: 97–107.

28.

Hanish

Cherian

Baumann

, et al. Reducing complex words and sentence length improves readability of sports medicine materials. Arthrosc Sports Med Rehabil 2023; 5: e45–e52.

29.

Hancı

Ergün

Gül

, et al. Assessment of readability, reliability, and quality of AI chatbot responses on palliative care. Medicine (Baltimore) 2024; 103: e00061.

30.

Hanci

Otlu

Bıyıkoğlu

. Readability of online patient education materials of intensive care societies. Crit Care Med 2024; 52: e126–e134.

31.

Weiss

. Help patients understand. Manual for Clinicians. AMA Foundation. 2007 May (accessed 14 May 2025).

32.

Agar

Sahin

. Kyphosis-related information on the internet is the quality, content and readability sufficient for the patients? Global Spine J 2022; 12: 476–482.

33.

Zhou

Pan

Zhang

, et al. Evaluating AI-generated patient education materials for spinal surgeries: comparative analysis of readability and DISCERN quality across ChatGPT and deepseek models. Int J Med Inform 2025; 198: 105871.

34.

Ozduran

Hancı

Erkin

, et al. Assessing readability and quality of AI responses regarding low back pain. PeerJ 2025; 13:e18847.

35.

Garcia

Taylor

, et al. Online resources for shoulder instability: what are patients reading?. J Bone Joint Surg Am 2014; 96(20): e177. doi:https://doi.org/10.2106/JBJS.M.01228

36.

Collins

Giammanco

Guirgus

, et al. Evaluating generative AI chatbot responses in Achilles tendon rupture management. Cureus 2025; 17: e78313.

37.

Leopold

Fahy

Perka

, et al. Assessment of quality and readability of information provided by ChatGPT in relation to developmental dysplasia of the hip and periacetabular osteotomy. J Hip Preserv Surg 2025; 12: 242–247. doi.org/10.1093/jhps/hnaf025

38.

Foundation BW-M for CliniciansA, 2007 undefined. Help patients understand. mercycareaz.org, https://www.mercycareaz.org/content/dam/mercycare/pdf/ahec_health_clinicians_manual.pdf (accessed 14 May 2025).

Readability,reliability,and quality of kyphosis-related information provided by artificial intelligence chatbots: A cross-sectional study

Abstract

Background

Methods

Results

Conclusions

Keywords

Introduction

Materials and methods

Readability assessment

Reliability and quality of answers

Statistics

Results

Discussion

Conclusions

Footnotes

Abbreviations

ORCID iDs

Ethical approval

Contributorship

Funding

Declaration of conflicting interests

Institutional review board statement

Guarantor

Informed consent

References