Sage Journals: Discover world-class research

Abstract

Background:

Atrial fibrillation (AF) is the most prevalent arrhythmia and a significant cause of morbidity. Artificial intelligence (AI)-based language models represent a novel tool for searching for medical information; however, there is still uncertainty regarding their reliability and readability in different languages.

Objective:

To assess the reliability and readability of information provided by AI-based models for patients with AF.

Methods:

A cross-sectional study was conducted to assess the reliability and readability of the responses generated by ChatGPT, YouChat, Gemini and Perplexity on AF in English and Spanish. Thirty standardised questions were posed in both languages. The quality of the responses was then assessed by 2 independent reviewers via a standardised tool. Readability was assessed via the Flesch–Szigrist formula. The results were then compared by tool and language.

Results:

ChatGPT demonstrated the highest interrater agreement (PA = 0.73 in Spanish, 0.80 in English), followed by Gemini in English (PA = 0.66). In Spanish, ChatGPT generated the highest percentage of complete responses (80%), followed by Perplexity (73%) and Gemini (47%). In English, Perplexity demonstrated the strongest performance, with a score of 93%, followed by ChatGPT, with 73%, and Gemini, with 53%. A readability analysis revealed significant differences between the models (P < .01). The ChatGPT demonstrated the highest performance, although its content was moderately challenging in Spanish and highly challenging in English.

Conclusion:

ChatGPT and Perplexity emerged as the most reliable models, although readability remains a concern. There is a clear need for improvements to optimise the accuracy and accessibility of AI-generated medical information.

Keywords

atrial fibrillation artificial intelligence large language models readability

Introduction

Atrial fibrillation (AF), the most prevalent type of sustained arrhythmia, has been determined to be a substantial risk factor for adverse events, including mortality, cerebrovascular disease, and heart failure.¹ In 2020, approximately 50 million individuals worldwide were estimated to be affected by this condition.^2,3 This increase is influenced by multiple factors, including an ageing population, increased case detection, and its association with various cardiovascular diseases.^1,4 Given the impact of AF, providing patients and their caregivers with clear, accurate, and accessible information that allows them to understand the disease and actively participate in its management is essential.

Furthermore, ChatGPT, developed by OpenAI in 2022, was designed with reinforcement learning with human feedback to deliver more useful, confident and conversational responses, marking a milestone in natural language processing.^5,6 Other artificial intelligence (AI) platforms, such as YouChat, Perplexity and Gemini, were subsequently introduced. These AI tools are capable of generating text, translating languages, producing creative content and providing answers to questions in an informative manner.^7
-9

Moreover, the utilisation of AI in the domain of medical information seeking has markedly increased, with language models such as ChatGPT, YouChat, Gemini and Perplexity^7
-9 attaining a notable degree of popularity. Nevertheless, the reliability and readability of the medical information they provide remains a matter of debate, especially in chronic diseases such as AF.

Reliability is defined as the veracity and consistency of information on the basis of scientific evidence, whereas readability is measured as the level of understanding of the general public.^10
-12 Studies have highlighted concerns regarding the quality of online medical information, which can influence clinical decisions and the doctor–patient relationship. Besides, this has been shown to result in deficiencies in communication and the critical evaluation of sources.^11,12 The aim of the present study is to evaluate the reliability and readability of the AF information generated by 4 AI models (ChatGPT, YouChat, Gemini and Perplexity) in both English and Spanish.

Methods

Design, Setting and Data Collection

A cross-sectional, analytical observational study was conducted to assess the reliability and readability of the information provided by ChatGPT, YouChat, Gemini and Perplexity in English and Spanish. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist for cross-sectional studies was used to guide reporting. Informed consent was not required given the nature of the study.^13,14 The study was considered safe and was approved by the ethics committee of the Pontificia Universidad Javeriana (CIE-319-2024).

Sample Size and Question Formulation

We computed the sample size required to detect discordant proportions of 0.1 and 0.4 (delta = 0.3) via a 2-sample paired-proportions test, with 80% power, a 2-sided approach, and a 5% significance level. The resulting calculation indicated a minimum of 30 questions. In line with this, our study employed 30 questions (15 in English and 15 in Spanish), which were distributed evenly across the 4 AI models.

The patients’ most frequently asked questions about AF were subsequently identified from queries made at various search engines. These were compiled from information sites and documents primarily aimed at patients, including Fundación Española del Corazón,¹⁵ Mayo Clinic,¹⁶ the Division of Cardiothoracic Surgery (WashU Medicine),¹⁷ the AFIB Clinic,¹⁸ UChicago Medicine,¹⁹ the American Heart Association factsheet,²⁰ and the guidelines of AF for patients of the Spanish Society of Cardiology²¹ (Appendix 1).

For the selection process, the ‘frequently asked questions’ (FAQ) sections and doubt-oriented chapters in patient guides and leaflets were reviewed.^15
-21 These sources were chosen for their accessibility to the general public, relevance, and wide dissemination in the field of health education. On the basis of these criteria, the adjustment, modification and formulation of 15 FAQs were guided, directly reflecting the most common concerns regarding definition, symptoms, causes, diagnosis, treatment options, lifestyle, complications, and prognosis.

These questions were formulated in simple language, and in the first person where possible, as this is the way patients tend to express themselves. The inclusion of the question about heart transplantation was considered relevant owing to the clinical importance of the topic and the frequency with which it arises in forums and consultations with patients with advanced heart failure and AF.

Questions that were ambiguous, contained unverifiable information or lacked clear and objective answers were excluded. Questions were considered ambiguous if their wording allowed for multiple interpretations. Questions were defined as unverifiable if there was no single or standardised answer supported by scientific evidence, or if they depended exclusively on individual clinical assessment, making it difficult to verify them against recognised sources. Finally, 15 standardised questions were selected by the researchers in each language, which were classified into 6 categories: (1) general information about the disease; (2) diagnosis; (3) nonpharmacological treatment; (4) pharmacological treatment; (5) prognosis; and (6) complications.

Response Generation

The formulation of the questions was conducted via the free research versions of ChatGPT, Perplexity, YouChat and Gemini. For the purposes of the study, a user was created in ‘incognito mode’. The questions were posed individually, with the options ‘New Chat, New Thread or New Conversation’ available. Importantly, only the initial response generated by the chatbot was taken into consideration during the evaluation process, without the need for further response generation.

Response Evaluation

These responses were then analysed independently by 2 specialists in internal medicine and cardiology (EJ and JB). To ascertain the reliability of the data, a scale that had previously been used in research to evaluate information generated by AI tools for patients with various diseases was employed.^22,23 This scale classifies the answers into 4 categories: (1) Complete: equivalent to the answer of an expert in the subject; (2) Correct but insufficient: accurate but with incomplete information; (3) Partially correct: with correct and incorrect elements; and (4) Completely incorrect.²⁴

Furthermore, the readability of the responses was analysed via the Flesch formula adapted to Spanish and the Flesch–Szigrist formula,^25,26 which is designed to assess the readability of health information for patients. In addition, to assess the quality of the responses generated to questions about AF, the simplified DISCERN scale was used together with the Global Quality Scale (GQS).²⁷

Statistical Analysis

The distribution of the data was examined via the Shapiro–Wilk test. Given that the assumption of normality was not satisfied, quantitative variables are presented as medians and their corresponding interquartile ranges (IQRs). Absolute and relative frequencies were used to describe categorical variables. Fisher’s exact test was employed to compare them, given that the expected frequencies were less than 5. To ensure consistency in the evaluation of the questions among the researchers, 3 questions were randomly selected and evaluated independently by 2 researchers. The group engaged in a thorough discussion of the discrepancies until a consensus was reached.

The reliability of responses was measured via the proportion of agreements (PA),²⁸ an extension of the simple agreement ratio, with the aim of correcting for potential biases introduced by chance or by the structure of the data (such as ties). The levels of agreement were categorised in accordance with the methodology established by Landis and Koch.²⁸

The Mann–Whitney U test was used to compare the medians obtained in the readability assessment between pairs. The Kruskal–Wallis test was used to compare the medians of more than 3 groups. A P-value of P < .05 was used as the criterion for statistical significance. The statistical analysis was conducted using R software, version 4.3.3.

Results

A total of 30 responses (15 in Spanish and 15 in English) were analysed for each AI model.

Interrater Agreement

Table 1 presents the interrater agreement in the assessment of the reliability of the AF information for each model, which was discriminated by language. The agreement was substantial for both ChatGPT (PA = 0.73 in Spanish and 0.80 in English) and Gemini (PA = 0.66 in English). The other models presented lower values.

Table 1.

Interrater Concordance for the Assessment of the Quality of Patient Information on Atrial Fibrillation in English and Spanish.

Language of evaluation	AI model
Language of evaluation	ChatGPT¹	Gemini¹	Perplexity¹	YouChat¹
Spanish	0.73	0.60	0.46	0.40
English	0.80	0.66	0.40	0.33

Proportion of agreements (PA).

Reliability of Information

In the reliability assessment in Spanish, significant differences were observed between the models evaluated (P = .01). As demonstrated in Table 2, ChatGPT obtained the highest percentage of complete responses, with 12 responses (80%), followed by Perplexity, with 11 complete responses (73%), Gemini, with 7 responses (47%), and finally YouChat, with 5 complete responses (33%). The YouChat model was the only model that recorded responses classified as ‘some correct and some incorrect’, with 4 responses (27%). It was evident that none of the models produced completely incorrect responses (Figure 1).

Table 2.

Assessment of the Reliability of Patient Information on Atrial Fibrillation in Spanish and English Languages.

Language of evaluation	Reliability	AI model
Language of evaluation	Qualification	ChatGPT¹	Gemini¹	Perplexity¹	YouChat¹	P value²
Spanish	Complete	12 (80%)	7 (47%)	11 (73%)	5 (33%)	0.01
	Correct but insufficient	3 (20%)	8 (53%)	4 (27%)	6 (40%)
	Some correct and some incorrect	-	-	-	4 (27%)
	Completely incorrect	-	-	-	-
English	Complete	11 (73%)	8 (53%)	14 (93%)	3 (20%)	<0.001
	Correct but insufficient	4 (27%)	7 (47%)	1 (6.70%)	10 (67%)
	Some correct and some incorrect	-	-	-	2 (13%)
	Completely incorrect	-	-	-	-

n (%).

Fisher exact test.

Figure 1.

Assessing the reliability of Spanish-language information for patients on atrial fibrillation.

In the English language, the disparities between models were even more pronounced (P < .001). Perplexity was identified as the model with the highest level of performance, achieving 14 complete responses, equivalent to 93% of the total. In second place was ChatGPT, with 11 responses, representing 73% of the total. In third place was Gemini, with 8 responses, equivalent to 53% of the total. Finally, YouChat obtained only 3 complete responses, equivalent to 20% of the total. YouChat was the only model that produced responses classified as ‘some correct and some incorrect’, with 2 responses (13%; Figure 2).

Figure 2.

Assessing the reliability of patient information on atrial fibrillation in the English language.

Table 3 provides an overview of the reliability of the AF information generated in English by ChatGPT and Perplexity, categorised by completeness level.

Table 3.

Reliability Assessment of Patient Information on Atrial Fibrillation in English Language.

Language of evaluation	Reliability	AI model
Language of evaluation	Qualification	ChatGPT¹	Perplexity¹	P value²
English	Complete	11 (73%)	14 (93%)	0.3
	Correct but insufficient	4 (27%)	1 (6.70%)
	Some correct and some incorrect	-	-
	Completely incorrect	-	-

n (%).

Fisher exact test.

Although Perplexity was associated with a greater percentage of complete responses (93% vs 73% for ChatGPT), this difference was not statistically significant (P = .3). Both tools avoided completely incorrect or seriously flawed responses, which is positive in terms of information security. However, ChatGPT had a greater proportion of responses considered ‘correct but insufficient’ (27% vs 6.7% for Perplexity), suggesting that it may provide more conservative or less detailed content. Therefore, although both models offer an acceptable level of reliability, Perplexity could be more effective in providing complete information, which could be important for patient education.

The following study compares the reliability of the responses that were categorised as ‘comprehensive’ across the most accomplished models. As demonstrated in Table 4, ChatGPT and Perplexity in English for AF. Despite the fact that Perplexity demonstrated a greater percentage of complete responses in all the categories evaluated (93% in total vs 73% in ChatGPT), the differences were not statistically significant in any of the categories, according to Fisher’s exact test.

Table 4.

Reliability: Comparison of ChatGPT and Perplexity-Generated Responses Rated as ‘Comprehensive’ in Atrial Fibrillation in English Language.

Answers categorised as ‘Comprehensive’	AI model
Category	ChatGPT (n = 15)	Perplexity (n = 15)	P value¹
General information, n = 5 (%)	4 (80%)	5 (100%)	0.33
Diagnosis, n = 2 (%)	2 (100%)	2 (100%)	0.1
Non-pharmacological treatment, n = 2 (%)	1 (50%)	2 (100%)	0.33
Pharmacological treatment, n = 3 (%)	2 (67%)	3 (100%)	0.7
Complications, n = 3 (%)	2 (67%)	2 (67%)	0.53
Total	11 (73%)	14 (93%)

Fisher exact test.

Notably, both tools provided complete responses for 100% of the diagnosis-related questions. The most significant disparities were observed in nonpharmacological treatment (50% in ChatGPT vs 100% in perplexity). The findings of this study indicate that while Perplexity demonstrates a propensity to furnish more comprehensive responses, particularly in less structured domains such as nonpharmacological treatment, both models exhibit acceptable reliability. Nevertheless, the absence of statistical significance may be attributable to the limited sample size within each category, which compromises the analytical power. This underscores the necessity for subsequent studies to employ larger sample sizes, thereby substantiating the reliability of the observed trends.

Readability

In the Spanish language readability assessment, the AI models demonstrated significant disparities in their performance (P = .009). ChatGPT obtained the highest median score of 55.8 (range: 47.5-65.1; moderately difficult rating), followed by Gemini with 53.4 (43.2PA = 0.73-61.5). The differences between ChatGPT and Gemini were not significant (P = .54), whereas the differences between ChatGPT and YouChat (P = .03) and between Gemini and Perplexity (P = .02) were statistically significant (see Table 5 and Figure 3).

Table 5.

Readability Assessment of Patient Information on Atrial Fibrillation in English and Spanish Languages.

Language of evaluation	AI model
Language of evaluation	ChatGPT¹	Gemini¹	Perplexity¹	YouChat¹	P value²
Spanish	55.8 (47.5, 65.1)	53.4 (43.2, 61.5)	47.9 (38.6, 57.1)	50.0 (35.6, 61.3)	0.009
English	42.9 (28.4, 49.3)	31 (12, 41)	32 (18, 51)	27 (18, 49)	0.002

n (%).

Fisher exact test.

Figure 3.

Readability assessment of the information for patients on atrial fibrillation in Spanish language.

Furthermore, a significant discrepancy was identified between the models evaluated in the English language (P = .002). Once more, ChatGPT attained the highest median score, with a value of 42.9 (28.4-49.3; hard rating). YouChat clearly obtained the lowest score, with a total of 27 points (out of a possible 49). The differences between Gemini and Perplexity were not significant (P = .35), but notable differences were identified between ChatGPT and Gemini (P = .0032), as well as between ChatGPT and YouChat (P = .0023; Figure 4).

Figure 4.

Readability assessment of patient information on atrial fibrillation in English language.

Figure 5 shows the comparison of the readability scores obtained by the ChatGPT and Perplexity models in English. Although the ChatGPT model obtained a visibly higher median score than the perplexity model did, this difference did not reach statistical significance (P = .13). This finding suggests a tendency for ChatGPT-generated responses to be more readable. However, it cannot be stated with statistical certainty that there is a real difference between the 2 models.

Figure 5.

Readability comparison between ChatGPT and Perplexity models for patients on atrial fibrillation in English language.

Of particular significance is the observation of outliers in Perplexity, which are indicative of heightened variability in the quality of readability of responses. This, in turn, has the potential to adversely impact the user experience. These findings could have implications for the practical use of these tools, especially in clinical settings where the clarity of language is crucial for patient comprehension. However, it is imperative to acknowledge the need for a more substantial sample size to substantiate these trends with greater certainty.

Quality

In the Spanish language, according to the DISCERN scale,²⁷ a moderate degree of answer quality was obtained from evaluations of both reviewers (EJ and JB; median = 3, IQR 3-3). In addition, in English, both reviewers give a moderate degree of quality for the answers (median = 3, IQR 3-3), according to the DISCERN scale.²⁷

Discussion

This study provides an inaugural comparative evaluation of the information generated by 4 AI-based language models for AF patients in both Spanish and English. The study’s objective is to compare the accuracy of responses generated in English and Spanish for medical information.

Our findings demonstrate significant variations in the reliability of information provided by disparate AI models on AF in both English and Spanish. ChatGPT and Perplexity demonstrated superior performance in the provision of complete and accurate information, whereas Gemini and YouChat exhibited greater variability in the quality of their responses.

In the Spanish language, ChatGPT emerged as the model that provided the most complete responses (80%), followed by Perplexity (73%). In contrast to Gemini and YouChat, which exhibited significant disparities and inaccuracies, they presented a higher frequency of responses classified as ‘correct but insufficient’. Notably, 27% of the YouChat platform responses contained partially incorrect information, indicating reduced reliability in providing information in Spanish.

Conversely, in the English language, Perplexity demonstrated exceptional performance, with a 93% complete response rate, surpassing those of ChatGPT (73%) and Gemini (53%). Moreover, YouChat resulted in a lower percentage of complete responses (20%) and a higher percentage of correct but insufficient responses (67%). These findings underscore discrepancies in the performance of AI models, particularly in the context of health-related information.

The analysis revealed significant discrepancies between the models in both languages. In the English language, the difference was statistically significant (P < .001), suggesting that the discrepancies in the quality of information provided by the different models are consistent and not due to chance.

The utilisation of Perplexity and ChatGPT within the models clearly engenders a greater degree of reliability, particularly within the context of the English language. Conversely, it is important to acknowledge the limitations inherent in YouChat and Gemini, which should be duly considered by patients requiring AF information. In addition, the presentation of erroneous answers by models necessitates the supervision of medical personnel to prevent the dissemination of misinformation, which could influence patients’ clinical decisions.

The results of the evaluation of the readability of AF information generated by the different AI models show statistically significant differences in both languages, with p values of .009 in Spanish and .002 in English. In the Spanish context, ChatGPT emerged as the most readable option, with a median score of 55.8, closely followed by Gemini (53.4), YouChat (50.0), and Perplexity (47.9). Conversely, in the English language, considerably lower values were observed, with ChatGPT (42.9), Gemini (31), Perplexity (32) and YouChat (27) demonstrating explanations and answers that were more challenging for nonmedical staff to comprehend.

Considering the aforementioned points, it is posited that the comprehension of AF information generated in the Spanish language is superior to that produced in English. The lower readability of English-language texts has been demonstrated to impede the comprehension of medical information, thus emphasising the necessity of adjusting such texts to the reading level of patients to facilitate health education and informed decision-making.

Finally, considering our findings, several recommendations can be made. First, for those seeking reliable information on AF in Spanish across these AI models, ChatGPT 3.5 appears to provide the most reliable responses. Moreover, if information that is easier to read and understand is desired, ChatGPT also seems to offer the most accessible content. However, if reliable information is desired in English, perplexity may be the most reliable AI tool; however, for those seeking easier-to-read answers, ChatGPT could be the most useful AI tool.

Limitations

The present study is subject to certain limitations. One of the issues that has been identified is that the interpretation of the responses generated by the AI models was carried out from the perspective of health personnel, which may differ from patients’ perceptions. The levels of health literacy vary between individuals. Consequently, patients may have divergent interpretations of the information. To address this limitation, responses were selected on the basis of a broad population and discussed as a team to ensure that they reflected a comprehensive view of information quality.

In addition, factors such as age, educational attainment and technological literacy may influence patients’ comprehension and utilisation of these tools. However, given that the assessment concentrated on overall reliability and readability, the results offer a comprehensive and representative perspective of the population under study.

A further limitation is that ChatGPT does not provide references to information sources unless explicitly requested, which places it at a disadvantage in comparison to other models that do cite their sources. In addressing this limitation, the analysis acknowledged it, thus indicating the necessity for subsequent iterations of these models to provide references in a more accessible manner.

Moreover, it should be noted that the AI models utilised in this study represent evolving technologies; consequently, the responses generated by these models may vary depending on the specific version of the model, the context of the query, and the manner in which the questions are articulated. Nevertheless, to minimise these variations, the responses of each model should be evaluated in its most recent version available at the time of the study. Furthermore, the evaluation was limited to the initial response generated by each model, thus restricting the analysis by excluding the potential for regenerated responses that could have corrected or complemented the initial information. This limitation should be considered, and additional responses should be evaluated in future studies to obtain a more complete picture of the consistency and accuracy of the responses over time.

A salient aspect of this study pertains to the interrater agreement observed in select AI models, as evidenced by the values presented in Table 1. While ChatGPT demonstrated higher levels of consistency in the evaluation of information quality in both Spanish (0.73) and English (0.80), alternative models exhibited lower degrees of agreement. This variability suggests the presence of inconsistencies in responses between raters, which may be attributable to differences in natural language processing algorithms. These findings underscore the necessity of developing more standardised assessment methods, which will yield more robust results in future research.

Conclusions

The present study revealed variations in the reliability and readability of AI-generated AF information. ChatGPT and Perplexity demonstrated the highest levels of accuracy, whereas YouChat exhibited lower accuracy and the presence of erroneous responses. In terms of readability, ChatGPT demonstrated the highest score, although its content was found to be moderately challenging in Spanish and highly challenging in English. These findings underscore the necessity of enhancing the precision and accessibility of these instruments for utilisation in educational and clinical contexts.

Footnotes

Appendix 1 Acknowledgements

To the Pontifical Xavierian University for promoting research.

ORCID iDs

Emilio Jose Juan Juan-Guardela

Jesús Andrés Beltrán-España

Cristian Orlando Porras-Bueno

Oscar Muñoz-Velandia

Ángel Alberto García-Peña

Ethical Considerations

This study was approved by the institutional ethics committee (CIE-319-2024).

Author Contributions

Emilio Jose Juan Juan-Guardela: conceptualisation, data curation, formal analysis, investigation, methodology, project administration, supervision, validation, writing (original draft), writing review, and editing.

Jesús Andres Beltrán-España: conceptualisation, data curation, formal analysis, investigation, writing (original draft), review, and editing.

María Paula Ravagli-Baquero: conceptualisation, data curation, formal analysis, investigation, writing (original draft), review, and editing.

Cristian Orlando Porras-Bueno: conceptualisation, data curation, formal analysis, investigation, writing (original draft), review, and editing.

Edward Cáceres-Méndez: conceptualisation, data curation, formal analysis, investigation, writing (original draft), review, and editing.

Daniel Fernandez Ávila: conceptualisation, data curation, formal analysis, investigation, writing (original draft), review, and editing.

Oscar Muñoz-Velandia: conceptualisation, data curation, formal analysis, investigation, methodology, project administration, supervision, validation, writing (original draft), writing review, and editing.

Ángel Alberto García-Peña: conceptualisation, data curation, formal analysis, investigation, methodology, project administration, supervision, validation, writing (original draft), writing review, and editing.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This article has been funded by the Pontifical Xavierian University.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request and in compliance with the General Data Protection Regulation.

References

Joglar

Chung

Armbruster

. 2023 ACC/AHA/ACCP/HRS guideline for the diagnosis and management of atrial fibrillation: a report of the American College of Cardiology/American Heart Association Joint Committee on Clinical Practice Guidelines [published correction appears in Circulation. 2024 Jan 2;149(1):e167. doi: 10.1161/CIR.0000000000001207.] [published correction appears in Circulation. 2024 Feb 27;149(9):e936. doi: 10.1161/CIR.0000000000001218.] [published correction appears in Circulation. 2024 10.1161/CIR.0000000000001263.]. Circulation. 2024;149(1):e1-e156.

Tsao

Aday

Almarzooq

. Heart disease and stroke statistics—2023 update: a report from the American Heart Association [published correction appears in 440 441 442 443 447 451 Circulation. 2023 Feb 21;147(8):e622. doi: 10.1161/CIR.0000000000001137.] [published correction appears in Circulation. 10.1161/CIR.0000000000001167.]. Circulation. doi:10.1161/CIR.0000000000001123. Circulation. 2023;147(8):e93-e621.

Chugh

Havmoeller

Narayanan

, et al. Worldwide epidemiology of atrial fibrillation: a Global Burden of disease 2010 study. Circulation. 2014;129(8):837-847.

García-Peña

ÁA

Ospina-Buitrago

Rico-Mendoza

JDP

Fernández-ávila

Muñoz-Velandia

ÓM

Suárez

. Prevalencia de fibrilación auricular en Colombia según información del sistema integral de Información de la Protección Social (SISPRO). Rev Colomb Cardiol. 2022;29(2):170-176.

Salah

Alhalbusi

Ismail

Abdelfattah

Chatting with ChatGPT: decoding the mind of chatbot users and unveiling the intricate connections between user perception, trust and stereotype perception on self-esteem and psychological well-being. Curr Psychol. 2024;43:7843-7858. Published online 2023. doi:10.1007/s12144-023-04989-0

OpenAI. Introducing ChatGPT® [Internet]. 2022. [cited 2025 January 28]. Accessed January 28, 2025. https://openai.com/blog/chatgpt

OpenAI. About ChatGPT® [Internet]. 2022 [cited 2025 January 28]. Accessed January 28, 2025. https://openai.com

YC, Features of YouChat® [Internet]. 2022 [cited 2025 January 28]. Accessed January 28, 2025. https://you.com

Google DeepMind. Introducing Gemini®: DeepMind’s latest language model [Internet]. 2023 [cited 2025 Jan 28]. https://deepmind.com

10.

Rooney

Santiago

Perni

, et al. Readability of patient education materials from high-impact medical journals: a 20-year analysis. J Patient Exp. 2021;8:7. Published 2021 Mar 3.

11.

Boroumand

Sedghi

Adibi

Panahi

Rahimi

Patients’ perspectives on the quality of online patient education materials: a qualitative study. J Educ Health Promot. 2022;11:402. Published 2022 Dec 28.

12.

De la Torre Barbero

MJ.

Evaluación de las páginas web de enfermería y hospitales del Sistema Público de Andalucía. Thesis. Universidad de Córdoba; 2012. Accessed January 28, 2025. https://helvia.uco.es/bitstream/handle/10396/10659/774.pdf.

13.

von Elm

Altman

Egger

, et al. STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med. 2007;16;147(8):573‑577. doi: 10.7326/0003‑4819‑147‑8‑200710160‑00010

14.

Vandenbroucke

von Elm

Altman

Gøtzsche

Mulrow

Pocock

, et al. Mejorar la comunicación de estudios observacionales en epidemiología (STROBE): explicación y elaboración. Gac Sanit. 2009;23(2):158.e1-158.e28.

15.

Fundación Española

del Corazón

. Frequently asked questions about atrial fibrillation [Internet]. n.d. [cited 2025 August 22]. https://fundaciondelcorazon.com/informacion-para-pacientes/enfermedades-cardiovasculares/fibrilacion-auricular/preguntas-frecuentes.html

16.

Clinic

Fibrilación auricular –síntomas y causas [Internet]. Mayo Clinic; (fecha de última actualización: 8 March 2024). Accessed August 21, 2025. https://www.mayoclinic.org/es/diseases-conditions/atrial-fibrillation/symptoms-causes/syc-20350624/

17.

Division of Cardiothoracic Surgery, Washington University School of Medicine. Atrial fibrillation Q&A [Internet]. Washington University School of Medicine; n.d. [cited 2025 August 21]. https://cardiothoracicsurgery.wustl.edu/patient-care/cardiac/atrial-fibrillation-center/atrial-fibrillation-qa/

18.

The AFIB Clinic. FAQs about AFib [Internet]. n.d. [cited 2025 August 21]. https://www.theafibclinic.com/patient-resources/faqs-about-afib/

19.

Upadhyay

GA MD

. Answers to Common Questions About AFib [Internet]. University of Chicago Medicine; 5 Feb 2020. Accessed August 21, 2025. https://www.uchicagomedicine.org/forefront/heart-and-vascular-articles/common-questions-and-concerns-about-afib

20.

American Heart Association. What Is Atrial Fibrillation? American Heart Association; 2025. https://www.heart.org/en/health-topics/atrial-fibrillation/what-is-atrial-fibrillation-afib-or-af

21.

Mont

Guasch

Pérez-Villacastín

Fibrilación auricular: guía para pacientes. Publicaciones Permanyer; 2009.

22.

Yeo

Samaan

, et al. Assessing the performance of ChatGPT in answering questions regarding cirrhosis and hepatocellular carcinoma. Clin Mol Hepatol. 2023;29(3):721-732.

23.

Aydinbelge-Dizdar

Dizdar

Evaluación de la fiabilidad y legibilidad de las respuestas de los chatbots como recurso de información al paciente para las exploraciones PET-TC más comunes. Rev Esp Med Nucl Imagen Mol. 2025;44:500065. Published online September 28, 2024. doi:10.1016/j.remn.2024.500065

24.

Samaan

Yeo

Rajeev

, et al. Assessing the accuracy of responses by the Language Model ChatGPT to questions regarding bariatric surgery. Obes Surg. 2023;33(6):1790-1796.

25.

Rios Hernandez

. Un acercamiento a la legibilidad de textos relacionados con el campo de la salud. Chasqui Rev Latinoam Comun. 2017;135:253. doi:10.16921/chasqui.v0i135.2892

26.

Lorza

DPQ

Morales

Méndez

NDD

. Herramienta para análisis de la legibilidad lingüística de contenido web en español como apoyo a procesos de evaluación de accesibilidad. Brazilian Creat Ind J. 2021;1:71-87.

27.

Charnock

Shepperd

Needham

Gann

DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health. 1999;53(2):105-111.

28.

Landis

Koch

GG.

The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-174.

Reliability and Readability Assessment of Atrial Fibrillation Patient Information Delivered by Artificial Intelligence-Based Language Models (ChatGPT,YouChat,Gemini,and Perplexity AI) in English and Spanish

Abstract

Background:

Objective:

Methods:

Results:

Conclusion:

Keywords

Introduction

Methods

Design, Setting and Data Collection

Sample Size and Question Formulation

Response Generation

Response Evaluation

Statistical Analysis

Results

Interrater Agreement

Reliability of Information

Readability

Quality

Discussion

Limitations

Conclusions

Footnotes

Appendix 1

Acknowledgements

ORCID iDs

Ethical Considerations

Author Contributions

Funding

Declaration of Conflicting Interests

Data Availability Statement

References