Artificial intelligence (AI)-based chatbots are increasingly used as sources of medical information. Given the high prevalence of neck pain as a musculoskeletal symptom, patients may commonly consult such tools for health-related guidance.
Objective
To evaluate and compare the performance of ChatGPT 4.0 and Google Gemini in addressing commonly asked patient questions and clinical case scenarios related to neck pain, focusing on their accuracy, quality, understandability, readability, reliability, and usability.
Methods
Twenty-four patient-oriented questions and four clinical case scenarios regarding neck pain were submitted to ChatGPT 4.0 and Google Gemini. Responses were evaluated using validated tools: modified DISCERN (mDISCERN) for reliability, Global Quality Scale (GQS) for quality, PEMAT-P for understandability and actionability, and Flesch Reading Ease (FRE) and Flesch-Kincaid Grade Level (FKGL) for readability. Case-based responses were assessed for accuracy, safety, and usability on a 7-point Likert scale by two experienced physicians.
Results
Gemini demonstrated significantly higher reliability (mDISCERN, p < 0.001), whereas ChatGPT 4.0 had slightly higher, though statistically insignificant, GQS and PEMAT-P scores. Readability metrics were similar: ChatGPT's FRE was 48.78 and FKGL 9.08; Gemini's FRE was 47.12 and FKGL 9.11. Both models’ outputs were considered difficult to read. In clinical scenarios, both chatbots showed comparable accuracy, safety, and usability, with minor omissions noted.
Conclusion
ChatGPT 4.0 and Google Gemini provided similar performance in addressing neck pain-related queries. While both may support patient.
PopescuALeeH. Neck pain and lower back pain. Med Clin North Am2020; 104: 279–292.
2.
KazeminasabSNejadghaderiSAAmiriP, et al.Neck pain: global epidemiology, trends and risk factors. BMC Musculoskelet Disord2022; 23: 26.
3.
GerardTNayeFDecaryS, et al.Prognostic factors of pain, disability, and poor outcomes in persons with neck pain - an umbrella review. Clin Rehabil2024; 38: 1658–1676.
4.
ChisariEBCubisinoGD'ArmaF, et al.Temporomandibular disorders and cervical spine: a systematic review. Ann Stomatol (Roma)2018; 9: 95–105.
5.
IshaqIMehtaPSkinnerIW, et al.Treatment classifications and interventions for neck pain: a scoping review. J Clin Epidemiol2023; 159: –9.
6.
VosTLimSSAbbafatiC, et al.Global burden of 369 diseases and injuries in 204 countries and territories, 1990–2019: a systematic analysis for the global burden of disease study 2019. Lancet2020; 396: 1204–1222.
7.
Valenza-PeñaGMartín-NúñezJHeredia-CiuróA, et al.Effectiveness of self-care education for chronic neck pain: a systematic review and meta-analysis. Healthcare (Basel)2023; 11.
8.
van der NoordRReezigtRRPaapD, et al.Unhelpful information about low back and neck pain on physiotherapist's websites. Eur J Pain2025; 29: e4782.
9.
CegeEEComertHAkalN, et al.Evaluation of the performance of artificial intelligence based chatbots in providing first aid information on dental trauma according to the ToothSOS application. Dent Traumatol2025; 41: 696–705.
10.
LiZYanCCaoY, et al.Evaluating performance of large language models for atrial fibrillation management using different prompting strategies and languages. Sci Rep2025; 15: 19028.
11.
TackC. Artificial intelligence and machine learning | applications in musculoskeletal physiotherapy. Musculoskelet Sci Pract2019; 39: 164–169.
12.
Gonzalez ArocaJVergara-MerinoLEscobar LiquitayCM, et al.Role of artificial intelligence-powered conversational agents (chatbots) in musculoskeletal disorders: a scoping review protocol. BMJ Open2025; 15: e092982.
13.
DastaniMMardanehJRostamianM. Large language models’ capabilities in responding to tuberculosis medical questions: testing ChatGPT, Gemini, and Copilot. Sci Rep2025; 15: 18004.
14.
Yilmaz MulukSOlcucuN. The role of artificial intelligence in the primary prevention of common musculoskeletal diseases. Cureus2024; 16: e65372.
15.
MilnerJDQuinnMSSchmittP, et al.Performance of artificial intelligence in addressing questions regarding the management of pediatric supracondylar humerus fractures. J Pediatr Soc North Am2025; 11: 100164.
16.
AlhurA. Redefining healthcare with artificial intelligence (AI): the contributions of ChatGPT, Gemini, and Co-pilot. Cureus2024; 16: e57795.
17.
GrzybowskiAPawlikowska-ŁagódKLambertWC. A history of artificial intelligence. Clin Dermatol2024; 42: 221–229.
18.
DedeBTOguzMAlyanakB, et al.Competencies of large language models about Piriformis syndrome: quality, accuracy, completeness, and readability study. HSS J2025: 15563316251340697.
19.
HassanMGAbdelazizAAAbdelrahmanHH, et al.Performance of AI-Chatbots to common temporomandibular joint disorders (TMDs) patient queries: accuracy, completeness, reliability and readability. Orthod Craniofac Res2025; 28: 127–134.
20.
GirodMMSanieiSUlrichMN, et al.Artificial intelligence in the diagnosis and prognostication of the musculoskeletal patient. HSS J2025: 15563316251339660.
21.
GursesOAOzudogruATuncayF, et al.The role of artificial intelligence large language models in personalized rehabilitation programs for knee osteoarthritis: an observational study. J Med Syst2025; 49: 73.
22.
DaherMKoaJBoufadelP, et al.Breaking barriers: can ChatGPT compete with a shoulder and elbow specialist in diagnosis and management?JSES Int2023; 7: 2534–2541.
23.
ZhangLTashiroSMukainoM, et al.Use of artificial intelligence large language models as a clinical tool in rehabilitation medicine: a comparative test case. J Rehabil Med2023; 55: jrm13373.
24.
MendelTSinghNMannDM, et al.Laypeople's use of and attitudes toward large language models and search engines for health queries: survey study. J Med Internet Res2025; 27: e64290.
25.
YunHSBickmoreT. Online health information-seeking in the era of large language models: cross-sectional web-based survey study. J Med Internet Res2025; 27: e68560.
26.
GezerMCArmangilM. Assessing the quality of ChatGPT's responses to commonly asked questions about trigger finger treatment. Ulus Travma Acil Cerrahi Derg2025; 31: 389–393.
27.
CorpNMansellGStynesS, et al.Evidence-based treatment recommendations for neck and low back pain across Europe: a systematic review of guidelines. Eur J Pain2021; 25: 275–295.
OnderCEKocGGokbulutP, et al.Evaluation of the reliability and readability of ChatGPT-4 responses regarding hypothyroidism during pregnancy. Sci Rep2024; 14: 243.
30.
BernardALangilleMHughesS, et al.A systematic review of patient inflammatory bowel disease information resources on the world wide web. Am J Gastroenterol2007; 102: 2070–2077.
31.
ShoemakerSJWolfMSBrachC. Development of the patient education materials assessment tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns2014; 96: 395–403.
32.
KincaidPFishburneRPRogersRL, et al.Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel1975.
33.
FleschR. A new readability yardstick. J Appl Psychol1948; 32: 221–233.
34.
GuvenYOzdemirOTKavanMY. Performance of artificial intelligence chatbots in responding to patient queries related to traumatic dental injuries: a comparative study. Dent Traumatol2025; 41: 338–347.
35.
KorogluEYErsoyRSacikaraM, et al.Evaluation of the impact of ChatGPT support on acromegaly management and patient education. Endocrine2025; 87: 1141–1149.
36.
OzduranEHanciVErkinY, et al.Assessing the readability, quality and reliability of responses produced by ChatGPT, gemini, and perplexity regarding most frequently asked keywords about low back pain. PeerJ2025; 13: e18847.
37.
AlyanakBDedeBTBagcierF, et al.Parental education in pediatric dysphagia: A comparative analysis of three large language models. J Pediatr Gastroenterol Nutr2025; 81: 18–26.
38.
CollinsCEGiammancoPAGuirgusM, et al.Evaluating the quality and readability of generative artificial intelligence (AI) Chatbot responses in the management of achilles tendon rupture. Cureus2025; 17: e78313.
39.
KaraMOzduranEKaraMM, et al.Evaluating the readability, quality, and reliability of responses generated by ChatGPT, Gemini, and Perplexity on the most commonly asked questions about ankylosing spondylitis. PLoS One2025; 20: e0326351.
40.
OzduranEAkkocIBuyukcobanS, et al.Readability, reliability and quality of responses generated by ChatGPT, Gemini, and Perplexity for the most frequently asked questions about pain. Medicine (Baltimore)2025; 104: e41780.
41.
ZureMMenekşeoğluAK. Assessment of the artificial intelligence- generated fibromyalgia information: beyond the hype. Arch Rheumatol2025; 40: 358–364.
42.
KolumanACÇiftçiMUÇiftçiEA, et al.Balancing accuracy and readability: comparative evaluation of AI Chatbots for patient education on rotator cuff tears. Healthcare (Basel)2025; 13.
43.
KolacUCKarademirOMAyikG, et al.Can popular AI large language models provide reliable answers to frequently asked questions about rotator cuff tears?JSES Int2025; 9: 390–397.
44.
DongCQiuXDengJ, et al.Comparative evaluation of large language models in delivering guideline-compliant recommendations for topical NSAID use in musculoskeletal pain: a multidimensional analysis. Clin Rheumatol2025; 44: 4703–4710.
45.
ScaffSPSReisFJJFerreiraGE, et al.Assessing the performance of AI chatbots in answering patients’ common questions about low back pain. Ann Rheum Dis2025; 84: 143–149.
46.
AndreaoFFMoura NascimentoMDe FariaAM, et al.Assessing diagnostic precision and therapeutic guidance using artificial intelligence in functional neurosurgery cases. Cureus2025; 17: e83592.
47.
StevensonEWalshCHibberdL. Can artificial intelligence replace biochemists? A study comparing interpretation of thyroid function test results by ChatGPT and google bard to practising biochemists. Ann Clin Biochem2024; 61: 143–149.
48.
Gomez-CabelloCABornaSPressmanSM, et al.Large language models for intraoperative decision support in plastic surgery: a comparison between ChatGPT-4 and Gemini. Medicina (Kaunas)2024; 60.
