Assessing the Usability of ChatGPT Responses Compared to Other Online Information in Hand Surgery

Abstract

Background:

ChatGPT is a natural language processing tool with potential to increase accessibility of health information. This study aimed to: (1) assess usability of online medical information for hand surgery topics; and (2) evaluate the influence of medical consensus.

Methods:

Three phrases were posed 20 times each to Google, ChatGPT-3.5, and ChatGPT–4.0: “What is the cause of carpal tunnel syndrome?” (high consensus), “What is the cause of tennis elbow?” (moderate consensus), and “Platelet-rich plasma for thumb arthritis?” (low consensus). Readability was assessed by grade level while reliability and accuracy were scored based on predetermined rubrics. Scores were compared via Mann–Whitney U tests with alpha set to .05.

Results:

Google responses had superior readability for moderate-high consensus topics (P < .0001) with an average eighth-grade reading level compared to college sophomore level for ChatGPT. Low consensus topics had poor readability throughout. ChatGPT-4 responses had similar reliability but significantly inferior readability to ChatGPT-3.5 for low medical consensus topics (P < .01). There was no significant difference in accuracy between sources. ChatGPT-4 and Google had differing coverage of cause of disease (P < .05) and procedure details/efficacy/alternatives (P < .05) with similar coverage of anatomy and pathophysiology.

Conclusions:

Compared to Google, ChatGPT does not provide readable responses when providing reliable medical information. While patients can modulate ChatGPT readability with prompt engineering, this requires insight into their health literacy and is an additional barrier to accessing medical information. Medical consensus influences usability of online medical information for both Google and ChatGPT. Providers should remain aware of ChatGPT limitations in distributing medical information.

Keywords

large language model artificial intelligence hand surgery ChatGPT medical information

Get full access to this article

View all access options for this article.

References

Fox

. The Social Life of Health Information. Pew Research Center; 2011. Accessed March 19, 2025. https://www.pewresearch.org/2011/05/12/the-social-life-of-health-information-2011/

Increase the Proportion of Adults Who Use IT to Track Health Care Data or Communicate with Providers — HC/HIT‑07. Office of Disease Prevention and Health Promotion. U.S. Department of Health and Human Services; n.d. Accessed March 19, 2025. https://odphp.health.gov/healthypeople/objectives-and-data/browse-objectives/health-it/increase-proportion-adults-who-use-it-track-health-care-data-or-communicate-providers-hchit-07

Menendez

Mudgal

Jupiter

, et al. Health literacy in hand surgery patients: a cross-sectional survey. J Hand Surg Am. 2015;40(4):798-804. doi:10.1016/j.jhsa.2015.01.010

Tarabochia

Menendez

Ring

Health literacy and decisional preferences in hand surgery. J Hand Surg Am. 2021;46(1):70.e1-70.e8. doi:10.1016/j.jhsa.2020.08.008

Stiggelbout

Weijden

TVD

Wit

MPTD

, et al. Shared decision making: really putting patients at the centre of healthcare. BMJ. 2012;344:e256-e256. doi:10.1136/bmj.e256

Vranceanu

Cooper

Ring

Integrating patient values into evidence-based practice: effective communication for shared decision-making. Hand Clin. 2009;25(1):83-96. doi:10.1016/j.hcl.2008.09.003

Dardas

Stockburger

Boone

, et al. Preferences for shared decision making in older adult patients with orthopedic hand conditions. J Hand Surg Am. 2016;41(10):978-987. doi:10.1016/j.jhsa.2016.07.096

Vallurupalli

Shah

Vyas

. Optimizing Readability of Patient-Facing Hand Surgery Education Materials Using Chat Generative Pretrained Transformer 3.5. J Hand Surg. 2024;49(10):986-991. doi:10.1016/j.jhsa.2024.05.007

Weiss

Health Literacy: A Manual for Clinicians. American Medical Association Foundation and American Medical Association; 2003.

10.

How to Write Easy-to-Read Health Materials . National Institute of Health; 2016

11.

Pohl

Derector

Rivlin

, et al. A quality and readability comparison of artificial intelligence and popular health website education materials for common hand surgery procedures. Hand Surg Rehabil. 2024;43(3):101723. doi:10.1016/j.hansur.2024.101723

12.

Browne

Gull

Hurley

, et al. ChatGPT-4 can help hand surgeons communicate better with patients. J Hand Surg Glob Online. 2024;6(3):441-443. doi:10.1016/j.jhsg.2024.03.008

13.

Kayastha

Lakshmanan

Valentine

, et al. A readability study of carpal tunnel release in 2023. HAND. Published online February 27, 2024. doi:10.1177/15589447241232095

14.

Management of Carpal Tunnel Syndrome . American Academy of Orthopaedic Surgeons; 2024. Accessed March 19, 2025. https://www.aaos.org/quality/quality-programs/upper-extremity-programs/carpal-tunnel-syndrome/

15.

Marigi

Dancy

Alexander

, et al. Lateral epicondylitis: critical analysis review of current nonoperative treatments. JBJS Rev. 2023;11(2):e22.00170. doi:10.2106/JBJS.RVW.22.00170

16.

Lapner

Alfonso

Hebert-Davies

, et al. Nonoperative treatment of lateral epicondylitis: a systematic review and meta-analysis. JSES Int. 2022;6(2):321-330. doi:10.1016/j.jseint.2021.11.010

17.

Sims

SEG

Miller

Elfar

, et al. Non-surgical treatment of lateral epicondylitis: a aystematic review of randomized controlled trials. HAND. 2014;9(4):419-446. doi:10.1007/s11552-014-9642-x

18.

Swärd

Wilcke

Effects of intra-articular Platelet-Rich Plasma (PRP) injections on osteoarthritis in the thumb basal joint and scaphoidtrapeziotrapezoidal joint. PLoS ONE. 2022;17(3):e0264203. doi:10.1371/journal.pone.0264203

19.

Malahias

Roumeliotis

Nikolaou

, et al. Platelet-rich plasma versus corticosteroid intra-articular injections for the treatment of trapeziometacarpal arthritis: a prospective randomized controlled clinical trial. Cartilage. 2021;12(1):51-61. doi:10.1177/1947603518805230

20.

Concentration of Autograft PRP for Regeneration Purposes Delivered as an Intra-Articular Injection . American Academy of Orthopaedic Surgeons; 2020. Accessed March 19, 2025. https://www.aaos.org/quality/biologics/biologics-dashboard/

21.

Hasiba-Pappas

Kamolz

Luze

, et al. Regenerative therapies for basal thumb arthritis—a systematic review. Int J Mol Sci. 2023;24(19):14909. doi:10.3390/ijms241914909

22.

Kung

Cheatham

Medenilla

, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digit Health. 2023;2(2):e0000198. doi:10.1371/journal.pdig.0000198

23.

Pan

Musheyev

Bockelman

, et al. Assessment of Artificial Intelligence Chatbot Responses to top searched queries about cancer. JAMA Oncol. 2023;9(10):1437. doi:10.1001/jamaoncol.2023.2947

24.

OpenAI Achiam

Adler

, et al. GPT-4 technical report. Published online March 2023. doi:10.48550/ARXIV.2303.08774

25.

Flesch

A new readability yardstick. J Appl Psychol. 1948;32(3):221-233. doi:10.1037/h0057532

26.

Gunning

. The Technique of Clear Writing. Rev. ed. Mcgraw-Hill; 1968.

27.

Smith

Senter

. Automated readability index. AMRL-TR Aerosp Med Res Lab US. Published online May 1967:1-14.

28.

Falagas

Korbila

Giannopoulou

, et al. Informed consent: how much and what do patients understand? Am J Surg. 2009;198(3):420-435. doi:10.1016/j.amjsurg.2009.02.010

29.

Grady

Enduring and emerging challenges of informed consent. N Engl J Med. 2015;372(9):855-862. doi:10.1056/NEJMra1411250

30.

Cook

Sasor

Tholpady

, et al. Hand surgery resources exceed American health literacy. Hand. 2018;13(5):547-551. doi:10.1177/1558944717725384

31.

Pierson

Shanmugam

Movva

, et al. Using large language models to promote health equity. NEJM AI. 2025;2(2):AIp2400889. doi:10.1056/AIp2400889

32.

PAI Staff. Managing the risks of AI research: six recommendations for responsible publication. Accessed March 19, 2025. https://partnershiponai.org/paper/responsible-publication-recommendations/.

33.

Wang

Capo

Orillaza

Readability and comprehensibility of patient education material in hand-related web sites. J Hand Surg Am. 2009;34(7):1308-1315. doi:10.1016/j.jhsa.2009.04.008

34.

Introducing GPT-4o and More Tools to ChatGPT Free Users . OpenAI; 2024. Accessed March 19, 2025. https://openai.com/index/gpt-4o-and-more-tools-to-chatgpt-free/

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.26 MB

0.00 MB