Sage Journals: Discover world-class research

Abstract

Background:

The development and potential of artificial intelligence (AI) is remarkable. Its application in all medical disciplines, including rheumatology, is attracting attention. To what extent AI is already used in clinical routine in rheumatology is unknown. In addition, the perceived barriers, potentials, and expectations regarding AI by rheumatologists have not yet been studied.

Objectives:

To examine the current usage and perceived barriers and facilitators of AI, including large language models (LLMs), among rheumatologists.

Design:

National, observational, non-interventional, and cross-sectional web-based study.

Methods:

A web-based survey was developed by the Working Group Young Rheumatology (AGJR) of the German Society for Rheumatology. The survey was distributed at the Congress of the German Society for Rheumatology and via social media, QR code, and email from August 30 until November 4, 2023.

Results:

Responses from 172 rheumatologists (55% female; mean age 43 years) were analyzed. The majority stated that they did not previously use AI (73%) in their daily practice. Eighty-eight percent of rheumatologists rated their AI knowledge as low to intermediate and 84% would welcome dedicated training on LLMs. The majority of rheumatologists anticipated AI implementation to improve patient care (60%) and reduce daily workload (62%). Especially for diagnosis (73%), writing medical reports (70%), and data analysis (70%), rheumatologists reported a potential positive benefit of AI. Main AI concerns addressed the responsibility for medical decisions (64%) and data security (58%).

Conclusion:

Overall, the results indicate that rheumatologists currently have little AI knowledge and make very little use of AI in clinical routine. However, the majority of rheumatologists anticipate positive AI effects and would welcome increased AI implementation and dedicated training programs.

Keywords

artificial intelligence education implementation research quality improvement rheumatology

Introduction

The speed of development and the potential of artificial intelligence (AI) in medicine is striking.¹ Beyond the applications of machine learning for diagnosis, prognostic factors as well as biomarkers,² the use of large language models (LLMs) has gained a lot of momentum in medicine. Major medical journals (e.g., New England Journal of Medicine or the Lancet) released subjournals for AI and digital health highlighting the significance of this technology. The release of ChatGPT in November 2022 has additionally accelerated the interest and usage of LLM in medicine.³ ChatGPT-4 correctly answered 94% of the rheumatology questions from the Spanish medical training access exam⁴ and recently performed diagnostically better than experienced rheumatologists, at least in terms of sensitivity.⁵ However, as with any new technology, there are concerns regarding the benefits and risks of AI. While there is hope that AI can lead to shorter disease delays, cover shortages of specialists, and aid in research development, there is also a fear of overdiagnosis, uncontrollable data, misinformation, and loss of transparency.^3,6
–9

The European Alliance of Associations for Rheumatology (EULAR) anticipated the potential of AI for rheumatology and set up a dedicated task force in 2018.¹⁰ This task force systematically screened the literature¹⁰ and published points to consider for the use of big data in rheumatic and musculoskeletal diseases.¹¹ To ensure the effective implementation¹² of AI in rheumatology care, it is essential to also systematically analyze the perspectives of key stakeholders, including rheumatologists. It is currently unclear to what extent AI is used in rheumatology clinical routine. In addition, the perceived barriers, facilitators, and expectations regarding AI by rheumatologists have not yet been evaluated.

This study aimed to close these knowledge gaps by conducting a web-based national survey to understand the current use and perceptions of AI and LLM by rheumatologists.

Methods

A web-based survey was designed by the Working Group Young Rheumatology (AGJR) of the German Society for Rheumatology (Supplemental Material 1). An expert panel identified three main areas of interest to investigate: (1) AI and LLM usage, (2) AI knowledge, and (3) AI and LLM barriers and potential.

A formal inquiry was submitted to the ethics committee of the Philipps University Marburg, Germany (23-264 ANZ). Due to the anonymous and non-interventional nature of the study, it was deemed exempt from IRB review.

After individual literature research, a first survey draft was designed by the expert panel and then adapted according to feedback from the other members of the AGJR. All 17 items of the questionnaire were mandatory once the questionnaire was started. With participation in the open, anonymous, and voluntary survey, agreement was requested and later on interpreted as informed consent. No cookies or IP addresses were used. Results were stored anonymously with time stamps and were only accessible to the principal investigators. The survey was performed with Google Forms and was distributed at the Congress of the German Society for Rheumatology and additionally accessible from August 30 until November 4, 2023, via email, social media, and QR code. All rheumatologists (specialists and trainees) were eligible. For each successfully completed questionnaire, the AGJR donated 5€ to the Deutsche Rheumastiftung.

The study population was divided into two groups according to the median of 38 years (⩽38 and >38 years) to analyze age differences. A Fisher’s exact test was calculated for subgroup analysis to investigate sex or age differences in single-answer questions. We used a Chi-square test for subgroup analysis for age and sex and multiple-answer questions as the item asking for barriers of AI. Descriptive analysis and statistics were calculated with IBM SPSS Statistics Version 28.0.1.0. Windows (SPSS Inc., Chicago, IL, USA). Graphics were created with GraphPad Prism version 9.0.2 (GraphPad, San Diego, CA, USA). Reporting was conducted according to the checklist for reporting results of internet e-surveys (CHERRIES; Supplemental Material 2).¹³

Results

Study population

In all, 172 rheumatologists completed the survey, of whom 55.2% (95/172) were female. The mean participant age was 42.5 years (23–72 years, median 38 years). Two-thirds (107/172, 62.2%) of rheumatologists were board-certified. The largest group of rheumatologists participating worked in university hospitals (71/172, 36.2%) (Table 1).

Table 1.

Characteristics of rheumatologists.

Characteristics	Rheumatologists, n = 172
Age, years (mean ± standard deviation)	42.5 ± 12.8
Women, n (%)	95 (55.2)
Training status, n (%)
Resident	65 (37.8)
Board-certified rheumatologist	107 (62.2)
Type of workplace, n (%)
Solo practice	15 (7.7)
Group practice	27 (13.8)
Medical care center	16 (8.2)
Outpatient specialist care	21 (10.7)
Primary care clinic	27 (13.8)
Non-university hospital	19 (9.7)
University hospital	71 (36.2)

AI usage

Most rheumatologists had never used AI (126/172, 73.3%) in a work-related context (Figure 1). Daily use of AI was reported by 2.3% (2/172). No significant gender- or age-adopted differences (younger vs older than the median of 38 years) in rheumatologists regarding usage frequency of AI were identified (data not shown).

Figure 1.

Reported frequency of AI use in clinical practice according to sex.

LLMs were used by 60/172 (34.9%), with significantly more men stating the usage of LLM (meaning monthly, weekly, or daily usage; 34 men vs 25 women, p = 0.015 by Fisher’s exact test), whereas age-adopted analysis did not show significant differences.

LLMs were used for scientific writing (29/60, 48.3%), patient correspondence (28/60, 46.7%), and diagnosis (27/60, 45.0%) (Figure 2a). Assistance with writing medical reports (20/60, 33.3%), documentation, and correspondence to health insurance companies (17/60, 28.3%) was also reported. Only 8/60 (13.3%) reported that they used LLMs for treatment decisions and only 4/60 (11.7%) reported the use of LLMs for treatment monitoring.

Figure 2.

(a) Reported areas of application of LLMs and (b) reported potential future areas of application of AI.

Presumed future AI application areas (Figure 2b) were reported by rheumatologists as follows: diagnosis (125/172, 72.7%), writing medical reports (121/172, 70.3%), data analysis (120/172, 69.8%), documentation (119/172, 69.2%), optimizing structural processes (111/172, 64.5%), and health insurance correspondence (110/172, 64.0%). Patient correspondence, treatment decisions, patient monitoring as well as scientific writing were reported less frequently (58.7%, 53.5%, 51.2%, and 45.3%, respectively). Younger rheumatologists (⩽38 years) stated more often structural, workload-related applications like AI usage in the documentation and optimizing structural processes (64 vs 55 and 68 vs 43), whereas rheumatologists older than 38 years presumed future AI usage more in diagnosis (66 vs 59).

Most rheumatologists (141/172, 82%) preferred a transparent AI-based diagnostic decision support compared to an AI-based diagnostic decision support with improved diagnostic accuracy but an untransparent decision algorithm.

AI knowledge

Most rheumatologists declared their AI knowledge as very low to intermediate (151/172, 87.8%) and 83.7% (144/172) welcomed dedicated training on LLMs (Figure 3a and b). No differences regarding age or gender were detected (data not shown).

Figure 3.

(a) Perceived AI knowledge, (b) acceptance of AI training, and (c–f) anticipated impact of AI on clinical care.

Potential and barriers to AI

AI’s influence on patient care was seen as positive or very positive by the majority of rheumatologists (104/172, 60.4%) (Figure 3c). Similarly, the majority assumed a positive or very positive influence of AI on the daily workload (106/172, 61.6%) (Figure 3d).

Regarding the impact on the future patient–physician relationship, 52.9% (91/172) stated that AI would have no effect, while a (major) improvement was seen by 32.6% (56/172) and a (major) deterioration by 14.5% (25/172) (Figure 3e). 84.9% (146/172) did not believe that their medical work could become redundant due to AI (Figure 3f).

Interestingly, younger rheumatologists (⩽38 years) presumed an even more positive influence of AI on patient care and the daily workload (p = 0.002 and p = 0.003, respectively, by Fisher’s exact test). Furthermore, gender subanalysis revealed that women suspected a more positive influence of AI on patient care than men (p = 0.018 by Fisher’s exact test). Otherwise, no significant differences in the age-stratified or gender-stratified analysis were found.

Reduction of repetitive tasks was rated most often at 84.9% (146/172) as a benefit of implementing AI (Figure 4a). An improvement in evidence-based medicine (106/172, 61.6%) and diagnosis as well as treatment decisions (99/172, 57.6%) were also perceived benefits of AI implementation.

Figure 4.

(a) Reported potential advantages of AI and (b) potential barriers. Legend (a) Others: improvement of administrative aspects, paring down personnel. Legend (b) Others: less personal contacts, unreliable sources for AI results, and physicians themselves becoming redundant.

Major barriers seen by rheumatologists were the responsibility for AI-based decisions (110/172, 64%) and data security issues (99/172, 57.6%) (Figure 4b). Other barriers mentioned were insufficient knowledge about the use of AI (84/172, 48.8%), fear of the influence of big technology companies (77/172, 44.8%), and the fear that AI could become the “new gold standard” (54/172, 31.4%). Low trust in AI in general was reported by a quarter of rheumatologists (43/172, 25%). No significant differences in age or gender were detected regarding presumed barriers (data not shown).

Discussion

To the best of our knowledge, this is the first study to examine rheumatologists’ current use and perceived barriers and facilitators of AI and LLM. Our findings show that most German rheumatologists would welcome the increased use of AI in clinical practice due to its anticipated benefits. Current AI use and AI knowledge were low, and rheumatologists welcomed dedicated training on LLMs as one of the most recent developments of AI, most likely reflecting a general wish for AI training.

These results are in line with findings in the other medical fields. For example, in a recent survey in China, over 90% of healthcare workers were found to be very willing to learn and study AI.¹⁴ Radiology, as an AI implementation frontrunner, offers several AI training programs and conferences; however, academic institutions are rarely involved, professional institutions as radiologic societies and commercial companies dominate the offer.^15,16 European medical imaging professionals recently called for a university-led systematic curriculum.¹⁶ To accelerate AI implementation, German university hospitals are increasingly offering structured digital clinician scientist programs. During this time, physicians have dedicated and protected research time for digital health (often AI) studies closely linked to the respective clinical work.

Ideally, AI would already be integrated into the medical curriculum to sustainably increase the AI competency of the medical profession. A recent review highlighted how important precise LLM prompt engineering is to use LLMs most effectively for academic rheumatology. Webinars and online courses could be offered by rheumatology societies to foster the safe and effective adoption of LLMs.¹⁷

Our results are consistent with a survey among imaging-focused clinicians.¹⁸ The authors found that 71% of clinicians believed AI to improve their clinical work (61% in this study). Similarly, our study identified AI diagnostic tools as a top use case for AI implementation. First pilot studies suggested that LLMs like ChatGPT-4⁵ or AI-based symptom checker apps for laypersons¹⁹ may identify rheumatic diseases even more reliably than experienced rheumatologists. A prospective randomized controlled trial however revealed limited accuracy of symptom-based diagnostic tools (symptom checkers) for rheumatic diseases as the correct diagnosis was only given among the top five suggested diagnoses in 17%–26% of the cases. The authors suggest that this might be due to the complexity of the usual rheumatologic diagnostic workup comprising imaging, laboratory, and clinical examination.²⁰ Even more elegant than patients using AI-based screening apps would be to make use of already collected patient data. For instance, van Leeuwen et al. recently demonstrated how AI was successfully used to identify rare vasculitis patients from electronic health records.²¹

Interestingly, in our study, rheumatologists preferred an inferior accuracy but transparent AI reasoning over intransparent diagnostic decisions with higher accuracy. Combined with the stated potential barriers of final responsibility and safety, this emphasizes the need to ensure trustworthiness.²² A racial and gender bias of ChatGPT-4 has recently been demonstrated.23 Current governmental guidance and regulations try to ensure such trustworthiness. The European AI Act, for example, encompasses comprehensive rules to guarantee trustworthy and fair AI.²⁴ Of note, our survey did not encounter gender-related differences in use or expectancies regarding AI in rheumatology.

Moreover, the preference for a transparent AI algorithm may indicate additional desires regarding AI development. Future research should investigate whether rheumatologists seek detailed information about the specific data models on which the AI was trained and the exact data it utilized. These questions were not addressed in our current survey.

Rheumatologists also anticipate improvement with AI regarding treatment and patient monitoring. In a landmark study, Gossec et al. demonstrated that disease flares could be reliably predicted by AI based on passively collected step-count data from patients with rheumatoid arthritis and spondyloarthritis.²⁵ A recent study found that using an AI-based flare risk tool led to more consistent treatment decisions and a reduction in treatment by rheumatologists.²⁶ Another benefit of AI, was the reduction of workload, stated by 61.6% of the rheumatologists. AI itself by interpreting images, values, and parameters may help in many clinical settings. For example, the workload of visual interpretation of antinuclear antibodies (ANA) immunofluorescence can be reduced when using an AI-driven visual analyzer.²⁷ In addition, the implementation of LLMs in electronic medical reports may reduce the bureaucracy and therefore might ameliorate the daily workload of any physician.³ Furthermore, LLMs may aid in answering frequently asked patient questions.^28,29

While 52.9% of the rheumatologists assumed a neutral influence on the patient–physician relationship, a deterioration was expected in 14.5%. Interestingly, a cross-sectional study by Ayers et al. was able to demonstrate that ChatGPT answers to patients’ questions were rated as better and more empathic than physicians’ answers.²⁹ Easy access to LLM will certainly empower patients, reducing information asymmetry, leading to more autonomy and elimination of any residue of medical paternalism. Lorenzini et al. predicted a paradigm shift toward a three-way relationship between patient, physician, and AI that can improve shared decision-making when the autonomy of both patients and physicians in this process is maintained.³⁰ Training on both sides on AI’s influence on shared decision-making as well as on patient–physician relationships including potential risks and benefits of AI is therefore warranted.³¹

A major limitation of this work includes a possible sampling bias. Due to the promotion of the survey at the annual German rheumatology congress, a high proportion of academically active rheumatologists participated. No significant difference between university-hospital and non-university hospital-employed rheumatologists was however observed (data not shown). In addition, no cookies or IP addresses were used and participants could theoretically answer the questionnaire multiple times. The survey length was limited and that is no open-ended questions were included; thus the scope of responses and perceptions of rheumatologists is limited.

Conclusion

The majority of rheumatologists expect AI including LLM to improve many aspects of their work and patient care. To overcome current low use and adequately address the dangers and barriers of AI, high-quality training programs are needed. Furthermore, the aforementioned potential barriers regarding the safety and trustworthiness of AI imply the need for ethical and legal guidelines for physicians to enforce its implementation in clinical practice.

Supplemental Material

sj-docx-1-tab-10.1177_1759720X241275818 – Supplemental material for Artificial intelligence in rheumatology: status quo and quo vadis—results of a national survey among German rheumatologists

Supplemental material, sj-docx-1-tab-10.1177_1759720X241275818 for Artificial intelligence in rheumatology: status quo and quo vadis—results of a national survey among German rheumatologists by Marie-Therese Holzer, Anna Meinecke, Felix Müller, Isabell Haase, Harriet Morf, Thorben Witte, Hannah Labinsky, Philipp Klemm, Johannes Knitza and Martin Krusche in Therapeutic Advances in Musculoskeletal Disease

Supplemental Material

sj-docx-2-tab-10.1177_1759720X241275818 – Supplemental material for Artificial intelligence in rheumatology: status quo and quo vadis—results of a national survey among German rheumatologists

Supplemental material, sj-docx-2-tab-10.1177_1759720X241275818 for Artificial intelligence in rheumatology: status quo and quo vadis—results of a national survey among German rheumatologists by Marie-Therese Holzer, Anna Meinecke, Felix Müller, Isabell Haase, Harriet Morf, Thorben Witte, Hannah Labinsky, Philipp Klemm, Johannes Knitza and Martin Krusche in Therapeutic Advances in Musculoskeletal Disease

Footnotes

Acknowledgements

We would like to thank all participants as well as all members of the Working Group Young Rheumatology.

Declarations

ORCID iD

Marie-Therese Holzer

Supplemental material

Supplemental material for this article is available online.

References

Wang

Casalino

Khullar

Deep learning in medicine-promise, progress, and challenges. JAMA Intern Med 2019; 179: 293–294.

Hügle

Omoumi

van Laar

, et al. Applied machine learning and artificial intelligence in rheumatology. Rheumatol Adv Pract 2020; 4: rkaa005.

Hügle

The wide range of opportunities for large language models such as ChatGPT in rheumatology. RMD Open 2023; 9: e003105.

Madrid-García

Rosales-Rosado

Freites-Nuñez

, et al. Harnessing ChatGPT and GPT-4 for evaluating the rheumatology questions of the Spanish access exam to specialized medical training. Sci Rep 2023; 13: 22129.

Krusche

Callhoff

Knitza

, et al. Diagnostic accuracy of a large language model in rheumatology: comparison of physician and ChatGPT-4. Rheumatol Int 2024 Feb; 44(2): 303–306.

McMaster

Bird

Liew

DFL

, et al. Artificial intelligence and deep learning for rheumatologists. Arthritis Rheumatol 2022; 74: 1893–1905.

Hügle

Kalweit

Künstliche Intelligenz-unterstützte Behandlung in der Rheumatologie Grundlagen, aktueller Stand und Ausblick. Z Rheumatol 2021; 80: 914–927.

Singhal

Azizi

, et al. Large language models encode clinical knowledge. Nature 2023; 620: 172–180.

Raman

Transparency in research: an analysis of ChatGPT usage acknowledgment by authors across disciplines and geographies. Account Res 2023; Oct 25: 1–22.

10.

Kedra

Radstake

Pandit

, et al. Current status of use of big data and artificial intelligence in RMDs: a systematic literature review informing EULAR recommendations. RMD Open 2019; 5: e001004.

11.

Gossec

Kedra

Servy

, et al. EULAR points to consider for the use of big data in rheumatic and musculoskeletal diseases. Ann Rheum Dis 2020; 79: 69–76.

12.

Bauer

Damschroder

Hagedorn

, et al. An introduction to implementation science for the non-specialist. BMC Psychol 2015; 3: 32.

13.

Eysenbach

Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res 2004; 6: e34.

14.

Xiang

Zhao

Liu

, et al. Implementation of artificial intelligence in medicine: status analysis and development suggestions. Artif Intell Med 2020; 102: 101780.

15.

Schuur

Rezazade Mehrizi

Ranschaert

Training opportunities of artificial intelligence (AI) in radiology: a systematic review. Eur Radiol 2021; 31: 6021–6029.

16.

Walsh

Stogiannos

van de Venter

, et al. Responsible AI practice and AI education are central to AI implementation: a rapid review for all medical imaging professionals in Europe. BJR Open 2023; 5: 20230033.

17.

Venerito

Lalwani

Del Vescovo

, et al. Prompt engineering: the next big skill in rheumatology research. Int J Rheum Dis 2024; 27: e15157.

18.

Scheetz

Rothschild

McGuinness

, et al. A survey of clinicians on the use of artificial intelligence in ophthalmology, dermatology, radiology and radiation oncology. Sci Rep 2021; 11: 5193.

19.

Gräf

Knitza

Leipe

, et al. Comparison of physician and artificial intelligence-based symptom checker diagnostic accuracy. Rheumatol Int 2022; 42: 2167–2176.

20.

Knitza

Mohn

Bergmann

, et al. Accuracy, patient-perceived usability, and acceptance of two symptom checkers (Ada and Rheport) in rheumatology: interim results from a randomized controlled crossover trial. Arthritis Res Ther 2021; 23: 112.

21.

van Leeuwen

Penne

Rabelink

, et al. Using an artificial intelligence tool incorporating natural language processing to identify patients with a diagnosis of ANCA-associated vasculitis in electronic health records. Comput Biol Med 2023; 168: 107757.

22.

Smuha

NA.

The EU approach to ethics guidelines for trustworthy artificial intelligence. Comput Law Rev Int 2019; 20: 97–106.

23.

Zack

Lehman

Suzgun

, et al. Assessing the potential of GPT-4 to perpetuate racial and gender biases in health care: a model evaluation study. Lancet Digit Health 2024; 6(1): e12–e22.

24.

Aktuelles. Artificial Intelligence Act: deal on comprehensive rules for trustworthy AI, https://www.europarl.europa.eu/news/de/press-room/20231206IPR15699/artificial-intelligence-act-deal-on-comprehensive-rules-for-trustworthy-ai (2023, accessed 31 December 2023).

25.

Gossec

Guyard

Leroy

, et al. Detection of flares by decrease in physical activity, collected using wearable activity trackers in rheumatoid arthritis or axial spondyloarthritis: an application of machine learning analyses in rheumatology. Arthritis Care Res (Hoboken) 2019; 71: 1336–1343.

26.

Labinsky

Ukalovic

Hartmann

, et al. An AI-powered clinical decision support system to predict flares in rheumatoid arthritis: a pilot study. Diagnostics (Basel) 2023; 13: 148.

27.

Alsuwaidi

Dollinger

Fleck

, et al. The reliability of a novel automated system for ANA immunofluorescence analysis in daily clinical practice. Int J Rheumatol 2016; 2016: 6019268.

28.

Goodman

Patrinely

Stone

, et al. Accuracy and reliability of chatbot responses to physician questions. JAMA Netw Open 2023; 6: e2336483.

29.

Ayers

Poliak

Dredze

, et al. Comparing physician and artificial intelligence chatbot responses to patient questions posted to a public social media forum. JAMA Intern Med 2023; 183: 589–596.

30.

Lorenzini

Arbelaez Ossa

Shaw

, et al. Artificial intelligence and the doctor-patient relationship expanding the paradigm of shared decision making. Bioethics 2023; 37: 424–429.

31.

Hryciw

Fortin

Ghossein

, et al. Doctor-patient interactions in the age of AI: navigating innovation and expertise. Front Med (Lausanne) 2023; 10: 1241508.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB