This study examines the comparative efficacy of three AI large language models (LLMs)—ChatGPT-4, Gemini, and Microsoft Copilot—in clinical social work.
Method
By presenting scenarios of varying complexities, the study assessed their performance using the Ateşman Readability Index and a Likert-type accuracy scale.
Results
Results showed that Gemini had the highest accuracy, while Microsoft Copilot excelled in readability. Significant differences were found in accuracy scores (p = .003), although readability differences were not statistically significant (p = .054). No correlation was found between case complexity and either accuracy or readability.
Discussion
Despite the differences, none of the models fully met all accuracy standards, indicating areas for further improvement. The findings suggest that while LLMs offer promise in social work, they require refinement to better meet the field's needs.
Al-AshwalF.ZawiahM.GharaibehL.Abu-FarhaR.BitarA. N. (2023). Evaluating the sensitivity, specificity, and accuracy of ChatGPT-3.5, ChatGPT-4, Bing AI, and Bard against conventional drug-drug interactions clinical tools. Drug, Healthcare and Patient Safety, 15, 137–147. https://doi.org/10.2147/DHPS.S425858
2.
AlhurA. (2024). Redefining healthcare with artificial intelligence (AI): The contributions of ChatGPT, Gemini, and Copilot. Cureus, 16(4), Article e57795. https://doi.org/10.7759/cureus.57795
3.
AsakuraK.OcchiutoK.ToddS.LeitheadC.ClappertonR. (2020). A call to action on artificial intelligence and social work education: Lessons learned from a simulation project using natural language processing. Journal of Teaching in Social Work, 40(5), 501–518. https://doi.org/10.1080/08841233.2020.1813234
4.
AteşmanE. (1997). Türkçede okunabilirliğin ölçülmesi. Dil Dergisi, 58, 71–74.
5.
BronfenbrennerU. (1979). The ecology of human development. Harvard University Press.
6.
BrownT. B.MannB.RyderN.SubbiahM.KaplanJ.DhariwalP.NeelakantanA.ShyamP.SastryG.AskellA.AgarwalS.Herbert-VossA.KruegerG.HenighanT.ChildR.RameshA.ZieglerD.WuJ.WinterC. … AmodeiD. (2020). Language models are few-shot learners. In LarochelleH.RanzatoM.HadsellR.BalcanM. F.LinH. (Eds.), Advances in neural information processing systems 33(NeurIPS 2020) (pp. 1877–1901). Curran Associates, Inc.
7.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed). Routledge Academic.
CrawfordJ.CowlingM.Ashton-HayS.KelderJ.MiddletonR.WilsonG. S. (2023). Artificial intelligence and authorship editor policy: ChatGPT, Bard, Bing AI, and beyond. Journal of University Teaching & Learning Practice, 20(5), 1–11. https://doi.org/10.53761/1.20.01.01
10.
DuBayW. H. (2004). The principles of readability. Impact Information.
GiannakopoulosK.KavadellaA.Aaqel SalimA.StamatopoulosV.KaklamanosE. (2023). Evaluation of the performance of generative AI large language models, ChatGPT, Google Bard, and Microsoft Bing Chat in supporting evidence-based dentistry: Comparative mixed methods study. Journal of Medical Internet Research, 25, e51580. https://doi.org/10.2196/51580
13.
GuptaP. K.RaturiS.VenkateswarluP. (2023). Chatgpt for designing course outlines: A boon or bane to modern technology. Social Science Research Network. https://doi.org/10.2139/ssrn.4386113
14.
HaiderS. (2024). Exploring opportunities and challenges of artificial intelligence in social work education. In PrzeperskiJ.BaikadyR. (Eds.), The Routledge international handbook of social work teaching (pp. 46–62). Routledge.
15.
IbrahimA. T. H.SalehE. F.Al MamariW. S.ElsherbinyM. M. K., & MustafaM. M. (2023). Understanding the role of ChatGPT in social work: What we know and what we still need to discover. Social Issues, 1(1), 5–13.
16.
IsbannerS.O’ShaughnessyP.SteelD.WilcockS.CarterS. (2022). The adoption of artificial intelligence in health care and social services in Australia: Findings from a methodologically innovative national survey of values and attitudes (the AVA-AI study). Journal of Medical Internet Research, 24(8), e37611. https://doi.org/10.2196/37611
17.
KaftanA. N.HussainM. K.NaserF. H. (2024). Response accuracy of ChatGPT 3.5, Copilot, and Gemini in interpreting biochemical laboratory data: A pilot study. Scientific Reports, 14(1), Article 8233. https://doi.org/10.1038/s41598-024-58964-1
18.
KrauseD. (2023). Large language models and generative AI in finance: An analysis of ChatGPT, Bard, and Bing AI. Social Science Research Network.
19.
KuroiwaT.SarconA.IbaraT.YamadaE.YamamotoA.TsukamotoK.FujitaK. (2023). The potential of ChatGPT as a self-diagnostic tool in common orthopedic diseases: Exploratory study. Journal of Medical Internet Research, 25, e47621. https://doi.org/10.2196/47621
20.
MakrygiannakisM. A.GiannakopoulosK.KaklamanosE. G. (2024). Evidence-based potential of generative artificial intelligence large language models in orthodontics: A comparative study of ChatGPT, Google Bard, and Microsoft Bing. European Journal of Orthodontics. https://doi.org/10.1093/ejo/cjae017
NgoT. T.TranT. T.AnG. K.NguyenP. T. (2024). ChatGPT for educational purposes: Investigating the impact of knowledge management factors on student satisfaction and continuous usage. IEEE Transactions on Learning Technologies, 17, 1341–1352. https://doi.org/10.1109/TLT.2024.3383773
24.
ÖzçetinK.KarakuşN. (2020). 5. Sınıf türkçe ders kitaplarındaki metinlerin okunabilirlik yönünden incelenmesi. Türkiye Eğitim Dergisi, 5(1), 175–190.
25.
PattonD. U.LandauA. Y.MathiyazhaganS. (2023). ChatGPT for social work science: Ethical challenges and opportunities. Journal of the Society for Social Work and Research, 14(3), 553–562. https://doi.org/10.1086/726042
26.
ReamerF. (2023). Artificial intelligence in social work: Emerging ethical issues. International Journal of Social Work Values and Ethics, 20(2), 52–71. https://doi.org/10.55521/10-020-205
27.
RussellS.NorvigP. (2020). Artificial intelligence: A modern approach (4th edn.). Pearson.
28.
SallamM.Al-SalahatK.EidH.EggerJ.PuladiB. (2024). Human versus artificial intelligence: ChatGPT-4 outperforming Bing, Bard, ChatGPT-3.5, and humans in clinical chemistry multiple-choice questions. Advances in Medical Education and Practice,15, 857–871. https://doi.org/10.2147/AMEP.S479801
29.
SethI.LimD.XieY.CevikJ.RozenW.RozenW. M.LeeM. (2023). Comparing the efficacy of large language models ChatGPT, BARD, and Bing AI in providing information on rhinoplasty: An observational study. Aesthetic Surgery Journal Open Forum, 5, 1–9. Article ojad084. https://doi.org/10.1093/asjof/ojad084
30.
SingerJ. B.BáezJ. C.RiosJ. A. (2023). AI Creates the message: Integrating AI language learning models into social work education and practice. Journal of Social Work Education, 59(2), 294–302. https://doi.org/10.1080/10437797.2023.2189878
TepeM.EmekliE. (2024). Decoding medical jargon: The use of AI language models (ChatGPT-4, BARD, Microsoft Copilot) in radiology reports. Patient Education and Counseling, 126, 108307. https://doi.org/10.1016/j.pec.2024.108307
VećkalovB.van StekelenburgA.van HarreveldF.RutjensB. T. (2023). Who is skeptical about scientific innovation? Examining worldview predictors of artificial intelligence, nanotechnology, and human gene editing attitudes. Science Communication, 45(3), 337–366. https://doi.org/10.1177/10755470231184203
35.
VictorB. G.KubiakS.AngellB.PerronB. E. (2023a). Time to move beyond the ASWB Licensing exams: Can generative artificial intelligence offer a way forward for social work?Research on Social Work Practice, 33(5), 511–517. https://doi.org/10.1177/10497315231166125
36.
VictorB. G.SokolR. L.GoldkindL.PerronB. E. (2023b). Recommendations for social work researchers and journal editors on the use of generative aı and large language models. Journal of the Society for Social Work and Research, 14(3), 563–577. https://doi.org/10.1086/726021
