Purpose: The study checked the validity of the entry-level Chinese Social Work Certification Examination in 2024 and tested the social work knowledge of artificial intelligence (AI) by exploring the performance of two systems—ChatGPT 4.0 and ERNIE Bot 4.0. Method: We applied zero-shot learning to the testing models, determined the correctness of their answers based on the reference books, and analyzed their reasons behind. Result: The results show that both AIs passed the exam with scores around 80/100. The AIs answered most questions correctly with adequate reasoning. The reasons for incorrect answers included wrong knowledge, logical fallacies, misunderstanding of question contents, and neglecting the test instruction, which reflected the hallucinations in large-language models. ChatGPT had better reasoning abilities, while ERNIE Bot was more familiar with local contexts and policies. Discussion: The findings support the validity of the exam and also confirm the ability of AI in using social work knowledge.
AlbrightD. L.ThyerB. A. (2010). A test of the validity of the LCSW examination: Quis custodiet ipsos custodes?Social Work Research, 34(4), 229–234. https://doi.org/10.1093/swr/34.4.229
China Personnel Testing Authority. (2023b). 社会工作者职业水平考试 [Professional Qualification Examination for Social Workers]. http://www.cpta.com.cn/test/1450.html
5.
ChoiJ. H.HickmanK. E.MonahanA. B.SchwarczD. (2022). ChatGPT goes to law school. Journal of Legal Education, 71(3), 387–400. https://doi.org/10.2139/ssrn.4335905
6.
DwivediY. K.KshetriN.HughesL.SladeE. L.JeyarajA.KarA. K.BaabdullahA. M.KoohangA.RaghavanV.AhujaM.AlbannaH.AlbashrawiM. A.Al-BusaidiA. S.BalakrishnanJ.BarletteY.BasuS.BoseI.BrooksL.BuhalisD.…WrightR. (2023). “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management, 71, Article 102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642
7.
FreglyC.BarthA.EigenbrodeS. (2024). 生成式AI入门与AWS实战 [Generative AI on AWS: Building context-aware multimodal reasoning applications] (Generative AI Technology Interest Group, Trans.). The People’s Posts and Telecommunications Press. (Original work published 2023).
8.
FriederichsH.FriederichsW. J.MarzM. (2023). ChatGPT in medical school: how successful is AI in progress testing?Medical Education Online, 28(1). Article 2220920. https://doi.org/10.1080/10872981.2023.2220920
9.
GaoJ. G.YanM. C. (2015). Social work in the making: The state and social work development in China. International Journal of Social Welfare, 24(1), 93–101. https://doi.org/10.1111/ijsw.12089
10.
HuangL.YuW.MaW.ZhongW.FengZ.WangH.LiuT. (2025). A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2), 1–55. https://doi.org/10.1145/3703155
11.
KaplanA.HaenleinM. (2019). Siri, Siri, in my hand: Who’s the fairest in the land? On the interpretations, illustrations, and implications of artificial intelligence. Business Horizons, 62(1), 15–25. https://doi.org/10.1016/j.bushor.2018.08.004
12.
LiY.QinH.ZhaoY. (2025). Effectiveness of China’s social work professional level examination: Based on evaluating professional competence. Asia Pacific Journal of Social Work and Development, 35(1), 37–58. https://doi.org/10.1080/29949769.2024.2326468
13.
LiuB. Z. (2025a). The 2025 editions of Social Work Practice (Entry-Level): Real Exam Questions and Detailed Solutions (2018–2024) (1st ed, Vol. 1). East China University of Science and Technology press.
14.
LiuB. Z. (2025b). The 2025 editions of Social Work Practice (Entry-Level): Real Exam Questions and Detailed Solutions (2018–2024) (1st ed., Vol. 2). East China University of Science and Technology press.
15.
MarkovicD. (2024). Current options and limits of digital technologies and artificial intelligence in social work. SHS Web of Conferences, 184, Article 05003. https://doi.org/10.1051/shsconf/202418405003
16.
RodriguezM. Y.GoldkindL.VictorB. G.HiltzB.PerronB. E. (2024). Introducing generative artificial intelligence into the MSW curriculum: A proposal for the 2029 educational policy and accreditation standards. Journal of Social Work Education, 60(2), 174–182. https://doi.org/10.1080/10437797.2024.2340931
17.
ShenL.LiuD. (2025a). The 2025 editions of Social Work Comprehensive Competence (Entry-Level): Real Exam Questions and Detailed Solutions (2018–2024) (1st ed., Vol. 1). East China University of Science and Technology press.
18.
ShenL.LiuD. (2025b). The 2025 editions of Social Work Comprehensive Competence (Entry-Level): Real Exam Questions and Detailed Solutions (2018–2024) (1st ed., Vol. 2). East China University of Science and Technology press.
19.
TanZ.WangR.LianB. (2025). Professional education or certificate: The factors influencing child maltreatment recognition in China. Research on Social Work Practice, 35(6), 655–670. https://doi.org/10.1177/10497315241269839
20.
ThyerB. A. (2011). LCSW Examination pass rates: Implications for social work education. Clinical Social Work Journal, 39, 296–300. https://doi.org/10.1007/s10615-009-0253-x
21.
VictorB. G.KubiakS.AngellB.PerronB. E. (2023). Time to move beyond the ASWB licensing exams: Can generative artificial intelligence offer a way forward for social work?Research on Social Work Practice, 33(5), 511–517. https://doi.org/10.1177/10497315231166
22.
WangW.FuJ.ZhangY.HuK. (2025). A comparative analysis of GPT-4o and ERNIE Bot in a Chinese radiation oncology exam. Journal of Cancer Education. Advance online publication. https://doi.org/10.1007/s13187-025-02652-9
23.
WeiX. (2024). The use of large language models for translating Buddhist texts from classical Chinese to modern English: An analysis and evaluation with ChatGPT 4, ERNIE Bot 4, and Gemini Advanced. Religions, 15(12). Article 1559. https://doi.org/10.3390/rel15121559
24.
WengT. L.WangY. M.ChangS.ChenT. J.HwangS. J. (2023). ChatGPT failed Taiwan’s family medicine board exam. Journal of the Chinese Medical Association, 86(8), 762–766. https://doi.org/10.1097/JCMA.0000000000000946
ZengS. C.XuX. y.ZhangH. H. (2024). 社会工作师职业水平考试的内容效度研究[A preliminary test of the content validity of social worker’s professional level examination]. Social Work and Management, 24(3), 12–21+68.
27.
ZengS.LiX.ChenW. (2020). China Social work professional level examination pass rates: Chronological change and regional difference. Research on Social Work Practice, 30(3), 344–352. https://doi.org/10.1177/1049731519875792
