Purpose: This study sought to replicate a previous investigation of construct-irrelevant variance on the Association of Social Work Boards (ASWB) clinical licensing exam completed by Albright and Thyer over a decade ago. Method: The performance of ChatGPT was assessed on a modified version of 50 newly developed clinical exam questions currently distributed by the ASWB, where only the four multiple-choice options for each item were presented without the question. Results: ChatGPT achieved an average accuracy rate of 73.3% across three rounds of testing, providing strong evidence of construct-irrelevant variance. Discussion: These results raise concerns about the construct validity of the clinical exam and emphasize the need for reassessment of its structure and content to ensure fairness and accuracy. Based on the findings, state legislators and regulators are encouraged to temporarily discontinue the use of the ASWB exam in the clinical licensure process until its validity flaws are resolved.
AlbrightD. L.ThyerB. A. (2010). A test of the validity of the LCSW examination: Quis custodiet ipsos custodes?Social Work Research, 34(4), 229–234. https://doi.org/10.1093/swr/34.4.229
2.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (Eds.) (2014). Standards for educational and psychological testing. American Educational Research Association.
DeCarloM. P. (2022). Racial bias and ASWB exams: A failure of data equity. Research on Social Work Practice, 32(3), 255–258. https://doi.org/10.1177/10497315211055986
11.
GilsonA.SafranekC. W.HuangT.SocratesV.ChiL.TaylorR. A.ChartashD. (2023). How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Medical Education, 9(1), e45312. https://doi.org/10.2196/45312
PursnaniV.SermetY.DemirI. (2023). Performance of ChatGPT on the U.S. Fundamentals of Engineering Exam: Comprehensive assessment of proficiency and potential implications for professional environmental engineering practice. arXiv. https://arxiv.org/abs/2304.12198.
17.
VictorB. G.KubiakS.AngellB.PerronB. E. (2023). Time to move beyond the ASWB licensing exams: Can generative artificial intelligence offer a way forward for social work?Research on Social Work Practice, 33(5), 511–517. https://doi.org/10.1177/10497315231166125
