Artificial Intelligence (AI)-Assisted Coding for Competency-Based Skill Improvement: A Case Study of Motivational Interviewing Skill Development in Behavioral Health Providers-in-Training
Restricted accessResearch articleFirst published online 2025
Artificial Intelligence (AI)-Assisted Coding for Competency-Based Skill Improvement: A Case Study of Motivational Interviewing Skill Development in Behavioral Health Providers-in-Training
Coding of session transcripts is a key strategy in assessing counselors’ and social workers’ fidelity to the motivational interviewing (MI) approach. Artificial intelligence (AI) shows promise in identifying patterns in session transcripts associated with MI proficiency. This study utilizes a multiple instrumental case study approach to compare and contrast intercoder reliability for Atlas.ti, versus human researchers, for six session transcripts using the Motivational Interviewing Competency Assessment (MICA). Atlas.ti’s scores and qualitative feedback were compared to human researchers, and interrater reliability calculated using the Intraclass Correlation Coefficient. Results indicate significant agreement between researchers and AI scoring of intention and strategy, as well as the composite score. The difference in reflection-to-question ratio scoring highlights a difference in AI’s ability to attend to the nuanced nature of utilizing MI skills and strategies to enhance client/patient motivation.
AhmadiA.NoetelM.SchellekensM.ParkerP.AntczakD.BeauchampM.DickeT.DiezmannC.MaederA.NtoumanisN.YeungA.LonsdaleC. (2021). A systematic review of machine learning for assessment and feedback of treatment fidelity. Psychosocial Intervention, 30(3), 139–153. https://doi.org/10.5093/pi2021a4
2.
AlmansourM.AlQurmalahS. I.Abdul RazackH. I. (2023). Motivational interviewing: An evidence-based, collaborative, goal-oriented communication approach in Lifestyle Medicine—A comprehensive review of the literature. Journal of Taibah University Medical Sciences, 18(5), 1170–1178. https://doi.org/10.1016/j.jtumed.2023.03.011
3.
ArthurG. L.GfroererK. P. (2002). Training and supervision through the written word: A description and intern feedback. The Family Journal, 10(2), 213–219.
BernardJ. M.GoodyearR. K. (2019). Fundamentals of clinical supervision (6th ed.). Allyn and Bacon.
7.
BishopC. M. (2007). Pattern recognition and machine learning. Springer.
8.
CampbellS. D.AdamsonS. J.CarterJ. D. (2010). Client language during motivational enhancement therapy and alcohol use outcome. Behavioural and Cognitive Psychotherapy, 38(4), 399–415. https://doi.org/10.1017/S1352465810000263
9.
CarconeA. I.HasanM.AlexanderG.DongM.EgglyS.HartliebK. B.NaarS.MacDonellK.KotovA. (2019). Developing machine learning models for behavioral coding. Journal of Pediatric Psychology, 44(3), 289–299. https://doi.org/10.1093/jpepsy/jsy113
10.
CioffiV.RagozzinoO.MoscaL. L.MorettoE.TortoraE.AcocellaA.MontanariC.FerraraA.CrispinoS.GiganteE.LommatzschA.PizzimentiM.TemporinE.BarlacchiV.BilliC.SaloniaG.SperandeoR. (2025). Can AI technologies support automating clinical supervision? Assessing the potential of ChatGPT. Informatics, 12, 29. https://doi.org/10.3390/informatics12010029
11.
EriksenK.McAuliffeG. (2011). Constructing the counseling skills course. In EriksenK.McAuliffeG. (Eds.), Handbook of counselor preparation: Constructivist, developmental, and experiential approaches (pp. 91–110). Sage.
12.
EwbankM. P.CumminsR.TablanV.CatarinoA.BuchholzS.BlackwellA. D. (2020). Understanding the relationship between patient language and outcomes in internet-enabled cognitive behavioural therapy: A deep learning approach to automatic coding of session transcripts. Psychotherapy Research, 31(3), 300–312. https://doi.org/10.1080/10503307.2020.1788740
13.
FalenderC. A.ShafranskeE. P. (2011). The importance of competency-based clinical supervision and training in the Twenty-First Century: Why bother?Journal of Contemporary Psychotherapy, 42(3), 129–137. https://doi.org/10.1007/s10879-011-9198-9
14.
FangJ.SchoolerL.ShenghuaL. (2022). Machine learning strategy identification: A paradigm to uncover decision strategies with high fidelity. Behavioral Research Methods, 55, 263–284. https://doi.org/10.3758/s13428-022-01828-1
15.
Finefter-RosenbluhI.LevinsonM. (2015). What is wrong with grade inflation (if anything)?Philosophical Inquiry in Education, 23(1), 3–21. https://doi.org/10.7202/1070362ar
16.
FrankelR. M. (1989). Microanalysis and the medical encounter. In HelmD. T. (Ed.), The interactional order: New directions in the study of social order (pp. 21–49). Irvington Publishers.
17.
GabardaA.ButterworthS.LiangO.BeckiordE. (2023). Pilot study of a motivational interviewing training on practitioners’ skill set for patient centered communication. American Journal of Health Promotion, 37(8), 1070–1077. https://doi.org/10.1177/08901171231191130
18.
GibsonJ.AtkinsD. C.CreedT.ImelZ.GeorgiouP.NarayananS. (2023). Multi-label multi-task deep learning for behavioral coding. IEEE Transactions on Affective Computing, 13(1), 508–518. https://doi.org/10.1109/taffc.2019.2952113
19.
GrodniewiczJ. P.HoholM. (2023). Waiting for a digital therapist: Three challenges on the path to psychotherapy delivered by artificial intelligence. Frontiers in Psychiatry, 14, 1190084. https://doi.org/10.3389/fpsyt.2023.1190084
20.
HershbergerP. J.PeiY.BrickerD. A.CrawfordT. N.ShivakumarA.CastleA.ConwayK.MedaramittaR.RechtinM.WilsonJ. F. (2024). Motivational interviewing skills practice enhanced with artificial intelligence: ReadMI. BMC Medical Education, 24(1), 237. https://doi.org/10.1186/s12909-024-05217-4
21.
HershbergerP. J.PeiY.BrickerD. A.CrawfordT. N.ShivakumarA.VasoyaM.MedaramittaR.RechtinM.BosittyA.WilsonJ. F. (2021). Advancing motivational interviewing training with artificial intelligence: ReadMI. Advancement of Medical Educational Practice, 12, 613–618. https://doi.org/10.2147/AMEP.S312373
22.
HohmanM. (2015). Motivational interviewing: An evidence-based practice for improving student practice skills. Journal of Social Work Education, 51(2), 287–297.
23.
HohmanM.DoranN.KoutsenokI. (2019). Motivational interviewing in social work practice: A systematic review. Research on Social Work Practice, 29(7), 739–752. https://doi.org/10.1177/1049731519827377
JephcoteC.MedlandE.Lygo-BakerS. (2020). Grade inflation versus grade improvement: Are our students getting more intelligent?Assessment & Evaluation in Higher Education, 46(4), 547–571. https://doi.org/10.1080/02602938.2020.1795617
26.
JewellR. T.McPhersonM. A.TieslauM. A. (2013). Whose fault is it? Assigning blame for grade inflation in higher education. Applied Economics, 45, 1185–1200. https://doi.org/10.1080/00036846.2011.621884
27.
JonesB. L.CarlsonJ. (2022). Training incoming social work students in motivational interviewing skills: An experiential lab model for advancing social work education. Advances in Social Work, 22(1), 144–160. https://doi.org/10.18060/25478
28.
KarpetisG. (2019). In-depth learning in field education: Evaluating the effectiveness of process recording. Journal of Social Work Practice, 33(1), 95–107. https://doi.org/10.1080/02650533.2017.1400956
29.
KooT. K.LiM. Y. (2016). A guideline of selecting and reporting Intraclass Correlation Coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
30.
KoprowskaJ.KedgeS. (2023). Implementing motivational interviewing in adult mental health social work. Social Work Education, 43(1), 73–89. https://doi.org/10.1080/02615479.2023.2239838
31.
LeeG.-G.LatifE.WuX.LiuN.ZhaiX. (2024a). Applying large language models and chain-of-thought for automatic scoring. Computers and Education: Artificial Intelligence, 6, 100213. https://doi.org/10.1016/j.caeai.2024.100213
32.
LeeY.SonK.KimT. S.KimJ.Young ChungJ. J.AdarE.KimJ. (2024b, June5). One vs. many: Comprehending accurate information from multiple erroneous and inconsistent AI generations [Conference session]. FAccT’ 24: Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pp. 2518–2531. https://doi.org/10.1145/3630106.3662681
33.
LevensonJ. S.WillisG. M. (2024). Motivational interviewing: A physician and social worker collaboration. Health & Social Work, 49(3), 200–208. https://doi.org/10.1093/hsw/hlae005
34.
LundahlB. W.KunzC.BrownellC.TollefsonD.BurkeB. L. (2010). A meta-analysis of motivational interviewing: Twenty-five years of empirical studies. Research on Social Work Practice, 20(2), 137–160.
35.
MartinoS.ParisM.AñezL.NichC.Canning-BallM.HunkeleK.OlmsteadT. A.CarrollK. M. (2016). The effectiveness and cost of clinical supervision for motivational interviewing: A randomized controlled trial. Journal of Substance Abuse Treatment, 68, 11–23. https://doi.org/10.1016/j.jsat.2016.04.005
36.
McAuliffeG.EriksenK. (2010). Handbook of counselor education: Constructivist, developmental, and exponential approaches. Sage.
37.
McLeodJ.BalamoutsouS. (2001). A method for qualitative narrative analysis of psychotherapy transcripts. Psychologische Beitrage, 43(3), 128–152.
38.
MillerW. R.RollnickS. (2023). Motivational interviewing: Helping people change. The Guilford Press.
39.
MillsA. J.DureposG.WiebeE. (2009). Encyclopedia of case study research, volume 1. Sage.
40.
MoyersT. B.RowellL. N.ManuelJ. K.ErnstD.HouckJ. M. (2016). The Motivational Interviewing Treatment Integrity code (MITI 4): Rationale, preliminary reliability and validity. Journal of Substance Abuse Treatment, 65, 36–42. https://doi.org/10.1016/j.jsat.2016.01.001
PellemansM.SalmiS.MérelleS.JanssenW.van der MeiR. (2024). Automated behavioral coding to enhance the effectiveness of motivational interviewing in a chat-based suicide prevention helpline: Secondary analysis of a clinical trial. Journal of Medical Internet Research, 26, e53562. https://doi.org/10.2196/53562
43.
RollnickS.MillerW. R.ButlerC. C.AloiaM. S. (2009). Motivational interviewing in health care: Helping patients change behavior. COPD: Journal of Chronic Obstructive Pulmonary Disease, 5(3), 203.
44.
ShaikhA.DavidsonD. (2021). Motivational interviewing in social work practice: A strength-based approach to client change. Journal of Social Work Practice, 35(2), 189–201. https://doi.org/10.1080/1533256X.2021.1973826
45.
SvenssonM.WagnssonS.GustafssonH. (2021). Can motivational interviewing be a helpful professional tool? investigating teachers’ experiences. Educational Research, 63(4), 440–455. https://doi.org/10.1080/00131881.2021.1989318
46.
TavabiL.TranT.StefanoK.BorsariB.WoolleyJ. D.SchererS.SoleymaniM. (2021, June). Analysis of behavior classification in motivational interviewing [Conference session]. Seventh Workshop on Computational Linguistics and Clinical Psychology: Improving Access, Association for Computational Linguistics, pp. 110–115.
47.
TsaiM.LuF.FrankelR. M. (2013). Learning to listen: Effects of using conversational transcripts to help medical students improve their use of open questions in soliciting patient problems. Patient Education and Counseling, 93(1), 48–55. https://doi.org/10.1016/j.pec.2013.03.022
48.
UguzH.ArslanA.TurkogluI. (2007). A biomedical system based on hidden Markov model for diagnosis of the heart valve diseases. Pattern Recognition Letters, 28(4), 395–404.
WalshS. E.FeighK. M. (2022). Understanding human decision processes: Inferring decision strategies from behavioral data. Journal of Cognitive Engineering and Decision Making, 16(4), 301–325. https://doi.org/10.1177/15553434221122899
51.
YinR. K. (2018). Case study research and applications: Design and methods (6th ed.). Sage.
