Chatbot Demonstrates Moderate Interrater Reliability in Billing for Hand Surgery Clinic Encounters

Abstract

Background:

Artificial intelligence offers opportunities to improve the burden of health care administrative tasks. Application of machine learning to coding and billing for clinic encounters may represent time- and cost-saving benefits with low risk to patient outcomes.

Methods:

Gemini, a publicly available large language model chatbot, was queried with 139 de-identified patient encounters from a single surgeon and asked to provide the Current Procedural Terminology code based on the criteria for different encounter types. Percent agreement and Cohen’s kappa coefficient were calculated.

Results:

Gemini demonstrated 68% agreement for all encounter types, with a kappa coefficient of 0.586 corresponding to moderate interrater reliability. Agreement was highest for postoperative encounters (n = 43) with 98% agreement and lowest for new encounters (n = 27) with 48% agreement. Gemini recommended billing levels greater than the surgeon’s billing level 31 times and lower billing levels 10 times, with 4 wrong encounter type codes.

Conclusions:

A publicly available chatbot without specific programming for health care billing demonstrated moderate interrater reliability with a hand surgeon in billing clinic encounters. Future integration of artificial intelligence tools in physician workflow may improve the accuracy and speed of billing encounters and lower administrative costs.