Examining the Efficacy of Large Language Models for Injury Surveillance

Abstract

Accurate and efficient injury coding is critical for effective injury surveillance. Machine learning (ML) models trained on manually-coded historical injury records have been used to predict the injury codes efficiently and with reasonable accuracy but their accuracy has been limited for complex narratives and rare causes of injury. In this study, we examined performance of Large Language Models (LLMs) to predict three injury codes: cause-of-injury, product-involved, and nature-of-injury, on 100 injury cases randomly selected from the National Electronic Injury Surveillance System database. The prediction performance of LLM (ChatgGPT-3.5) was compared with a traditional ML (Logistic Regression) and a neural network model (Multilayer Perceptron). We observed that LLM was better than the other two models in terms of effectively (a) extracting syntactic relationships, (b) handling misspellings and common acronyms, and (c) deciphering semantic information from the text with reasonable accuracy, even when the narratives were not in a proper grammatical format.