Retrieval Augmented Generation-Based Large Language Models for Bridging Transportation Cybersecurity Legal Knowledge Gaps

Abstract

With the advancement of connected and automated transportation systems, a growing need has emerged for regulatory authorities to update existing laws and statutes, as well as create new ones, to address future implications of connectivity and automation. Specifically, ensuring proper engagement with cybersecurity and data privacy challenges for connected and automated transportation systems will require a comprehensive legal framework at both the federal and state levels. To help policymakers achieve this, a retrieval augmented generation (RAG)-based large language model (LLM) framework, Transportation Cybersecurity and Resiliency (TraCR) AI, has been developed in this study, focusing on extracting pertinent information from existing legislation based on inquiries and crafting LLM-generated responses to highlight potential loopholes for further scrutiny. This study primarily aims to mitigate the hallucinations caused in the domain of LLMs by developing a curated knowledge base of legislative documents and an associated question–answer dataset to improve the effectiveness of query results. This RAG-based framework extracts relevant information to improve the specificity of answers and aids the LLM in providing factually accurate responses with increased reliability. Our analyses reveal that the presented RAG-based framework can aid legislative analysis by generating queries for particular questions and responses. We also compare our RAG framework-generated responses with commercially available LLMs to demonstrate the effectiveness of our approach. TraCR AI outperforms leading commercial LLMs across four distinct metrics, that is, AlignScore, ParaScore, BERTScore, and ROUGE score. This highlights that integrating RAG allows LLMs to produce more factually accurate and up-to-date responses than standalone LLMs. This approach to domain-specific LLM development will improve the quality of legislative analysis that can be used to aid policymakers in meeting the challenge of updating legal codes in accordance with emerging technologies.