Exploring the potential of large language models (LLMs) in analyzing passengers’ perceptions of transit service quality

Abstract

Public transit systems are essential to urban mobility, serving millions of daily commuters. To develop a more responsive, equitable, and efficient public transportation system, it is crucial for transportation planners and policymakers to gain a comprehensive understanding of the diverse travel experiences of transit users. Social media platforms offer valuable, continuous feedback, enabling transit providers to identify issues, make real-time adjustments, and plan long-term improvements. Recently, large language models (LLMs) have attracted significant attention in the urban planning field due to their exceptional performance in natural language processing (NLP) tasks. Using a Weibo dataset related to the Shenzhen metro system (2018–2019) in China, this study developed a two-stage analysis framework to evaluate LLMs’ capabilities in transit service management acting as customer experience analyst and transport planner respectively. We employed LLMs including GPT-3.5 and GPT-4o, utilizing zero-shot, few-shot, and chain-of-thought prompting techniques. Our findings demonstrate that LLMs consistently excel in the classification task and the policy recommendation task when benchmarked against the traditional Bag of Words (BOW) model. The systematic error analysis revealed three types of hallucinations: overthinking, contextual reasoning error, and ambiguity error. Despite these challenges, this research underscores the potential of LLMs in enhancing transit service quality assessment and emphasizes the importance of domain-specific expert rationale in designing prompts and interpreting results. Our study provides valuable insights for transportation planners aiming to leverage advanced NLP techniques for more responsive and data-driven service improvements.