Thermal Forecast: Traffic Trajectory Prediction in Challenging Nighttime Conditions Using Thermal Imaging

Abstract

Accurate trajectory prediction of traffic agents is crucial for the advancement of intelligent transportation systems (ITS), especially under challenging conditions such as nighttime and adverse weather. Although significant advances have been made in trajectory prediction for ITS, existing methods often fail in low-visibility environments because of their reliance on traditional red-green-blue cameras. Such cameras face significant limitations in low-light environments because of high glare, reflections, and reduced visibility. To address these challenges, this study presents a novel deep learning framework that leverages thermal cameras for robust trajectory prediction. We propose a vision transformer–based dual gated recurrent unit network for future trajectory prediction using thermal videos. The proposed network efficiently captures the complex interactions among object-specific and scene-level rich information in the temporal sequence from very limited cues in thermal images. Moreover, a custom loss function is proposed, which is designed to emphasize long-term prediction accuracy by penalizing errors more heavily in distant future predictions. The proposed method is evaluated in a challenging thermal video dataset, which is captured on rainy nights in different traffic intersections. Experimental results demonstrate promising performance across various prediction horizons for trajectory prediction of traffic agents in challenging nighttime conditions.