This paper presents an on-board multi-sensor system which is able to detect obstacles and estimate their distances in railway scenes in different illumination conditions. The system was developed within the H2020 Shift2Rail project SMART (Smart Automation of Rail Transport) and aims at increasing the safety of rail transport by detecting obstacles on the rail tracks ahead of a moving train in order to reduce the number of collisions. The system hardware consists of cameras of different types integrated into a specially designed housing, mounted on the front of the train. Multiple vision sensors complement each other in order to handle different illumination and environmental conditions. The system software uses a novel machine learning-based method that is suited to a particular challenge of railway operations, the need for long-range obstacle detection and distance estimation. The development of this method used a long-range railway dataset, which was specifically generated for this project. Evaluation results of reliable obstacle detection in various environmental conditions using the SMART RGB camera in day light illumination conditions and using the SMART Night Vision sensor in poor (night) illumination conditions are presented. The results demonstrate both the potential of the on-board SMART obstacle detection system in the operational railway environment and the benefit of the use of different cameras to be more independent of light and environmental conditions.
BarneyDGeorgeDNikandrosG. Calculating train braking distance, safety critical systems and software. In: 2001 sixth Australian workshop on safety-related programmable systems (SCS2001), St Lucia, Queensland, Australia, 6 July 2001.
BergAÖfjällKAhlbergJ, et al. Detecting rails and obstacles using a train-mounted thermal camera. In: PaulsenRPedersenK (eds) Image analysis. SCIA 2015. Lecture notes in computer science. vol 9127. Cham: Springer, 2015, pp. 492–503.
5.
PinggeraPFrankeUMesterR. High-performance long range obstacle detection using stereo vision. In: IEEE/RSJ international conference on intelligent robots and systems (IROS2015), Hamburg, Germany, 28 September–2 October 2015.
HaseebMAGuanJRistić-DurrantD, et al. DisNet: a novel method for distance estimation from monocular camera. In: 2018. IEEE/RSJ international conference on intelligent robots and systems – IROS 2018, 10th workshop on planning, perception and navigation for intelligent vehicles (PPNIV), Madrid, Spain, October 1st, 2018.
8.
HaseebMARistić-DurrantDGräserA.Long-range obstacle detection from a monocular camera. In: ACM computer science in cars symposium (CSCS 2018), ECCV Munich, Germany, September 13th-14th, 2018.
9.
HaseebMARistić-DurrantDGräserA, et al. Multi-DisNet: machine learning-based object distance estimation from multiple cameras. In: 12th international conference on computer vision systems, ICVS2019 (Tzovaras D, et al. (eds.)). LNCS 11754. Switzerland: Springer Nature AG, 2019, pp.457–469.
10.
LeuAAiteanuDGräserA. High speed stereo vision based automotive collision warning system. In: Precup RE., Kovács S., Preitl S., Petriu E. (eds), Applied computational intelligence in engineering and information technology. Volume 1. Heidelberg: Springer Verlag, 2012, pp.187–199.
11.
WeichselbaumJZinnerCGebauerO, et al. Accurate 3D-vision-based obstacle detection for an autonomous train. Computers in Industry2013; 64: 1209–1220.
12.
Heng, L., Choi, B., Cui, Z., Geppert, M., Hu, S., Kuan, B., Liu, P., Nguyen, R., Yeo, Y.C., Geiger, A. and Lee, G.H., 2019. Project AutoVision: Localization and 3D Scene Perception for an Autonomous Vehicle with a Multi-Camera System. In International Conference on Robotics and Automation (ICRA 2019) (pp. 4695–4702). IEEE.
13.
DuvieubourgLCabestaingFAmbellouisS, et al. Long distance vision sensor for driver assistance. IFAC Proc Vol2007; 40: 330–336.
14.
SaxenaASchulteJNgAY. Depth estimation using monocular and stereo cues. In: IJCAI, Hyderabad, India, 6–12 January 2007.
15.
MaddernWPascoeGLinegarC, et al. 1 Year, 1000 km: the oxford RobotCar dataset. Int J Robot Res2017; 36: 3–15.
16.
Joel Janai, Fatma Güney, Aseem Behl and Andreas Geiger (2020), “Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art”, Foundations and Trends® in Computer Graphics and Vision 2020; 12(1-3): 1–308.
17.
GeigerALenzPStillerC, et al. Vision meets robotics: the KITTI dataset. Int J Robot Res (IJRR)2013; 32: 1231–1237.
18.
Scheidegger S, Benjaminsson J, Rosenberg E, et al., Mono-camera 3d multi-object tracking using deep learning detections and pmbm filtering. In: 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, Suzhou, China, 26-30 June 2018, pp. 433–440.
19.
ChenZKhemmarRDecouxB, et al. Real time object detection, tracking, and distance and motion estimation based on deep learning: application to smart mobility. In: Eighth international conference on emerging security technologies (EST), UK, 22–24 July 2019.
20.
ZendelOMurschitzMZeilingerM, et al. RailSem19: a dataset for semantic rail scene understanding. In: The IEEE conference on computer vision and pattern recognition (CVPR) workshop, Long Beach, CA, 16–20 June 2019.
XuYGaoCYuanLTang, et al. Real-time obstacle detection over rails using deep convolutional neural network. In: IEEE intelligent transportation systems conference (ITSC), Auckland, NZ, 27–30 October 2019.
23.
YeTWangBSongP, et al. Railway traffic object detection using differential feature fusion convolution neural network.IEEE Trans Intell Transport Syst PP2020; 22: 1–13.
24.
KapoorRGoelRSharmaA.Deep learning based object and railway track recognition using train mounted thermal imaging system. j Comput Theor Nanosci2020; 17: 5062–5071.
RedmonJDivvalaSGirshickR, et al. You only look once: unified, real-time object detection. In: IEEE conference on computer vision and pattern recognition, Las Vegas, NV, 27–30 June 2016.
30.
ZhouXWangDKrähenbühlP. Objects as points, 2019. arXiv preprint arXiv:1904.07850, https://arxiv.org/abs/1904.07850v2 (accessed 1 July 2021).
31.
RedmonJFarhadiA. YOLOv3: an incremental improvement, arXiv, 2018.
