The combination of steering angle prediction and control of autonomous vehicles (AVs) is a challenging task. To improve the real-time steering angle prediction accuracy and the effectiveness of steering control, a novel steering angle prediction YOLOv5-based end-to-end adaptive neural network control for AVs is proposed. Firstly, since most of the lane line datasets are simulated images and lack of diversity, a novel lane dataset derived from the real roads are made manually to train the You Only Look Once version 5 (YOLOv5) network model. To improve the detection accuracy of the network model, the Generalized Intersection over Union (GIoU) of the bounding box regression loss function is updated to a Complete Intersection over Union (CIoU) with a better convergence effect. Furthermore, the neural network-based controller and disturbance observer are proposed to effectively control the steering angle predicted by YOLOv5 and estimate the lumped uncertainty. Meanwhile, a composite adaptive updating law is constructed by utilizing the tracking error and modeling error to improve steering performance. Finally, the system stability is proved by Lyapunov theory and the effectiveness of the proposed method is verified with experiments.
YaoQTianY. A model predictive controller with longitudinal speed compensation for autonomous vehicle path tracking. Appl Sci2019; 9(22): 4739.
2.
TampuuASemikinMMuhammadN, et al. A survey of end-to-end driving: architectures and training methods. IEEE Trans Neural Netw Learn Syst. Epub ahead of print 29December2020. DOI: 10.1109/TNNLS.2020.3043505.
3.
GeigerALenzPStillerC, et al. Vision meets robotics: the kitti dataset. Int J Rob Res2013; 32(11): 1231–1237.
4.
YurtseverELambertJCarballoA, et al. A survey of autonomous driving: common practices and emerging technologies. IEEE Access2020; 8: 58443–58469.
5.
ThrunSMontemerloMDahlkampH, et al. Stanley: the robot that won the DARPA grand challenge. J Field Robot2006; 23(9): 661–692.
6.
MontemerloMBeckerJBhatS, et al. Junior: the stanford entry in the urban challenge. J Field Robot2008; 25(9): 569–597.
7.
VanBJOBrienMGruyerD, et al. Autonomous vehicle perception: the technology of today and tomorrow. Transp Res Part C Emerg Technol2018; 89: 384–406.
8.
MaYWangZYangH, et al. Artificial intelligence applications in the development of autonomous vehicles: a survey. IEEE/CAA J Automatica Sin2020; 7(2): 315–329.
9.
ZengWLuoWSuoS, et al. End-to-end interpretable neural motion planner. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Long Beach, CA, USA, 16–20 June 2019, pp.8652–8661. New York, NY: IEEE.
10.
ChenCSeffAKornhauserA, et al. Deepdriving: learning affordance for direct perception in autonomous driving. In: 2015 IEEE international conference on computer vision (ICCV), Santiago, Chile, 7–13 December 2015, pp.2722–2730. New York: IEEE.
11.
ChenZHuangX. End-to-end learning for lane keeping of self-driving cars. In: 2017 IEEE intelligent vehicles symposium (IV), Los Angeles, CA, USA, 11–14 June 2017, pp.1856–1860. New York, NY: IEEE.
12.
PomerleauDA. Alvinn: an autonomous land vehicle in a neural network. Report, Carnegie Mellon University, USA, January1989.
13.
LeCunYMullerUBenJ, et al. Off-road obstacle avoidance through end-to-end learning. In: Advances in neural information processing systems (NIPS), Vancouver, BC, Canada, 5–8 December 2005, pp.739–746. Cambridge, MA: MIT Press.
14.
BojarskiMDelTDDworakowskiD, et al. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316, 2016.
15.
XuHGaoYYuF, et al. End-to-end learning of driving models from large-scale video datasets. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017, pp.3530–3538. New York, NY: IEEE.
16.
CodevillaFSantanaELpezAM, et al. Exploring the limitations of behavior cloning for autonomous driving. In: Proceedings of the IEEE/CVF international conference on computer vision (ICCV), Seoul, Korea (South), 20–26 October 2019, pp.9329–9338. New York, NY: IEEE.
17.
RiedmillerMMontemerloMDahlkampH. Learning to drive a real car in 20 minutes. In: IEEE 2007 frontiers in the convergence of bioscience and information technologies, Jeju, Korea (South), 11–13October2007, pp.645–650. New York, NY: IEEE.
18.
MnihVKavukcuogluKSilverD, et al. Human-level control through deep reinforcement learning. Nature2015; 518(7540): 529–533.
19.
SallabAEAbdouMPerotE, et al. Deep reinforcement learning framework for autonomous driving. Electron Imaging2017; 2017(19): 70–76.
20.
KendallAHawkeJJanzD, et al. Learning to drive in a day. In: 2019 IEEE international conference on robotics and automation (ICRA), Montreal, Canada, 20–24 May 2019, pp.8248–8254. New York, NY: IEEE.
21.
KuuttiSBowdenRJinY, et al. A survey of deep learning applications to autonomous vehicle control. IEEE trans Intell Transp Syst2020; 2(22): 712–733.
22.
HuangXZhangHZhangG, et al. Robust weighted gain-scheduling H∞ vehicle lateral motion control with considerations of steering system backlash-type hysteresis. IEEE Trans Control Syst Technol2014; 22(5):1740–1753.
23.
HuangCNaghdyFDuH. Observer-based fault-tolerant controller for uncertain steer-by-wire systems using the delta operator. IEEE ASME Trans Mechatron2018; 23(6): 2587–2598.
24.
HuangCNaghdyFDuH. Fault tolerant sliding mode predictive control for uncertain steer-by-wire system. IEEE Trans Cybern2017; 49(1): 261–272.
25.
HuangCNaghdyFDuH. Sliding mode predictive tracking control for uncertain steer-by-wire system. Control Eng Pract2019; 85: 194–205.
26.
WangHKongHManZ, et al. Sliding mode control for steer-by-wire systems with AC motors in road vehicles. IEEE Trans Ind Electron2013; 61(3): 1596–1611.
27.
SunZZhengJManZ, et al. Robust control of a vehicle steer-by-wire system using adaptive sliding mode. IEEE Trans Ind Electron2015; 63(4): 2251–2262.
28.
SunZZhengJWangH, et al. Adaptive fast non-singular terminal sliding mode control for a vehicle steer-by-wire system. IET Control Theory Appl2016; 11(8): 1245–1254.
29.
WuXZhangMXuM. Active tracking control for steer-by-wire system with disturbance observer. IEEE Trans Veh Technol2019; 68(6): 5483–5493.
30.
WangHManZKongH, et al. Design and implementation of adaptive terminal sliding-mode control on a steer-by-wire equipped road vehicle. IEEE Trans Ind Electron2016; 63(9): 5774–5785.
31.
SunZZhengJManZ, et al. Nested adaptive super-twisting sliding mode control design for a vehicle steer-by-wire system. Mech Syst Signal Process2019; 122: 658–672.
32.
GidadoUMChiromaHAljojoN, et al. A survey on deep learning for steering angle prediction in autonomous vehicles. IEEE Access2020; 8: 163797–163817.
33.
HeKZhangXRenS, et al. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, New York, NY, USA, 26 June–1 July 2016, pp.770–778. New York, NY: IEEE.
34.
BochkovskiyAWangCYLiaoHYM. YOLOv4: optimal opeed and occuracy of object oetection. arXiv preprint arXiv:2004.10934, 2020.
KisantalMWojnaZMurawskiJ, et al. Augmentation for small object detection. arXiv preprint arXiv:1902.07296, 2019.
37.
RenSHeKGirshickR, et al. Faster r-cnn: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell2016; 39(6):1137–1149.
38.
IoffeSSzegedyC. Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd international conference on machine learning, ICML 2015, Lille, France, 6–11 July 2015, pp. 1093–1102. New Jersey: International Machine Learning Society (IMLS).
39.
WangCYLiaoHYMWuYH, et al. CSPNet: a new backbone that can enhance learning capability of CNN. In: 2020 IEEE/CVF conference on computer vision and pattern recognition workshops (CVPRW), Seattle, WA, USA, 16–18 June 2020, pp. 1571–1580. New York, NY: IEEE.
40.
HeKZhangXRenS, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans Pattern Anal Mach Intell2015; 37(9): 1904–1916.
41.
SivicJZissermanA. Video google: a text retrieval approach to object matching in videos. In: Proceedings Ninth IEEE international conference on computer vision, Nice, France, 13–16 October 2003, pp. 1470–1477. New York, NY: IEEE.
42.
LinTYDollrPGirshickR, et al. Feature pyramid networks for object detection. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017, pp. 936–944. New York, NY: IEEE.
43.
LiuSQiLQinH, et al. Path aggregation network for instance segmentation. In: 2018 IEEE/CVF conference on computer vision and pattern recognition, Salt Lake City, UT, USA, 18–23 June 2018, pp. 8759–8768. New York, NY: IEEE.
44.
RedmonJFarhadiA. Yolov3: an incremental improvement. arXiv preprint arXiv:1804.02767, 2018.
45.
YuJJiangYWangZ, et al. Unitbox: an advanced object detection network. In: Proceedings of the 24th ACM international conference on multimedia, Amsterdam, The Netherlands, 15–19 October 2016, pp. 516–520. New York, NY: ACM.
46.
RezatofighiHTsoiNGwakJ, et al. Generalized intersection over union: a metric and a loss for bounding box regression. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Long Beach, CA, USA, 15–20 June 2019, pp. 658–666. New York, NY: IEEE.
47.
ZhengZWangPLiuW, et al. Distance-IoU loss: faster and better learning for bounding box regression. In: Thirty-second conference on innovative applications of artificial intelligence (AAAI), 2020, Hilton Midtown, New York, NY, USA, 7–12 February 2020, pp. 12993–13000. Palo Alto, CA: AAAI Press.
48.
VisaSRamsayBRalescuAL, et al. Confusion matrix-based feature selection. MAICS2011; 710: 120–127.
49.
NaJChenQRenX, et al. Adaptive prescribed performance motion control of servo mechanisms with friction compensation. IEEE Trans Ind Electron2013; 61(1): 486–494.
50.
LiYQiangSZhuangX, et al. Robust and adaptive backstepping control for nonlinear systems using RBF neural networks. IEEE Trans Neural Netw2004; 15(3): 693–701.
51.
WangYWangYChenG. Robust composite adaptive neural network control for air management system of PEM fuel cell based on high-gain observer. Neural Comput Appl2020; 32(14): 10229–10243.
52.
DoMTManZZhangC, et al. Robust sliding mode-based learning control for steer-by-wire systems in modern vehicles. IEEE Trans Veh Technol2014; 63(2): 580–590.
