Navigation in complex urban environments is very difficult, mainly due to perceptual uncertainty caused by obstructions in the field of view. The perceptual uncertainty is mainly due to sensors obstructed by obstacles. This has a significant impact on the safety of autonomous vehicles. Existing approaches based on partially observable Markov decision processes or reinforcement learning for the uncertainty problem may lead to conservative planning and expensive computation. We propose caution-driven networks combined with distributional reinforcement learning fully parameterized quantile function, which termed Intrinsic caution module – fully parameterized quantile function (ICM-FQF). The method is also applied in the continuous action space. The method is used to evaluate two challenging scenarios, pedestrians crossing with occlusion and a unsignalized intersection with a limited field of view. The algorithm is trained and evaluated using the CARLA simulator. Compared to conventional RL algorithms, the method makes smarter decisions and reduces the rate of collisions.
YanSWelscheholdTBuscherD, et al. Courteous behavior of automated vehicles at unsignalized intersections via reinforcement learning. IEEE Robot Autom Lett2022; 7(1): 191–198.
2.
WangGHuJLiZ, et al. Harmonious lane changing via deep reinforcement learning. IEEE Trans Intell Transp Syst2022; 23(5): 4642–4650.
3.
ChenLHeYWangQ, et al. Joint optimization of sensing, decision-making and motion-controlling for autonomous vehicles: a deep reinforcement learning approach. IEEE Trans Veh Technol2022; 71(5): 4642–4654.
4.
WangHYuanSGuoM, et al. A deep reinforcement learning-based approach for autonomous driving in highway on-ramp merge. Proc IMechE, Part D: J Automobile Engineering2021; 235(10–11): 2726–2739.
5.
ZhangLZhangRWuT, et al. Safe reinforcement learning with stability guarantee for motion planning of autonomous vehicles. IEEE Trans Neural Netw Learn Syst2021; 32(12): 5435–5444.
6.
AradiS. Survey of deep reinforcement learning for motion planning of autonomous vehicles. IEEE Trans Intell Transp Syst2022; 23(2): 740–759.
7.
HubmannCSchulzJBeckerM, et al. Automated driving in uncertain environments: planning with interaction and uncertain maneuver prediction. IEEE Trans Intell Vehicles2018; 3(1): 5–17.
8.
ShuKYuHChenX, et al. Autonomous driving at intersections: a behavior-oriented critical-turning-point approach for decision making. IEEE/ASME Trans Mechatron2022; 27(1): 234–244.
9.
BernhardJPollokSKnollA. Addressing inherent uncertainty: risk-sensitive behavior generation for automated driving using distributional reinforcement learning. In: 2019 IEEE intelligent vehicles symposium (IV), Paris, France, 9–12 June 2019, pp.2148–2155. New York: IEEE.
10.
RezaeeKYadmellatPChamorroS. Motion planning for autonomous vehicles in the presence of uncertainty using reinforcement learning. In: 2021 IEEE/RSJ international conference on intelligent robots and systems (IROS), Prague, Czech Republic, 27 September–1 October 2021, pp.3506–3511. New York: IEEE.
11.
KamranDEngelgehTBuschM, et al. Minimizing safety interference for safe and comfortable automated driving with distributional reinforcement learning. In: 2021 IEEE/RSJ international conference on intelligent robots and systems (IROS), Prague, Czech Republic, 27 September–1 October 2021, pp.1236–1243. New York: IEEE.
12.
LeeMJoKSunwooM. Collision risk assessment for possible collision vehicle in occluded area based on precise map. In: 2017 IEEE 20th international conference on intelligent transportation systems (ITSC), Yokohama, Japan, 16–19 October 2017, pp.1–6. New York: IEEE.
13.
LiB. Occlusion-aware on-road autonomous driving: a trajectory planning method considering occlusions of lidars. Optik2021; 243: 167347.
14.
PatzBJPapelisYPillatR, et al. A practical approach to robotic design for the DARPA urban challenge. J Field Robot2008; 25(8): 528–566.
15.
GongJYuanSYanJ, et al. Intuitive decision-making modeling for self-driving vehicles. In: 17th international IEEE conference on intelligent transportation systems (ITSC), Qingdao, China, 8–11 October 2014, pp.29–34. New York: IEEE.
16.
NagerYCensiAFrazzoliE. What lies in the shadows? Safe and computation-aware motion planning for autonomous vehicles using intent-aware dynamic shadow regions. In: 2019 international conference on robotics and automation (ICRA), Montreal, QC, Canada, 20–24 May 2019, pp.5800–5806. New York: IEEE.
17.
WangZJChenXMWangP, et al. A decision-making model for autonomous vehicles at urban intersections based on conflict resolution. J Adv Transport2021; 2021: 1–12.
18.
PathakDAgrawalPEfrosAA, et al. Curiosity-driven exploration by self-supervised prediction. In: 2017 IEEE conference on computer vision and pattern recognition workshops (CVPRW), Honolulu, HI, USA, 21–26 July 2017, pp.488–489. New York: IEEE.
19.
BellemareMGDabneyWMunosR. A distributional perspective on reinforcement learning. In: Proceedings of the 34th international conference on machine learning, Sydney, NSW, Australia, pp.449–458.
20.
DabneyWRowlandMBellemareM, et al. Distributional reinforcement learning with quantile regression. Proc AAAI Conf Artif Intell2018; 32: 11791.
21.
DabneyWOstrovskiGSilverD, et al. Implicit quantile networks for distributional reinforcement learning. In: Proceedings of the 35th international conference on machine learning, Stockholm, Sweden, pp.1096–1105.
22.
YangDZhaoLLinZ, et al. Fully parameterized quantile function for distributional reinforcement learning. Adv Neural Inf Process Syst2019; 32.
DuanJGuanYLiSE, et al. Distributional soft actor-critic: off-policy reinforcement learning for addressing value estimation errors. IEEE Trans Neural Netw Learn Syst2022; 33: 6584–6598.
25.
ChoiJDanceCKimJE, et al. Risk-conditioned distributional soft actor-critic for risk-sensitive navigation. In: 2021 IEEE international conference on robotics and automation (ICRA), Xi’an, China, 30 May–5 June 2021, pp.8337–8344. New York: IEEE.
26.
ChenXMXuSYWangZJ, et al. A decision-making model for autonomous vehicles at intersections based on hierarchical reinforcement learning. Unmanned Syst2023; 11: 1–12.
27.
XuSLiuQHuY, et al. Decision-making models on perceptual uncertainty with distributional reinforcement learning. Green Energy Intell Transp2023; 2(2): 100062.
28.
KanazawaTWangHGuptaC. Distributional actor-critic ensemble for uncertainty-aware continuous control. In: 2022 international joint conference on neural networks (IJCNN), Padua, Italy, 18–23 July 2022, pp.1–10. New York: IEEE.
29.
WangPLiYShekharS, et al. Uncertainty estimation with distributional reinforcement learning for applications in intelligent transportation systems: a case study. In: 2019 IEEE intelligent transportation systems conference (ITSC), Auckland, New Zealand, 27–30 October 2019, pp.3822–3827. New York: IEEE.
30.
LeeKIseleDTheodorouEA, et al. Risk-sensitive MPCs with deep distributional inverse RL for autonomous driving. In: 2022 IEEE/RSJ international conference on intelligent robots and systems (IROS), Kyoto, Japan, 23–27 October 2022, pp.7635–7642. New York: IEEE.
31.
LillicrapTPHuntJJPritzelA, et al. Continuous control with deep reinforcement learning. arXiv preprint arXiv:150902971, 2015.
32.
PatelUKumarNKSSathyamoorthyAJ, et al. DWA-RL: dynamically feasible deep reinforcement learning policy for robot navigation among mobile obstacles. In: 2021 IEEE international conference on robotics and automation (ICRA), Xi’an, China, 30 May–5 June 2021, pp.6057–6063. New York: IEEE.
33.
KästnerLZhaoXBuiyanT, et al. Connecting deep-reinforcement-learning-based obstacle avoidance with conventional global planners using waypoint generators. In: 2021 IEEE/RSJ international conference on intelligent robots and systems (IROS), Prague, Czech Republic, 27 September–1 October 2021, pp.1213–1220. New York: IEEE.
