Foggy environments present significant challenges to autonomous driving owing to the effects of attenuation and backscattering, which often compromise the performance of LiDAR-camera fusion-based perception systems. In this study, we introduce FogFusion, a novel 3D object detection network specifically designed to operate effectively under foggy conditions by leveraging a synergistic camera-LiDAR fusion approach. Our approach integrates a Depth Completion network with Fog Convolution (DCFC) to generate virtual point clouds that enhance the original sparse LiDAR data. These enhanced point clouds are then processed using a Flexible Cylindrical Voxel (FCV) encoding method. To ensure robust multi-modal feature integration, we employ a Cylindrical Fusion Module (CFM) during the fusion process. Experimental evaluations on the KITTI and KITTI-C datasets reveal that FogFusion improves detection performance in foggy conditions by at least 3.32% compared to the baseline model and surpasses the performance of advanced 3D object detection models. These results highlight FogFusion’s potential to significantly enhance the environmental perception capabilities of autonomous vehicles operating in foggy weather conditions.
HuangKShiBLiX, et al. Multi-modal sensor fusion for auto driving perception: a survey. arXiv preprint arXiv:2202.02703, 2022.
2.
FattalR.Single image dehazing. ACM Trans Graph2008; 27: 1–9.
3.
HeKSunJTangX.Single image haze removal using dark channel prior. IEEE Trans Pattern Anal Mach Intell2011; 33: 2341–2353.
4.
Van RossumMCW. Multiple scattering of classical waves: microscopy, mesoscopy, and diffusion. Rev Mod Phys1999; 71: 313–371.
5.
RasshoferRHSpiesMSpiesH.Influences of weather phenomena on automotive laser radar systems. Adv Radio Sci2011; 9: 49–60.
6.
Jose ValanarasuJMYasarlaRPatelVM. TransWeather: transformer-based restoration of images degraded by adverse weather conditions. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, New Orleans, LA, USA, 18–24 June 2022, pp.2343–2353. New York: IEEE.
7.
HeinzlerRPiewakFSchindlerP, et al. CNN-based lidar point cloud de-noising in adverse weather. IEEE Robot Autom Lett2020; 5: 2514–2521.
8.
BijelicMGruberTMannanF, et al. Seeing through fog without seeing fog: deep multimodal sensor fusion in unseen adverse weather. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, Seattle, WA, USA, 13–19 June 2020, pp.11679–11689. New York: IEEE.
9.
ChaturvediSSZhangLYuanX.Pay ‘attention’ to adverse weather: weather-aware attention-based object detection. In: Proceedings - international conference on pattern recognition, Montreal, QC, Canada, 21–25 August 2022, pp.4573–4579. New York: IEEE.
10.
MaiNAMDuthonPKhoudourL, et al. 3D object detection with SLS-fusion network in foggy weather conditions. Sensors2021; 21: 6711.
11.
LiuZCaiYWangH, et al. Robust target recognition and tracking of self-driving cars with radar and camera information fusion under severe weather conditions. IEEE Trans Intell Transp Syst2022; 23: 6640–6653.
12.
ChuSYLeeMS.MT-DETR: robust end-to-end multimodal detection with confidence fusion. In: Proceedings - 2023 IEEE winter conference on applications of computer vision, WACV 2023, Waikoloa, HI, USA, 2–7 January 2023, pp.5241–5250. New York: IEEE.
13.
ChaeYKimHYoonK-J.Towards robust 3D object detection with LiDAR and 4D radar fusion in various weather conditions. In: 2024 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Seattle, WA, USA, 16–22 June 2024, pp.15162–15172. New York: IEEE.
14.
DongYKangCZhangJ, et al. Benchmarking robustness of 3D object detection to common corruptions in autonomous driving. In: 2023 IEEE/CVF conference on computer vision and pattern recognition (CVPR), Vancouver, BC, Canada, 17–24 June 2023, pp.1022–1032. New York: IEEE.
15.
PangSMorrisDRadhaH. CLOCs: camera-LiDAR object candidates fusion for 3D object detection. In: IEEE international conference on intelligent robots and systems, Las Vegas, NV, USA, 24 October 2020–24 January 2021, pp.10386–10393. New York: IEEE.
16.
BraunMRaoQWangY, et al. Pose-RCNN: joint object detection and pose estimation using 3D object proposals. In: 2016 IEEE 19th international conference on intelligent transportation systems (ITSC), Rio de Janeiro, Brazil, 1–4 November 2016, pp.1546–1551. New York: IEEE.
17.
ChenXMaHWanJ, et al. Multi-view 3D object detection network for autonomous driving. In: Proceedings - 30th IEEE conference on computer vision and pattern recognition, CVPR 2017, Honolulu, HI, USA, 21–26 July 2017, pp.6526–6534. New York: IEEE.
18.
KuJMozifianMLeeJ, et al. Joint 3D proposal generation and object detection from view aggregation. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), Madrid, Spain, 1–5 October 2018, pp.5750–5757. New York: IEEE.
19.
VoraSLangAHHelouB, et al. Pointpainting: sequential fusion for 3D object detection. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, Seattle, WA, USA, 13–19 June 2020, pp.4603–4611. New York: IEEE.
20.
YinTZhouXKrähenbühlP.Multimodal virtual point 3D detection. Adv Neural Inf Process Syst2021; 34: 16494–16507.
21.
HuangTLiuZChenX, et al. EPNet: enhancing point features with image semantics for 3D object detection. In: VedaldiABischofHBroxT, et al. (eds) Computer vision - ECCV 2020 16th European conference proceedings. Lecture Notes in Computer Science (LNCS 12360). Cham: Springer, 2020, pp.35–52.
22.
ZhangZShenYLiH, et al. MAFF-Net: filter false positive for 3D vehicle detection with multi-modal adaptive feature fusion. In: IEEE conference on intelligent transportation systems, proceedings, ITSC, Macau, China, 8–12 October 2022, pp.369–376. New York: IEEE.
23.
LiYYuAWMengT, et al. DeepFusion: lidar-camera deep fusion for multi-modal 3D object detection. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, New Orleans, LA, USA, 18–24 June 2022, pp.17161–17170. New York: IEEE.
24.
ChenZLiZZhangS, et al. AutoAlign: pixel-instance feature aggregation for multi-modal 3D object detection. arXiv preprint arXiv:2201.06493, 2022.
25.
HeYPengLYiQ, et al. Multi-scale representation learning for image restoration with state-space model. arXiv preprint arXiv:2408.10145, 2024.
26.
SheenyMde PellegrinEMukherjeeS, et al. Radiate: a radar dataset for automotive perception in bad weather. In: Proceedings - IEEE international conference on robotics and automation, Xi’an, China, 30 May–5 June 2021, pp.5617–5623. New York: IEEE.
27.
PitropovMGarciaDRebelloJ, et al. Canadian adverse driving conditions dataset. Int J Rob Res2020; 40: 681–690.
28.
HahnerMSakaridisCBijelicM, et al. LiDAR snowfall simulation for robust 3D object detection. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, New Orleans, LA, USA, 18–24 June 2022, pp.16343–16353. New York: IEEE.
29.
KilicVHegdeDSindagiV, et al. Lidar Light Scattering Augmentation (LISA): physics-based simulation of adverse weather conditions for 3D object detection. arXiv preprint arXiv:2107.07004, 2021.
30.
HahnerMSakaridisCDaiD, et al. Fog simulation on real LiDAR point clouds for 3D object detection in adverse weather. In: Proceedings of the IEEE international conference on computer vision, Montreal, QC, Canada, 10–17 October 2021, pp.15263–15272. New York: IEEE.
31.
HuMWangSLiB, et al. PENet: towards precise and efficient image guided depth completion. In: Proceedings - IEEE international conference on robotics and automation, Xi’an, China, 30 May–5 June 2021, pp.13656–13662. New York: IEEE.
32.
KongLLiuYLiX, et al. Robo3D: towards robust and reliable 3D perception against corruptions. In: Proceedings of the IEEE international conference on computer vision, Paris, France, 1–6 October 2023, pp.19937–19949. New York: IEEE.
33.
ZhuXZhouHWangT, et al. Cylindrical and asymmetrical 3D convolution networks for LiDAR-based perception. IEEE Trans Pattern Anal Mach Intell2022; 44: 6807–6822.
34.
KimIIMcarthurBKorevaarE. Comparison of laser beam propagation at 785 nm and 1550 nm in fog and haze for optical wireless communications. In: Optical wireless communications III, Boston, MA, USA, 5–8 November 2000, vol. 4214, pp.26–37. Bellingham, WA, USA: SPIE.
35.
KoppalSJ. Lambertian reflectance. In: IkeuchiK (ed.) Computer vision: a reference guide. Boston, MA: Cham: Springer International Publishing, pp.729–731.
36.
DengJShiSLiP, et al. Voxel R-CNN: towards high performance voxel-based 3D object detection. Proc AAAI Conf Artif Intell2021; 35: 1201–1209.
LangAHVoraSCaesarH, et al. Pointpillars: fast encoders for object detection from point clouds. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, Long Beach, CA, USA, 15–20 June 2019, pp.12689–12697. New York: IEEE.
39.
ShiSWangXLiH.PointRCNN: 3D object proposal generation and detection from point cloud. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, Long Beach, CA, USA, 15–20 June 2019, pp.770–779. New York: IEEE.
40.
ShiSWangZShiJ, et al. From points to parts: 3D object detection from point cloud with part-aware and part-aggregation network. IEEE Trans Pattern Anal Mach Intell2021; 43: 2647–2664.
41.
ShiSGuoCJiangL, et al. PV-RCNN: point-voxel feature set abstraction for 3D object detection. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, Seattle, WA, USA, 13–19 June 2020, pp.10526–10535. New York: IEEE.
42.
YangZSunYLiuS, et al. 3DSSD: point-based 3D single stage object detector. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, Seattle, WA, USA, 13–19 June 2020, pp.11037–11045. New York: IEEE.
43.
WuHWenCLiW, et al. Transformation-equivariant 3D object detection for autonomous driving. Proc AAAI Conf Artif Intell2023; 37: 2795–2802.
44.
LiuZWuZTothR.SMOKE: single-stage monocular 3D object detection via keypoint estimation. In: IEEE computer society conference on computer vision and pattern recognition workshops, Seattle, WA, USA, 14–19 June 2020, pp.4289–4298. New York: IEEE.
45.
WangTZhuXPangJ, et al. Probabilistic and geometric depth: detecting objects in perspective. Proc Mach Learn Res2022; 164: 1475–1485.
46.
RukhovichDVorontsovaAKonushinA. ImVoxelNet: image to voxels projection for monocular and multi-view general-purpose 3D object detection. In: Proceedings - 2022 IEEE/CVF winter conference on applications of computer vision, WACV 2022, Waikoloa, HI, USA, 3–8 January 2022, pp.1265–1274. New York: IEEE.
47.
ChenYLiYZhangX, et al. Focal sparse convolutional networks for 3D object detection. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, New Orleans, LA, USA, 18–24 June 2022, pp.5418–5427. New York: IEEE.
48.
LiYQiXChenY, et al. Voxel field fusion for 3D object detection. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, New Orleans, LA, USA, 18–24 June 2022, pp.1110–1119. New York: IEEE.
49.
LiXMaTHouY, et al. LoGoNet: towards accurate 3D object detection with local-to-global cross-modal fusion. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, (CVPR), Vancouver, Canada, 18–22 June 2023, pp.17524–17534. New York: IEEE
