Light Detection And Ranging (LiDAR) sensors can generate a number of sequential 3D point clouds, which are widely deployed in many real-world systems. 3D object detection in point clouds, is one of the most fundamental tasks. Unfortunately, the existing 3D object detection methods degrade in snowy weather, because in that situation the annotated samples are difficult to collect. To solve this issue, we propose a novel GAN-based Snowfall Point-cloud AugmentOR (GANspaor) to generate high-quality synthetic snowfall point clouds as augmentations. The basic idea of GANspaor is to transfer annotated point clouds to snowfall versions by simultaneously learning the global style of real snowfall point clouds and the local details of physics-induced ones. Our framework fuses data-driven and physical modeling methods for rapidly generating data in snowy weather. To evaluate the effectiveness of GANspaor, we employ a number of recent 3D object detection methods and train them by using the synthetic samples of GANspaor as auxiliary augmentations. Moreover, we conduct a comparative analysis of the characteristics of the data distributions of the snowy point clouds synthesized by GANspaor. Experimental results demonstrate that GANspaor can improve the performance of 3D object detection methods compared with other existing snowfall point cloud simulators.
RorizRCabralJGomesT. Automotive LiDAR technology: a survey. IEEE Trans Intell Transp Syst2022; 23(7): 6282–6297.
2.
ShiSWangXLiH. PointRCNN: 3D object proposal generation and detection from point cloud. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR), Long Beach, CA, USA, 15–20 June 2019, pp.770–779. New York: IEEE.
3.
YinTZhouXKrähenbühlP. Center-based 3D object detection and tracking. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR), Nashville, TN, USA, 20–25 June 2021, pp.11779–11788. New York: IEEE.
4.
YangZSunYLiuS, et al. 3DSSD: point-based 3D single stage object detector. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR), Seattle, WA, USA, 13–19 June 2020, pp.11037–11045. New York: IEEE.
DengJShiSLiP, et al. Voxel R-CNN: towards high performance voxel-based 3D object detection. Proc AAAI Conf Artif Intell2021; 35: 1201–1209.
7.
ShiSWangZShiJ, et al. From points to parts: 3D object detection from point cloud with part-aware and part-aggregation network. IEEE Trans Pattern Anal Mach Intell2021; 43(8): 2647–2664.
8.
WangHTaoLPengY, et al. Voxel RCNN-HA: a point cloud multi-object detection algorithm with hybrid anchors for autonomous driving. IEEE Trans Transp Electrif2023; 10: 7286–7296.
9.
PiroliADallabettaVWalessaM, et al. Robust 3D object detection in cold weather conditions. In: IEEE intelligent vehicles symposium (IV), Aachen, Germany, 4–9 June 2022 pp.287–294. New York: IEEE.
10.
LinJYinHYanJ, et al. Improved 3D object detector under snowfall weather condition based on LiDAR point cloud. IEEE Sens J2022; 22(16): 16276–16292.
11.
CharronNPhillipsSWaslanderSL. De-noising of lidar point clouds corrupted by snowfall. In: 15th conference on computer and robot vision (CRV), Toronto, ON, Canada, 8–10May2018, pp.254–261. New York: IEEE.
12.
BaeGKimBAhnS, et al. SLiDE: self-supervised LiDAR de-snowing through reconstruction difficulty. In: AvidanSBrostowGCisséM, et al. (eds) European conference on computer vision (ECCV). Cham: Springer, 2022, pp.283–300.
13.
ZhangYCarballoAYangH, et al. Perception and sensing for autonomous vehicles under adverse weather conditions: a survey. ISPRS J Photogramm Remote Sens2023; 196: 146–177.
14.
FangJZhouDYanF, et al. Augmented LiDAR simulator for autonomous driving. IEEE Robot Autom Lett2020; 5(2): 1931–1938.
15.
ZhaoJLiYZhuB, et al. Method and applications of lidar modeling for virtual testing of intelligent vehicles. IEEE Trans Intell Transp Syst2021; 22(5): 2990–3000.
16.
KilicVHegdeDSindagiV, et al. Lidar light scattering augmentation (LISA): physics-based simulation of adverse weather conditions for 3D object detection. arXiv preprint arXiv: 210707004, 2021.
17.
HahnerMSakaridisCBijelicM, et al. LiDAR snowfall simulation for robust 3D object detection. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR), New Orleans, LA, USA, 18–24 June 2022, pp.16343–16353. New York: IEEE.
18.
LiCLZaheerMZhangY, et al. Point cloud GAN. arXiv preprint arXiv: 181005795, 2018.
19.
NakashimaKKurazumeR. Learning to drop points for LiDAR scan synthesis. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Prague, Czech Republic, 27 September–1 October 2021, pp.222–229. New York: IEEE.
20.
LuoSHuW. Diffusion probabilistic models for 3D point cloud generation. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR), Nashville, TN, USA, 20–25 June 2021, pp.2836–2844. New York: IEEE.
21.
GuillardBVempralaSGuptaJK, et al. Learning to simulate realistic LiDARs. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Kyoto, Japan, 23–27 October 2022, pp.8173–8180. New York: IEEE.
22.
LiWPanCZhangR, et al. AADS: augmented autonomous driving simulation using data-driven algorithms. Sci Robot2019; 4(28): eaaw0863.
23.
YangGHuangXHaoZ, et al. PointFlow: 3D point cloud generation with continuous normalizing flows. In: IEEE/CVF international conference on computer vision (ICCV), Seoul, South Korea, 27 October–2 November 2019, pp.4540–4549. New York: IEEE.
24.
CacciaLVan HoofHCourvilleA, et al. Deep generative modeling of LiDAR data. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Macau, China, 3–8 November 2019, pp.5034–5040. New York: IEEE.
25.
ManivasagamSWangSWongK, et al. LiDARsim: realistic LiDAR simulation by leveraging the real world. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR), Seattle, WA, USA, 13–19 June 2020, pp.11164–11173. New York: IEEE.
26.
RonnebergerOFischerPBroxT. U-Net: convolutional networks for biomedical image segmentation. In: NavabNHorneggerJWellsW, et al. (eds) Medical image computing and computer-assisted intervention (MICCAI). Cham: Springer, 2015, pp.234–241.
27.
VacekPJašekOZimmermannK, et al. Learning to predict lidar intensities. IEEE Trans Intell Transp Syst2022; 23(4): 3556–3564.
28.
XiaoAHuangJGuanD, et al. Transfer learning from synthetic to real LiDAR point cloud for semantic segmentation. Proc AAAI Conf Artif Intell2022; 36: 2795–2803.
29.
BijelicMGruberTMannanF, et al. Seeing through fog without seeing fog: deep multimodal sensor fusion in unseen adverse weather. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR), Seattle, WA, USA, 13–19 June 2020, pp.11679–11689. New York: IEEE.
30.
GeigerALenzPUrtasunR. Are we ready for autonomous driving? The KITTI vision benchmark suite. In: IEEE/CVF conference on computer vision and pattern recognition (CVPR), Providence, RI, USA, 16–21 June 2012, pp.3354–3361. New York: IEEE.
