In recent years, Sound Event Classification (SEC) is a remarkable field of computer audition research. The abilities to automatically capture and classify sound events are meaningful for expanding the auditory ability of autonomous vehicle in a complex driving environment. The Autonomous vehicles will encounter some special road conditions during driving, such as large truck occlusion, tunnels, and buildings that hinder radar detection. Therefore, the perception capabilities of on-board lidar, vision, and other sensors are impacted, resulting in a great threat to the safe driving of the car. At this time, auditory environment perception can provide auditory dimension information for the vehicle. However, with the increase of noise level, the SEC task of autonomous vehicle becomes seriously difficult. Traditional machine learning methods show weak robustness in the highly noisy environment. In this work, an autonomous vehicle Sound Event Classification (AVSEC) framework based on Spectrogram Image Features (SIFs) and Convolutional Neural Networks (CNN) algorithm is proposed to solve the SEC question. Further on, the ambulance, bus, civil defense siren, fire truck, screaming, etc. are taken as the research traffic sounds. Traffic Sound Datasets (TSD) are collected, augmented and customized. Then, directly connected channel (DCC) between convolutional blocks and Convolutional Block Attention Module (CBAM) are designed to optimize the neural network. Testing results show that the AVSEC system achieves an accuracy of 97.18%. Furthermore, the results of AVSEC and other machine learning methods are compared, which shows that the combination of SIFs and AVSEC-net provides better noise robustness and classification accuracy in AVSEC task. The sound classification algorithm designed in this paper, as an advanced auxiliary driving system for sound detection, can identify the sound types in the current traffic environment of the Autonomous vehicles, that is, the traffic scene category. The vehicle computer can remind the driver to pay attention to driving safety through the above sound types, so as to improve the safety of the driver and the vehicle.
LiLOtaKDongM. Humanlike driving: empirical decision-making system for Autonomous Vehicles. IEEE Trans Vehicular Technol2018; 67(8): 6814–6823.
2.
DasSRMishraS. A review on vision based control of autonomous vehicles using artificial intelligence techniques. In: 2019 International conference on information technology (ICIT), Bhubaneswar, 2019, pp. 500–504. New York: IEEE.
3.
KuuttiSBowdenRJinY, et al. A survey of deep learning applications to autonomous vehicle control. IEEE Trans Intell Transp Syst2021; 22(2): 712–733.
4.
ShiJZhaoLWangX, et al. A novel deep Q-learning-based air-assisted vehicular caching scheme for safe autonomous driving. IEEE Trans Intell Transp Syst2021; 22(7): 4348–4358.
5.
OsórioAPintoA. Information, uncertainty and the manipulability of artificial intelligence autonomous vehicles systems. Int J Hum Comput Stud2019; 130: 40–46.
6.
YangPDuanDChenC, et al. Multi-sensor multi-vehicle (MSMV) localization and mobility tracking for autonomous driving. IEEE Trans Vehicular Technol2020; 69(12): 14355–14364.
7.
MwaffoVCurryJSIudiceFL, et al. Pause-and-Go self-balancing formation control of autonomous vehicles using vision and ultrasound sensors. IEEE Trans Control Syst Technol2021; 29(6): 2299–2311.
8.
YiCZhangKPengN. A multi-sensor fusion and object tracking algorithm for self-driving vehicles. Proc IMechE, Part D: J Automobile Engineering2019; 233(9): 2293–2300.
9.
Van BrummelenJO’BrienMGruyerD, et al. Autonomous vehicle perception: the technology of today and tomorrow. Transp Res Part C Emerg Technol2018; 89: 384–406.
10.
GloaguenJRCanALagrangeM, et al. Road traffic sound level estimation from realistic urban sound mixtures by non-negative matrix factorization. Appl Acoust2019; 143: 229–238.
11.
KongQXuYSobierajI, et al. Sound event detection and time–frequency segmentation from weakly labelled data. IEEE Trans Audio Speech Lang Process2019; 27(4): 777–787.
12.
BroughtonGMajerFRoučekT, et al. Learning to see through the haze: multi-sensor learning-fusion system for vulnerable traffic participant detection in Fog. Robot Auton Syst2021; 136: 1–12.
13.
MesarosADimentAElizaldeB, et al. Sound event detection in the DCASE 2017 Challenge. IEEE Trans Audio Speech Lang Process2019; 27(6): 992–1006.
14.
KapilPEkbalA. A deep neural network based multi-task learning approach to hate speech detection. Knowl Based Syst2020; 210: 106458.
15.
KongQXuYWangW, et al. Sound event detection of weakly labelled data with CNN-transformer and automatic threshold optimization. IEEE Trans Audio Speech Lang Process2020; 28: 2450–2460.
16.
HilalARSayedelahlATabibiazarA, et al. A distributed sensor management for large-scale IoT indoor acoustic surveillance. Future Gener Comput Syst2018; 86: 1170–1184.
17.
TuncerTDoganSAcharyaUR. Automated accurate speech emotion recognition system using twine shuffle pattern and iterative neighborhood component analysis techniques. Knowl Based Syst2021; 211: 106547.
18.
SongHDengSHanJ. Exploring inter-node relations in CNNs for environmental sound classification. IEEE Signal Process Lett2022; 29: 154–158.
19.
XiaXTogneriRSohelF, et al. Auxiliary classifier generative adversarial network with soft labels in imbalanced acoustic event detection. IEEE Trans Multimedia2019; 21(6): 1359–1371.
20.
VesperiniFGabrielliLPrincipiE, et al. Polyphonic sound event detection by using Capsule Neural Networks. IEEE J Sel Top Signal Process2019; 13(2): 310–322.
21.
WangCYWangJCSantosoA, et al. Sound event recognition using auditory-receptive-field binary pattern and hierarchical-diving deep belief network. IEEE Trans Audio Speech Lang Process2018; 26(8): 1336–1351.
DengMMengTCaoJ, et al. Heart sound classification based on improved MFCC features and convolutional recurrent neural networks. Neural Netw2020; 130: 22–32.
24.
LiLMaLJiaoL, et al. Complex Contourlet-cnn for polarimetric SAR image classification. Pattern Recognit2020; 100: 107110.
25.
LiSZhangBFeiL, et al. Joint discriminative feature learning for multimodal finger recognition. Pattern Recognit2021; 111: 107704.
26.
SangwanPDeshwalDDahiyaN. Performance of a language identification system using hybrid features and ANN learning algorithms. Appl Acoust2021; 175: 107815.
27.
WangZRDuJWangJM. Writer-aware CNN for parsimonious HMM-based offline handwritten Chinese text recognition. Pattern Recognit2020; 100: 107102.
28.
ItoHYoshidaMYanoY, et al. Comparison of ESD damage test for common-mode chokes with ESD gun and TLP-HMM. IEEE Lett Electromagn Compat Pract Appl2023; 5(1): 10–15.
29.
XueJZhengTHanJ. Exploring attention mechanisms based on summary information for end-to-end automatic speech recognition. Neurocomputing2021; 465: 514–524.
30.
ZhangHDongYLiJ, et al. Dynamic time warping under product quantization, with applications to time-series data similarity search. IEEE Internet Things J2022; 9(14): 11814–11826.
31.
HuangSKangZXuZ, et al. Robust deep k-means: an effective and simple method for data clustering. Pattern Recognit2021; 117: 107996.
32.
DinkelHWuMYuK. Towards duration robust weakly supervised sound event detection. IEEE Trans Audio Speech Lang Process2021; 29: 887–900.
33.
CakirEParascandoloGHeittolaT, et al. Convolutional recurrent neural networks for Polyphonic Sound Event Detection. IEEE Trans Audio Speech Lang Process2017; 25(6): 1291–1303.
34.
EspinosaRPonceHGutiérrezS. Click-event sound detection in automotive industry using machine/deep learning. Appl Soft Comput2021; 108: 107465.
35.
YangMPengLLiuL, et al. LCSED: A low complexity CNN based SED model for IoT devices. Neurocomputing2022; 485: 155–165.
ManhertzGBereczkyA. STFT spectrogram based hybrid evaluation method for rotating machine transient vibration analysis. Mech Syst Signal Process2021; 154: 107583.
38.
MekhaPTeeyasuksaetNSompowloyT, et al. Honey bee sound classification using spectrogram image features. In: 2022 Joint International Conference on Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT & NCON), Chiang Rai, Thailand, 2022, pp. 205–209.
39.
WangPFanEWangP. Comparative analysis of image classification algorithms based on traditional machine learning and deep learning. Pattern Recognit Lett2021; 141: 61–67.
40.
BernardiniMRomeoLMisericordiaP, et al. Discovering the Type 2 diabetes in electronic health records using the sparse balanced support vector machine. IEEE J Biomed Health Inform2020; 24: 235–246.
41.
YanCLuoMLiuH, et al. Top-k multi-class SVM using multiple features. Inf Sci2018; 432: 479–494.
42.
AvolioMFuduliA. A semiproximal support vector machine approach for binary multiple instance learning. IEEE Trans Neural Netw Learn Syst2021; 32(8): 3566–3577.
43.
DingCTaoD. Trunk-branch ensemble Convolutional Neural Networks for video-based face recognition. IEEE Trans Pattern Anal Mach Intell2018; 40(4): 1002–1014.
44.
LeeHEumSKwonH. ME R-CNN: multi-expert R-CNN for Object Detection. IEEE Trans Image Process2020; 29: 1030–1044.
45.
ArandjelovicRGronatPToriiA, et al. NetVLAD: CNN Architecture for weakly supervised place recognition. IEEE Trans Pattern Anal Mach Intell2018; 40(6): 1437–1451.
46.
WangXBYingTTianW. Spectrum representation based on STFT. In: 2020 13th international congress on image and signal processing, biomedical engineering and informatics (CISP-BMEI), Chengdu, China, 2020, pp. 435–438.
47.
GuoMKuenzleB. Obtaining narrow transition region in STFT domain processing using subband filters. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 2019, pp. 970–974.
48.
KiranaMCLubisMZAmaliaED. The effect of sound manipulation to know response rate in autism children using FFT. In: 2018 International conference on applied engineering (ICAE), Batam, Indonesia, 2018, pp. 1–5.
XuJLiZDuB, et al. Reluplex made more practical: Leaky ReLU. In: 2020 IEEE symposium on computers and communications (ISCC), Rennes, France, 2020, pp. 1–7.
51.
XueYZhuXZhengC, et al. Lactating sow postures recognition from depth image of videos based on improved faster R-CNN. Trans Chin Soc Agric Eng2018; 34: 189–196.
52.
XuKFZhangYWZangJ, et al. Integration of vibration control and energy harvesting for whole-spacecraft: Experiments and theory. Mech Syst Signal Process2021; 161: 107956.
53.
ZhuQHZhengTRHanJQ. Acoustic event classification based on binary spectrogram in driving environment. In: Proceedings of national conference on man-machine speech communication, 2013.
54.
HanJQKongHY. Robust acoustic event classification method in Driving Environment. Harbin, China: Harbin Institute of Technology, 2013.
55.
PeiXZZhengTRHanJQ. An acoustic feature extraction approach based on frequency selectivity of human auditory under driving noisy environments. Intell Comput Appl2015; 5(3): 16–18.
56.
MaoJLiLCLiuK, et al. Robust acoustic feature extraction algorithm for driving environment of Driverless cars. China J Highw Transp2019; 32(6): 169–175.
57.
WangRPLiRRChenDL, et al. Low signal to noise ratio traffic environment acoustic recognition based on improved wavelet packet denoising and Mel cepstrum coefficient. Sci Technol Eng2019; 19(36): 290–295.
