Visual semantic segmentation is a key technology to realize scene understanding for autonomous driving and its accuracy is affected by the light changes in images. This paper proposes a novel multi-exposure fusion approach to visual semantic enhancement of autonomous driving. Firstly, a multi-exposure image sequence is aligned to construct a stable image input. Secondly, high contrast regions of multi-exposure image sequences are evaluated by context aggregation network (CAN) to predict image weight map. Finally, the high-quality image is generated by weighted fusion of multi-exposure image sequences. The proposed approach is validated by using Cityscapes’ HDR dataset and real environment data. The experimental results show that the proposed method effectively restores lost features in the light changing images and enhances accuracy of subsequent semantic segmentation.
LvZHuangXLiangY, et al. An improved efficient model for structure-aware lane detection of unmanned vehicles. Proc IMechE, Part D: J Automobile Engineering2021; 235(9): 2496–2508.
2.
TonguçGAkçayİHGürbüzH. Improvement of the visual warning system for various driving and road conditions in road transportation. Proc IMechE, Part D: J Automobile Engineering2018; 232(2): 175–187.
3.
CaoWHuangXShuF. Location recognition of unmanned vehicles based on visual semantic information and geometric distribution. Proc IMechE, Part D: J Automobile Engineering2021; 235(2–3): 552–563.
4.
YiCZhangKPengN. A multi-sensor fusion and object tracking algorithm for self-driving vehicles. Proc IMechE, Part D: J Automobile Engineering2019; 233(9): 2293–2300.
5.
RomeraEAlvarezJMBergasaLM, et al. ERFNet: Efficient residual factorized convNet for real-time semantic segmentation. IEEE Trans Intell Transp Syst2018; 19(1): 263–272.
6.
GirshickRDonahueJDarrellT, et al. Rich feature hierarchies for accurate object detection and semantic segmentation. In: IEEE Conference on computer vision and pattern recognition, Columbus, OH, 23–28 June 2014, pp.580–587.
7.
ChenLCPapandreouGKokkinosI, et al. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans Pattern Anal Mach Intell2018; 40(4): 834–848.
8.
ChenLZhuYPapandreouG, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation. In: 15th European conference on computer vision, Munich, Germany, 8–14 September 2018, pp.833–851.
9.
PaszkeAChaurasiaAKimS. ENet: A deep neural network architecture for real-time semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, 2016.
10.
BadrinarayananVKendallACipollaR. SegNet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pattern Anal Mach Intell2017; 39(12): 2481–2495.
11.
TungFChenJMengL, et al. The raincouver scene parsing benchmark for self-driving in adverse weather and at night. IEEE Robot Autom Lett2017; 2(4): 2188–2193.
12.
ZengHSuHZhangT. High dynamic range image acquisition based on multiplex cameras. Proceedings of SPIE: Young Scientists Forum2017; 10710N: 1–8.
13.
ChenGWangHChenK. A Survey of the four pillars for small object detection: multiscale representation, contextual information, super-resolution, and region proposal. IEEE Trans Syst Man Cybern Syst2020; 99:1–18.
14.
ChenGCaoHConradtJ. Event-based neuromorphic vision for autonomous driving: a paradigm shift for bio-inspired visual sensing and perception. IEEE Signal Process Mag2020; 37(4): 34–49.
15.
ValadaAMohanRBurgardW. Self-supervised model adaptation for multimodal semantic segmentation. Int J Comput Vis2020; 128(5): 1239–1285.
16.
LevinkovE. Scene segmentation in adverse vision conditions. Pattern Recognit2014; 8753:750–756.
17.
RomeraEBergasaLYangK, et al. Bridging the day and night domain gap for semantic segmentation. In: 30th IEEE intelligent vehicles symposium, Paris, France, 9–12 June 2019, pp.1312–1318.
18.
TakikawaTAcunaDJampaniV. Gated-SCNN: Gated shape CNNs for semantic segmentation. In: International conference on computer vision, Seoul, 27–28 October 2019, pp.5228–5237.
19.
LiXLiXZhangL. Improving semantic segmentation via decoupled body and edge supervision. In: ECCV 2020: 16th European conference, Glasgow, UK, 23–28 August 2020.
20.
HaQWatanabeKKarasawaT. MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes. In: IEEE/RSJ International conference on intelligent robots and systems, Vancouver, BC, Canada, 24–28 September 2017, pp.5108–5115.
21.
PfeufferADietmayerK. Robust semantic segmentation in adverse weather conditions by means of sensor data fusion. In: 22nd International conference on information fusion, Ottawa, ON, Canada, 2–5 July 2019.
22.
GaldranA. Image dehazing by artificial multiple-exposure image fusion. Signal Process2018; 149:135–147.
23.
LiuQLeungH. Variable augmented neural network for decolorization and multi-exposure fusion. Inf Fusion2019; 46:114–127.
24.
KePJungCFangY. Perceptual multi-exposure image fusion with overall image quality index and local saturation. Multimed Syst2017; 23(2): 239–250.
25.
LiuYWangZ. Dense SIFT for ghost-free multi-exposure fusion. J Vis Commun Image Represent2015; 31:208–224.
26.
KinoshitaYShiotaSKiyaH. Multi-exposure image fusion based on exposure compensation. In: IEEE International conference on acoustics, speech, & signal processing, Calgary, AB, Canada, 15–20 April 2018, pp.1388–1392.
XingLCaiLZengH, et al. A multi-scale contrast-based image quality assessment model for multi-exposure image fusion. Signal Process2018; 145:233–240.
29.
LiXLiFZhuoL, et al. Layered-based exposure fusion algorithm. IET Image Process2013; 7(7): 701–711.
30.
ShaoMHuM. Parallel feature based calibration method for a trinocular vision sensor. Opt Express2020; 28(14): 20573–20586.
31.
HayatNImranM. Ghost-free multi exposure image fusion technique using dense SIFT descriptor and guided filter. J Vis Commun Image Represent2019; 62:295–308.
32.
LeungTJiaYSukthankarR, et al. MatchNet: unifying feature and metric learning for patch-based matching. In: IEEE computer vision & pattern recognition, Boston, MA, 7–12 June 2015, pp.3279–3286.
33.
YanQGongDShiQ. Attention-guided network for ghost-free high dynamic range imaging. In: IEEE computer vision and pattern recognition, Long Beach, CA, 15–20 June 2019, pp.1751–1760.
34.
YanQZhuYZhouY, et al. Enhancing image visuality by multi-exposure fusion. Pattern Recognit Lett2019; 127:66–75.
35.
ChenQJiaXKoltunV. Fast image processing with fully-convolutional networks. In: IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017, pp.2516–2525.
36.
DingH. Context contrasted feature and gated multi-scale aggregation for scene segmentation. In: IEEE/CVF conference on computer vision & pattern recognition, Salt Lake City, UT, 18–23 June 2018, pp.2393–2402.
37.
MaKDuanmuZZhuH, et al. Deep guided learning for fast multi-exposure image fusion. IEEE Trans Image Process2020; 29:2808–2819.
MariusCMohamedOSebastianR. The Cityscapes dataset for semantic urban scene understanding. In: IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, NV, 27–30 June 2016 pp.3213–3223.
40.
PrabhakarKRSrikarVSBabuRV. DeepFuse: a deep unsupervised approach for exposure fusion with extreme exposure image pairs. In: IEEE international conference on computer vision, Venice, Italy, 22–29 October 2017, pp.4724–4732.
41.
FuntBShiL. The rehabilitation of MaxRGB. Color Imag Conf2010; 10:145–154.
