SLAM (simultaneous localization and mapping) is becoming a significant technology in driverless vehicles nowadays. Sometimes the optical flow method can be used to track feature points to reduce the amount of calculation. But the traditional optical flow method is based on the assumption that the luminosity will not change and has high requirements for the environment. Therefore, an optical flow calculation method based on neural network called Optical-Net is proposed in this paper to match and track feature and corner points. We use the multi-head attention module to extract and fuse the features of different scales. The training results show that the Optical-Net designed in this study effectively improves the robustness and accuracy of optical flow estimation when the object has large displacement and small displacement. Meanwhile, to address the problem of reduced matching accuracy during video input due to dynamic blurring, a video dynamic deblurring algorithm based on the one-dimensional Wineman filter is adopted to preprocess the input video. Finally, we replaced the tracking thread in traditional SLAM. The performance of the improved algorithm is verified by experiments on our own driverless car.
MohamedMARashwanHAMertschingB, et al. Illumination-robust optical flow using a local directional pattern. IEEE Trans Circuits Syst Video Technol2014; 24: 1499–1508.
3.
ChenJCaiZLaiJ, et al. A filtering-based framework for optical flow estimation. IEEE Trans Circuits Syst Video Technol2019; 29(5): 1350–1364.
4.
BarronJLFleetDJBeaucheminSS. Performance of optical flow techniques. Int J Comput Vis1994; 12(1): 43–77.
5.
NagelHHEnkelmannW. An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Trans Pattern Anal Mach Intell1986; 8: 565–593.
6.
BroxTBruhnAPapenbergN, et al. High accuracy optical flow estimation based on a theory for warping. In: Proceedings of the 8th European conference on computer vision, Prague, Czech Republic, 2004, vol. 5, pp.25–36.
7.
BroxTMalikJ. Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans Pattern Anal Mach Intell2011; 33(3): 500–513.
8.
KimYHKakAC. Error analysis of robust optical flow estimation by least median of squares methods for the varying illumination model. IEEE Trans Pattern Anal Mach Intell2006; 28(9): 1418–1435.
9.
YuanYZhaoYWangQ. Action recognition using spatial-optical data organization and sequential learning framework. Neurocomputing2018; 315(11): 221–333.
10.
DosovitskiyAFischerPIlgE, et al. FlowNet: learning optical flow with convolutional networks. In: 2015 IEEE international conference on computer vision (ICCV), Santiago, Chile, 2015, vol. 9, pp.2758–2766.
11.
IlgEMayerNSaikiaT, et al. FlowNet 2.0: evolution of optical flow estimation with deep networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Venice, Italy, 2017, vol. 7, pp.1647–1655.
12.
RanjanABlackMJ. Optical flow estimation using a spatial pyramid network. In: IEEE conference on computer vision and pattern recognition (CVPR), Venice, Italy, 2017, vol. 7, pp.2720–2729.
13.
HuiTWTangXLoyCC. LiteFlowNet: a lightweight convolutional neural network for optical flow estimation. In: IEEE conference on computer vision and pattern recognition (CVPR), Salt Lake City, USA, 2018, vol. 3, pp.223–235.
14.
HuiTWTangXLoyCC. A lightweight optical flow CNN – revisiting data fidelity and regularization. IEEE Trans Pattern Anal Mach Intell2020; 3: 2048–2051.
15.
AhmadiAPatrasI. Unsupervised convolutional neural networks for motion estimation. In: IEEE international conference on image processing (ICIP), Phoenix, Arizona, USA, 2016, vol. 9, pp.1629–1633.
16.
YuJJHarleyAWDerpanisKG. Back to basics: unsupervised learning of optical flow via brightness constancy and motion smoothness. In: European conference on computer vision, Amsterdam, Netherlands, 2016, vol. 1, pp.3–10.
17.
RenZYanJNiB, et al. Unsupervised deep learning for optical flow estimation. In: Proceedings of the AAAI conference on artificial intelligence, San Francisco, California, USA, 2017, vol. 2, pp.1–7.
18.
ZhuYNewsamS. Learning optical flow via dilated networks and occlusion reasoning. In: Proceedings of the 25th IEEE international conference on image processing (ICIP), Athens, Greece, 2018, vol. 8, pp.3333–3337.
19.
WangYYangYYangZ, et al. Occlusion aware unsupervised learning of optical flow. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), Salt Lake City, USA, 2018, vol. 6, pp.4884–4893.
20.
WuPChenSMetaxasD. MotionNet: joint perception and motion prediction for autonomous driving based on bird’s eye view maps. arXiv:2003.06754, 2020.
21.
MeisterSHurJRothS. UnFlow: unsupervised learning of optical flow with a bidirectional census loss. arXiv:1711.07837, 2017.
22.
DingMWangZZhouB, et al. Every Frame counts: joint learning of video segmentation and optical flow. In: Proceedings of the AAAI conference on artificial intelligence, New York, USA, 2020, vol. 34, pp.10713–10720.
23.
ShiLSunHZhangB. Restoration of motion-blurred images. Opt Precis Eng2007; 15(5): 767–772.
24.
FuC-YLiuWRangaA, et al. DSSD: deconvolutional single shot detector. arXiv:1701.06659, 2017.
25.
SimonyanKZissermanA. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014.
26.
MayerNIlgEHausserP, et al. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In: IEEE conference on computer vision and pattern recognition (CVPR), Las Vegas, NV, USA, 2016, vol. 6, pp.4040–4048.
27.
SunDYangXLiuM-Y, et al. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), Salt Lake City, USA, 2018, vol. 6, pp.8934–8943.
28.
JiaYShelhamerEDonahueJ, et al. Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM international conference on multimedia, Orlando, FL, USA, 2014, vol. 2, pp.675–678.
29.
CamposCElviraRRodríguezJJG, et al. ORB-SLAM3: an accurate open-source library for visual, visual-inertial and multi-map SLAM. arXiv:2007.11898, 2020.
