Semantic segmentation–aided visual odometry for urban autonomous driving

Abstract

Visual odometry plays an important role in urban autonomous driving cars. Feature-based visual odometry methods sample the candidates randomly from all available feature points, while alignment-based visual odometry methods take all pixels into account. These methods hold an assumption that quantitative majority of candidate visual cues could represent the truth of motions. But in real urban traffic scenes, this assumption could be broken by lots of dynamic traffic participants. Big trucks or buses may occupy the main image parts of a front-view monocular camera and result in wrong visual odometry estimation. Finding available visual cues that could represent real motion is the most important and hardest step for visual odometry in the dynamic environment. Semantic attributes of pixels could be considered as a more reasonable factor for candidate selection in that case. This article analyzed the availability of all visual cues with the help of pixel-level semantic information and proposed a new visual odometry method that combines feature-based and alignment-based visual odometry methods with one optimization pipeline. The proposed method was compared with three open-source visual odometry algorithms on Kitti benchmark data sets and our own data set. Experimental results confirmed that the new approach provided effective improvement both on accurate and robustness in the complex dynamic scenes.

Introduction

Visual odometry (VO) is the most important part of visual simultaneous location and mapping (V-SLAM) algorithm and has already been widely used in the optical mouses, small mobile robot, and unmanned aerial vehicles (UAVs). As a kind of relative affordable and lightweight solution, VO plays a more and more important role in visual-based navigation system for autonomous driving cars.

The VO term had been first used by Nistér in 2004, ¹ but the relevant researches in this area had been focused over 30 years. ^2,3 VO uses camera as main sensor and takes the current image frames as input and compares with previous frame, then estimates camera’s pose transformation and the trajectory like wheel odometry does. ² Moravec proposed first visual motion estimation pipeline and used it for NASA Mars rover in 1980. ⁴ Nistér found efficient five-point algorithm and built the first real-time VO pipeline. ¹ A common VO pipeline included four steps: capture, matching, estimation, and optimization or filtering. The capture step grabbed and rectified the images taken from cameras. The matching step computed point-wise or patches-wise correspondences. It could be achieved by a feature point-level matching ⁵ or directly used raw pixel/subregion values for patches alignment. ^6,7 Optical flow (OF) ⁸ and tracking method ⁹ could also be integrated in this step and reduce the computation cost of feature extraction and matching. The estimation step often solved a perspective from n points (PnP) problem to recover camera pose transformation and 3-D points’ structure, ¹⁰ and finally optimization or filtering step recovered the part or whole trajectory a local/global optimization process or a filter method. ^{11 –13}

Most of the VO methods worked fine in static indoor scene. However, in the outdoor environment, especially in dynamic urban scenes, too many factors impacted accuracy and availability of VO. Lots of moving objects were the biggest challenges. High-level object recognition or segmentation methods could provide semantic information for better environment understanding. This could be done by cameras only or by fusion multiple sensors. ^{14 –19} But some distance sensors, like sonar and lidar, needed accumulated multiple data frames for effective recognition, ²⁰ which limited their robustness in dynamic environments. Buczko and Willert present a feature-adaptive scaling method for outliers removal. ²¹ Engel et al. proposed a direct sparse odometry (DSO) approach that jointly optimizes the full likelihood for all involved model parameters. ²² These methods still tried to overcome the problem in measurement level. Recently, deep learning had been used successful in object detection and image semantic segmentation ²³ and spatial semantics learning. ²⁴ These semantic information could provide more causal factors for visual motion estimation and helped to improve robustness in complex environment. ²⁵ Mohanty et al. proposed a deep VO method that estimated the odometry vectors between any arbitrary image pair by a trained convolutional neural network (CNN). ²⁶ These efforts could provide a better way to look inside how VO using features or pixels and make evaluation method of VO toward the way that humans could easily understand.

This article focused on dynamic scene VO problem for the urban autonomous driving cars. A robustness VO system could reflect a kind of cognitive process how to understand dynamic scenes correctly and discover the real motions from not only low pixel-level matching but also high-level semantic understanding. This work analyzed the accurateness and robustness impacts of different semantic segmentations from a statistical point of view. Then a deep learning neural network was used for preprocessing pixel-level semantics. These semantic information were used to select reasonable visual cues and remove outliers in matching step. After that, a new VO pipeline was provided to combine feature-based method and alignment-based VO method. The contributions of this article were a novel semantic-aided probabilistic model for outliers removal and alignment patches selection in dynamic scenes and a new feature-based and alignment-based combined VO pipeline.

This article was organized as follows: In section “Related work,” we reviewed the relative VO works. In section “System overview,” we introduced the semantic segmentation by a deep learning network and described our algorithm model. In section “Experimental results,” we evaluate the accuracy and robustness for the three different models on a Kitti benchmark data set and our own real-world data sets. Finally, in section “Conclusions,” we concluded the method and lined out future work.

Related work

A lot of efforts and researches had been focused on developing usable VO systems which show successful applications. Parallel tracking and mapping (PTAM) ²⁷ was a first feature-based VO method running in real time. It had two parallelized threads computing motion estimation and mapping, respectively. PTAM ran an efficient bundle adjustment (BA) on all keyframes, which limits it could only be used in small environment. Civera et al. proposed EKF-based method using one-point random sample consensus (RANSAC) and reduced the size of the data subset to instantiate a hypothesis to one point. ²⁸ Dense tracking and mapping (DTAM) ¹³ was a typical direct method and computed pose transformation by whole image alignment on a depth map. Semi-direct visual odometry (SVO) algorithm proposed by Forster et al. ⁶ used a sparse model-based image alignment algorithm for motion estimation, which tracks some corner points and uses the 4 × 4 patches around them for direct alignment. Geiger et al. presented VISO ²⁹ to compute six-DOF motion of a moving stereo/monocular camera and tested it in urban scene data set Kitti. ³⁰ DSO puts intensity resident, exposure time, attenuation, and irradiance in one energy function and optimized motion estimation, geometrical, and photometric calibration in a joint framework simultaneously. A lot of V-SLAM system contained VO part. ORB_SLAM2 had a feature-based VO thread and a loop closure thread ³¹ and could compute the camera trajectory real time in a wide variety of environments.

Most of the successful VO or V-SLAM systems focused on static environment. In dynamic scene, removing outliers became a more necessary step for the accurate motion estimation. Choosing correct pixels or keypoints, which represents the real camera motion, would help to improve VO robustness in the complex scene. As a common tool, RANSAC-based method is often used to reject outliers. ³² Given an expected rate of success P, the necessary iteration times of RANSAC N could be computed by the number of data points s and outliers rate ε

$N=\frac{\text{log}(1-p)}{\text{log}(1-{(1-\epsilon )}^s)}$

1

With higher ε, N could reach thousand times in many cases. Preemptive RANSAC ³³ tried to fix N using motion hypotheses. The progressive sample consensus ³⁴ computed similarities of the correspondences for ranking and sampling to increase the convergence process. Using RANSAC-based method to reject outliers has an assumption that the noise samples are far less than the correct samples. They only tried to find a probabilistic stable set of inliers by growing iterations. In dynamic urban traffic, this assumption would not be always hold. A front-view monocular camera with limited field-of-view (FOV) lens could suffer from the occlusions and disturbances from the moving vehicle nearby. RANSAC-based methods could not guarantee to choose the pixels or keypoints that belong to static object in real world and could lead to hard data association for motion estimation in dynamic complex scene. Recently, Buczko and Willert proposed a normalized reprojection error method ²¹ which shows an increased error for outliers and a constant offset for inlier. But this method focused on high-speed scene with an assumption of small rotation, longitudinal motion only and didn’t consider semantic attributions of all points.

A robust VO in dynamic scenes should have the ability that distinguishes static object and moving traffic participants. In this semantic level, scene understanding could help VO by a higher level visual cues selection process. Civera et al. proposed a semantic SLAM using a monocular extended kalman filter (EKF) SLAM and inserted 3-D objects into geometric map. ³⁵ Anand et al. trained a graphical model for contextually guided semantic labeling. ³⁶ Yang et al. proposed a method to solve navigation and vehicle distance estimation simultaneously and used dynamic object tracking to divide view field of camera into static and dynamic parts. ³⁷ This method would be hard to distinguish a moving object which has the same speed to observer. Geiger et al. provided a probabilistic model combining semantic scene labels, occupancy grid, vanishing points, and moving object tracklets to discover the intersection model. ³⁸ In his work, the semantic labels provide a probability of label class given a road layout. In that work, the labels are three simple classes, foreground, background, and sky, and contribute little for motion estimation. Pop-up SLAM proposed by Yang used pop-up model and large-scale direct monocular SLAM (LSD) ³⁹ to predict depth and demonstrated that scene understanding improves state estimation and dense mapping. ⁴⁰

System overview

This section introduced semantic segmentation–aided VO (SAVO) method. The whole pipeline is shown in Figure 1. The VO system took monocular RGB image sequence I ₀, ..., I _k−1, I_k , ...,I_n as input, followed by a feature detection pipeline and deep learning segmentation network. The feature points in current frame were computed by point-wise matching to previous image and weighted by the segmentation category labels, which depended on their contribution to reduce reprojection errors. Then the inlier points were sampled by a RANSAC process with the semantic weights and used to estimate camera pose translation. The selected segmentation patch, which had semantic meaning of static, was used to direct alignment between previous frame and current one. The two motion assumptions from two paralleled methods were fused for output as final pose estimation.

Figure 1.

Semantic segmentation–aided visual odometry pipeline.

Throughout this work, the image at time step k was I_k , and the pose of camera was represented by T_k ∈ SE(3). The transformation between two consecutive frames I _k−1 and I_k could be T_k = T_k ′T _k−1 ⁴¹

$T_k=\left(\begin{array}{cc}{R}_k& t_k\\ 0& 1\end{array}\right)$

2

with rotation R_k ∈ SO(3) and translation $t\in R_k^3$ . A 2-D pixel was represented by u = [u, v]^T, u ∈ ℝ². A 3-D world point was X = [x, y, z]^T, X ∈ ℝ³. The project function π: ℝ²↦ℝ³ mapped 2-D pixel u to 3-D point X

$X=\pi (u,d)$

3

d was an inverse depth of pixel u. If d = 1, the pixel would be projected into a 3-D unit sphere surface.

Semantic segmentation

Semantic segmentation was widely used in autonomous driving for scene parsing and understanding. This work used a modified SegNet with a pretrained driving model proposed by Badrinarayanan et al. ²³ SegNet was a deep learning encoder network with 13 convolutional layers of VGG16 model ⁴² and had 12 segmentation categories, including Sky, Building, Pole, Road Marking, Road, Pavement, Tree, Sign Symbol, Fence, Vehicle, Pedestrian, and Bike. These category information provided a kind of semantic understanding for an urban road scene, which could help us to distinguish the object is movable or not. To find static pixels was a very important factor for VO in dynamic urban traffic scene. A moving object would bring too much uncertainty in motion estimation process. This work assumed that each semantic category would have different contributions to VO. The contribution was relative to the errors of motion estimation brought by the pixel’s category. For example, in a dynamic urban traffic scene, a pixel from Building category could be more reliable than a pixel from a Car for motion estimation and should be sampled as a candidate with higher probability in VO process. The contribution of category c_i was represented by a probability variable $P_{c_i}$ in this article

$P_{c_i}=\frac{1}{Z}\frac{n_{c_i}}{{\displaystyle {\sum }_{i=0}^{n_{c_i}}{r}_{c_i}}}$

4

Z was a normalized factor. $r_{c_i}$ was the reprojection error of category c_i

$r_{c_i}={\displaystyle \sum }({\widehat{u}}_{j,I_k}-{u^{\prime }}_{j,I_k})={\displaystyle \sum }({\widehat{u}}_{j,I_k}-E*u_{j,I_{k-1}})$

5

E was the essential matrix

$E_{k-1,k}=K^{-1}*[t_{k-1,k}{]}_x*R_{k-1,k}*K$

6

In some open benchmark data set, the ground truth of transformation between frames T _k−1,k was provided. So the reprojection errors of every semantic categories could be computed by cumulation of its pixels’ reprojection errors. In some ways, these errors implicitly provided the level of contributions to correct motion estimation.

Visual odometry

Feature-based method

This part was similar to a traditional feature-based approach. In the current image frame I_k , the feature points and their rich descriptors were extracted. A k-nearest neighbor (KNN)-based method matched them to the keypoints from the previous frame I _k−1. The correspondence of these 2-D points was refined by direction symmetry check and ratio check. Given the 2-D correspondences, the essential matrix E could be computed by epipolar constrain equations and PnP method. The main difference of the proposed method was the sampling step. Rather than choosing the correspondent 2-D point pairs randomly in the RANSAC iteration, this work sampled them depending on their contribution probabilities which came from pixels’ semantic segmentation described as before.

The transformation T _{feature-based,k} = [R|t] with rotation R and translation t was computed by SVD(E) and optimized by a window BA. The 2-D keypoint u_j in previous image I _k−1 had a correspondent keypoint ${\widehat{u}}_j$ in current image I_k (Figure 2)

Figure 2.

2-D points u₁ , u₂ , u₃ in the frame I _k−1 could be projected to 3-D world points X₁ , X₂ by π and reprojected to u′ ₁ , u′ ₂ in frame I_k with optimization variable T _k−1,k . The reprojection error could be computed by u′ and its ground truth point V. Here not all of the matching pairs should be taken into account. For example, u₃ belongs to a movable object category and should have lower contribution to real motion estimation or just dropped out.

$u^{\prime }={\pi }^{-1}(X^{\prime })={\pi }^{-1}(T_{k}X)={\pi }^{-1}(T_k\pi (u,d))$

7

u′ was the reprojected 2-D point in I_k from 2-D point u in I _k−1. The correspondent point was $\widehat{u}$ in I_k from point-wise matching of u. So the reprojection error r _reprojection was

$r_{\text{reprojection}}(T_{k-1,k},u)=\parallel u^{\prime }-\widehat{u}\parallel$

8

Then the cost function J _reprojection was built by

$J_{\text{reprojection}}={\displaystyle \sum _i}\parallel r(T_{k-1,k},p_i){\parallel }^2$

9

And every transformation T _k−1,k between two continuous frames was solved by least-squares (LS) minimization method

$T_{k-1,k}=\underset{T}{{\displaystyle \text{arg} \text{min}}}{J}_{\text{reprojection}}$

10

Alignment-based method

This process used a semi-dense image alignment framework. Comparing with traditional direct method, there were three main differences: Firstly, instead of using whole image pixels for alignment, the proposed method only used partial image that had specific segmentation labels. These patches had the semantic priori knowledge that they were motionless objects. Secondly, these patches indicated a set of objects that belonged to one planar surface, which was a basic assumption for image alignment. Thirdly, the depth from these candidate patches or pixels from mono-camera should be estimated easily and could be used for weighting or sorting residual blocks in LS minimization process. In this work, the pixels, which labeled with Road Marking, Road, and Pavement, were selected for indirect VO estimation. The 3-D points belonged to these patches were regarded as being static to global coordinate system and assumed to lay on a rough road plane.

The candidate patch set Ω that belongs to category c was represented by $I_{{\Omega }_c}$ . And every pixel in the set was filtered by an intensity gradient threshold, which provided a semi-dense 2-D point and dropout the trivial points with less texture gradients. The pose transformation T _k−1,k between images $I_{{\Omega }_c,k-1}$ and $I_{{\Omega }_c,k}$ was computed by minimizing the intensity residuals r _intensity

$r_{\text{intensity}}(T_{k-1,k},u)=I_{{\Omega }_c,k-1}(u)-I_{{\Omega }_c,k}(u^{\prime })$

11

Then the cost function J _intensity was built by

$J_{\text{intensity}}={\displaystyle \sum _{i\in {\Omega }_c}}{\parallel r(T_{k-1,k},u_i)\parallel }^2$

12

And the transformation T _k−1,k was formed as a LS minimization optimization problem

$T_{k-1,k}=\underset{T}{{\displaystyle \text{arg} \text{min}}}{J}_{\text{intensity}}$

13

The whole problem cost function J combined the costs of two parts

$J=\alpha J_{\text{reprojection}}+(1-\alpha )J_{\text{intensity}}$

14

α was a hand-tuned parameter by experience. Lie group representation and dual number method were used to compute Jacobian. The pose was represented by ξ = [ρ φ]^T ∈ R ⁶, ρ ∈ R ³ was 3-D translation parameter, and φ ∈ R ³ was the rotation parameter including yaw, pitch, and roll. The Jacobians of transformation in J _reprojection and J _intensity part had the same formulation

$\begin{array}{c}\frac{\partial r}{\partial \delta \xi }=\left[\frac{\partial \rho }{\partial \delta \xi }|\frac{\partial \phi }{\partial \delta \xi }\right]\\ =\left[\begin{array}{llllll}\frac{f_x}{z^{\prime }}\hfill & 0\hfill & -\frac{f_x{x}^{\prime }}{{z^{\prime }}^2}\hfill & -\frac{f_x{x}^{\prime }{y}^{\prime }}{{z^{\prime }}^2}\hfill & f_x+\frac{f_x{x}^2}{{z^{\prime }}^2}\hfill & -\frac{f_x{y}^{\prime }}{z^{\prime }}\hfill \\ 0\hfill & \frac{fy}{z^{\prime }}\hfill & -\frac{fy{y}^{\prime }}{{z^{\prime }}^2}\hfill & -f_y-\frac{f_y{y^{\prime }}^2}{{z^{\prime }}^2}\hfill & \frac{f_y{x}^{\prime }{y}^{\prime }}{{z^{\prime }}^2}\hfill & \frac{f_y{x}^{\prime }}{z^{\prime }}\hfill \end{array}\right]\end{array}$

15

X = [x, y, z]^T and X′ = [x′, y′, z′]^T were the same 3-D point in different coordinate systems of I _k−1 and I_k . f_x and f_y were camera focal lengths. The J _intensity also had a partial derivative factor of $A\frac{\partial I}{\partial u}$ , A was a constant projection matrix for bird view of road surface, $\frac{\partial I}{\partial u}$ was the gradient of intensity in the image. Then a classical LS method could be used to solve the problem.

Experimental results

Data set

Proposed model was tested on open benchmark Kitti odometry data set. ²⁰ It contained 20 rectified stereo image sequences with calibration file and was recorded from a car traveling in urban blocks. We selected first 11 sequences which had pose ground truth and only used the monocular data of left camera. The proposed method was also tested on our autonomous driving data set. The intelligent vehicle platform was retrofitted from a Changan Raeton car, equipped with two AVT® 1394 Pike F-200c cameras capturing front view stereo images, one OxTS inertial IMU and Novatel RTK-GPS, and Velodyne VLP-16 LIDAR on the top of vehicle. We collected about 20 km real road data on Wuhuan Road in Beijing with different traffic conditions.

Semantic segmentation contribution analysis

Eleven sequences in Kitti odometry data set were selected to evaluate semantic segmentation’s reprojection error. First, segmentation process calculated the categories of every pixel on all Kitti odometry data set. Then, in each sequence, ground truth pose file was used to compute the transformation matrix T _k−1,k between every two consecutive image frames. SIFT features were extracted and used for neighbor frame matching. All matching keypoints were filtered by consistence check and ratio check and were projected from previous frame to next one. The reprojection errors were computed with the ground truth and used to evaluate segmentation’s contribution to odometry estimation. The feature point count in each semantic category and the impact of planar distance on image center of each feature point were also considered.

The reprojection errors were accumulated at each feature point and were sorted by planar distance to image center. As shown in Figure 3(a), horizontal axis represented normalized distance to image center and the vertical coordinates showed average value of reprojection error. For rectified images, this average error was increased with the growth of distance. In general, the border points in an image always had bigger errors. And nearer to image center, smaller were the errors. These errors were relative to inherent physical characteristics of camera lens and imaging sensors and hardly eliminated even after image rectification.

Figure 3.

Examples of Kitti data set sequence (a) 00, (b) 01, and (c) 02 and their semantic segmentation results.

Figure 4(b) shows the normalized reprojection errors and keypoint counts about 12 semantic segmentation categories, such as Sky, Building, Pole, Road Marking, Road, Pavement, Tree, Sign Symbol, Fence, Vehicle, Pedestrian, and Bike. The higher count of keypoints means that image patches of this category had more pixels and stronger texture than other categories. The reprojection error represented the uncertainty for each patch used in motion estimation. In the figure, the categories of Building and Tree have more keypoints and lower error. In Kitti data set, the buildings and trees usually appeared in the middle part of images. And they also occupied more area with various textures. These pixels were motionless objects and they did not stay in one plane. So they could be suitable for feature-based motion estimation. The categories of movable objects, Vehicle, Pedestrian, and Bike, showed low reprojection error too. The reason was most of the vehicles in Kitti data set were parking cars, and the moving pedestrian and bike were not presented in most of the sequences. The Bike had higher speed than Pedestrian and brought higher errors. In a dynamic urban traffic scene, dropping out the pixels of cars, pedestrian, and bikes would reduce dynamic disturbance and avoid the difficulties of moving object tracking and motion judgment. Though the Road pixels were static, they were hard to extract feature points and led to higher error. On the opposite side, Road Marker pixels had lower error because they had more corner points than Road and could be easily tracked and matched.

Figure 4.

Reprojection errors. (a) Considering coordinate distance to image center. (b) Errors distribution on 12 semantic segmentation categories. All statistic results were computed on Kitti odometry data set.

VO performance analysis

This work was tested on both data sets and evaluated the performances compared with VISO, DSO, and ORB_SLAM2. FAST corner points and SURF rich descriptors were computed. The threshold of ratio check was 0.7. The contribution probabilities of segmentation categories were computed as given in section “System Overview,” and the probabilities of Sky, Vehicle, Pedestrian, and Bike were tuned to zero. For monocular VO performance test only, ORB_SLAM2’s loop closure thread was turned off and its feature number was set to 3000. ORB_SLAM2 used default vocabulary. DSO was set to pinhole model with regular FOV lens, and its gamma and vignette configure were ignored.

Kitti odometry data set

The results of proposed method on average translational errors and rotational errors were shown in Tables 1 and 2. ORB_SLAM2 showed best rotation precision on most of the sequences which had little moving cars or other traffic participants. In sequence01, there were several moving cars running on neighboring lanes. Proposed method showed better robustness and precision than other algorithm. The proposed method showed distinct improvements in translation estimation on all Kitti sequences. Most of the monocular VO couldn’t recover real scale. This work used a prior knowledge of camera position with fixed height to the ground and assumed the road surface was a plane. These two factors made proposed method benefit a lot from Kitti data set. Figure 5 shows rotational and translational errors relative to travel distance and speed in Kitti sequence00. The proposed method had the lowest errors on both rotation and translation. More details about the evaluation method could be found in VISO. ^29,43 Some of the final trajectories are shown in Figure 6.

Table 1.

Rotation error.

Sequence	SAVO	VISO	DSO	ORB_SLAM2
0	0.000156	0.000702	0.005729	0.000056
1	0.000621	0.001238	0.002207	0.001579
2	0.000174	0.000519	0.004907	0.000049
3	0.000166	0.000495	0.005719	0.000033
4	0.000126	0.000285	0.000204	0.000046
5	0.000163	0.000803	0.005072	0.000041
6	0.000109	0.000498	0.001381	0.000043
7	0.000372	0.001375	0.009974	0.000057
8	0.000172	0.000657	0.005437	0.000053
9	0.000231	0.000548	0.005485	0.000061
10	0.000343	0.000867	0.004496	0.000214

SAVO: semantic segmentation–aided visual odometry; DSO: direct sparse odometry.

Table 2.

Translation error.

Sequence	SAVO	VISO	DSO	ORB_SLAM2
0	0.017481	0.168392	0.638761	0.525636
1	0.159909	0.424336	0.962693	0.955000
2	0.019727	0.237248	0.704573	0.672709
3	0.013100	0.293990	0.981139	0.913303
4	0.008170	0.166139	0.980135	0.980062
5	0.018440	0.140762	0.605002	0.504923
6	0.011571	0.116012	0.579200	0.546418
7	0.033184	0.163079	0.605035	0.513819
8	0.022087	0.121467	0.622131	0.563724
9	0.033228	0.162112	0.747357	0.704958
10	0.039921	0.182698	0.927605	0.804431

SAVO: semantic segmentation–aided visual odometry; DSO: direct sparse odometry.

Figure 5.

The transformation errors of proposed method SAVO on Kitti sequence00. SAVO showed lowest values on both rotational and translational errors. (a) Rotational error versus sequence length. (b) Rotational error versus driving speed. (c) Translational error versus sequence length. (d) Translational error versus driving speed. SAVO: semantic segmentation–aided visual odometry.

Figure 6.

Trajectories of SAVO on Kitti odometry data set sequences (a) 00, (b) 01, (c) 02, (d) 09. SAVO: semantic segmentation–aided visual odometry.

Beijing Wuhuan data set

As shown in Figure 7, the data set was collected in a sunny day afternoon with normal traffic flow condition. Two sequences in the data set were used for performance experiment. The first one contained 313 pictures and took 936 s. It started at a ring road and stopped at an open expressway and had 7.8 km distance. The second one contained 212 pictures and took 455 s. This one covered 5.9 km distance of the ring road. Tens of moving vehicles could be found in these sequences. This made VISO, DSO, and ORB_SLAM2 fail to estimate a reasonable trajectory. As shown in Figure 8, proposed method could recover the traces with noises. On one hand, semantic segmentation provided a prior probability to sample correct candidate points and image patches. This helped VO try to avoid the disturbance of moving object in the limit FOV. It made possible to use traditional VO in dynamic urban traffic environment. Making full use of fixed camera position information and assumption of road plane not only could provide a scale estimation for monocular camera but also covered the shortage of reduction of available points. On the other hand, limited by the precision of segmentation, the sampling process could not guarantee the sampled pixels were all static. The experimental result showed that dynamic objects are still one of the biggest factors to robustness of VO.

Figure 7.

Trajectories of SAVO on Beijing Wuhuan data set sequences 00, 01. ORB_SLAM2, and DSO failed directly, and VISO could not provide any reasonable trajectories. SAVO’s recovered trajectories with very big noise. The noises always came with those contained many dynamic vehicles. (a) seq00 raw image; (b) seq00 segmentation; (c) seq01 raw image; (d) seq01 segmentation. SAVO: semantic segmentation–aided visual odometry.

Figure 8.

Trajectories, rotational, and translational errors of SAVO on Beijing Wuhuan data set sequences 00, 01. Both errors in this data set had more higher value levels than Kitti sequences. (a) Trajectory of sequence 00; (b) trajectory of sequence 01; (c) average rotational error on sequence 00; (d) average rotational error on sequence 01; (e) average translational error on sequence 00; (f) average translational error on sequence 01. SAVO: semantic segmentation–aided visual odometry.

Conclusions

This article proposed a new semantic segmentation–aided VO pipeline. The new method used a deep learning network to segment input image with 12 semantic categories. Then a probabilistic model about categories and reprojection errors was computed for each pixel and used to weighing and sampling pixel candidates for feature-based VO pipeline. And semantic segmentation results also helped to select road plane for alignment-based VO pipeline. These two pipelines brought cost functions of reprojection and intensity errors, respectively, and were combined into a joint optimization in motion estimation process. These helped VO to reduce impacts of moving objects and make full use of motionless pixels by their geometry and physical characters. The experimental results on dynamic urban traffic scene data sets showed that new method provided higher precision and robustness than three state-of-the-art VO solutions. To improve the pipeline real-time performance and study how the VO impact segmenting procedure would be useful works in the future.