Sage Journals: Discover world-class research

Abstract

Estimating the camera trajectories is very important for the performance of visual simultaneous localization and mapping. However, visual simultaneous localization and mapping systems based on RGB image are generally not robust in complex situations such as low-textures or large illumination variations. In order to solve this problem, more environmental information is added by introducing depth information, and a feature extraction and matching algorithm combining depth information is proposed. In this article, firstly, the intrinsic mechanism that depth image is used to extract and match feature points is discussed. Then depth information and appearance information are comprehensively considered to extract and describe feature points. Finally, the matching problem of feature points is transformed into a regression and classification problem, with which a matching model is presented in a data-driven way. Experimental results show that our algorithm has better distribution uniformity and matching accuracy and can effectively improve the trajectory accuracy and drift degree of the simultaneous localization and mapping system.

Keywords

Visual SLAM feature extraction feature matching depth information regression and classification

Introduction

Visual simultaneous localization and mapping (SLAM) has been greatly developed in the past decades. Klein et al.¹ proposed the parallel tracking and mapping (PTAM) algorithms in 2007, and since then many visual SLAM systems have been proposed. These visual SLAM systems can be divided into direct methods and feature-based methods according to the difference in utilizing image information. The former directly uses the pixel intensity for matching and performs pose estimation by calculating the gradient and direction values of the local pixels, such as large-scale direct monocular SLAM (LSD-SLAM),² and direct sparse odometry (DSO).³ The latter extracts feature points and builds descriptors from the 2D image, and then computes and optimizes the camera pose by matching the feature points, such as real-time single camera SLAM (MonoSLAM),⁴ oriented FAST (Features from Accelerated Segment Test⁵) and rotated BRIEF (Binary Robust Independent Elementary Features⁶) SLAM (ORB-SLAM),⁷ and 3-D mapping with an RGB-D camera (RGBD-SLAM).⁸

The feature-based methods have the advantages of high robustness to complex scenes, and is currently a relatively mature solution,⁹ such as scale invariant feature transform (SIFT),¹⁰ speed up robust feature (SURF),¹¹ and oriented fast and rotated brief (ORB).¹² These methods are generally based on gray images, and these algorithms have relatively stable performance in the case of obvious environmental contrast.¹³ However, the feature points cannot be extracted in low-texture and blurred places which often appear in practical applications, in these situations feature point clustering and unevenness are prone to appear. Moreover, descriptors of feature points are indistinguishable when the structures of objects are similar, which easily leads to mismatches.⁹ Therefore, it is particularly important and necessary to develop more robust feature point extraction and matching algorithm for visual SLAM. Some researchers devote to this issue from different viewpoints. Points and line segments are combined to increase the mutuality between features in Zhou et al.,¹⁴ and thus obtaining a more accurate camera pose estimation. To obtain more environmental information, the feature extraction method based on color images is applied to visual SLAM in Wang et al.¹⁵ In Kim et al.,¹⁶ the matching risk is added to each feature point, and only feature points with lower matching risk are used in triangulation to avoid mismatching problems caused by excessive environmental changes. However, most of these methods only consider the appearance information and pay no attentions on the depth information which is insensitive to environmental changes.

Depth estimation is usually fundamental in a SLAM system.¹⁷ Many monocular and stereo SLAM systems take advantage of camera parallax from multiple images to estimate the depth, such as triangulation. Since Kinect camera has been presented by Microsoft, the RGB-D cameras have been gradually applied to SLAM systems. Besides the appearance information of the scene, depth information containing rich environmental information can also be obtained with the depth camera, and thus the process of 3D reconstruction is simplified. The method of using RGB-D camera to perform three-dimensional reconstruction of indoor environment was first proposed in Henry et al.,¹⁸ extracting SIFT features from color images and searching for corresponding depth information on depth images. Newcombe et al.¹⁹ obtained the camera pose by minimizing the distance measurement of each pixel in each frame through the depth image obtained by Kinect. Kerl et al.²⁰ simply combined the intensity error and depth error of the pixel as an error function to obtained the optimal camera pose.

Although depth is widely used in visual SLAM, it simply extends 2D points to 3D points and does not comprehensively consider the role of depth information as with appearance information. In addition, it is less used in the feature-based method. When some appearance information cannot be effectively distinguished or is insufficient for feature extraction and matching, the addition of depth information may give us another solution. Depth information and appearance information are combined to construct descriptors in Liu et al.,²¹ but it simply linearly adds the Hamming distances of the two types of descriptors during matching. Moreover, it does not consider depth information when extracting feature points. On the basis of fully researching the intrinsic mechanism of depth information, this article further utilizes the depth information and applies it to the entire process of feature point extraction and matching. The advantage of high speed makes ORB algorithm widely used in visual SLAM system, so this article adds depth information on the basis of the ORB algorithm, and a new feature extraction and matching algorithm is proposed. The main contributions of this article can be summarized as follows:

Depth information and appearance information are comprehensively considered during the extraction and description of feature points, which solves the problem that the feature points cannot be extracted under the condition of blurry and low-texture in RGB pictures, and has a better distribution uniformity.

An ORB feature matching model was established, which transformed the matching of feature points into a regression and classification problem, and improved the accuracy of feature point matching.

Our algorithm is integrated into ORB-SLAM2’s visual odometer, and its performance is tested using the Mikolajczyk data set²² and RGB-D data set.²³ The results prove that our algorithm can effectively improve the performance of ORB-SLAM2.²⁴

Related work

ORB algorithm

The ORB¹² algorithm consists of two parts: feature point extraction algorithm (oFAST, oriented features from accelerated segment test⁵) and feature descriptor calculation algorithm (rBRIEF, rotated binary robust independent elementary features⁶).

oFAST: FAST keypoint orientation

The ORB algorithm uses FAST-9 (circular radius of 9) to detect keypoints because of its good performance and uses Harris²⁵ corner to order the keypoints. Aiming at the disadvantage that FAST do not have scale invariance, the ORB algorithm builds a scale pyramid and extracts FAST features (filtered by Harris) at each level of the pyramid. When N keypoints need to be extracted, the ORB algorithm first sets a lower threshold to obtain more than N keypoints, and then selects the top N keypoints according to the results of Harris corner sorting.

Because the keypoints of FAST are not directional, the ORB algorithm adds an efficiently computed orientation. It uses intensity centroids²⁶ to achieve rotation invariance of features. In a local image patch of feature points, the moments of a patch are defined as

m_{u v} = \sum_{x, y} x^{u} y^{v} I (x, y)

where x, y are the coordinates of the pixel. The value of u, v is 0 or 1. I(x, y) is the grayscale value of the image at x, y. With these moments the centroid can be defined as

C = (\frac{m_{10}}{m_{00}}, \frac{m_{01}}{m_{00}})

The direction vector $\vec{O} C$ can be obtained by connecting the geometric center O and the centroid C of the image block. The orientation of the patch is

θ = a tan 2 (m_{01}, m_{10})

rBRIEF: Rotation-aware BRIEF

BRIEF is a binary description vector consisting of 0s and 1s. For an image patch, R, this description vector can be defined as

E (R; x, y) = {\begin{array}{r} 1 : & R (x) < R (y) \\ 0 : & R (x) \geq R (y) \end{array}

where $R (x)$ is the gray value of the image patch at point x. A binary description vector can be generated by selecting m pairs of test points and it is defined as

f_{m} (R) = \sum_{1 \leq m \leq N} 2^{m - 1} E (R; x, y)

Matching performance of BRIEF falls off sharply as the degrees of in-plane rotation increase. The rBRIEF defines a $2 \times m$ matrix for the m-dimensional binary sequence generated in the image block

S = (\begin{matrix} x_{1},....., x_{m} \\ y_{1},....., y_{m} \end{matrix})

Rotate the matrix S toward the main direction $θ$ , the description matrix $S_{θ}$ of the following formula can be obtained

S_{θ} = (\begin{matrix} cos θ & sin θ \\ sin θ & cos θ \end{matrix}) (\begin{matrix} x_{1},....., x_{m} \\ y_{1},....., y_{m} \end{matrix})

The rotated BRIEF descriptor can be expressed as

g (E, θ) : = f_{m} (E) | (x_{i}, y_{i}) \in S_{θ}

Qtree_ORB algorithm

In visual SLAM, to solve the problem of uneven distribution of feature points in the standard ORB algorithm, a quadtree method is used to homogenize the feature points, and at the same time an adaptive threshold method to improve the ORB algorithm in Mur-Artal et al.,⁷ which is called the Qtree_ORB algorithm. The specific steps are as follows:

Construct an eight-layer image pyramid.

Grid division of each layer of image pyramid.

Use the initial threshold (iniTHFAST) to extract FAST-9 corner points for each grid. If no feature points can be extracted in the grid, lower the threshold and use the minimum threshold (minTHFAST) to extract FAST-9 corner points.

Select the required FAST-9 corner points uniformly based on the quadtree. The feature point with the highest Harris response value is retained as the final feature point.

Calculate the intensity centroid.

Calculate the rBRIEF descriptor of the feature point.

Depth information prediction or completion

Depth information is not easy to obtain compared with appearance information. Monocular and stereo SLAM systems need to use camera parallax from multiple images to estimate the depth, and the data collected by the depth camera often has errors and missing. As deep neural networks have been developed with better backbone networks, huge RGB-D data sets, richer training labels, and complex loss functions, data-driven approach provides another option for obtaining depth information. In Eigen and Fergus,²⁷ a supervised deep neural network is designed to estimate depth from a single image without any processing. In Xie et al.,²⁸ a deep neural network is used to estimate the viewing angle difference in the image, which is then applied to the conversion of 2D to 3D video. In addition, Zoran et al.,²⁹ Chen et al.,³⁰ and others also proposed various depth estimation networks, which significantly improved the depth perception of a single RGB image. Meanwhile, a variety of depth complement algorithms have also been developed to complement the incomplete depth information collected by the depth camera, such as the joint bilateral filter method,³¹ the median filter method,³² joint rendering method of color and depth,³³ machine learning method,³⁴ and so on. These methods can effectively complement the incomplete depth image.

Sparse or incomplete depth images cannot be used to extract feature points, so it is necessary to predict the depth value of RGB images or complement the incomplete depth images for subsequent feature extraction and matching. In this article, Aleotti’s method³⁵ is used to predict the depth value of the RGB image, and Alexandru’s method³⁶ is used to fill in the incomplete depth image. In Figure 1, (a) is an RGB image, and (c) is a depth image predicted by (a). (b) is the incomplete depth image collected by the depth camera, and (d) is the completed depth image obtained after inputting (b) into the depth completion model.

Figure 1.

Depth information prediction or completion. (a) RGB image, (b) incomplete depth image, (c) predicted depth image, (d) completed depth image.

Feature extraction with depth information

The feature point extraction and matching algorithm can be divided into three parts, as shown in Figure 2. The “Depth information prediction or completion” section gives an introduction to the method of acquiring depth information. This section will introduce the feature extraction method combined with depth information. The feature matching algorithm that combines depth information is shown in detail in the “Feature matching with depth information” section.

Figure 2.

The algorithm flow of this article.

Depth information preprocessing

The relationship between the depth value obtained from the depth camera and the true depth value is not linear (pinhole camera model).³⁷ The mapping function can be obtained through the camera model³⁸

f (d) = \frac{Z_{r}}{1 + \frac{z_{r}}{f b} d}

where $f (d)$ represents the actual depth from camera to the measuring object, Z_r is the depth of camera to the reference plane, b is the length of baseline, f is the focal length of camera, and d is the disparity measured by the camera.

The pixel value range of an RGB image is 0–255(0–2⁸). But the pixel value of the depth image may exceed 255, so it needs to be converted into the range of 0–255 using the formula (9). The measurement distance of ordinary commercial depth cameras is limited, and the measurement accuracy is not high beyond a certain limit, so only the pixel value within a fixed range needs to be converted. This article sets the maximum distance of 4 m. That is, the pixel value of 4 m is 255, and the pixel value greater than 4 m is filled with the surrounding pixel values.

ORB algorithm adopts a Harris corner to score each feature point. The higher the score, the stronger the distinguishability of the feature points, and the better the subsequent matching effect. We randomly select 200 ORB feature points extracted from the RGB image, the completed depth image and the predicted depth image, and calculate their Harris response values, as shown in Figure 3. The initial threshold (iniTHFAST) is set to 20. Most of the Harris response values of the predicted depth image and the RGB image lie in the interval 20–80, indicating that the feature points have relatively strong distinguishability. The Harris response value of the completed depth image is mainly in the interval 120–160, which is more distinguishable.

Figure 3.

Harris response.

Since the Harris response values of the three types of image is not uniform, define a normalization coefficient $γ$

γ = \frac{\sum_{1 \leq n \leq N} G_{n}}{\sum_{1 \leq n \leq N} h_{n}}

where h_n is the Harris response of the nth feature point on the depth image, and G_n is the Harris response of the nth feature point on the gray image. The Harris response of each feature point of the depth image needs to be multiplied by a normalization coefficient to get the final Harris response

H_{n} = γ h_{n}

The mean value of the three types of pictures is calculated. Figure 4 shows the spread of means for rBRIEF pattern of 256 bits over 1 k sample keypoints. There are many feature points in the three pictures whose average value falls between 0.05 and 0.2, in which 0.05 corresponds to the most number of bits, indicating that the average value of each of the feature points is very close to 0.5. A high mean value means that the feature descriptor has a better discriminative degree.

Figure 4.

Distribution of means for feature vectors. The X-axis is the distance to a mean of 0.5.

Extraction

In reality, there are many places with low texture and blur. If feature points are only extracted from RGB images, it is easy to have no feature points in a large area, resulting in too clustered feature points (as shown in Figure 5(a)). After adding the depth information, the feature points can be extracted even if the gray level changes little where the depth changes, making the distribution of feature points more uniform (as shown in Figure 5(b)). Moreover, the feature points with higher Harris response values are selected to have higher accuracy in the subsequent matching. We improve the Qtree_ORB algorithm to extract the feature points of the image. Firstly, construct eight-layer image pyramid on RGB image and depth image. Secondly, grid division of each layer of image pyramid. Thirdly, extract FAST-9 corner points and put together the points extracted in the RGB image and the depth image. Finally, calculate the Harris response and normalize it, and keep the point with the largest Harris response as the final feature point. The specific steps are as shown in Algorithm 1.

Figure 5.

The colored circles on the image are the extracted feature points. (a) Feature points extracted on RGB images. (b) Feature points extracted on RGB image and depth image.

Algorithm 1.

Feature extraction.

Require: RGB_image;N;//N is the target number of feature points
Ensure: Keypoints;
1: if RGB-D SLAM then
Require: Depth_image;
2: Depth_image=Completion(Depth_image);//Depth completion
3: else
4: Depth_image=Predict(RGB_image);//Depth prediction
5: end if
6: RGB_image⁸=ComputePyramid(RGB_image);
7: Depth_image⁸=ComputePyramid(Depth_image);//Build an 8-layer Image Pyramid
8: RGB_image[30×30]⁸=GridPyramid(RGB_image⁸);
9: Depth_image[30×30]⁸=GridPyramid(Depth_image⁸); //Each layer of the image pyramid is divided into a 30×30 grid
10: for

i = 0

t0 8 do
11: for

u = 0

t0 30×30 do
12: Depth_Keypoints=FAST(Depth_image[u][i], iniTHFAST);//Use initial threshold to extract feature points
13: if Depth_Keypoints.number<1 then
14: Depth_Keypoints=FAST(Depth_image[u][i], minTHFAST);//Use minimum threshold to extract feature points
15: end if
16: RGB_Keypoints=FAST(RGB_image[u][i], iniTHFAST);
17: if RGB_Keypoints.number<1 then
18: RGB_Keypoints=FAST(RGB_image[u][i], minTHFAST);
19: end if
20: Depth_AllKeypoints.push_back(Depth_Keypoints);
21: RGB_AllKeypoints.push_back(RGB_Keypoints);
22: end for
23: end for
24: for

m = 0

t0 Depth_AllKeypoints.number do
25: Depth.Harris=Depth.Harris+Depth_AllKeypoints[m].Harris;
26: end for
27: for

m = 0

t0 RGB_AllKeypoints.number do
28: RGB.Harris=RGB.Harris+RGB_AllKeypoints[m].Harris;
29: end for
30:

γ

=Depth.Harris/RGB.Harris;
31: for

m = 0

t0 Depth_AllKeypoints.number do
32: Depth_AllKeypoints[m].Harris=Depth_AllKeypoints[m].Harris

\times γ

;//Normalized
33: end for
34: allKeypoints=Depth_Keypoints+RGB_Keypoints;
35: while Node.number<N do
36: Node=DivideNode();//Use quadtree to divide nodes
37: if Node.number>allKeypoints.number then
38: break;;
39: end if
40: if Node.empty then //If the quadtree can no longer split
41: deleteNode;
42: end if
43: end while
44: Keypoints=push_back(OctTree(allKeypoints).MaxHarris);//Save the feature point with the largest Harris response value

Feature matching with depth information

Descriptor

According to formula (8), the rBRIEF descriptor calculated on the depth image can be characterized as

g (E, β) : = f_{n} (E) | (x_{i}, y_{i}) \in S_{β}

In this article, a feature point has two descriptors, which are calculated on the RGB image (formula (8)) and the depth image (formula (12)), respectively. It can be expressed as

g (E, θ, β) : = {\begin{array}{r} f_{m} (E) | (x_{i}, y_{i}) \in S_{θ} : & R G B \\ f_{n} (E) | (x_{i}, y_{i}) \in S_{β} : & D e p t h \end{array}

where $θ$ is the main direction of the feature points calculated on the gray image. $β$ is the main direction of the feature points calculated on the depth image; $f_{n} (P)$ and $f_{m} (P)$ are the feature descriptors on the grayscale image and the depth image; and $S_{θ}$ and $S_{β}$ are the description matrix rotated.

Matching

The traditional feature point matching algorithm calculates the Hamming distance of the feature descriptor, finds the initial match according to the principle of the smallest Hamming distance and less than the preset distance, and then votes to eliminate the feature points with large rotation angles to obtain the final match. In the above process, the preset Hamming distance threshold has a greater impact on the matching result. In the case of large changes in the environment, a fixed threshold may eliminate the correct match. In the case of small environmental changes, a fixed threshold may lead to an increase in the number of false matches. Moreover, using the rotation angle as a condition for removing the initial match may also remove the correct match, because the larger rotation angle may be the correct match. In the question of whether the two feature points match, the contribution of Hamming distance and rotation angle is not clear. Therefore, this article transforms the matching between two feature points into a classification and regression problem, and uses a data-driven idea to learn how much they contributes. Since this article adds depth information, there are four input variables, namely the Hamming distance of the descriptor on the gray image (H_g ), the Hamming distance of the descriptor on the depth image (H_d ), the difference in the rotation angle of the descriptor on the gray image (A_g ), and the difference in the rotation angle of the descriptor on the depth image (A_d ).

Random forest algorithm is a classification regression algorithm that has the advantages of fast training speed, strong robustness, high accuracy, and simple implementation. In this article, a random forest algorithm is used to establish a mathematical model to calculate the matching probability between two feature points. The random forest algorithm uses $G i n i (D)$ to indicate the purity of the node sample data

G i n i (D) = 1 - \sum_{n = 1}^{N} p_{n}^{2}

where D is the data set; N is the number of sample categories; p_n is the proportion of the ith sample in D.

The training sample set used to build the matching model comes from the RGB-D data set and the KITTI data set. The training sample set is divided into two parts, one part is 100,000 point pairs that are marked as correct. The images in the RGB-D data set and the KITTI data set are changed to different degrees of rotation, viewpoint, light, blur, and so on, and the feature points before and after transformation are marked as correct point pairs. The other part is 100,000 point pairs that are marked as wrong. Matching experiments are performed on the two data sets using traditional algorithms, and the mismatched point pair are marked as the wrong point pair.

Adjusting the number and depth of decision trees of the random forest model can make the model faster and more accurate. This article uses the forward search method to find the optimal number of trees, and the depth of the decision tree can be ignored because of fewer variables. The fitting performance is evaluated by the accuracy of the test sample set. The results are shown in Figure 6. Considering the prediction time of each frame and the prediction accuracy, it was decided to build the final prediction model from 150 decision trees. The prediction accuracy rate is 96.13%, and the prediction time per frame is about 20 ms, which meets the real-time requirements. Algorithm 2 shows the steps of establishing a matching model.

Figure 6.

Parameter adjustment of random forest number.

Algorithm 2.

Construct feature point matching model.

Require: TrainingSamples[

H_{g}, A_{g}, H_{d}, A_{d}

];
Ensure: MatchingModel;
1: for

n = 0

t0 150 do
2: TrainingSet[n]=BootstrapResample(TrainingSamples);
3: model[n]=DecisionTree(TrainingSet[n]);
4: MatchingModel += model[n];
5: end for

The area under curve value of the receiver operating characteristic curve (ROC curve) in Figure 7 is 0.94, indicating that the model has high sensitivity and accuracy, and can effectively judge whether the two feature points match. According to the ROC curve, although point A (the false positive rate is 0.61%, the true positive rate is 84.3%) has the largest sum of sensitivity and specificity, point B (the false positive rate is 1.63% and the true positive rate is 94.54%) is selected as the optimal cut off point. Because it only needs to increase the false positive rate by about 1%, the true positive rate can be increased by about 10%. In the feature importance histogram in Figure 7, the importance ratios of the four variables ( $H_{g}, A_{g}, H_{d}, A_{d}$ ) are 0.31, 0.22, 0.26, 0.20, respectively. Although the feature descriptors on RGB images still occupy high importance, the overall difference is not big. It is reasonable for us to put the Hamming distance and the rotation angle at the same priority.

Figure 7.

Random forest model. (a) ROC curve; (b) feature importance.

The matching model is tested on the RGB-D data set. Algorithm 3 shows the process of matching two feature points. In Figure 8, (a) is the matching result obtained by the traditional algorithm. There are many mismatches, and the accuracy is 85.06%. (b) is the matching result obtained by our algorithm, and the accuracy is 93.48%. It shows that the algorithm in this article eliminates many incorrect matches under traditional algorithms and increases the correct matches eliminated under traditional algorithms.

Algorithm 3.

The process of feature point matching.

Require: Keypoints1;Keypoints2;
Ensure: MatchResult;
1: Keypoints1.RGBdescriptors=ComputeRGBDescriptors(RGB1);
2: Keypoints1.Depthdescriptors=ComputeRGBDescriptors(Depth1);
3: keypoint1.RGBangle=computeOrientation(RGB1)
4: keypoint1.Depthangle=computeOrientation(Depth1)
5: Keypoints2.RGBdescriptors=ComputeRGBDescriptors(RGB2);
6: Keypoints2.Depthdescriptors=ComputeRGBDescriptors(Depth2);
7: keypoint2.RGBangle=computeOrientation(RGB2)
8: keypoint2.Depthangle=computeOrientation(Depth2)
9: H_g =HammingDistance(Keypoints1.RGBdescriptors, Keypoints2.RGBdescriptors);
10: H_d =HammingDistance(Keypoints1.Depthdescriptors, Keypoints2.Depthdescriptors);
11: A_g =keypoint1.RGBangle-keypoint2.RGBangle;
12: A_d =keypoint1.Depthangle-keypoint2.Depthangle;
13: MatchResult=MatchingModel(

H_{g}, A_{g}, H_{d}, A_{d}

);

Figure 8.

Feature point matching. (a) Matches obtained by traditional algorithms. There are 241 pairs of matching points (the accuracy rate is 85.06%). (b) Matches obtained by our algorithms. There are 230 pairs of matches (the accuracy rate is 93.48%).

Experiments

All the experiments in this article are run on a computer with i5-10400F, memory 8 GB, graphics card GTX960, and operating system ubuntu18.04. The data sets are the widely used RGB-D data set and Mikolajczyk data set. The experimental results are the average of 10 calculations. The matching performance is evaluated by correct rate P

P = \frac{T_{p}}{T_{P} + F_{p}} \times 100 %

where T_P is the number of matches detected correctly; F_P is the number of matches that the true result is not a match but the detection result is a match. This article shows median results after running each test 10 times.

Matching test of feature points

Monocular mode: The Mikolajczyk data set is a standard data set mainly used for image matching and is commonly used for region detector evaluation and region descriptor evaluation. Mikolajczyk data set gives varying degrees of rotation change, viewpoint change, light change, and blur change, and the changed H matrix is given, which is very convenient for us to test. We first use the Mikolajczyk data set to test the matching performance of our algorithm under different variations, and then use the RGB-D data set to test the matching performance of our algorithm in practical applications.

In the Mikolajczyk data set, each set of sequence changes five times, and the degree of change is getting larger and larger. Figure 9 shows the results of four sets of sequence matching performances. In the tests of blur change (Bike sequence) and lighting change (Leuven sequence), as the degree of change increases, the rate of change gradually decreases. The accuracy of our algorithm is always higher than that of the Qtree_ORB algorithm, and more correct matches can be obtained. In the tests of viewpoint change (Graf sequence), the Qtree_ORB algorithm quickly fails, but our algorithm can maintain a relatively high accuracy rate. ORB features are sensitive to rotation, so the accuracy of rotation change (Bark sequence) fluctuates greatly. By adding depth information and random forest algorithm, the accuracy of traditional algorithms can be improved. There are a total of eight sequences in the Mikolajczyk data set, Table 1 shows the average of the correct rate of the five degree changes. In general, the algorithm in this article is more robust and accurate than the Qtree_ORB algorithm, and is more able to adapt to complex and changeable realities.

Figure 9.

The abscissa represents the degree of change, and the ordinate represents the correct rate (%). (a) Rotation changes (Bark sequence); (b) viewpoint changes(Graf sequence); (c) lighting changes (Leuven sequence); (d) blur changes (Bike sequence).

Table 1.

Average correct rate (%).

Change	Sequence	Qtree_ORB	Our algorithm
Blur	Bike	70.48	77.99
Blur	Trees	56.01	57.59
Viewpoint	Graf	55.42	74.01
Viewpoint	Wall	3.82	6.95
Rotation	Bark	54.61	61.52
Rotation	Boat	62.32	73.97
Light	Leuven	86.63	80.73
Light	Ubc	96.52	97.86

In monocular vision SLAM, the initialization of the system uses brute-force matching, and then uses DBoW2 (bags of binary words for fast place recognition) matching. The fr2/rpy sequence in the RGB-D data set is used to test the performance of the two matches. Randomly select 150 adjacent frames to test their matching performance, and the number of feature points extracted in each frame is 1000. When using brute-force matching, only the feature points of the first level of the image pyramid are matched. The result is shown in Figure 10, it can be seen that the accuracy of the initial matching and DBoW2 matching of our algorithm is higher than that of the traditional algorithm, indicating that our system is more robust in the same environment. When DBoW2 matching, the number of matches obtained by the two algorithms is similar, but our algorithm has a higher number of matches in the initial matching. Since the system initialization requires a sufficient number of matches, our algorithm is easier to initialize. In some sequence, the algorithm in this article has been initialized, but the traditional algorithm is waiting because there is not enough number of correctly matched feature points, and a lot of information will be lost.

Figure 10.

Matching performances in the fr2/rpy sequence. The horizontal axis represents the number of frames. (a) Number of matches obtained by brute-force matching. (b) Correct rate of brute-force matching. (c) Number of matches obtained by DBoW2 matching. (d) Correct rate of DBoW2 matching.

Integrating our algorithm into the ORB-SLAM2's visual odometer, we test the number of frames required for the two algorithms to be initialized successfully in the ORB-SLAM2's monocular mode. Table 2 shows that our algorithm is always faster to initialize than the traditional algorithm, which can save a lot of time, because the algorithm in this article has more correct matches and better matching accuracy in the initial brute-force matching. Similarly, when the visual SLAM system fails to track and relocate, our algorithm will be more accurate and faster.

Table 2.

Number of initial matching frames.

Sequence	RGB	RGB-D + Rain forest
fr1/360	54	655
fr2/360_hemisphere	44	61
fr1/large_with_loop	60	355

Depth mode: Deep vision SLAM does not require complex initial matching, so only the performance of DBoW2 matching is tested on the fr2/360_hemisphere sequence. As shown in Figure 11, the number of matches and the correct matching rate will appear lower than the ORB-SLAM2 algorithm, but the number of correct matches is always higher than the ORB-SLAM2 algorithm.

Figure 11.

Matching performance in the fr2/360_hemisphere sequence. A positive number means our algorithm is better. (a) The difference in number of matches. (b) The difference in correct rate. (c) The difference in number of correct matches.

Trajectory error

This experiment is tested on the open source algorithm ORB-SLAM2, and its evaluation standard is the absolute trajectory error (ATE). The sequences in the RGB-D data set are used to evaluate the accuracy and stability of the SLAM system in dynamic scenes.

As shown in Table 3, the root mean square error (RMSE), the sum of squares due to error (SSE), and standard deviation (STD) of our algorithm in the four sequences are all smaller than the traditional algorithm, indicating that the trajectory of the algorithm proposed in this article is closer to the true value. In addition, our algorithm can effectively suppress the maximum value of trajectory error by adding depth, which has high accuracy and robustness, and the fluctuation of trajectory error is relatively gentle.

Table 3.

Absolute trajectory error (ATE).

Sequence	Algorithm	Max	Mean	Min	RMSE	SSE	STD
fr1/360	ORB-SLAM2	0.477	0.238	0.032	0.27	54.283	0.127
fr1/360	Our algorithm	0.271	0.11	0.009	0.125	11.163	0.059
fr2/360_kidnap	ORB-SLAM2	0.607	0.092	0.018	0.109	9.181	0.058
fr2/360_kidnap	Our algorithm	0.256	0.09	0.011	0.1	8.39	0.05
fr2/pioneer_slam2	ORB-SLAM2	1.166	0.104	0.026	0.136	12.186	0.087
fr2/pioneer_slam2	Our algorithm	0.483	0.09	0.01	0.11	7.251	0.044
fr2/pioneer_slam3	ORB-SLAM2	0.298	0.118	0.044	0.124	30.883	0.039
fr2/pioneer_slam3	Our algorithm	0.207	0.093	0.01	0.101	20.586	0.038

Table 4 shows the accuracy comparison results in a subset of sequences where most RGB-D methods are usually evaluated. In the sequence fr2/desk, fr2/xyz, fr2/office, our algorithm has a slight improvement, because the ORB feature of the RGB image is obvious in these sequences. But in sequences (fr1/desk, fr1/desk2, fr1/room, fr3/nst) with low-texture and blurry scenes, the RMSE of this article is significantly smaller. In this case, the ORB-SLAM2 algorithm sometimes fails to extract feature points, or fails to obtain correct matching, which further leads to a large drift in pose estimation. As shown in Figure 12, in the sequence fr3/nst, few feature points and matches are obtained through appearance information, only 41 matches with a correct rate of 73.17%. However, our algorithm can obtain a sufficient number of matches, with 109 matches and a correct rate of 83.49%.

Table 4.

Comparison of translation RMSE.

Algorithm	fr1/desk	fr1/desk2	fr1/room	fr2/desk	fr2/xyz	fr3/office	fr3/nst
ORB-SLAM2	0.017	0.021	0.051	0.010	0.004	0.010	0.023
Our algorithm	0.014	0.017	0.036	0.010	0.004	0.009	0.021

Figure 12.

Feature point matching in sequence fr3/nst. (a) Qtree_ORB algorithm (there are 30 correct matches). (b) Our algorithm (there are 91 correct matches).

In order to properly validate the effectiveness and robustness of our algorithm in low-texture and blurred places, tracking performance is tested on the sequence fr2/pioneer_360 with many blurry and low-texture images. For the case as shown in Figure 13 and Figure 14, the ORB-SLAM2 algorithm only gets few correct matches, which further results in partial path loss depicted in Figure 15(b). However, our algorithm can get more matches by adding depth information, which is enough to guarantee a relatively complete path, depicted as Figure 15(c).

Figure 13.

Feature point matching in sequence fr2/pioneer_360. (a) The above picture shows the matching obtained by the Qtree_ORB algorithm, and there are 107 matches with a correct rate of 93.46%. The figure below shows the match obtained by our algorithm, and there are 200 matches with a correct rate of 97%. (b) The above picture shows the matching obtained by the Qtree_ORB algorithm, and there are 56 matches with a correct rate of 71.42%. The figure below shows the match obtained by our algorithm, and there are 172 matches with a correct rate of 94.18%.

Figure 14.

Pose error trajectories of the two algorithms in the fr1/pioneer_360 sequence. (a) ORB-SLAM2. (b) Our algorithm. (c) ORB-SLAM2. (d) Our algorithm.

Figure 15.

Comparison of the estimated trajectories of ORB-SLAM2 and our algorithm with the ground-truth in sequence fr2/pioneer_360. (a) The ground-truth trajectory. (b) ORB-SLAM2. (c) Our algorithm.

Timing results

In general, the average tracking time per frame is below the inverse of the camera frame rate for each sequence, meaning that a system is able to work in real time.²⁴ The experiments were run on a computer with i5-10400F and graphics card GTX960.

Table 5 shows the average tracking time per frame of our algorithm and ORB-SLAM2 algorithm in different data sets and different modes. Because depth information is considered in this article, it takes more time to acquire and process depth information. The completed depth information is not always directly obtained. In monocular mode, the algorithm used in this article is able to predict depth information at 100 FPS (frames per second). In depth mode, the time to fill the depth mainly depends on the degree of corruption of the depth image. The smaller the incomplete part of the depth image, the less time it takes. In this case, our algorithm consumes 1–10 ms more than ORB-SLAM2 algorithm. Besides, it takes no more than 1 ms time to process depth information in the matching stage for our algorithm compared to the ORB-SLAM2 algorithm. Generally speaking, our algorithm can run in real time when camera FPS is 30HZ. In addition, a more lightweight network can obtain depth information at a faster speed. For example, network proposed by Wofk³⁹ can infer depth predictions at 178 FPS. If a better hardware is adopted, the run time of our algorithm can further decrease.

Table 5.

Average tracking time (ms).

Sequence	Mode	ORB-SLAM2	Our algorithm
fr1/desk	Monocular mode	23.58	35.74
fr1/desk	Depth mode	26.02	34.08
fr1/room	Monocular mode	26.61	35.23
fr1/room	Depth mode	30.51	32.92
fr2/desk	Monocular mode	25.34	34.12
fr2/desk	Depth mode	26.57	29.54

Conclusion

This article proposes a method of feature point extraction and matching, which combines depth images. It is mainly divided into four steps: processing depth images, extracting feature points, calculating feature point descriptors, and using models for feature point matching. Experimental results prove that our algorithm has better uniformity and higher accuracy than traditional algorithms. Especially in some places with low texture and blur, the algorithm in this article can effectively increase the stability of the visual SLAM system.

Our algorithm needs to rely on GPU for accelerated computing. Further research can focus on reducing the difficulty of acquiring depth information and apply it to CPU or embedded systems. Moreover, the predicted depth image may be helpful to the triangulation of monocular SLAM, and the depth information can be dig deeper in the future.

Footnotes

Author contributions

Data processing, code writing, hardware and software preparation were handled by Yunpeng Sun. Yunpeng Sun and Xiaoli Li jointly wrote the first draft and made the second revision. All authors read and approved the final manuscript.

Code or data availability

Custom code (C, C++, python). The data are all from public data sets.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Natural Science Foundation of Shanghai (under grants nos16ZR1446700, 17ZR1400100) and Shanghai Pujiang Program (under grants no. 18PJ1400100).

ORCID iD

Yunpeng Sun

References

Klein

Murray

. Parallel tracking and mapping for small AR workspaces. In: 2007 6th IEEE and ACM international symposium on mixed and augmented reality, Nara, Japan, 13–16 November 2007, pp. 225–234. IEEE.

Engel

Schöps

Cremers

. LSD-SLAM: large-scale direct monocular SLAM. In European conference on computer vision. Berlin: Springer, 2014, pp. 834–849.

Engel

Koltun

Cremers

. Direct sparse odometry. IEEE Trans Pattern Anal Mach Intell 2017; 40(3): 611–625.

Davison

Reid

Molton

, et al. MonoSLAM: real-time single camera SLAM. IEEE Trans Pattern Anal Mach Intell 2007; 29(6), 1052–1067.

Rosten

Porter

Drummond

.Faster and better: a machine learning approach to corner detection. IEEE Trans Pattern Anal Mach Intell 2008; 32(1): 105–119.

Calonder

Lepetit

Strecha

, et al. BRIEF: binary robust independent elementary features. In: European conference on computer vision, Berlin: Springer, 2010, pp. 778–792.

Mur-Artal

Montiel

JMM

Tardos

. ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Trans robot 2015; 31(5): 1147–1163.

Endres

Hess

Sturm

, et al. 3-D mapping with an RGB-D camera. IEEE trans robot 2013; 30(1): 177–187.

Meixiang

Songhao

Guo

. An overview of visual SLAM. CAAI Trans Intell Technol 2016; 11(6): 768–776.

10.

Lowe

. Distinctive image features from scale-invariant keypoints. Int J Comput Vis 2004; 60(2): 91–110.

11.

Bay

Tuytelaars

Van Gool

. SURF: speeded up robust features. In: European conference on computer vision, Berlin: Springer, 2006, pp. 404–417.

12.

Rublee

Rabaud

Konolige

, et al. ORB: an efficient alternative to SIFT or SURF. In: 2011 international conference on computer vision, Piscataway, NJ, 6–13 November 2011, pp. 2564–2571. IEEE.

13.

Yousif

Bab-Hadiashar

Hoseinnezhad

. An overview to visual odometry and visual SLAM: applications to mobile robotics. Intell Ind Syst 2015; 1(4): 289–311.

14.

Zhou

Zhang

Deng

, et al. Improved point-line feature based visual SLAM method for complex environments. Sensors 2021; 21(13): 4604.

15.

Wang

Liang

Zhao

. Research on extraction of RGB feature points for visual SLAM. Proceedings of the 2021 1st International Conference on Control and Intelligent Robotics, 2021, pp. 276–280.

16.

Kim

C-H

Lee

T-J

Dan Cho

D-I

. Matching risk for feature selection in visual SLAM. IFAC-Papers OnLine 2018; 51(22): 144–147.

17.

Wang

Ongoing evolution of visual SLAM from geometry to deep learning: challenges and opportunities. Cogn Comput 2018; 10(6): 875–889.

18.

Henry

Krainin

Herbst

, et al. RGB-D mapping: using depth cameras for dense 3D modeling of indoor environments. In: Experimental robotics, Berlin: Springer, 2014, pp. 477–491.

19.

Newcombe

Izadi

Hilliges

, et al. Kinectfusion: real-time dense surface mapping and tracking. In: 2011 10th IEEE international symposium on mixed and augmented reality, Basel, Switzerland, 26–29 October 2011, pp. 127–136. IEEE.

20.

Kerl

Sturm

Cremers

. Dense visual SLAM for RGB-D cameras. In: 2013 IEEE/RSJ international conference on intelligent robots and systems, Tokyo, Japan, 3–7 November 2013, pp. 2100–2106. IEEE.

21.

Liu

Yang

, et al. Semi-direct tracking and mapping with RGB-D camera. Int Conf Robot Artif Intell Robot Applicat 2019; 11744: 461–472.

22.

Barandiaran

Cortes

Nieto

, et al. A new evaluation framework and image dataset for keypoint extraction and feature descriptor matching. In:VISAPP 2013; 1: 252–257.

23.

Sturm

Engelhard

Endres

, et al. A benchmark for the evaluation of RGB-D SLAM systems. In: 2012 IEEE/RSJ international conference on intelligent robots and systems, Vilamoura-Algarve, Portugal, 7–12 October 2012, pp. 573–580. IEEE.

24.

Mur-Artal

Tardós

. ORB-SLAM2: an open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Trans Robot 2017; 33(5): 1255–1262.

25.

Harris

Stephens

. A combined corner and edge detector. In: Alvey vision conference, vol. 15, Manchester, 31 August–2 September, 1988, pp. 10–5244. Citeseer.

26.

Rosin

. Measuring corner properties computer vision and image understanding. Comput Vis Image Underst 1999; 73(2): 291–307.

27.

Eigen

Fergus

. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In: Proceedings of the IEEE international conference on computer vision, Washington, NW, 7–13 December 2015, pp. 2650–2658. IEEE Computer Society.

28.

Xie

Girshick

Farhadi

. Deep3D: fully automatic 2D-to-3D video conversion with deep convolutional neural networks. In: European conference on computer vision, Berlin: Springer, 2016, pp. 842–857.

29.

Zoran

Isola

Krishnan

, et al. Learning ordinal relationships for mid-level vision. In: Proceedings of the IEEE international conference on computer vision, Santiago, Chile, 7–13 December 2015, pp 388–396. IEEE.

30.

Chen

Yang

, et al. Single-image depth perception in the wild. Adv Neural Inf Process Syst 2016; 29: 730–738.

31.

Camplani

Salgado

. Efficient spatio-temporal hole filling strategy for kinect depth maps. In: Three-dimensional image processing (3DIP) and Applications II, vol. 8290, 2012, p. 82900. Bellingham, Washington, USA: International Society for Optics and Photonics.

32.

Matyunin

Vatolin

Berdnikov

, et al. Temporal filtering for depth maps generated by kinect depth camera. In: 2011 3DTV conference: the true vision-capture, transmission and display of 3D video (3DTV-CON), Antalya, Turkey, 16–18 May 2011, pp. 1–4. IEEE.

33.

Wang

Jin

Yang

, et al. Stereoscopic inpainting: joint color and depth completion from stereo images. In: 2008 IEEE conference on computer vision and pattern recognition, Anchorage, AK, USA, 23–28 June 2008, pp. 1–8. IEEE.

34.

Zhang

Funkhouser

. Deep depth completion of a single RGB-D image. In: Proceedings of the IEEE conference on computer vision and pattern recognition, Salt Lake City, UT, USA, 18–23 June 2018, pp. 175–185. IEEE.

35.

Aleotti

Zaccaroni

Bartolomei

, et al. Real-time single image depth perception in the wild with handheld devices. Sensors 2020; 21(1): 15.

36.

Telea

. An image inpainting technique based on the fast marching method. J Graphics Tools 2004; 9(1): 23–34.

37.

Tian

Gao

Wei

, et al. Spatial uncertainty model based on scale-space for RGBD-SLAM system. In: 2019 5th international conference on control, automation and robotics (ICCAR), China, 19–22 April 2019, 2019, pp. 58–62. IEEE.

38.

Khoshelham

Elberink

. Accuracy and resolution of Kinect depth data for indoor mapping applications. Sensors 2012; 12(2): 1437–1454.

39.

Wofk

Yang

T-J

, et al. Fastdepth: fast monocular depth estimation on embedded systems. In: 2019 international conference on robotics and automation (ICRA), Montreal, Canada, 20–24 May 2019, pp. 6101–6108. IEEE.

Feature extraction and matching combined with depth information in visual simultaneous localization and mapping

Abstract

Keywords

Introduction

Related work

ORB algorithm

oFAST: FAST keypoint orientation

rBRIEF: Rotation-aware BRIEF

Qtree_ORB algorithm

Depth information prediction or completion

Feature extraction with depth information

Depth information preprocessing

Extraction

Feature matching with depth information

Descriptor

Matching

Experiments

Matching test of feature points

Trajectory error

Timing results

Conclusion

Footnotes

Author contributions

Code or data availability

Declaration of conflicting interests

Funding

ORCID iD

References