High-precision and robust localization system for mobile robots in complex and large-scale indoor scenes

Abstract

High-precision and robust localization is the key issue for long-term and autonomous navigation of mobile robots in industrial scenes. In this article, we propose a high-precision and robust localization system based on laser and artificial landmarks. The proposed localization system is mainly composed of three modules, namely scoring mechanism-based global localization module, laser and artificial landmark-based localization module, and relocalization trigger module. Global localization module processes the global map to obtain the map pyramid, thus improve the global localization speed and accuracy when robots are powered on or kidnapped. Laser and artificial landmark-based localization module is employed to achieve robust localization in highly dynamic scenes and high-precision localization in target areas. The relocalization trigger module is used to monitor the current localization quality in real time by matching the current laser scan with the global map and feeds it back to the global localization module to improve the robustness of the system. Experimental results show that our method can achieve robust robot localization and real-time detection of the current localization quality in indoor scenes and industrial environment. In the target area, the position error is less than 0.004 m and the angle error is less than 0.01 rad.

Keywords

AGV automated guided vehicle global localization high-precision localization relocalization trigger mechanism landmark

Introduction

Mobile robot localization is a key issue for long-term and autonomous navigation of various mobile robots, such as service robots, logistics robots, industrial automated guided vehicle (AGV), and self-driving cars. The localization problem of mobile robots is generally divided into global localization and pose tracking problems. Global localization mainly achieves the pose correction of mobile robots by searching in global map, especially when robots are powered on or kidnapped. Global localization is of significant importance to ensure the robust operation of mobile robots. Pose tracking mainly solves the problems of robust localization in dynamic environments and high-precision localization in target areas. For example, for industrial logistics robots, to accurately move to the loading area and accurately complete the consignment of objects, the localization error should be less than 1 cm. Due to the change or renewal of shelves in the dynamic industrial environment, the scene will change frequently. It is necessary to ensure robust localization of the mobile robots to operate in such highly dynamic environment. Therefore, for indoor mobile robots, especially industrial AGVs, the whole localization process is crucial, from global localization when robots are powered on or kidnapped to robust localization when robots moving in dynamic environments, to high-precision localization in target loading area.

For global localization, global positioning system (GPS) is often used to provide global position, but GPS will fail in occluded environments or indoor scenes. In indoor scenes, global localization could be achieved by retrieving global maps data or features. Recently, many place recognition methods have been proposed based on visual information to achieve global localization.^1

–5 Camera can provide a lot of detailed information about the surrounding environment, which is useful for place recognition or global localization. However, for constantly changing industrial scenes or vision degraded scenes, which will have a fatal impact on the place recognition or global localization, vision-based localization methods usually fail. In addition to vision-based methods, laser-based global localization is another popular method. Laser-based global localization methods usually record some keyframes in the process of map building and then use descriptors to match in the processing of global localization.⁶ However, this method consumes a large amount of computation resources.

For robust localization in dynamic environment, recently some researchers have designed various methods to detect and remove the dynamic obstacles from each laser scan.⁷ However, this method is not effective for long-term changing objects such as frequently changing shelves in factory and may make localization quality worse. Some researchers have also proposed methods to localize mobile robots in dynamic environment by updating maps dynamically.⁸ This method can be effective for improving the robustness of localization but is relatively weak in precision, and it requires updating maps frequently in a highly dynamic environment, which will waste resources.

For high-precision localization, mobile robots should achieve centimeter-level or even millimeter-level localization in the target areas. In the past two decades, the adaptive Monte Carlo localization (AMCL) method based on particle filter is widely used in two-dimensional (2D) localization of mobile robots.⁹ AMCL can achieve high robustness but it can only achieve centimeter-level localization precision. That is far from the requirement of industrial scenes, since millimeter-level localization is required to ensure safe industrial production in modern industrial scenes.

In industrial application scenes, especially for industrial AGVs, robust localization and high-precision localization are essential. To satisfy the requirements of precision and robustness, global localization and high-precision localization must be achieved simultaneously. Mobile robot localization system faces many challenges in indoor scenes, especially in the complex industrial environment. For example, long-term changes in the environment such as the movement of shelves and goods, short-term changes such as pedestrians occluding sensors and artificial landmarks, and the existence of high-reflective objects in the environment such as glass and smooth metal will bring great challenges to the mobile robot localization system. Moreover, due to the detection accuracy of the laser itself, the errors of detecting and recognizing artificial landmarks as well as the accuracy of the algorithm, the superposition of these errors will cause certain deviations of robot localization and even pose jumps.

To address the above problems, we propose a high-precision and robust localization system for mobile robots in complex and large-scale indoor scenes. The system can robustly achieve the complete working process from global localization to robust localization in dynamic environments and to high-precision localization in target loading area. The proposed localization framework is mainly composed of three modules, namely scoring mechanism-based global localization module, laser and artificial landmark-based localization module, and relocalization trigger module. The global localization module based on the scoring mechanism processes the global map according to the idea of image pyramid, scores the possible position of the robot, and then finds the pose with the lowest score as the global robot pose. High-precision and robust localization based on laser and artificial landmarks fuses AMCL¹ and artificial landmarks, which can achieve robust localization in dynamic environments and long corridor environments and achieve millimeter-level localization in target areas. The relocalization trigger module is used to detect and monitor the localization quality in real time. Moreover, the speed of localization quality evaluation is improved by building a K-Dimensional (KD)-Tree of the global map and then the localization evaluation result is fed back to the global localization module, ensuring the feedback and closed-loop of the entire localization system, which makes the mobile robot localization system more robust. Overall, the main contributions of this article are as follows:

We propose a joint localization system, which can achieve global localization and high-precision localization in indoor scenes.

A high-precision and robust localization method based on laser and artificial landmarks is proposed. Our proposed method achieves the position error of less than 0.004 m and the angle error of less than 0.01 rad in the target area. Experiment results show that our proposed method has a better performance compared to other existing methods.

We present a robust localization system for mobile robots that provides localization quality feedback. The relocalization trigger mechanism can detect the localization quality of the mobile robot in real time and feed it back to the global localization module, which could improve the robustness of the whole localization system in long-term navigation.

Qualitative and quantitative experiments in real environments show the feasibility of the proposed localization system, and our proposed method has a state-of-the-art performance in indoor scenes. Moreover, we also carry out experiments in a real industrial environment, which validates the effectiveness of our proposed method.

The rests of this article are organized as follows. In the second section, we discuss the related work. The third section describes the detail of the proposed method. We validate the performance of our methods in indoor scene and industrial environments in the fourth section and we conclude in the fifth section.

Related works

The localization problem of mobile robots can generally be divided into global localization and pose tracking.

Global localization

Vision-based global localization

In the vision-based global localization method, the image captured by mobile robots is compared with the images stored in the database and then the most similar images are identified. Some researchers have proposed the bag of words¹⁰ to describe the similarity between images, such as methods based on the speeded up robust features (SURF) descriptor¹¹ or the oriented features from accelerated segment test and rotated binary robust independent elementary features descriptor.¹² However, vision-based methods are sensitive to changes in the environment. With the continuous development of deep learning, more and more researchers begin introducing convolutional neural networks into the field of mobile robot localization to solve the problem of place recognition based on vision. Compared with the traditional descriptor method, the learning-based descriptor method has better robustness to environmental changes.^13,14 However, the disadvantage of this method is that it has a large consumption of hardware resources.

Laser-based global localization

In indoor scenes, the use of three-dimensional (3D) light detection and ranging (LiDAR) for mobile robot localization will lead to a substantial increase in hardware costs. To improve the practicability of the method and reduce the hardware cost, this article uses a 2D laser to achieve mobile robot localization. Wang et al.¹⁵ proposed a dense 2D signature and one-dimensional (1D) registration method. By establishing the search database for 2D positions and combining it with fast search technology, the 2D search space is compressed and then the heading angle is calculated by converting scanning points into 1D space and calculating the difference of scanning profiles. Millane et al.¹⁶ proposed a global localization method based on geometric features of surfaces and free regions. However, these methods require establishing the submap search database in advance when building the map, which means more resources will be occupied. In addition, Yin et al.¹⁷ proposed a LiDAR sensor information representation using the Siamese network—LocNet and then designed a Monte Carlo localization framework for global localization. Su et al.⁶ proposed a global descriptor and a local descriptor and combined the laser scanner with the camera to achieve global localization. Sun et al.¹⁸ proposed a global localization method in large scenes based on learning and filtering methods, and mobile robot pose could be estimated by a depth probability model. Ratz et al.¹⁹ proposed a global localization method—OneShot, which can estimate the global pose by relying on the point cloud description based on the learning method. However, these methods require high hardware resources and training for a long time to achieve a better global localization result.

Robust localization in dynamic environment

As a popular method for robust localization in dynamic environment, dynamic objects occurring in a short period of time are often treated as outliers, and semidynamic objects occurring in a long period of time are often treated by dynamically updating maps. Sun et al.^20,21 proposed a motion removal approach for reliable RGB-D simultaneous localization and mapping in dynamic environments. Yao et al.²² proposed a novel visual odometry based on edges and points for RGB-D cameras to achieve robust and precise localization in dynamic environments. Fox et al.⁷ applied the method of distance filtering to remove unmodeled objects. However, this method is only applicable to short-term dynamic objects. Gallagher et al.²³ proposed a system for localization and mapping, adding dynamic characteristics of the environment based on semantic tags, and dividing objects in the environment into multiple moving levels, from static level (e.g. wall) to high dynamic level (e.g. human). With the change of object position in the map, the map will also change with time. Sun et al.²⁴ proposed a method that combining particle filter, distance filter, and scan matching to achieve robust localization in dynamic environments. And dynamically updating maps can be achieved by using distance filter to incorporate dynamic factors into the sensor model. Zhang et al.⁸ proposed a hybrid signed distance function framework for laser-based localization. However, these methods need to update the global map dynamically, and there is no feedback information about the current localization quality, which makes them not robust enough for long-term operation. Meyer-Delius et al.²⁵ tracked the observations caused by unknown objects in the environment by using a temporary local map and proposed a robust localization method that relies on both these temporary local maps and the reference map of the environment to estimate the robot pose in nonstatic environment. Valencia et al.²⁶ proposed a localization method using dual time scales—DT normal distributions transform Monte-Carlo localization (NDT-MCL) to cope with environmental changes. DT NDT-MCL is a localization method based on particle filter, which uses both a prior known static map and a short-term map to achieve pose tracking. However, constantly updating maps will decrease the stability of long-term navigation.

High-precision localization

High-precision localization is mainly to achieve centimeter-level or even millimeter-level localization. Censi proposed an iterative closest point (ICP) variant—PL-ICP algorithm,²⁷ which uses point-to-line to achieve accurate scan registration. Saarinen et al.²⁸ proposed a NDT-MCL localization method that uses the NDT model to represent map and sensor data. However, the localization accuracy of this method is only about 5 cm, which cannot satisfy the accuracy requirements in industrial scenes. Goeddel et al.²⁹ propose a method for robot localization based on 3D structure and 2D map. Although many advantages of 3D matching are retained and the space and computing requirements are greatly reduced, the hardware cost of mobile robots equipped with 3D LiDAR will greatly increase.

There are also some researchers who started considering adding artificial landmarks to improve the localization accuracy of mobile robots. For example, Olson³⁰ proposed a visual system using 2D bar code style tags—AprilTag, which allows the robot to obtain six degrees of freedom (DOF) pose by using a single image. However, vision is too sensitive to the change of illumination, which reduces the robustness of the whole localization system. Sobreira et al.³¹ improved the accuracy and robustness of the whole localization system by combining 2D laser with reflectors, achieving the position error within 1 cm and the angle error within 0.2. Guo et al.³² proposed a method for improving the localization accuracy of the mobile manipulator by using dual laser rangefinders and many artificial landmarks. The method is based on trilateration, and an adaptive unscented Kalman filter method is used to improve the localization accuracy. For most experiment environments, the localization accuracy can be around 8 mm. However, the proposed localization method only works well in small experimental scenes, and more than three reflectors must be detected simultaneously to localize the robot, which makes deploying the localization system complicated.

Localization approach

The overall pipeline of the proposed high-precision and robust localization system is shown in Figure 1. The proposed localization framework is mainly composed of three modules, namely scoring mechanism-based global localization module, laser and artificial landmark-based localization module, and relocalization trigger module. When the mobile robot is powered on, the real-time laser point cloud is matched with the global map through the global localization module, and the possible robot pose is scored. Finally, the pose with the lowest score is selected as the global pose of the mobile robot. Then, the global pose output by the global localization module is sent to the laser and artificial landmark-based localization module. This module detects possible artificial landmarks from the current laser scan, removes false detections, smoothes the current pose using the Gaussian filter, and finally achieves robust localization in dynamic environment and high-precision localization in target areas. Moreover, to ensure the robustness and reliability of the whole localization system, we propose a novel relocalization trigger module. The localization quality is monitored in real time by comparing the current laser scan with the global map. Additionally, a global map KD-Tree is built to improve the evaluation speed of localization quality. Then, the localization evaluation result is fed back to the global localization module to achieve closed-loop of the entire localization system, which makes the mobile robot localization system more robust.

Figure 1.

The overview of the proposed system. The proposed localization system framework is composed of three modules: scoring mechanism-based global localization module, laser and artificial landmark-based localization module, and relocalization trigger module.

Scoring mechanism-based global localization

Processing of global map

Inspired by the idea of image pyramid in computer vision,³³ this article applies the same idea to process the global map. We use Gmapping to build a 2D global map,³⁴ and the resulting global map is essentially a gray image. Similar to the global localization algorithm,³⁵ to speed up the search speed, the 3 × 3 template block is used to downsample the global map at a rate of 2 iteratively until the map resolution is 1 × 1. Therefore, at the bottom of the pyramid is the original global map and at the top of the pyramid is a map with a resolution of 1 × 1. A global map pyramid is built by layering the processed global maps. In this article, the global map is processed into small maps with different resolutions by using the sliding window method as shown in Figure 2, which can speed up the global pose search.

Figure 2.

The diagram of the global map pyramid.

Scoring mechanism

According to the global map pyramid, the robot pose score is retrieved from the lowest resolution global map. To speed up the process of retrieval, a certain sampling point is set in the global map. When receiving the latest real-time laser point, the matching degree between the current laser point cloud and the global map under the current pose is calculated at the same time, and the score of the current real-time laser point p is calculated by Eq. (1). Every sampling point in the global map is modeled through Eq. (1) and can obtain the probability score possible robot pose

\begin{array}{l} p = & \sum_{i = 1}^{N_{scan}} (- ln (z_{hit} \frac{1}{σ \sqrt{2 π}} exp (\frac{d_{m_{i}}^{2}}{2 σ^{2}}) + z_{rand} p_{rand})) * k_{1} \\ + \frac{N_{inlier}}{N_{scan}} * k_{2} \end{array}

where $z_{hit} and z_{rand}$ are mixing weights, $d_{m_{i}}$ is the distance between the laser point i and the nearest occupied area in the map under the current pose, the definition of $d_{m_{i}}$ is as follows

d_{m_{i}} = {\begin{array}{l} {max}_{dist}, & free
 
space
 
or
 
unknown \\ 0, & occupied \end{array}

And $p_{rand}$ is unexplained random measurement. The random measurement noise is assumed to be a uniform distribution

p_{rand} = \frac{z_{rand}}{z_{max}}

And $N_{inlier}$ is the count value of scan points that can search key points with a given radius in the global map under a given sampling pose, and $N_{scan}$ is the total number of laser points in the current laser point cloud. k ₁ and k ₂ are weighting factors. Note that in practical implementation, the weighting factors k ₁ and k ₂ should be one positive and one negative.

Laser and artificial landmark-based localization

To solve the problem of mobile robots localization in dynamic long corridor environments, we propose a robust localization method based on laser and artificial landmark.

Detection and extraction of artificial landmarks

Considering that the reflector is low-cost, simple to manufacture, and has little impact on the original environment, we use it primarily as artificial landmarks for hybrid localization. During the initial stage of detection, we achieve the preliminary detection of reflectors through the intensity values of different objects in the environment. Similar to the method,³² we assume that the reflector i is detected, and d_i and $ϕ_{i}$ representing the distance and azimuth angle of the reflector in the laser coordinate system, respectively. As shown in Figure 3, we can get a series of reflector data points $P_{m}, P_{m + 1} ...... P_{n}$ for a certain reflector. Considering the parity of $n - m$ corresponding to the detected reflector i, the distance d_i and azimuth $ϕ_{i}$ of the detected reflector data can be described in the following equations

d_{i} = {\begin{array}{l} (d_{(\frac{n + m + 1}{2})} + d_{(\frac{n + m - 1}{2})}) / 2 + r, (n - m) is
 
odd \\ d_{\frac{n + m}{2}} + r, (n - m) is
 
even \end{array}

ϕ_{i} = {\begin{array}{l} (ϕ_{(\frac{n + m + 1}{2})} + ϕ_{(\frac{n + m - 1}{2})}) / 2, (n - m) is
 
odd \\ ϕ_{\frac{n + m}{2}}, (n - m) is
 
even \end{array}

Figure 3.

Detection and recognition of artificial landmarks. (a) The real scene where the mobile robot detects the artificial landmark. (b) The schematic diagram that the mobile robot detects the artificial landmark.

Pairwise consistency check

To prevent wrong data association caused by other objects with smooth surfaces, such as floor-to-ceiling glass walls (Figure 4(a)), we propose the pairwise consistency check method. The high reflective laser points will be clustered according to the distance relationship between adjacent points. We denote $N_{real}$ and $N_{theory}$ as the real laser points and the theoretical laser points of each clustered reflector. The theoretical laser points can be calculated by the following equation

N_{theory} = (2 * arctan (\frac{r / d_{i}}{\sqrt{d_{i}^{2} - r^{2}} / d_{i}})) / δ θ

where r is the radius of the reflector, d_i is the distance between the robot and the center of reflector, and $δ θ$ is LiDAR angle resolution.

Figure 4.

The diagram of pairwise consistency check. (a) The real localization scene and (b) the RViz display effectiveness. Due to the smooth surface of the glass wall on the right side, it will lead to false detection of artificial landmarks.

If the difference between $N_{real}$ and $N_{theory}$ is less than the threshold and the distance between the detected reflector and the ground truth is less than the threshold value, the current reflector is considered to be a true reflector. The false detection situation encountered by the real robot during operation is shown in Figure 4. When the mobile robot moves to this area, the smooth surface of the glass wall will cause outliers to appear in laser detection, which will lead to false detection of artificial landmarks. The pairwise consistency check will eliminate false detection and achieve accurate identification of artificial landmarks.

Observation model of artificial landmarks

Ground mobile robot has three DOF; here, we only consider x-coordinate and y-coordinate for robot position information. We propose the observation model (Eq. (7)), and the probability distribution of the current pose is updated by the following equation

\begin{array}{l} p = & {(2 π σ_{1} σ_{2} \sqrt{(1 - ρ^{2})})}^{- 1} * exp (- \frac{1}{2 (1 - ρ^{2})} * (\frac{{(x - μ_{1})}^{2}}{σ_{1}^{2}} \\ - \frac{2 ρ (x - μ_{1}) (y - μ_{2})}{σ_{1} σ_{2}} + \frac{{(y - μ_{2})}^{2}}{σ_{2}^{2}})) \end{array}

where x and y are predicted position and $μ_{1}$ and $μ_{2}$ are the ground truth x-coordinate and y-coordinate of the reflector, respectively. $σ_{1}$ and $σ_{2}$ are the standard deviation (SD) of the ground truth x-coordinate and y-coordinate of the reflector, respectively. These two parameters determine the level of trust in the observation of the reflector. $ρ$ is the covariance between the x-coordinate and y-coordinate of the reflector. Since the x-coordinate and y-coordinate of the reflector are independent each other, here the covariance $ρ$ is set to 0.

In the past two decades, the AMCL method based on particle filter could achieve good localization performance.⁹ Similar to AMCL, we use the wheel odometry to predict the robot pose and then update the particle distribution by the reflector observation model (Eq. 7), and we can obtain the robot pose by updated particle distribution. If the robot detects one or two reflectors, the reflectors are modeled by Eq. (7) to achieve the robust localization in long corridors or dynamic environments. If the mobile robot detects more than three reflectors, the method, which is in next section is used to process the reflectors, which could achieve the high-precision localization in the target areas.

High-precision localization based on artificial landmarks and laser

In this section, we use the singular value decomposition method to solve the transformation between the ground truth and the detected reflector. The idea of ICP is applied to solve pose estimation between two groups of points.³⁶ We have two sets of reflector point, which are the ground truth of reflector $P^{*} = {p_{1}^{*}, p_{2}^{*},..., p_{n}^{*}}$ and the detected reflector $P = {p_{1}, p_{2},..., p_{n}}$ . We need to find a Euclidean transformation $R, t$ to meet the following equation

\forall i, p_{i}^{*} = R p_{i} + t

where $R and t$ are the rotation matrix and translation matrix, respectively.

The errors (Eq. 9) of the i pairs of points are defined. To minimize the sum of squares of errors, we need to construct the least-squares equation (Eq. 10) and find $R, t$ that could minimize it

e_{i} = p_{i}^{*} - (R p_{i} + t)

min_{R, t} J = \frac{1}{2} \sum_{i = 1}^{n} | | (p_{i}^{*} - (R p_{i} + t {)) | |}_{2}^{2}

Then, the obtained R and t are applied to the current robot pose, and a more accurate robot pose can be obtained (Eq. 11)

x_{t}^{'} = R x_{t} + t

Robust localization based on Gaussian filter in dynamic environment

Dynamic obstacles have a great impact on localization due to their complexity and uncertainty, especially when they occlude artificial landmarks. Occluded artificial landmarks will cause robot pose to deviate and even cause pose jump. To solve this problem, we propose a robust localization method based on the Gaussian filter (Eq. 12). In this method, a Gaussian filter is used to filter and smooth the original pose, thus avoid large pose jump

G (x) = \frac{1}{\sqrt{2 π} σ} * exp (- \frac{{(x - μ)}^{2}}{2 σ^{2}})

where $μ$ is the mean value of the Gaussian template. To ensure the maximum ratio of the current pose, we set the weight of the latest pose as maximum weight. $σ$ is the SD of Gaussian distribution.

As the x-coordinate, y-coordinate, and heading angle $θ$ of the reflector are independent of each other, we use a 1D Gaussian filter to process them respectively. Figure 5 shows the schematic diagram of Gaussian filtering on the robot pose. To achieve a good filtering effect, it is necessary to select the appropriate $σ$ and convolution kernel size. Finally, the Gaussian filtered data are taken as the current robot pose.

Figure 5.

The diagram of the Gaussian filter.

Relocalization trigger module

We propose a relocalization trigger module to evaluate the current localization quality. The evaluation result is fed back to the global localization module to achieve the feedback about current robot localization quality, which makes the proposed localization method more robust.

We design and integrate the relocalization trigger module (Figure 6) into the whole localization system. The module evaluates the localization quality in real time by comparing the global map M and the real-time laser scan S_t under the current pose x_t . A nearest neighbor search method is used to find the corresponding features between the global map and the laser point cloud in real time. We search for each real-time laser point in the global map with a certain radius. If the key point can be found in the global map and the ${Num}_{keypoint}$ adds 1, the localization quality evaluation score p_i for the laser point is set $\frac{1}{{Num}_{scan}}$ . Otherwise, the score p_i for the laser point is set 0 (Eq. 13). And we could achieve the final localization quality evaluation result through the following equation

p_{i} = {\begin{array}{l} \frac{1}{{Num}_{scan}}, & {Num}_{keypoint} >1 \\ 0, & Others \end{array}

p = \sum_{i = 1}^{{Num}_{scan}} p_{i}

Figure 6.

The overview of relocalization trigger module.

The relocalization trigger mechanism feeds back to the global localization module. If the current localization quality is below a threshold value, it means that the current robot localization quality is very poor and the global localization is required to recalibrate the current robot pose. Otherwise, the global localization part will not be triggered and the current robot localization algorithm will continue to operate. The relocalization trigger mechanism provides feedback of the current localization quality, making the localization system more robust in the long-term operation.

Experiments and result

In this section, we evaluate our method using qualitative and quantitative experiments in indoor scenes. All experiments were performed on a system equipped with an Intel Core i7 CPU, 8G memory, and a 2D laser (PEPPERL+FUCHS R2000). The speed of the robot is 0.5 m/s, and the maximum rotation speed is 0.25 rad/s. The main challenges in the experimental environment are as follows: movement of semidynamic obstacles such as tables and chairs (Figure 7), occlusion of artificial landmarks by dynamic objects (Figure 8), occlusion of artificial landmarks by pedestrians, and false detection of artificial landmarks owing to transparent and high reflective objects such as glass (Figure 4). The speed of dynamic obstacles such as pedestrians is normal walking speed, which is almost 1.5 m/s. For semidynamic obstacles such as tables and chairs, the speed is considered to be 0. The whole set of experiments can be divided into global localization experiments, robust experiments in dynamic environment, high-precision experiments, and whole system operation experiments. Our experiment video is available at https://youtu.be/3LRVUiczOzc.

Figure 7.

Robust localization experiments in real dynamic environment. (a) and (b) The real robot operation scene, (c) and (d) the localization results of AMCL, and (e) and (f) the localization results of the proposed method. As shown in the blue boxes, during robot mapping, tables and chairs are close to the wall. And when localization is performed, tables and chairs are moved away. As shown in the red boxes, AMCL localization will cause large pose deviation when the environment changes, while the proposed method could still achieve robust localization. AMCL: adaptive Monte Carlo localization.

Figure 8.

Dynamic occlusion of artificial landmarks. (a) A real scene of dynamic occlusion of artificial landmarks, (b) the RViz display effectiveness of dynamic occlusion of artificial landmarks, and (c) the error curves of robot localization in the whole occlusion process, including before, during, and after the dynamic occlusion of the artificial landmarks, wherein the red curve is the position error without the Gaussian filter and the blue curve is the position error using the Gaussian filter.

Global localization

Because the high-precision localization experiment in this article has validated that laser and artificial landmark-based localization module can achieve millimeter-level localization, the ground truth in the global localization experiments comes from the laser and artificial landmark-based localization module. Similar to the article,²⁴ if the position error is below 0.3 m and the angle error is below 10, we consider that the global localization is successful. Otherwise, we consider that the global localization is failed. In this experiment, the robot pose was changed 30 times and global localization was performed every time in the entire experimental area (Figure 9). All the results are presented in Table 1. The proposed localization method could achieve the 100% localization success rate in the current experiment environment.

Figure 9.

The experimental scene of global localization. The red arrow in (a) shows the pose changement when testing the global localization, and the blue boxes represent the real position of the mobile robots when testing global localization, (b) shows the real experimental scene at point A1, (c) shows the real experimental scene at point A2, and (d) shows the real scene at point A3.

Table 1.

Global localization success rate.

Scene	Method	Success times	Number of experiment	Success rate
A1	AMCL	0	30	0%
A1	Our method	30	30	100%
A2	AMCL	2	30	6.67%
A2	Our method	30	30	100%
A3	AMCL	1	30	3.33%
A3	Our method	30	30	100%

AMCL: adaptive Monte Carlo localization.

Robust localization in dynamic environment

To evaluate the localization performance of our method in dynamic environment, we carried out several experiments in real dynamic scene. During robot mapping, tables and chairs are close to the wall. And when localization is performed, tables and chairs are moved away. As shown in the blue boxes in Figure 7(c) to (f), it can be seen that the environment around the robot has changed a lot. The red boxes in Figure 7(c) and (d) show that the AMCL localization method appears to have a large localization deviation, while the red boxes in Figure 7(e) and (f) show that the proposed method could still achieve robust localization. The main reason is that AMCL relies too much on static environmental information, such as fixed walls and unmoved tables. However, the tables and chairs in front of the mobile robot have been moved, which will make AMCL fail. Our proposed method uses artificial landmarks to improve the robustness of localization in such dynamic scenes.

We also consider the uncertainty of dynamic obstacles and the influence of pedestrian movement on robot localization, especially the occlusion of artificial landmarks. We evaluate the robustness of the method using the Gaussian filter or not. Figure 8 shows that the pose error calculated by using the Gaussian filter is smaller. As presnted in Table 2, if the Gaussian filter is not used, the calculated pose SD becomes larger. This is because that, when artificial landmarks are occluded, due to the detection error of the laser itself and the errors of detecting artificial landmarks, the superposition of these errors may cause pose deviation of mobile robots and even cause the pose jump. If the Gaussian filter is used, the previous pose will have a certain impact on the current pose, which will make the robot pose change smaller and the pose will be smoother.

Table 2.

The comparison of position SD with the Gaussian filter and without the Gaussian filter.

	The occlusion of first landmark	The occlusion of second landmark	The occlusion of third landmark	The occlusion of fourth landmark
Without the Gaussian filter	0.0055 m	0.0047 m	0.0044 m	0.0067 m
With the Gaussian filter	0.0034 m	0.0029 m	0.0023 m	0.0048 m

SD: standard deviation

High-precision localization

The lack of GPS or real-time kinematic signal indoors makes it very difficult to obtain the ground truth of localization in large-scale indoor scenes. Moreover, compared with absolute localization accuracy, it is more concerned about repeated precision for localization based on global map. Therefore, we selected randomly two positions in the area containing artificial landmarks and then remotely control the robot to move cyclically from one position to another and record the robot pose data obtained at points B1 and B2, respectively. The results are shown in Figure 10. Our proposed method can achieve a very small pose drift. As presented in Table 3, our proposed method could achieve a position error of less than 0.004 m and an angle error of less than 0.01 rad.

Figure 10.

The experiment of repeated localization precision. (a) and (b) The real experiment scene at points B1 and B2, (c) and (d) the display effectiveness in the RViz, (e) and (f) the scatter plot of the position error by using our proposed method, and (g) and (h) the scatter plot of the angle error by using our proposed method.

Table 3.

Repeated localization precision.

	B1	B2
SD of position	0.0036 m	0.0040 m
SD of angle	0.004 rad	0.0102 rad

SD: standard deviation.

Whole system operation experiments

In this section, we compare the performance of our method with AMCL. As shown in Figure 11, we use the same robot and the same path tracking algorithm to control the robot to track the same trajectory (black curve in Figure 11). The tracking trajectories are denoted as red line and blue line, respectively. As shown in Figure 11 and blue boxes in Figure 7, the tables and chairs by the wall in the real environment are moved to another position. AMCL is unable to accurately locate the robot, and there is a large pose change, which makes the robot trajectory deviate. However, our proposed method can still accurately locate the robot, and the trajectory is relatively smooth and can still ensure the stable operation of the mobile robot.

Figure 11.

The comparison of the trajectory by using AMCL or our proposed method. The black curve represents the target trajectory for path tracking, the blue curve is the trajectory using AMCL for localization, and the red curve is the trajectory using the proposed method for localization. In the zoom-in area, the AMCL causes deviation of the trajectory because of environment changement, while our method can still achieve smooth and robust localization of the mobile robot. AMCL: adaptive Monte Carlo localization.

To better validate the performance of our method, we test our method in a real industrial environment (Figure 12). The main challenges of the industrial environment are as follows: semidynamic objects, machining devices with smooth surfaces, and large scenes with a scale of 218.4 m × 109.7 m. Moreover, there is a long corridor area of 88 m, and there are dynamic machining devices on both sides of the long corridor, which brings great challenges to robot localization. Figure 13 shows a comparison of the robot localization results of the proposed method and AMCL when passing through a long corridor. As shown in Figure 13, there will be a large longitudinal error using AMCL, while our method can achieve better localization performance. The main reason is that AMCL needs to estimate the robot pose by matching the real-time laser scan with the global map. However, there is no constraint in the longitudinal direction in the long corridor, which will increase the uncertainty of the robot pose and also lead to error in the longitudinal direction. In this article, we use the observation model of artificial landmarks to update the particle distribution, which is not affected by environmental changes or the longitudinal unconstraint of the long corridor. Therefore, the robot can still achieve robust localization. Figure 14 shows the results of AMCL and the proposed method for overall localization in the 218.4m × 109.7 m scene. As shown in Figure 14, the proposed method can achieve accurate and robust localization in such challenging environments.

Figure 12.

Industrial experimental environment.

Figure 13.

The comparison of the robot localization results between the AMCL and the proposed method when passing through the long corridor. (a) AMCL localization results and (b) the localization results of the proposed method. As shown in the blue boxes, there is a large longitudinal error using AMCL, while our proposed method can achieve better localization performance. AMCL: adaptive Monte Carlo localization.

Figure 14.

The localization results of the proposed method and AMCL in large-scale scenes. (a) The running trajectory of AMCL in the scene, where (C1) and (C2), respectively, represent zoom-in views of localization results of AMCL. (b) The running trajectory of the proposed method in the scene, where (D1) and (D2), respectively, represent zoom-in views of localization results of the proposed method. And the proposed method can still achieve robust localization in large-scale indoor scenes, especially when passing through the long corridor areas and dynamic scenes. AMCL: adaptive Monte Carlo localization.

Conclusion

In this article, we propose a localization system, composing of a scoring mechanism-based global localization module, a laser and artificial landmark-based localization module, and a relocalization trigger module, to achieve high-precision and robust robot localization in complex and large-scale indoor scenes. Experiments in indoor scenes and industrial environments show that the proposed method can achieve the whole localization process, from global localization when robots are powered on or kidnapped to robust localization when robots moving in dynamic environments and high-precision localization in target loading area. The position error is less than 0.004 m and the angle error is less than 0.01 rad in the target areas. The proposed method is robust in indoor scenes, especially in dynamic environment with smooth surface, but it is not suitable for outdoor scenes. In the future, we will carry out research on high-precision localization in outdoor scenes.

Footnotes

Author’s note

Jibo Wang is also affiliated with Science and Technology on Near-Surface Detection Laboratory, Wuxi, China.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Natural Science Foundation of China (62073066, U20A20197), the Science and Technology on Near-Surface Detection Laboratory (6142414200208), the Fundamental Research Funds for the Central Universities (N182608003), and the Major Special Science and Technology Project of Liaoning Province (No. 2019JH1/10100026), and the Major Scientific and Technological Innovation Project of Shandong Province (2019JZZY010128).

ORCID iDs

Jibo Wang

Zheng Fang

Supplemental material

Supplemental material for this article is available online.

Note

References

Lowry

Sünderhauf

Newman

, et al. Visual place recognition: a survey. IEEE Trans Robot 2016; 32(1): 1–19.

Kendall

Grimes

Cipolla

. PoseNet: a convolutional network for real-time 6-DOF camera relocalization. In: 2015 IEEE international conference on computer vision (ICCV), Santiago, Chile, 13–16 December 2015, IEEE, pp. 2938–2946.

Esfahani

Yuan

, et al. A new approach to train convolutional neural networks for real-time 6-DOF camera relocalization. In: 2018 IEEE 14th international conference on control and automation (ICCA), Anchorage, Alaska, USA, 12–15 June 2018, IEEE, pp. 81–85.

Glocker

Shotton

Criminisi

, et al. Real-time RGB-D camera relocalization via randomized ferns for keyframe encoding. IEEE Trans Visu Comput Graph 2015; 21(5): 571–583.

Chen

Lam

Jacobson

, et al. Convolutional neural network-based place recognition. In: Australasian conference on robotics and automation, ACRA, Melbourne, Australia, 2–4 December 2014.

Zhou

Cheng

, et al. Global localization of a mobile robot using lidar and visual features. In: 2017 IEEE international conference on robotics and biomimetics (ROBIO), Macau SAR, China, 5–8 December 2017, IEEE, pp. 2377–2383.

Fox

Burgard

Thrun

. Markov localization for mobile robots in dynamic environments. J Artif Intell Res 1999; 11: 391–427.

Zhang

Chen

. SDF-Loc: signed distance field based 2D relocalization and map update in dynamic environments. In: 2019 American control conference (ACC), Philadelphia, PA, USA, 10–12 July, 2019, pp. 1997–2004.

Fox

. KLD-sampling: adaptive particle filters. In: Advances in neural information processing systems, Vancouver, British Columbia, Canada, 3–6 December 2001, pp. 713–720.

10.

Galvez-López

Tardos

. Bags of binary words for fast place recognition in image sequences. IEEE Trans Robot 2012; 28(5): 1188–1197.

11.

Bay

Tuytelaars

Gool

. SURF: speeded up robust features. In: Proceedings of the 9th European conference on computer vision—volume part I, Graz, Austria, 7–13 May 2006, pp. 404–417.

12.

Rublee

Rabaud

Konolige

, et al. ORB: an efficient alternative to SIFT or SURF. In: IEEE international conference on computer vision (ICCV), Barcelona, Spain, 6–13 November 2011, pp. 2564–2571.

13.

Latif

Garg

Milford

, et al. Addressing challenging place recognition tasks using generative adversarial networks. In: 2018 IEEE international conference on robotics and automation (ICRA), Brisbane, Australia, 21–26 May 2018, IEEE, pp. 2349–2355.

14.

Arandjelović

Gronat

Torii

, et al. NetVLAD: CNN architecture for weakly supervised place recognition. IEEE Trans Pattern Anal Mach Intell 2018; 40(6): 1437–1451.

15.

Wang

Zhang

Zhao

, et al. Global localization with a single-line lidar by dense 2D signature and 1D registration. IEEE Sensors J 2021; 21(10): 11497–11506.

16.

Millane

Oleynikova

Nieto

, et al. Free-space features: global localization in 2D laser SLAM using distance function maps. In: 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS), Macau, China, 5–8 November 2019, IEEE, pp. 1271–1277.

17.

Yin

Tang

Ding

, et al. LocNet: global localization in 3D point clouds for mobile vehicles. In: 2018 IEEE intelligent vehicles symposium (IV), Changshu, China, 26–30 June 2018, IEEE, pp. 728–733.

18.

Sun

Adolfsson

Magnusson

, et al. Localising faster: efficient and precise lidar-based robot localisation in large-scale environments. In: 2020 IEEE international conference on robotics and automation (ICRA), Paris, France, 31 May–31 August 2020, IEEE, pp. 4386–4392.

19.

Ratz

Dymczyk

Siegwart

, et al. OneShot global localization: instant LiDAR-visual pose estimation. In: 2020 IEEE international conference on robotics and automation (ICRA). Paris, France, 31 May–31 August 2020, IEEE, pp. 5415–5421.

20.

Sun

Liu

Meng

. Improving RGB-D SLAM in dynamic environments: a motion removal approach. Robot Auton Syst 2017; 89: 110–122.

21.

Sun

Liu

Meng

. Motion removal for reliable RGB-D SLAM in dynamic environments. Robot Auton Syst 2018; 108: 115–128.

22.

Yao

Zhang

, et al. Robust RGB-D visual odometry based on edges and points. Robot Auton Syst 2018; 107: 209–220.

23.

Gallagher

Srinivasa

Bagnell

, et al. GATMO: a generalized approach to tracking movable objects. In: 2009 IEEE international conference on robotics and automation, Kobe, Japan, 12–17 May 2009, IEEE, pp. 2043–2048.

24.

Sun

Geißer

Nebel

. Towards effective localization in dynamic environments. In: 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS), Daejeon, Korea (South), 9–14 October 2016, IEEE, pp. 4517–4523.

25.

Meyer-Delius

Hess

Grisetti

, et al. Temporary maps for robust localization in semi-static environments. In: 2010 IEEE/RSJ international conference on intelligent robots and systems, Taipei, Taiwan, 18–22 October 2010, IEEE, pp. 5750–5755.

26.

Valencia

Saarinen

Andreasson

, et al. Localization in highly dynamic environments using dual-timescale NDT-MCL. In: 2014 IEEE international conference on robotics and automation (ICRA), Hong Kong, China, 31 May–7 June 2014, IEEE, pp. 3956–3962.

27.

Censi

. An ICP variant using a point-to-line metric. In: 2008 IEEE international conference on robotics and automation, Pasadena, CA, USA, 19–23 May 2008, IEEE, pp. 19–25.

28.

Saarinen

Andreasson

Stoyanov

, et al. Normal distributions transform Monte-Carlo localization (NDT-MCL). In: 2013 IEEE/RSJ international conference on intelligent robots and systems, Tokyo, Japan, 3–7 November 2013, IEEE, pp. 382–389.

29.

Goeddel

Kershaw

Serafin

, et al. FLAT2D: fast localization from approximate transformation into 2D. In: 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS), Daejeon, Korea (South), 9–14 October 2016, IEEE, pp. 1932–1939.

30.

Olson

. AprilTag: a robust and flexible visual fiducial system. In: 2011 IEEE international conference on robotics and automation, Shanghai, China, 9–13 May 2011, IEEE, pp. 3400–3407.

31.

Sobreira

Moreira

Costa

, et al. Robust mobile robot localization based on a security laser: an industry case study. Ind Robot 2016; 43(6): 596–606.

32.

Guo

Yan

Song

, et al. Improvement of localization with artificial landmark for mobile manipulator. Int J Adv Robot Syst 2019; 16(4): 1–11.

33.

Adelson

Anderson

Bergen

, et al. Pyramid methods in image processing. RCA Eng 1984; 29: 33–41.

34.

Grisetti

Stachniss

Burgard

. Improving grid-based SLAM with Rao-Blackwellized particle filters by adaptive proposals and selective resampling. In: Proceedings of the 2005 IEEE international conference on robotics and automation, Barcelona, Spain, April 2005, IEEE, pp. 2432–2437.

35.

https://github.com/GKIFreiburg/gki_global_localization (accessed 27 April 2021).

36.

Rusinkiewicz

Levoy

. Efficient variants of the ICP algorithm. In: Proceedings third international conference on 3-D digital imaging and modeling, Quebec City, QC, Canada, 28 May–1 June 2001, IEEE, pp. 145–152.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB