Sage Journals: Discover world-class research

Abstract

This article reports our research results on an autonomous forklift, with the focus on pallet recognition and localization using an RGB-D camera. It is a fundamental issue for unmanned storehouses, which enables the forklift to insert the forks within the pallet’s slots for loading and unloading packages. Particularly, a pallet recognition and localization approach is presented. The range image is firstly segmented into planar patches based on a region growing algorithm. Then, the segments are filtered heuristically according to the storehouse environment. Afterward, a template matching method is utilized to recognize pallets in the remained segments, based on calculating the degree of similarity at each location during sliding the templates on the segment. Once a pallet has been recognized, its pose is calculated straightforward. The article has three main contributions, that is, a low-cost RGB-D camera is employed for pallet recognition and localization, where only depth information has been utilized; using the proposed method, multiple kinds of pallets can be used at the same time, which provides a flexibility for the storehouse; and furthermore, the method has a good expansibility to allow the storehouse to adopt new pallets easily.

Keywords

Pallet recognition and localization autonomous forklift unmanned storehouse

Introduction

Unmanned storehouse is one of the fundamental issues to build the automatic logistics system, which has attracted increasing attention in recent years.^1
–3 In an unmanned storehouse, as an autonomous robot, the forklift has to deal with the basic navigation problems, for example, localizing itself, planing a safe path, and navigating to the goal point. Besides these navigation requirements, pallet recognition and localization is the most important issue, which enables the forklift to insert the forks within the pallet’s slots for loading and unloading packages.

Automation of pallet engagement is not only useful for unmanned forklift. It can also be integrated on manually guided forklifts, thus enabling to load and unload pallets effectively, especially helping to avoid damages caused by operator mistakes.

In this article, we propose an approach for pallet recognition and localization using an RGB-D camera, for example, a Microsoft Kinect 2.0 camera. The proposed algorithms can be also applied to other type of RGB-D cameras. It can also be applied to pure depth cameras since only the depth channel is involved in this study. Compared against existing works (see “Related work” section), the proposed approach has the following advantages. Firstly, there is no requirement for additional modification on the pallet, such as sticking markers on the pallet. Secondly, various kinds of pallet can be recognized and localized at the same time. Particularly, commonly used plastic pallets and wooden pallets have been employed for evaluation. Thirdly, there is no constraint on the environment illumination. Fourthly, the approach has an approximately linear computational complexity, which can be easily parallelized for real-time applications.

As a result, there are two main contributions in this article. Firstly, the approach allows to use multiple kinds of pallets, which means a flexibility for storehouses. Secondly, new pallets can be easily adopted to the system; in other words, the proposed approach has a good expansibility.

To evaluate the proposed approach, a series of tests have been carried out in a simulated storehouse, as illustrated in Figure 1. The pallet localization results have been utilized to guide the forklift to load the pallets with goods. Note that the QR codes fixed on the ground are used as landmarks for the visual navigation system.

Figure 1.

Photos of the simulated storehouse, as shown; different kind of pallets are used in the storehouse. The blue lines and QR codes fixed on the ground, which can be seen on the first two figures, are artificial landmarks for the visual navigation system.

This article is organized as follows. The related work on pallet recognition and localization was given in “Related work” section, followed by detailing our planar segment-based approach in “Planar segment-based pallet recognition and localization” section. Experiments and results come in “Experiments and results” section, and the last section concludes this article.

Related work

A number of pallet recognition methods have been proposed in the past years. According to the features being used, the pallet recognition methods can be divided into two categories.

First, approaches based on artificial features (fiducials) are attached to the pallets. In the study by Seelinger and Yoder,⁴ three fiducials are placed on each pallet: a white-centered fiducial in the middle and two black-centered fiducials on either side of the pallet; then a vision-based approach has been proposed to identify the fiducials. Two reflectors are fixed left and right on the short side of the pallet in the study by Lecking et al.,⁵ which are detected using a two-dimensional (2-D) laser range finder (LRF). This approach is independent of luminance conditions. However, the 2-D LRF can only scan a slice of the environment, which requires a prior knowledge of the vertical position of the pallet, if the involved pallet is from a stack. Basically, using artificial features simplifies the problem, at the cost of labeling of all pallets with such features. Additionally, the circulation of pallets among storehouses is very strong, which limits to use them locally. Therefore, to promote these approaches in real storehouses is very difficult.

Second, approaches are based on self-contained features of the pallets, that is, shape, color, and other possible characteristics. For example, Garibotto et al.^6,7 proposed a vision-based algorithm to detect the central holes of the pallet, where the geometry model of the pallet is projected onto the image plane for position estimation. Similarly, Byun and Kim⁸ also detect the holes in the image based on monocular vision. However, different from Garibotto et al.,^6,7 the geometric model is projected onto a virtual plane which is parallel to but not the image plane. Pagès et al.⁹ proposed an image segmentation method based on the color and geometric characteristics of the pallet. For this approach, the camera should be calibrated carefully and a good illumination condition is required. Another work can be found in the study by Walter et al.,¹⁰ where a closed-loop pallet manipulation system is proposed. For pallet recognition and localization, closest edges are detected in the region of interest (ROI) of a single 2-D laser scan. However, the ROI should be specified by a remote operator which decreases the automatic level of the system. Baglivo et al.¹¹ proposed a multisensor fusion method, where a 2-D LRF and a camera have been employed. The pallet is first identified from a one-dimensional model in the laser scan, with a coarse orientation estimation. However, false positives may occur in this step. To omit the false positives and locate the pallet accurately, the candidates are further filtered using the visual information, and the remained positions are optimized based on the geometric model. However, this method requires an accurate laser-camera calibration. Stereo cameras are employed in the study by Varga et al.,¹² where a sliding window-based algorithm is employed, and pallet recognition is modeled as a classification problem of the content in the sliding window. In order to classify the content, a boosting classifier based on aggregated channel features is trained. The drawback of this approach is the training phase that requires manually labeled training sets, which is time-consuming.

From the deployed sensor point of view, the abovementioned approaches can be classified into three groups, that is, vision based (including monocular vision and stereo vision), 2-D LRF based, and vision/2-D LRF hybrid system based. Both vision and 2-D LRF have their notable limitations: lighting condition is the main challenge for vision sensors; 2-D LRF can only scan objects in one plane, which introduces difficulties for efficiency. Different from them, the depth channel of RGB-D cameras has a similar field of view as normal CCD cameras and won’t be affected by variant illumination. Therefore, we explore the possibility to use RGB-D cameras for pallet recognition and localization in this article.

For pallet localization in the real storehouse applications, there are actually four degrees of freedom (DoF) to be determined, that is, the three-dimensional (3-D) position plus the yaw angle, as there is no change with regard to the pitch angle and roll angle. Particularly, the accuracy of yaw angle estimation is very important, in order to avoid collisions during inserting the forks into the pallets. If the deployed sensor can provide range information, for example, stereo cameras and LRFs, the localization can be calculated directly from the range information. However, for monocular vision-based methods, pallet localization is usually done by affine transformation.

Planar segment-based pallet recognition and localization

As shown in Figure 1, there are multiple kinds of pallets in the simulated storehouse, which are made of different materials and have different colors. Actually, they also have different geometric characteristics, see Figure 5. However, their end faces share a generality, that is, they are planar.

We assume the goods are carried in boxes (can be large or small in size). Although the boxes may be not aligned well, they still construct an approximate plane, which can be seen in Figure 1. Furthermore, the total width of the boxes is close to the width of the pallet. This is a weak constraint in a storehouse, since most goods are transported in boxes. Especially, the width of goods won’t be too narrow or too wide for the reason of keeping balance and space efficiency.

Based on the abovementioned properties and assumption, we propose a planar segment-based method to recognize and localize variant pallets. The point cloud gathered with Kinect 2.0 is first segmented into planar segments. The segments are then filtered heuristically based on prior knowledge about the environment. Afterward, each segment which may contain a pallet is assigned with a local coordinate system, and the segment is further discretized into a 2-D grid in the local coordinate system, with binary values in each cell indicating whether the corresponding cell has been occupied by the segment. After that, each template slides on the grid to search whether a corresponding pallet presents in the segment. If yes, the pallet pose is further determined by back-projecting the matched area to the 3-D space. See the following subsections for detail.

Coordinate systems definition

There are several coordinate systems involved in this article, that is, the forklift coordinate system, the forks coordinate system, the local coordinate system for each potential segment, and the image coordinate system for the pallet template and the segment. Below, an introduction to the first three coordinate systems is given. The local coordinate system and the image coordinate system will be described in “Template matching-based pallet recognition” section.

For clarity of presentation, we note the forklift coordinate system as C_r, the forks coordinate system as C_f, and the Kinect camera coordinate system as C_k. The default Kinect coordinate system has not been used in this article; instead, C_k is defined in Figure 2. C_f is defined in Figure 3. C_r has the same orientation as C_f. Therefore, there is only a translation vector ${}_{f}^{r}t$ between C_f and C_r. Furthermore, ${}_{f}^{r}t$ changes along with the forks motion. In order to know ${}_{f}^{r}t$ in real time, we have mounted linear position sensors to track the forks motion.

Figure 2.

The Kinect camera coordinate system.

Figure 3.

The forks coordinate system.

In order to guide the forklift to engage pallets, the pallet has to be localized in both C_r and C_f, which requires the relation between them and C_k. In other words, the external parameters of the Kinect camera have to be calibrated.

Since the relation between C_r and C_f is known, we only calibrate the camera with regard to C_f. To make the calibration easy, the Kinect is mounted with its x-axis parallel to that of C_f. In this case, the position and the pitch angle have to be determined. For this purpose, the position of the Kinect camera with regard to the origin point of C_f is measured manually. In order to get the orientation of the Kinect camera, the forklift is drove to the front of a wall, with the y-axis of C_f orthogonal to the wall. Then, a plane is fitted to the wall in the point cloud using RANSAC, whose normal direction can be used to calculate the pitch angle of the Kinect.

Point cloud plane segmentation

Plane detection in 3-D point clouds is a complex task which has attracted increasing attention from both the computer graphics and robotics community in recent years.^13
–15 In this article, the hybrid region growing (HRG) algorithm¹⁶ is employed for plane segmentation. For the sake of completeness, a brief introduction is given here, please see the original article for detail.

The Hessian form equation is chosen to represent planes in this article. It is described as

\hat{n} \cdot p = d

where $\hat{n}$ is the unit normal vector of the plane, p is an arbitrary point on the plane, and d is the distance from the origin to the plane.

As the name indicates, HRG is a region growing algorithm. As subwindows are employed for efficiency, the algorithm can only deal with organized point clouds which has an image-like data structure for the nearest neighbor searching. HRG processes a given organized point cloud Ψ as follows. It is decomposed into subwindows according to the image-like data structure. Thereafter, the subwindows are classified into planar, nonplanar, and sparse based on the corresponding scatter matrix. After appearance classification, planar subwindows are then put into a list Ω and the other subwindows are put into another list Θ. When there are still unidentified planar subwindows, the subwindow ω with the minimum plane fitting error among all unidentified planar subwindows is chosen as a new seed. A growing region G is initialized by ω, and its unidentified neighbors are put into a first in last out queue Q. Then, G is extended by investigating its neighbors. If the neighbor subwindow is planar, it will be directly determined whether to be added into G. Otherwise, each point in the subwindow will be investigated separately. This process will continue until no new neighbor of G can be found. The algorithm ends when every point has been assigned to a planar segment or not.

During the region growing phase, suppose that ω_c is the neighbor subwindow being investigated. If its appearance is planar, it is assigned to G if it meets the following criteria:

The dot product between the normal vectors of ω_c and G is greater than δ. Actually $arccos ({\hat{n}}_{G} \cdot {\hat{n}}_{ω_{c}})$ is the angle between G and ω_c, so this criterion is to ensure that the investigated subwindow has a similar surface normal direction to G.

To avoid adding a subwindow which is parallel but not coplanar to G, the distance from the mass center of ω_c to the optimal plane of G should be less than γ.

To guarantee an acceptable flatness of the resulting segment, the plane fitting error e of $G \cup ω_{c}$ should be less than ε.

Otherwise, each point in the ω_c will be investigated separately, and a point p_c will be added to G if it passes the distance and fitting error test in the above criteria.

After plane segmentation, each point cloud can be represented as an indexed planar patches using a set of plane parameters

P = {P_{i} 〈 {\hat{n}}_{i}, d_{i} 〉, i = 1, 2, ... N}

Segments filtering

There are many planar surfaces other than the pallets and boxes in the storehouse, foe example, ground, wall, and pillars, see Figure 1 for example. Since the pallet recognition algorithm has a relative high computational complexity, it will be helpful if some prior knowledge can be used to filter out the unconcerned segments. For this reason, the following heuristic cues are applied.

In normal cases, the end faces of the pallets should be orthogonal to the ground. In other words, the surface normal should be close to the ground plane. Since the orientation of the RGB-D sensor is known, the angle between the surface normal and the ground plane can be easily computed. In this work, if the angle is larger than a given threshold, the corresponding segment would be filtered out.

The width of the segment should be similar to a pallet. Therefore, We detect the left and right edge points of each segment. Then, RANSAC¹⁷ is chosen to fit a 3-D line for each edge. Note that, although the line is in 3-D, they are in the same 2-D plane as they are from the same planar segment. A typical result is illustrated in Figure 4. Afterward, the width of the segment can be estimated based on the two line equations. In the end, if the resulting width is very different from the width of the pallet, the corresponding segment is filtered out.

Figure 4.

Segment edge detection and line fitting. The segments have been colored randomly, and the line is shown in blue.

Template construction

For each kind of pallet, the scale parameters are measured, as shown in Figure 5. The end face is then discretized into a grid whose cell size is 1 cm. The grid is represented as a binary image, where the cells are set to 1 occupied by the pallet. The left cells which correspond to the hole of the pallet are set to 0. For example, the binary image corresponding to pallet in Figure 5(e) is illustrated in Figure 6.

Figure 5.

Pallets have been used in this article, with different appearances with regard to their end faces. The scale parameters are represented in centimeters. (a) Material: wood. (b) Material: wood. (c) Material: plastic. (d) Material: plastic. (e) Material: plastic.

Figure 6.

The binary image corresponding to the pallet in Figure 5(e).

Template matching-based pallet recognition

After segment filtering, each potential segment P _i is assigned with a local coordinate system C_i, with its z-axis along the surface normal, x-axis parallel to the horizontal plane of the storehouse at the same time orthogonal to the surface normal, and the y-axis is determined by the right-hand rule. There is no constraint on the origin point for C_i, because the z-coordinates are omitted for template matching. Therefore, only a rotation matrix $R_{i}$ is required for coordinate transformation. According to the above description, $R_{i}$ can be derived as

R_{i} = [{\hat{c}}_{0} {\hat{c}}_{1} c_{2}]^{T}

where

{\begin{array}{l} 2 mm c_{0} = [{\hat{n}}_{i} (1) - {\hat{n}}_{i} (0) {0]}^{T} \\ 2 mm c_{2} = {\hat{n}}_{i}^{T} \\ 2 mm c_{1} = (c_{2} \times c_{0})^{T} \end{array}

In equation (3) and (4), ^⁁ stands for a unit vector, and × stands for the cross product operator.

After obtaining the rotation matrix, all points on the segment are transformed from the Kinect coordinate system to the local coordinate system. Ideally, all points on the segment should have the same z coordinates; however, due to sensor noises, they form an approximate normal distribution around the fitted plane.

The rotated segment is then discretized into a grid with the same cell size as the pallet using a similar behavior, that is, the cells which are occupied by the segment are set to 1, while the others are set to 0. The z coordinates are omitted in this step. A typical discretization example has been illustrated in Figure 7 as a binary image, where 1 is represented as white and 0 is represented as black.

Figure 7.

A typical example for segment discretization. (a) The color image corresponding to the processing point cloud data. (b) and (c) The plane segmentation result shown with different view angles, and the segments are colored randomly for better visualization. (d) to (f) The three remained segments after segments filtering, which have been assigned with local coordinate systems and discretized into 2-D grids, are represented as binary images for the purpose of visualization. 2-D: two-dimensional.

Then, every discretized pallet slides on the discretized segment; at each location, the following value is calculated to measure how the underlay patch of the segment looks like the corresponding pallet

χ = \frac{θ - μ}{1 - μ}

where χ represents the degree of similarity between the template and the current patch. In equation (5), θ is the percentage of the number of identical pixels between the template and the current patch compared against the number of all pixels in the template, and μ is the percentage of the number of white pixels compared to the number of all pixels in the template. Based on this metric, the false positive of recognizing a black patch or a white patch as a template can be avoided, since both the black part and white part of the template and patch have been considered.

The maximum χ for each template is found during the above sliding process; at the same time, the position of the patch which reaches the maximum χ is marked. If only one pallet is employed in the storehouse, the next step is to determine whether the resulting patch belongs to the pallet. Otherwise, we pick up the largest χ among all templates and only the corresponding template enters the next step.

A threshold for χ is pretuned to decide whether the current patch is recognized as the pallet corresponding to the involved template. In this article, the threshold is manually tuned, which is time consuming. However, for further study, it could be determined using machine learning methods, such as boosting¹⁸ or SVM,¹⁹ based on manually labeled data sets.

Pallet localization

Once a pallet has been recognized using the abovementioned template matching method, its position and orientation is calculated using the following method. Assuming the recognized pallet is from $P_{i}$ . Firstly, the 2-D position $(x_{m}, y_{m})$ of the middle point of the recognized patch is calculated directly in the grid. Then, the mean value of z_m coordinates is calculated for the white points in the recognized patch. As a result, the 3-D position of the pallet in the Kinect coordinate system is

p = R_{i}^{- 1} {[x_{m}, y_{m}, z_{m}]}^{T}

The yaw angle β of the pallet in the Kinect coordinate system is derived as

β = arccos ({\hat{n}}_{x y} \cdot e_{y})

where $n_{x y} = [{\hat{n}}_{i} (0), {\hat{n}}_{i} {(1),0]}^{T}$ , and e_y stands for the 3-D unit vector along the y axis.

Experiments and results

The system setup and conducted experiments are presented in this section. The system will be further benchmarked in our future research.

System setup

To construct an autonomous forklift prototype, a Kinect 2.0 RGB-D camera has been mounted on the top of the carriage of a Komatsu electric forklift, which means the RGB-D camera will move along with the forks, as shown in Figure 8. The electric system of the forklift has been modified, whose motions can be controlled both manually by the driver or via an onboard computer. A laptop is used as the onboard computer for rapid development, which will be replaced with an industrial computer. For self-localization, printed QR codes are fixed on the ground, whose locations are measured manually. A web camera is then mounted on the rear top of the forklift, which is used to recognize and localize the QR codes. In order to see the pallets on the ground, the RGB-D camera looks down at about 10°.

Figure 8.

The autonomous forklift prototype, which is modified from a Komatsu forklift truck.

The algorithms have been implemented in C++ for evaluation and all the experiments are carried on an Intel Core 2 Duo 2.53 GHz, with 4 GB RAM under Ubuntu 14.04. Libfreenect2 (open source drivers for the Kinect for Windows v2 device, available at http://github.com/OpenKinect/libfreenect2) is used as the driver for Kinect 2.0. The efficiency of linear algebra is crucial, especially for calculating the eigenvalues and eigenvectors of a matrix, since it has to be performed every step of HRG. Therefore, the Eigen library²⁰ has been employed. Furthermore, the open source point cloud library (PCL)²¹ is employed for 3-D visualization.

Data preprocessing

Different from the structured light technology in Kinect 1.0, Kinect 2.0 uses ToF for measuring depth, which delivers more accuracy depth information. However, the data are still very noisy compared to LRFs. Therefore, the data have to be filtered before processing. For this purpose, a statistical filter implemented in PCL is applied, with a typical result shown in Figure 9.

Figure 9.

A typical result of sensor data filtering.

Plane segmentation

As mentioned in “Point cloud plane segmentation” section, there are several parameters in the HRG algorithm, which has to be tuned for each kind of range sensor. In this article, the parameters in Table 1 have been used for the Kinect 2.0 RGB-D camera.

Table 1.

HRG segmentation parameters for the Kinect 2.0 RGB-D camera.

Subwindow size	δ	γ	ε
6 ×6	0.866	0.10 m	0.0005 m²

Two typical scenarios and the corresponding segmentation results are shown in Figure 10. It can be seen that the major planar surfaces could be segmented using the HRG algorithm. The average segmentation time is 49.3 ms.

Figure 10.

Two typical scenarios and the corresponding segmentation results. The segments are colored randomly for visualization.

Pallet recognition

According to the Kinect camera technique specification, the sensor works well in the range of 0.5–4.5 m. The autonomous forklift prototype has two forks of 1.2-m long, which means the pallet should be recognized when the pallet is further than 1.2 m in order to guide the forks. Furthermore, based on the QR code-based visual navigation system, the path planning modular can control the forklift to a proper position and orientation such that it is in front of the pallet. Therefore, the pallet recognition algorithm works from 1.2 m to 3.5 m from the pallet. The threshold for χ in the experiments is 0.6, which is manually tuned.

Some typical pallet recognition results are shown in Figure 11. For visualization purpose, the patch being recognized as a pallet has been colored as red in the figure. As can be seen, the approach can deal with multiple kinds of pallets, although their end faces have different appearances (see “Template construction” section). The algorithm can also recognize the pallet which is put on other goods, as illustrated in Figure 11(c) and (d). Furthermore, the algorithm can recognize multiple pallets in one point cloud, as shown in Figure 11(e) and (f). The precondition is that the pallet should be fully observed in the point cloud. Otherwise, such as what shown in Figure 11(a), the two partially observed pallets cannot be recognized.

Figure 11.

Some typical pallet recognition results, where the patch being recognized as a pallet has been colored red. As shown, the proposed algorithm can deal with multiple kinds of pallets and pallets on other goods.

However, there are some exceptions shown in Figure 12. Actually, the scenario in Figure 12(a) is the same as Figure 11(e). From the experimental results, the left pallet cannot be stably recognized. It is made of wooden, and its color is much darker than others cause of rotten. Therefore, a part of it will be missed in the range information, which results failure recognition cases. We call this exception as “solid part missed.” The pallet in Figure 12(b) cannot be recognized due to a distinct reason. It can be seen that the pallet has been encysted together with the goods by plastic membranes. As a result, the hole of the pallet is filled in the point cloud. We call this exception as “hole filled.”

Figure 12.

Pallet recognition failures. The left pallet in (a) cannot be stably recognized because the pallet color is too dark due to the underlay wood is rotten, resulting data missing in the range data. The pallet in (b) cannot be recognized as its hole is filled by plastic membranes.

In the current implementation, about 33 ms is required to search the pallet for each potential segment. In most cases, there are less than three potential segments in one point cloud. In other words, the maximum time spend for pallet recognizing is 100 ms. It should be noted that the template matching process can be easily parallelized for further optimization.

Pallet localization

As mentioned above, the pallet is localized in the Kinect camera coordinate system; however, the result is used to guide the forks operation and forklift motion control. As there is no motion capture system with high accuracy in the simulated storehouse for obtaining the ground truth of pallet localization. The localization accuracy is then tested with guiding the pallet engagement. To make the evaluation easier, we eliminated the complex motion control from the engagement. For this purpose, the forklift is manually driven to the front of a pallet. In this case, the forklift can insert its forks by only going forward with little turnings and adjusting its forks, where the forks has two DoFs, that is, the up ⇔ down motion and left ⇔ right motion.

The experiment is then to insert the forks to the pallet slots automatically based on the pallet localization result. After each experiment, the bias from the forks’ end point to the middle of the pallet slots is measured as the localization error. Please note that, the forks motion control error has also been included in the measured bias. Even though, the maximum bias along the horizontal and vertical direction is below 0.03 m. Since angle is difficult to measure, the yaw angle estimation error has not been evaluated yet. However, the accuracy is enough for pallet engagement, as well as keeping the balance of goods.

Conclusion and discussion

In this article, we have presented a novel pallet recognition and localization approach. The range image is firstly segmented into planar patches based on a hybrid region growing algorithm. Then, the segments are filtered heuristically based on prior knowledge about the environment. Afterward, a template matching method is utilized to recognize pallets in the remained segments, based on calculating the degree of similarity at each location during sliding the templates on the segment. Once a pallet has been recognized, its position and orientation is calculated straightforward.

The article has three main contributions, that is, a low-cost RGB-D camera has been employed for pallet recognition and localization, where only depth information has been utilized. Using the proposed method, multiple kinds of pallets can be used at the same time, which provides a flexibility for the storehouse. Furthermore, the method allows the storehouse to adopt new pallet; in other words, it has good expansibility.

However, there are limitations in this study. Firstly, the accuracy of pallet localization has not been benchmarked. In real-world applications, it is coupled with the accuracy of forklift control, including the wheel movement control and the fork motion control. Secondly, the color channel of the RGB-D camera has not been employed yet, which is helpful for pallet recognition. Thirdly, the thresholds for segment filtering and pallet recognition are still manually tuned, which is time consuming. Furthermore, as Kinect has been employed, the proposed approach currently can be used indoor only.

Therefore, in the future, we will try to fuse the depth channel and color channel for pallet recognition, benchmark the localization accuracy using a motion capture system, and get the thresholds based on machine learning algorithms.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by National Science Foundation of China (no 61503401 and no 61403409) and China Postdoctoral Science Foundation (no 2014M562648).

References

Wurman

D’Andrea

Mountz

. Coordinating hundreds of cooperative, autonomous vehicles in warehouses. AI Mag 2008; 29(1): 9.

D’Andrea

. Guest editorial: a revolution in the warehouse. A retrospective on kiva systems and the grand challenges ahead. IEEE Trans Autom Sci Eng 2012; 4(9): 638–639.

Gadd

Newman

. A framework for infrastructure-free warehouse navigation. In: 2015 IEEE international conference on robotics and automation (ICRA), Seattle, WA, USA, 26–30 May 2015, pp. 3271–3278. IEEE, http://ieeexplore.ieee.org/document/7139650/

Seelinger

Yoder

. Automatic pallet engagment by a vision guided forklift. In: IEEE international conference on robotics and automation, Barcelona, Spain, Spain, 18–22 April 2005, pp. 4068–4073. IEEE, http://ieeexplore.ieee.org/document/1570744/

Lecking

Wulf

Wagner

. Variable pallet pick-up for automatic guided vehicles in industrial environments. In: IEEE conference on emerging technologies and factory automation, Prague, Czech Republic, 20–22 September 2006, pp. 1169–1174. IEEE, http://ieeexplore.ieee.org/document/4178257/

Garibotto

Masciangelo

Ilic

. Robolift: a vision guided autonomous fork-lift for pallet handling. In: IEEE/RSJ international conference on intelligent robots and systems, Osaka, Japan, Japan, 8 November 1996, pp. 656–663. IEEE, http://ieeexplore.ieee.org/document/571028/

Garibotto

Masciangelo

Bassino

. Computer vision control of an intelligent forklift truck. In: IEEE conference on intelligent transportation system, Boston, MA, USA, USA, 12 November 1997, pp. 589–594. IEEE, http://ieeexplore.ieee.org/document/571028/

Byun

Kim

. Real-time positioning and orienting of pallets based on monocular vision. In: 20th IEEE international conference on tools with artificial Intelligencei(ICTAI’08). Vol. 2, Dayton, OH, USA, 3–5 November 2008, pp. 505–508. IEEE, http://ieeexplore.ieee.org/document/571028/

Pagès

Armangué

Salvi

. A computer vision system for autonomous forklift vehicles in industrial environments. In: Proceedings of the 9th mediterranean conference on control and automation MEDS, Dubrovnik, Croatia, 27–29 June 2001, pp. 1–6. Mediterranean Control Association (MCA), http://med-control.org/index_Conferences.php

10.

Walter

Karaman

Frazzoli

. Closed-loop pallet manipulation in unstructured environments. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), Taipei, Taiwan, 18–22 October 2010, pp. 5119–5126. IEEE, http://ieeexplore.ieee.org/document/5652377/

11.

Baglivo

Biasi

Biral

. Autonomous pallet localization and picking for industrial forklifts: a robust range and look method. Meas Sci Technol 2011; 22(8): 5502–5512.

12.

Varga

Costea

Nedevschi

. Improved autonomous load handling with stereo cameras. In: IEEE international conference on intelligent computer communication and processing (ICCP), Cluj-Napoca, Romania, 3–5 September 2015, pp. 251–256. IEEE, http://ieeexplore.ieee.org/document/7312639/

13.

Kaushik

Xiao

. Accelerated patch-based planar clustering of noisy range images in indoor environments for robot mapping. Robot Auton Syst 2012; 60(4): 584–598.

14.

Qian

. Ncc-ransac: a fast plane extraction method for 3-d range data segmentation. IEEE Trans Cybern 2014; 44(12): 2771–2783.

15.

Borrmann

Elseberg

Lingemann

. The 3D hough transform for plane detection in point clouds: a review and a new accumulator design. 3D Res 2011; 2(2): 1–13.

16.

Xiao

Zhang

Adler

. Three-dimensional point cloud plane segmentation in both structured and unstructured environments. Robot Auton Syst 2013; 61(12): 1641–1652.

17.

Fischler

Bolles

. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 1981; 24(6): 381–395.

18.

Freund

Schapire

Abe

. A short introduction to boosting. J Japan Soc Artif Intell 1999; 14: 771–780.

19.

Suykens

Vandewalle

. Least squares support vector machine classifiers. Neural Process Lett 1999; 9(3): 293–300.

20.

Guennebaud

Jacob

., 2010. Eigen v3. http://eigen.tuxfamily.org/index.php?title=Main_Page

21.

Rusu

Cousins

. 3D is here: point cloud library (PCL). In: IEEE international conference on robotics and automation (ICRA). Shanghai, China, 9–13 May 2011, pp. 1–4. IEEE, http://ieeexplore.ieee.org/document/5980567/

Pallet recognition and localization using an RGB-D camera

Abstract

Keywords

Introduction

Related work

Planar segment-based pallet recognition and localization

Coordinate systems definition

Point cloud plane segmentation

Segments filtering

Template construction

Template matching-based pallet recognition

Pallet localization

Experiments and results

System setup

Data preprocessing

Plane segmentation

Pallet recognition

Pallet localization

Conclusion and discussion

Footnotes

Declaration of conflicting interests

Funding

References