A lightweight color and geometry feature extraction and fusion module for end-to-end 6D pose estimation

Abstract

Although advancements in red–green–blue-depth (RGB-D)-based six degree-of-freedom (6D) pose estimation methods, severe occlusion remains challenging. Addressing this issue, we propose a novel feature fusion module that can efficiently leverage the color and geometry information in RGB-D images. Unlike prior fusion methods, our method employs a two-stage fusion process. Initially, we extract color features from RGB images and integrate them into a point cloud. Subsequently, an anisotropic separable set abstraction network-like network is utilized to process the fused point cloud, extracting both local and global features, which are then combined to generate the final fusion features. Furthermore, we introduce a lightweight color feature extraction network to reduce model complexity. Extensive experiments conducted on the LineMOD, Occlusion LineMOD, and YCB-Video datasets conclusively demonstrate that our method significantly enhances prediction accuracy, reduces training time, and exhibits robustness to occlusion. Further experiments show that our model is significantly smaller than the latest popular 6D pose estimation models, which indicates that our model is easier to deploy on mobile platforms.

Introduction

Six degree-of-freedom (6D) object pose estimation aims to estimate the rigid transformation from the coordinate system of an object to the coordinate system of the employed camera. It is one of the core steps used in a wide range of robotic tasks, such as grasping objects ¹ and operating tools. ² A key challenge in 6D object pose estimation is occlusion, where objects of interest are partially or fully hidden, hindering accurate pose estimation due to missing or obscured features. Handling such common real-world scenarios is vital for robust pose estimation models. The occlusion problem has been historically approached in various ways. Early methods^3,4 relied on local feature matching, which could handle occlusion to some extent but was limited by viewpoint changes and performed poorly on texture-less objects. With the rapid development of deep learning in computer vision, several works use convolutional neural networks (CNNs) on red–green–blue (RGB) images to solve the occlusion problem. Two main categories of approaches have been developed: one-stage^5,6 and two-stage methods.^7–9 The former methods directly regress the 6D pose of an object from input images, and the latter methods first establish three-dimensional to two-dimensional (3D to 2D) correspondences and then use the Perspective-n-Point (PnP) algorithm to calculate the 6D pose of the object of interest. Two-stage methods generally have stronger robustness to occlusion, with a representative method being Pixel-wise Voting Network (PVNet). ⁷ It first uses a CNN to predict the directional vector from each pixel to the key points. Then it acquires the positions of the key points through vector voting. This voting mechanism enables it to learn the relationship between different parts of an object, thereby allowing occluded key points to be reliably recovered by visible parts. Despite having higher accuracy than traditional methods, these RGB-based methods have limited performance in occlusion scenes due to their lack of depth information.

With the emergence of consumer-level RGB-depth (RGB-D) cameras, more recent approaches have turned to using RGB-D images to address this problem. Incorporating depth information enables these methods to outperform those that only use RGB images in occluded environments. Early works^10–12 use depth information to refine previous estimations with the iterative closest point (ICP) algorithm or directly use depth information as an additional channel in their network architectures for RGB images. However, ICP is time-consuming, and due to the difference between the depth feature space and the color feature space, the depth information is not fully utilized. More recently, DenseFusion ¹³ uses different networks to extract color and geometry information and then densely fuses the extracted results at the pixel level to perform pixel-wise pose estimation, achieving great performance. However, this approach merely extracts features from each point individually, discarding the local information in the point cloud. Furthermore, Zhou et al. ¹⁴ and He et al. ¹⁵ used PointNet++ ¹⁶ to extract the local geometric features of the object, but all neighbor processing operations in PointNet++ are isotropic, which limits the geometric feature extraction performance of the network. Although the decent achievements of 6D pose estimation methods based on RGB-D images, a persisting challenge lies in effectively leveraging the color information in the RGB images and the geometric information in the depth images. Indeed, this represents a pivotal aspect in tackling the 6D object pose estimation under occlusion conditions.

Another challenge faced by deep learning-based 6D pose estimation algorithms is that they often have many model parameters and high computational complexity, which limits their deployment on mobile platforms with limited computing resources and small storage spaces. In addition, these methods often require hundreds of epochs to converge, which increases their time costs in practical applications.

In this paper, we introduce an end-to-end 6D object pose estimation method with a two-stage feature fusion module that can efficiently leverage the color and geometry information in RGB-D images. We find that when observing an object under heavy occlusion, human beings can simultaneously perceive the object’s appearance and geometric information and infer the whole pose from a local part. Therefore, we first extract color features from RGB images and integrate them into the target point cloud to obtain a point cloud with rich appearance information. Then, we process the point cloud with an anisotropic separable set abstraction network (ASSANet)-like network to obtain some local features and one global feature. The global feature is concatenated with each local feature to obtain the fusion features, and each fusion feature independently predicts the 6D pose with an associated confidence score. We choose the pose with the highest confidence as the final output. Finally, the pose is further improved via the iterative refinement module in Wang et al. ¹³ To further reduce the model size, we design a lightweight network based on a changed ShuffleNet V2 for color feature extraction. The proposed method is evaluated on three benchmark datasets: LineMOD, ¹⁷ Occlusion LineMODm ¹⁸ and YCB-Video. ¹¹ The experimental results show that our method exhibits better performance than competing approaches on all datasets. Moreover, our model is much smaller than the current popular 6D pose estimation methods, highlighting its suitability for deployment on mobile platforms.

The contributions of this work can be summarized as follows:

We design a two-stage fusion module that can fully leverage two complementary data sources from RGB-D images, which makes it robust for handling heavy occlusion in 6D object pose estimation tasks.

We propose a lightweight color feature extraction network that significantly reduces our model’s size and computational cost.

The experimental results obtained on the LineMOD, Occlusion LineMOD, and YCB-Video datasets show that our method achieves significantly boosted performance.

Related work

Pose estimation with RGB data

Since RGB images are easy to obtain, many studies use only RGB images for pose estimation. Traditional methods^1,19 generally perform pose estimation based on key point extraction and matching algorithms. Such methods are often sensitive to cluttered backgrounds and light changes. With the great success of deep learning technology in 2D vision, many tasks have used deep learning for estimating poses from RGB images. PoseNet ²⁰ was the first method to use a CNN to solve the 6D pose estimation problem and can adapt to the environment well. SSD-6D ⁶ discretizes the rotation space into a classifiable set of viewpoints and transforms the 6D pose estimation task into a classification problem. However, these two methods are highly dependent on time-consuming pose optimization operations to improve their performance. In contrast, EfficientPose ²¹ introduces the EfficientDet ²² structure to construct a pose estimation framework for directly predicting the 6D attitudes of objects. The utilization of a novel 6D enhancement strategy allows it to achieve better performance without optimization. Deep-6DPose ⁵ adds a pose estimation branch to a mask region-based CNN ²³ to directly estimate the 6D poses of objects without any postprocessing processes.

Compared with the direct prediction of 6D object poses, establishing a 2D to 3D correspondence and taking it as the indirect expression of a pose to calculate the corresponding 6D pose often yields more accurate results. BB8 ²⁴ predicts the 2D projections of the vertices of a 3D bounding box to construct the 2D to 3D correspondence. Similar to BB8, YOLO-6D ²⁵ uses a single-shot neural network to directly detect the key points of an object. PVNet ⁷ adopts a key point location strategy based on voting. It first trains a CNN to predict the direction vector from each pixel to the key point and then uses the vector of pixels belonging to the target object to vote on the key point position. Yu et al. ²⁶ proposed an effective loss based on PVNet to achieve more accurate vector field estimation results by incorporating the distances between pixels and key points into the target. However, these methods cannot be trained in an end-to-end manner. Therefore, Single-stage ²⁷ and GDR-Net ²⁸ both established 2D to 3D correspondences but attempted to learn the PnP algorithm in an end-to-end fashion. However, due to the loss of geometric information caused by perspective projection, RGB-based methods are sensitive to illumination changes, serious occlusion situations, and cluttered backgrounds.

Pose estimation with depth/point clouds

The emergence of inexpensive depth sensors has led to some methods that use point clouds or depth data. These methods take point clouds or depth images as inputs and utilize 3D CNNs or point cloud networks for geometry feature extraction to estimate 6D poses. VoxelNet ²⁹ and Frustrum PointNets ³⁰ are both PointNet-like structures that have achieved great performance on the KITTI benchmark dataset. Zhang et al. ³¹ first used PointNet to extract high-dimensional features from an input 3D point cloud, then performed feature dimensionality reduction and fusion, and finally regressed the corresponding 6D pose. However, the lack of object appearance information limits the performance of these methods in challenging scenes.

Pose estimation with RGB-D data

With the rapid development of hardware, the use of RGB-D information to estimate the 6D poses of objects has gradually become a hot research direction. PoseCNN ¹¹ estimates 6D poses from RGB images and then uses depth images to refine the poses. However, this refinement process is time-consuming and cannot be trained together with the pose estimation network. Li et al. ¹² used depth information as an additional channel of a network constructed for RGB images, ignoring the difference between the depth feature space and the color feature space and thus failing to fully utilize the RGB-D information. Recently, DenseFusion ¹³ uses a CNN and PointNet ³² to process the data acquired from RGB and depth images, respectively, and performs pixel-wise fusion. However, this approach discards the local information in the point cloud. A deep point-wise 3D keypoints voting network for 6 degrees of freedom pose estimation (PVN3D) ¹⁵ predicts the depth information for key points to convert these key points from the 2D space to the 3D space and optimizes the 6D pose estimation process according to the geometric information of the object of interest itself. Furthermore, Zhou et al. ¹⁴ and He et al. ¹⁵ used PointNet++ ¹⁶ to extract local geometric information from a point cloud. However, PointNet++ treats all local points equally, which limits the extraction of geometric information. In contrast, we use an ASSANet-like ³³ network to efficiently extract the geometric information from a point cloud and perform both color-depth and local–global fusion to obtain a better representation of the observed RGB-D information.

The proposed method

This paper aims to estimate the 6D pose of a known object from an RGB-D image. The 6D pose represents the camera coordinate system’s translation and rotation transformations relative to the original object’s world system. Generally, rotation and translation are represented by a rotation matrix $\mathbf{R}\in SO(3)$ and a translation matrix $\mathbf{t}\in R^3$ , respectively.

Overview of the network

Our overall network architecture, shown in Figure 1, is mainly divided into three stages. In the first stage, the input RGB-D image is semantically segmented to obtain the target region. During the second stage, we first use a lightweight network to extract color features from the RGB image and integrate the extracted color features into a point cloud (converted from depth maps) according to the corresponding relationships between the depth maps and color maps for color-depth fusion. Then, an ASSANet-like network is used to extract several local features and one global feature from the target point cloud, and the global features are copied and fused with each local feature for local–global fusion. The fusion features are sent to a pose predictor network to obtain the corresponding 6D poses with their confidence scores in the last stage. We choose the 6D pose with the highest score as the final output. In addition, we add a refinement network in Wang et al. ¹³ to obtain a better estimation. Each part of Figure 1 is described in detail in the subsequent sections.

Figure 1.

The overall framework of our approach. The input RGB-D image is segmented to obtain the target region. The color features extracted by a lightweight network are fused with the point cloud, and several local features and one global feature are extracted from the point cloud. Then, we fuse the global feature with the local features and send the fused features to the pose predictor network to obtain the corresponding 6D poses. Finally, we add a refinement network to obtain a better estimation. RGB-D: red–green–blue-depth; 6D: six degree-of-freedom; CNN: convolutional neural network; PSP: pyramid scene parsing; ASSANet: anisotropic separable set abstraction network.

Preprocessing

Before performing feature extraction, we first segment the target region derived from the input image. According to the segmentation results, we crop the RGB image and convert the depth image to a point cloud using the camera’s internal parameters. Since this part is not our focus, we directly adopt the segmentation network provided by Xiang et al. ¹¹

Color feature extraction and color-depth fusion

The cropped image is passed through a color feature extraction network to produce pixel-wise color embeddings. Unlike previous works,^13,14 we design a lightweight feature extraction network for color feature extraction, which reduces the number of parameters and the computational complexity of the model. This network comprises a variant of ShuffleNet V2, ³⁴ a pyramid scene parsing (PSP) module ³⁵ and four upsampling layers, as illustrated in Figure 2. The original ShuffleNet V2 unit for spatial downsampling, shown in Figure 3, uses DWConv and a channel random mixing strategy to reduce the model size. Among the calculations of ShuffleNet V2, the 1 × 1 convolution accounts for most of the complexity. The 1 × 1 convolutions before and after DWConv are mainly used to fuse the information between the channels or change the dimensions. Since we do not need to change the dimensionality here, we only need one 1 × 1 convolution to compensate for the lack of DWConv for conducting information fusion between channels. Therefore, we crop the 1 × 1 convolution after DWConv in branch 2 to further reduce the computational cost.

Figure 2.

Structure of our color feature extraction network. RGB: red-green-blue; PSP: pyramid scene parsing.

Figure 3.

The original ShuffleNet V2 unit for spatial downsampling. DWConv: depthwise convolution; BN: batch normalization; ReLU: rectified linear unit; Conv: convolution.

Specifically, the cropped RGB images of size $\mathbf{H}\times \mathbf{W}\times \mathbf{3}$ are extracted through the changed ShuffleNet V2 network to obtain a feature map of size ${\mathbf{H}}_{\mathbf{1}}\times {\mathbf{W}}_{\mathbf{1}}\times {\mathbf{d}}_{\mathbf{1}}$ , and then the PSP module is used to further fuse the information with different scales to obtain a feature map of size ${\mathbf{H}}_{\mathbf{2}}\times {\mathbf{W}}_{\mathbf{2}}\times {\mathbf{d}}_{\mathbf{2}}$ . Because we need to fuse color features and point clouds next, we use four upsampling layers to restore the feature map to the original input image size and obtain the final output feature map of size $\mathbf{H}\times \mathbf{W}\times {\mathbf{d}}_{\mathbf{r}\mathbf{g}\mathbf{b}}$ . Each pixel of the feature map is a ${\mathbf{d}}_{\mathrm{rgb}}$ -dimensional vector that represents the input image’s appearance information. Then, we perform the first fusion process. Specifically, we randomly select $\mathbf{N}$ points from the target point cloud and then find N corresponding ${\mathbf{d}}_{\mathrm{rgb}}$ -dimensional feature vectors according to the corresponding relationships between the depth and color maps. Finally, we splice the vectors together to obtain a point cloud of size $\mathbf{N}\times \mathbf{(}\mathbf{3}\mathbf{+}{\mathbf{d}}_{\mathrm{rgb}}\mathbf{)}$ with rich appearance information as the input for the next stage.

Geometric feature extraction and local–global fusion

Effectively extracting local geometric information from point clouds is a challenging task. Previous methods^14,15 typically employed PointNet++ to extract information from the point cloud, yet PointNet++ treats all local points in an isotropic manner. In contrast, we use a network based on the anisotropic separable set abstraction (ASSA) module to deal with the target point cloud. The ASSA module performs set abstraction operations in separate directions independently, facilitating the capture of anisotropic patterns and enhancing the performance of local information extraction. Furthermore, the ASSA module is more efficient and has a faster inference speed, which aligns with our intention to create a lightweight and efficient model. As shown in Figure 4, this module consists of five layers: a subsampling layer, a grouping layer, a geometry-aware anisotropic reduction layer, and two multilayer perceptron (MLP) layers. Specifically, taking the resulting point cloud of size $\mathbf{N}*\mathbf{3}$ , we first execute the MLPs on all $\mathbf{N}$ points directly. In the subsampling layer, we use the farthest point sampling algorithm to select ${\mathbf{N}}^{\mathrm{\prime }}$ points from the input points; these are the most distant points from each other, and they define the centers of the local areas. Then, the grouping layer uses the ball query method to find all points within a radius r of each center point and chooses the k nearest neighbor points from them. The point cloud is divided into n areas, and each area contains a center point and k neighbor points. Next, in the anisotropic reduction layer, we calculate the relative positions between the neighbor points and each center point in the x, y, and z dimensions. Then, the relative positions are used as scaling weights for aggregating the features across the neighborhood dimension, and they are normalized by the radius of the ball query r. The neighborhood features are individually scaled by the three corresponding relative positions. The three scaled neighborhood features are then combined and fed to another network for neighborhood information aggregation using a symmetric reduction function. Another MLP block then processes the output of the anisotropic reduction layer. Finally, we add a residual connection between the outputs of the two MLP parts. The design of this residual connection is inspired by deep separable convolution, which stabilizes the training process and provides better feature embeddings by fusing the aggregated local information with the point information.

Figure 4.

ASSA module. ASSA: anisotropic separable set abstraction; MLP: multilayer perceptron.

Our geometric feature extraction network consists of three ASSA modules. The second module outputs ${\mathbf{N}}_{\mathrm{local}}$ local features, and the last module further processes the local features to obtain one global feature. Then, we perform the local–global fusion. Specifically, we copy the ${\mathbf{N}}_{\mathrm{local}}$ global features and splice them with each local feature to obtain ${\mathbf{N}}_{\mathrm{local}}$ fused features. This fusion process combines the detailed local information with a broader contextual understanding of the global features, resulting in a more comprehensive representation. This representation is particularly effective in scenarios with severe occlusions, as it allows the model to infer the structure of partially visible objects by leveraging both local and global contextual cues. The effectiveness of this two-stage fusion in handling occlusions is further illustrated in our experimental results, demonstrating its superiority in capturing the complete essence of objects even when severely occluded.

Pose estimation and refinement

Pose estimation

We feed the obtained fusion features into a pose prediction network to obtain 6D poses. This network has three branches. Each branch consists of three 1 × 1 convolution layers and outputs the predicted rotation parameter R, translation parameter t, and confidence c. Each fusion feature independently predicts a 6D pose and a corresponding confidence score. We choose the pose with the highest confidence as the final output. This process enables the model to independently predict the pose from various local regions, each offering a distinct perspective. Thus, even when parts of the object are obscured, the model is capable of inferring the overall pose from regions that are less occluded. The final pose is determined based on the prediction with the highest confidence score, derived from these diverse local assessments. This method effectively leverages local information to navigate the challenges posed by severe occlusion, ensuring accurate pose estimation despite the object’s partial visibility.

Pose refinement

We further optimize the predicted pose to improve the resulting pose estimation accuracy. The standard ICP optimization algorithms are time-consuming and cannot be trained in an end-to-end manner. Therefore, we adopt the iterative refinement structure in Wang et al., ¹³ which is fast and can be jointly trained with our main network for end-to-end pose estimation. The refinement process is shown in Figure 5. The structure of the refinement network is similar to that of the main network except that our fusion module outputs only one global feature instead of fusion features. The global feature goes through two small regression networks composed of fully connected layers to return a single pose. Specifically, the input point cloud is first transformed according to the initial pose predicted by the pose prediction network. Then, the transformed input point is fed with the original color feature into a refinement network to obtain a residual pose. Finally, we transform the target point cloud again according to the residual pose and send it to the next iteration. After T iterations, we concatenate the results of each iteration to obtain the final pose estimate:

$$\tilde{p}=[R_T\mid t_T][R_{T-1}\mid t_{T-1}]\dots [R_0\mid t_0]$$

(1)

Figure 5.

The pose refinement procedure. ASSANet: anisotropic separable set abstraction network; 6D: six degree-of-freedom.

Loss functions

During the training phase, we define two different loss functions for symmetric and asymmetric objects. For asymmetric objects, to minimize the average offset between the point cloud randomly sampled on the object model for the real pose and the point cloud of the corresponding object in the predicted pose, the loss function is defined as:

$$L_i^p=\frac{1}{M}\sum _{j=1}^M\Vert (R{x}_j+t)-(\widehat{R_i}{x}_j+\widehat{t_i})\Vert$$

(2)

where $x_j$ denotes the jth point of the M randomly sampled 3D points from the object’s 3D computed-aided design model, $p=[R|t]$ is the ground-truth pose, and $\hat{p}=[\hat{R}|\hat{t}]$ is the pose predicted through the network.

Symmetric objects can have the same appearance under different poses in RGB-D images. Therefore, another loss function is introduced to minimize the average distance between each point in the predicted object model and the closest points in the object model for the real pose. The loss function is as follows:

$$L_i^p=\frac{1}{M}\sum _{j=1}^M\underset{0<k<M}{min}\Vert (R{x}_j+t)-(\widehat{R_i}{x}_j+\widehat{t_i})\Vert$$

(3)

We add a confidence term to the loss function to learn the confidence score of the prediction. The overall loss function is derived as:

$$L=\frac{1}{N_r}\sum _{i=1}^{N_r}(L_i^p{c}_i-\omega \log (c_i))$$

(4)

where $N_r$ is the number of local areas, ω is a balancing hyperparameter, and $c_i$ is the $i\mathrm{th}$ confidence score. We use the estimated pose with the highest confidence score as the final output.

Experiments

Datasets and metrics

The LineMOD dataset ¹⁷ contains 15,783 images of 13 different objects, with 1200 images for each object and the corresponding 6D pose annotations. The dataset includes challenging cluttered scenes, texture-less objects, and lighting condition variations, making accurate object pose prediction difficult. The LineMOD dataset has become the most commonly cited pose estimation dataset, and newly proposed estimation methods typically use it as their test benchmark.

The Occlusion LineMOD dataset ¹⁸ was generated from the LineMOD dataset to compensate for the lack of occlusion in the latter dataset. It consists of 1214 RGB images belonging to LineMOD’s bench vise sequence, and it additionally annotates the true 6D poses of another eight heavily occluded visible objects.

The YCB-Video dataset ¹¹ consists of 92 RGB-D video sequences, with 130,000 frames of real scene image data and 80,000 frames of synthetic model-rendered image data, covering 21 irregular objects. Each object has varying external influences, such as occlusion or poor lighting. The videos are annotated with 6D poses and instance semantic masks.

We adopt the average distance ADD ¹⁷ and ADD-S ¹¹ metrics, which are commonly used in 6D pose estimation, to evaluate our method. The ADD metric is defined as the mean pairwise distance between the model points transformed by the predicted pose $[R,t]$ and the ground-truth pose $[\hat{R},\hat{t}]$ .

$$\mathrm{ADD}=\frac{1}{M}\sum _{x\in M}\Vert (Rx+t)-(\hat{R}x+\hat{t})\Vert$$

(5)

For symmetric objects, the average distance is computed using the closest point distance:

$$\text{ADD-S}=\frac{1}{M}\sum _{x_1\in M}\underset{x_2\in M}{min}\Vert (R{x}_1+t)-(\hat{R}{x}_2+\hat{t})\Vert$$

(6)

For an evaluation conducted on the LineMOD and Occlusion LineMOD datasets, we report the accuracy in terms of ADD(-S) that are $<10\mathrm{\%}$ of the corresponding object’s diameter. For the YCB-Video dataset, we report the area under the ADD-S curve (AUC) and the percentage of ADD-S values that are smaller than 2 cm (2 cm is used for most robot grippers).

Implementation details

The color feature extraction network takes an RGB image of size $\mathbf{H}\times \mathbf{W}\times \mathbf{3}$ as its input, the ShuffleNet V2 network outputs a feature map of size $\mathbf{H}\mathbf{/}\mathbf{8}\times \mathbf{W}\mathbf{/}\mathbf{8}\times \mathbf{512}$ , and the PSP module outputs a feature map of size $\mathbf{H}\mathbf{/}\mathbf{8}\times \mathbf{W}\mathbf{/}\mathbf{8}\times \mathbf{1024}$ . Then, four upsampling layers restore the feature map to the original size and output a feature map of size $\mathbf{H}\times \mathbf{W}\times \mathbf{32}$ , which means that each pixel is a 32-dimensional vector. We select 500 points from a point cloud of size $\mathbf{N}\times \mathbf{3}$ ; then, we find the 32-dimensional color feature vector corresponding to each point, stitch the vectors together to obtain 500 35-dimensional vectors, and send them to the geometric feature extraction network. This network consists of three ASSA modules. The first module outputs 64 local features with 256 dimensions and the center coordinates of these 64 local areas, and then they are sent to the second module. The second module outputs 16 local features with 512 dimensions and the center coordinates of these 16 local areas, and then they are sent to the last module. The last module outputs one global feature with 1024 dimensions. Finally, we copy the 16 global features and splice them with each local feature to obtain 16 fusion features with 1536 dimensions. We use two iterations during the refinement procedure and choose $\omega =0.01$ for equation (4). The settings of the other hyperparameters are the same as those in Wang et al. ¹³

All experiments are performed on an Intel Core i9-10900X CPU@3.70 GHz × 20 with a single NVIDIA GeForce RTX3090 GPU. The training and test sets division is the same as the previous work. ¹⁴

Evaluation conducted on the LineMOD dataset

Table 1 lists the quantitative evaluation results obtained for all 13 objects in LineMOD dataset. As can be seen, the performance of RGB-D-based methods is generally better than that of the RGB-based methods. Our method achieves an accuracy of $97.77\mathrm{\%}$ and outperforms the baseline approach: DenseFusion ¹³ and the method of Zhou et al. ¹⁴ by $3.47\mathrm{\%}$ and $1.37\mathrm{\%}$ , respectively, indicating that our fusion module can leverage color and depth information better than competing methods. Although compared with the state-of-the-art methods, our method is $1.93\mathrm{\%}$ and $1.63\mathrm{\%}$ lower than a full flow bidirectional fusion network for 6d pose estimation. (FFB6D) ³⁷ and PVN3D, ¹⁵ respectively, the parameters of our model are much smaller than them.

Table 1.

Quantitative 6D pose evaluation results obtained on the LineMOD dataset in terms of the ADD(-S) metric. Objects with bold names are symmetric.

	RGB				RGB-D
	PVNet	PoseCNN+DeepIM	CPDN 36	DPOD 8	DenseFusion	Zhou et al.	PVN3D	FFB6D	Ours
ape	43.62	77.0	64.4	87.7	92.3	93.51	97.3	98.4	94.28
bench	99.90	97.5	97.8	98.5	93.2	95.15	99.7	100.0	97.28
cam	86.86	93.5	91.7	96.1	94.4	93.92	99.6	99.9	97.84
can	95.47	96.5	95.9	99.7	93.1	95.47	99.5	99.9	96.65
cat	79.34	82.1	83.8	94.7	96.5	98.6	99.8	99.9	98.30
driller	96.43	95.0	96.2	98.8	87.0	94.84	99.3	100.0	97.42
duck	52.58	77.7	86.3	89.58	92.3	95.96	98.2	98.4	95.96
eggbox	99.15	97.1	99.7	99.9	99.8	100.0	99.8	100.0	100.0
glue	95.66	99.4	99.6	99.3	100.0	99.71	100.0	100.0	100.0
hole	81.92	52.8	81.92	86.9	92.1	94.48	94.48	99.8	98.00
iron	98.88	98.3	97.9	100.0	97.0	97.54	99.7	99.9	98.47
lamp	99.33	97.5	93.38	96.8	95.3	98.94	99.8	99.9	98.66
phone	92.41	87.7	90.8	94.7	92.8	95.1	99.5	99.7	98.17
average	86.27	88.6	89.9	95.2	94.3	96.4	99.4	99.7	97.77

6D: six degree-of-freedom; RGB: red–green–blue; RGB-D: RGB-depth; PVNet: pixel-wise voting network; PoseCNN: pose convolutional neural network; DeepIM: deep iterative matching; CPDN: coordinates-based disentangled pose network; DPOD: 6D pose object detector and refiner; PVN3D: a deep point-wise 3D keypoints voting network for 6D pose estimation; FFB6D: a full flow bidirectional fusion network for 6D pose estimation.

Evaluation conducted on the Occlusion LineMOD dataset

The quantitative evaluation results obtained for all eight objects in the Occlusion LineMOD dataset are reported in Table 2. Hinterstoisser et al. ³⁸ only use depth images, while the other methods use RGB-D images. As the table shows, our method outperforms DenseFusion ¹³ and the approach of Zhou et al. ¹⁴ by $5.34\mathrm{\%}$ and $3.05\mathrm{\%}$ , respectively, indicating that our method is more robust to occlusion. In particular, our method outperforms DenseFusion by significant margins of $9.75\mathrm{\%}$ and $16.26\mathrm{\%}$ on the duck and hole puncher objects, respectively.

Table 2.

Quantitative 6D pose evaluation results obtained on the Occlusion LineMOD dataset in terms of the ADD(-S) metric. The bold value in each row represents the best performance achieved for one object category.

	Hinterstoisser et al. 38	Michel et al. 10	PoseCNN+ICP	DenseFusion	Zhou et al.	Ours
ape	81.4	80.7	76.2	73.20	68.43	79.08
can	94.7	88.5	87.4	88.64	92.65	93.93
cat	55.2	57.8	52.2	72.22	77.97	70.44
driller	86.0	94.7	90.3	92.50	95.13	97.37
duck	79.7	74.4	77.7	59.65	62.14	69.40
eggbox	65.5	47.6	72.2	94.24	96.06	95.96
glue	52.1	73.8	76.7	92.62	93.54	91.70
hole p.	95.5	96.3	91.4	78.77	83.64	95.03
average	76.3	76.7	78.0	81.27	83.56	86.61

6D: six degree-of-freedom; PoseCNN: pose convolutional neural network; ICP: iterative closest point.

Evaluation conducted on the YCB-Video dataset

Table 3 shows the results obtained for all 21 objects in the YCB-Video dataset. The percentages of ADD-S AUC values that are <0.1 m and ADD-S values that are <2 cm are used to measure the performance of these methods. All methods use the same segmentation masks as those in PoseCNN to ensure fairness. As seen from the table, our method outperforms PoseCNN + ICP and DenseFusion by $0.6\mathrm{\%}$ and $0.5\mathrm{\%}$ in terms of the AUC (<0.1 m) metric and by $4.2\mathrm{\%}$ and $0.6\mathrm{\%}$ in terms of the ADD-S<2 cm metric, respectively. Moreover, our method achieves the best results for most of the 21 objects. Some qualitative visualization results are illustrated in Figure 6.

Figure 6.

Visualization of the poses estimated by our method. The left panel shows the input images and the right panel shows the resulting 6D pose images.

Table 3.

Quantitative 6D pose evaluation results (ADD-S<2 cm and AUC) obtained on the YCB-Video dataset.

	PointFusion		PoseCNN + ICP		DenseFusion		Ours
	AUC	2 cm	AUC	2 cm	AUC	2 cm	AUC	2 cm
002 master chel can	90.9	99.8	95.8	100.0	96.4	100.0	96.2	100.0
003 ceacker box	80.5	62.6	92.7	91.6	95.5	99.5	96.3	100.0
004 sugar box	90.4	95.4	98.2	100.0	97.5	100.0	98.3	100.0
005 tomato soup can	91.9	96.9	94.5	96.9	94.6	96.9	94.7	96.9
006 mustard bottle	88.5	84.0	98.6	100.0	97.2	100.0	97.6	100.0
007 tuna fish can	93.8	99.8	97.1	100.0	96.6	100.0	97.2	100.0
008 pudding box	87.5	96.7	97.9	100.0	96.5	100.0	96.0	100.0
009 gelatin box	95.0	100.0	98.8	100.0	98.1	100.0	98.3	100.0
010 putted meat can	86.4	88.5	92.7	93.6	91.3	93.1	91.7	93.6
011 banana	84.7	70.5	97.1	99.7	96.6	100.0	96.9	100.0
019 pitcher base	85.5	79.8	97.8	100.0	97.1	100.0	97.6	100.0
021 bleach cleanser	81.0	65.0	96.9	99.4	95.8	100.0	96.8	100.0
024 bowl	75.7	24.1	81.0	54.9	88.2	98.8	88.3	97.8
025 mug	94.2	99.8	95.0	99.8	97.1	100.0	97.3	100.0
035 power drill	71.5	22.8	98.2	99.6	96.0	98.7	96.1	99.3
036 wood bolck	68.1	18.2	87.6	80.2	89.7	94.6	91.8	100.0
037 scissors	76.7	35.9	91.7	95.6	95.2	100.0	94.9	100.0
040 large marker	87.9	80.4	97.2	99.7	97.5	100.0	98.1	100.0
051 large clamp	65.9	50.0	75.2	74.9	72.9	79.2	72.3	78.5
052 extra large clamp	60.4	20.1	64.4	48.8	69.8	76.3	74.2	77.8
061 foam brick	91.8	100.0	97.2	100.0	92.5	92.5	93.4	100.0
Mean	83.9	74.1	93.0	93.2	93.1	96.8	93.6	97.4

6D: six degree-of-freedom; AUC: area under the ADD-S curve; PoseCNN: pose convolutional neural network; ICP: iterative closest point.

Time efficiency results

Training time

A 6D pose estimation algorithm based on deep learning often needs to train for hundreds of epochs to achieve good results. This excessively long training time limits the practical applications of these methods. Figure 7 shows the average accuracies and errors yielded by DenseFusion, ¹³ the approach of Zhou et al. ¹⁴ and our method on the LineMOD dataset during the first 10 training epochs. As we can see, our method can achieve a higher accuracy rate and a smaller error than those of the other two methods under the same number of training epochs. Most importantly, our approach can achieve 90+ $\mathrm{\%}$ accuracy after only seven iterations. On a GTX 3090 GPU, one epoch takes approximately 35 min, which means that our method can achieve acceptable accuracy in only a few hours of training. Therefore, our approach has a wide range of application scenarios, especially industrial applications where the accuracy requirements are not extremely strict but the time cost is essential.

Figure 7.

Training curves. (a) and (b) show the accuracy and average error on the LineMOD dataset over the first 10 training epochs, respectively.

Inference time

On the GTX 3090 GPU, our method takes 0.017 s for pose estimation and 0.007 s for refinement. With 0.03 s for prior instance segmentation, the overall runtime on the LineMOD dataset is approximately 0.054 s, which can meet the requirements of real-time applications.

Model size results

Table 4 shows the comparison between the size of our model and those of the latest popular 6D pose estimation models. As we can see, the parameters of our model are $66.6\mathrm{\%}$ and $71.2\mathrm{\%}$ less than those of FFB6D ³⁷ and PVN3D, ¹⁵ respectively. In particular, the storage space occupied by our model is only one-tenth of that occupied by PVN3D. This indicates that our method is easier to deploy on mobile platforms with limited computing resources and small storage spaces, such as mobile manipulators.

Table 4.

Comparison of our model with the latest popular models in terms of model size.

	PVN3D 15	FFB6D 37	Ours
Space (MB)	449	129	45.4
Parameters	39.2M	33.8M	11.3M

PVN3D: a deep point-wise 3D keypoints voting network for 6 degrees of freedom pose estimation; FFB6D: a full flow bidirectional fusion network for 6D pose estimation; MB: megabyte.

Ablation experiments

We conduct a series of ablation studies on the LineMOD dataset to verify the effects of different parts of our model.

Table 5 compares our model’s parameters, memory space, floating point operations (FLOPs), and accuracy under different color feature extraction networks. Compared to ResNet34+PSPNet and the color feature extraction network in DenseFusion, our color feature extraction network reduces the number of model parameters by $65.2\mathrm{\%}$ and $49.5\mathrm{\%}$ , the number of GFLOPs (when the input size is 120 × 120) by $55.3\mathrm{\%}$ and $39.5\mathrm{\%}$ , and the occupied memory space by $65.0\mathrm{\%}$ and $49.2\mathrm{\%}$ , respectively. Moreover, only a slight accuracy drop is observed. This shows that our color feature extraction network can substantially reduce the model complexity with a minor accuracy loss, thus facilitating mobile device deployment. Our feature fusion module divides the obtained point cloud into ${\mathbf{N}}_{\mathrm{local}}$ local areas and extracts ${\mathbf{N}}_{\mathrm{local}}$ local features from them, with each feature independently predicting a pose. We aimed to investigate the effectiveness of this region-dividing way. Therefore, we conducted a comparative analysis of the results obtained by partitioning the point cloud into varying numbers of areas, as shown in Table 6. The accuracy of our method is only $95.2\mathrm{\%}$ when only global features are used, which is much lower than the accuracy achieved with any number of local areas, which shows that the lack of local features significantly affects the resulting prediction accuracy. As the number of local regions increases from 8 to 16, the accuracy also increases from $97.3\mathrm{\%}$ to $97.75\mathrm{\%}$ , which shows that the increase in the number of local areas can help the model predict poses from more different angles to attain improved prediction accuracy. However, a further increase in the number of local areas does not yield improved accuracy. Our possible analysis is that these local areas already have too many overlapping parts. The increase in the number of local areas also increases the FLOPs of our model, so we choose ${\mathbf{N}}_{\mathrm{local}}=16$ for the experiment.

Table 5.

Comparison among different versions of our model using different color feature extraction networks on the LineMOD dataset in terms of accuracy and model complexity.

	ResNet34+PSPNet	DenseFusion’s	Ours
GFLOPs	8.91	6.58	3.98
Space (MB)	129.9	89.4	45.4
Parameters	32,385,075	22,284,339	11,274,627
accurary	97.8	97.8	97.75

GFLOP: giga floating point operation.

Table 6.

The changes induced in the FLOPs and accuracy produced on the LineMOD dataset according to the number of local areas.

	0	8	16	24
Accuracy	95.2	97.3	97.75	97.73
GFLOPs	3.91	3.94	3.98	4.02

FLOP: floating point operation; GFLOP: giga FLOP.

Conclusions

In this paper, we introduce an end-to-end network for estimating an object’s 6D pose from RGB-D images. We develop a two-stage feature fusion module to better leverage the color and geometry information in RGB-D images, which is particularly advantageous in dealing with occlusion environments. This module first extracts color features from the given RGB images and combines them into a point cloud for color-depth fusion. Then, it uses an ASSANet-like network to extract several local features and one global feature from the point cloud for local–global fusion. Furthermore, to reduce the complexity of our model, we developed a lightweight network based on ShuffleNet V2 for color feature extraction. Experimental results obtained on LineMOD, Occlusion LineMOD, and YCB-Video datasets demonstrate that the proposed approach can increase the overall accuracy of estimated 6D poses. Furthermore, our model is significantly smaller than the latest popular models for 6D pose estimation, making it suited for applications on mobile platforms with limited computational resources. In the future, we will apply the end-to-end advantages of our method to the field of self-supervised learning.