Quantitative motion detection of in-hand objects for robotic grasp manipulation

Abstract

In an actual grasp operation, grasp accidents (slant, rolling, turning over, and dropping) of in-hand objects occur frequently. Quantitative motion detection of in-hand objects is critical to optimize the grasp configuration and to improve the stability and dexterity of a grasp manipulation. In this article, an innovative method for quantitative measurement of the motions of in-hand objects is presented. Firstly, the slip information at object–finger interface between adjacent states is detected by three omnidirectional slip sensors; next, singular value decomposition method is applied to calculate the rotation and translation matrices according to the relative coordinate changes extracted from the slip information. Finally, Euler angles and the linear displacements which illustrate the motion of in-hand objects are quantitatively measured from the translation matrix and the rotation matrix, and the continuous motion track can be further established. Experiment results show that the proposed method is effective in detecting multiple motion information of in-hand objects.

Introduction

Stable and safe grasp in an unstructured environment is still one of the biggest challenges for grasp robotics. To get a successful grasp, most of the studies ^{1 –3} give their emphases on the grasp planning, namely, determining contact points of the finger on the object and formulating an appropriate gripper configuration. These researches attempt to acquire the optimal grasp planning based on the information of the object and the gripper obtained before grasp operation. But in fact, when exerting this optimal grasp planning in a real grasp operation, accidents, such as slip, slant, rolling, turning over, and even drop, frequently occur, because the grasp planning built in advance is based on ideal conditions and is inevitably affected by the actual working environment and factors. Therefore, most of the existing approaches are based on open-loop strategy, which is difficult to prevent the grasp accident and it is hard to analyze the reason for grasp failures. Conversely, if the in-hand object’s motions can be monitored and further be measured quantitatively, the gripper controller may guide the gripper to make corresponding response to prevent or alleviate the occurrence of accident. Furthermore, these quantitative sensing information can help the gripper to optimize the grasp planning for the grasp manipulations followed.

Besides, to impart the gripper more dexterity, many researchers ^{4 –7} recently explored a novel in-hand manipulation method named Extrinsic Dexterity, in which the environment is used to reorient the poses of in-hand objects. If the object is snatched up with a nonoptimal attitude, the gripper can actively use the environment to adjust its configuration to get a better grasp. To perform this extrinsic dexterity grasp scheme, the quantitative detection of the motions of in-hand objects is indispensable.

It can be concluded from the above research backgrounds that if the multiple motion information of an in-hand object, such as inclination, rolling, especially the precise linear displacement and rotation angles, can be detected, the stability and dexterity of a grasp manipulation is expected to greatly improve.

In this article, Motion Detection of in-hand objects refers to the measurement of the relative pose changes of the gripped object to the robot fingers, which are caused by the relative slip at the object–finger interface. The pose changes are quantitatively defined by linear slip along X/Y/Z directions and rotational slip around X/Y/Z axis.

Currently, the study on Motion Detection of in-hand objects is still insufficient. However, Pose Estimation of an in-hand object, a similar research area with Motion Detection, has been extensively studied. Pose estimation aims to determine the absolute location and the pose of the object so that the robot hand can accurately locate the object and correctly configure the grasp parameters. In earlier work, most systems ^{8 –11} rely solely on vision to locate an object and determine its pose. The performance of this strategy, however, tends to deteriorate during a manipulation task. This is due to the occlusions created by the robot hand and fingers, which conceal the object and hinder the accurate estimation of its pose. ¹² Recently, many other sensing modalities, such as force, torque, and tactile sensing information, were added to the robot hand to complement the insufficient of vision-based methods. Bimbo et al. proposed several methods that combined visual information with other sensor modalities. While the iterative optimization algorithm ¹³ fused information from vision, tactile sensors, and joint encoders, the object pose was solely estimated from tactile and force sensing, requiring visual data for the initialization of the object pose. ¹⁴ Bimbo et al. ¹² presented a principal component analysis (PCA)-based descriptor to find the position and orientation of a grasped object, by matching tactile data to an object’s geometric features. Similarly, Yitao et al. ¹⁵ proposed a particle filter that evaluates hypotheses of grasp poses by comparing tactile measurements and expected tactile information from computer aided design (CAD)-based haptic rendering models. Hebert et al. ¹⁶ and Martin et al. ¹⁷ proposed methods which fuse the stereo vision with force-torque and joint position measurements to determine the object pose.

Theoretically, the pose changes of in-hand objects can be easily computed through continuously estimating the absolute pose of a moving object. But in fact, there are at least two barriers for the existing pose estimation methods to be directly used for the motion measurement of the in-hand object. The first is that the physical information of the in-hand object, no matter the point clouds, ^12,13 or the CAD models, ¹⁵ or three-dimensional (3-D) scale invariant feature transform (SIFT) features, ¹⁶ or contour features, ¹⁷ must be known in advance, and it is initially provided by the vision in these papers, which means these methods are not suitable for robot hand without a vision system. The second is that the time-consuming optimization algorithms, such as particle filter, ¹⁵ Bayesian static multiple model (SMM), ^16,17 and covariance-based PCA methods, ¹² are used for data matching in these methods, which are more applicable to the applications with low real-time request, for example, to locate the object in a grasp manipulation, whereas do not work well for the detection of continuous changes of poses.

Although slip sensors or tactile sensors are now wildly equipped in a robotic hand, most of them are solely used ^18,19 to detect the occurrence of relative motion/slip at the object–finger interface. More recently, a few literatures have begun to study the motion patterns (linear or rotational slip) detection of in-hand objects. Su et al. ²⁰ applied neural network on the tactile sensing data to discriminate the in-hand object’s linear slip from rotational slip, which can guide the robot hand to control the finger force accordingly. Similarly, Meier et al. ^21,22 identified in-hand object’s translational, rotational slippage, and nonslip conditions from the tactile sensing data by using a convolutional neural networks. However, few literatures involve the quantitatively detection of the object’s motions, such as the linear displacements and rotational angles alone/around X/Y/Z axis, which can provide more sensing information for a stable and dexterous grasp manipulation.

In this article, we present a novel method to detect the multiple motion information of an in-hand object, specifically, (1) detecting the occurrence of slip, (2) measuring the object’s linear slip and rotational slip along/around X/Y/Z axis, and (3) computing pose changes and the continuous tracks of an in-hand object. The proposed motion detection scheme is based on the slip detection and Euclidean space transformation (singular value decomposition (SVD)) theory. Firstly, the slip information at the gripper–object interface is detected by omnidirectional slip sensors; then, the rotation and translation matrices which contain the motion changes between adjacent states can be computed using SVD theory; at last, Euler angles and the linear displacements along/around X/Y/Z axis of in-hand objects can be consequently obtained, and the continuous motion tracks of in-hand objects can also be rebuilt.

The major advantages of the proposed method are summarized as follows: (1) it can detect multiple motion information of in-hand objects; (2) it is less susceptible to the external factors, such as the unstable illumination and occlusions created by the work environment and the robot hand; (3) for gripper equipped with tactile sensors, tactile sensors can be used for the detection of slip information, no extra slip sensors are needed.

The remainder of the article is organized as follows. The second section is the principle and methods of proposed motion detection scheme, the third section explains the experiment setup and calibration procedures, the fourth section describes the experiment procedures and results for continuous track detection, the fifth section presents the verification experiments, the sixth section gives experiment results and analysis, and the seventh section is the discussion. In the end, conclusions are drawn in the last section.

Methods

Analysis model of motion changes

As shown in Figure 1, the cup is moving from state(n − 1) to state(n) and then to state(n + 1). If changes of displacement $\Delta x,\Delta y,\Delta z$ along X/Y/Z axis and rotation angles ${\theta }_x,{\theta }_y,{\theta }_z$ around X/Y/Z axis in Cartesian space can be detected during the grasp operation, the cup’s motion can be established accurately.

Figure 1.

Motion of the cup.

To facilitate analysis and calculation, as shown in Figure 2, m points on the surface are extracted to represent the cup, denoted by $p_i{x}_i,y_i,z_i, i=1,2\dots m$ . The cup under different states can be described by point set ${\mathit{P}}^{\text{state}\left(n\right)}$

${\mathit{P}}^{\text{state}\left(n\right)}=\left\{p_i^{\text{state}\left(n\right)}\right\}$

where state(1)…state(n) represent different space states when cup moves and changes its attitude.

Figure 2.

Cup’s moving model.

The relationship between adjacent states can be mathematically represented by

${\mathit{P}}^{\text{state}\left(n\right)}={\mathit{R}}^{n-1}{\mathit{P}}^{\text{state}\left(n-1\right)}+{\mathit{T}}^{n-1}$

1

where ${\mathit{T}}^{n-1}$ and ${\mathit{R}}^{n-1}$ are the translation matrix and the rotation matrix when cup moves from state(n − 1) to state(n), respectively.

${\mathit{T}}^{n-1}\hspace{0.17em}\text{is given by}\hspace{0.17em}{\mathit{T}}^{n-1}={\left[\Delta x^{n-1}\Delta y^{n-1}\Delta z^{n-1}\right]}^{\mathit{T}}$

where $\Delta x^{n-1}, \Delta y^{n-1},\Delta z^{n-1}$ are the linear displacements of point set ${\mathit{P}}^{\text{state}\left(n\right)}$ from state(n − 1) to state(n) along X/Y/Z axis; ${\mathit{R}}^{n-1}$ is a 3 × 3 matrix, denoted by ${\mathit{R}}^{n-1}=\left[\begin{array}{ccc}{r}_{11}^{n-1}& r_{12}^{n-1}& r_{13}^{n-1}\\ r_{21}^{n-1}& r_{22}^{n-1}& r_{23}^{n-1}\\ r_{31}^{n-1}& r_{32}^{n-1}& r_{33}^{n-1}\end{array}\right]$ .

Matrix ${\mathit{R}}^{n-1}$ contains rotation angles information of the point set ${\mathit{P}}^{\text{state}\left(n\right)}$ from state(n − 1) to state(n).

As shown in Figure 3, when only the rotational component is considered, the rotation of the cup can be described by ${\theta }_x^{n-1},{\theta }_y^{n-1},{\theta }_z^{n-1}$ which are called Euler angle and can be calculated from ${\mathit{R}}^{n-1}$ by

$\begin{array}{l}{\theta }_x^{n-1}=a\text{tan}2\left(r_{32}^{n-1},r_{33}^{n-1}\right)\\ {\theta }_y^{n-1}=a\text{tan}2\left(-r_{31}^{n-1},\sqrt{{\left(r_{32}^{n-1}\right)}^2+{\left(r_{33}^{n-1}\right)}^2}\right)\\ {\theta }_z^{n-1}=a\text{tan}2\left(r_{21}^{n-1},r_{11}^{n-1}\right)\end{array}$

2

where atan2() is arctangent function. ²³

Figure 3.

Cup’s rotation model.

As represented in equation (1), the rotational matrix R and the translation matrix T can be computed from the relationship between coordinates of ${\mathit{P}}^{\text{state}\left(n-1\right)}$ and ${\mathit{P}}^{\text{state}\left(n\right)}$ .

Detection of object’s continuous track

Based on the analysis model in the section “Analysis model of motion changes,” if the rotation matrix R and the translation matrix T between a sequential of adjacent states have been obtained, the coordinates in different states can be calculated and the track of each point on the cup’s surface can be figured out according to equation (1), as shown in Figure 4.

Figure 4.

Cup’s continuous tracks.

Calculating of R and T

As analyzed in the section “Analysis model of motion changes,” the solution of rotation matrix R and translation matrix T is the key procedure to calculate the coordinate changes at the object surface. In this article, SVD approach, which is based on the cross-covariance matrix, ²⁴ is used to compute R and T . The detailed procedures are as follows:

(1) Define variables and the object function.

The rotation vector is denoted as a unit quaternion

$\stackrel{\rightharpoonup }{q_{\mathit{R}}}=\left[q_0{q}_1{q}_2{q}_3\right]$

3

where $q_0\ge 0$ .

$\stackrel{\rightharpoonup }{q_{\mathit{R}}}$ which generates the 3 × 3 rotation matrix is denoted by

$\mathit{R}\left(\stackrel{\rightharpoonup }{q_{\mathit{R}}}\right)=\left[\begin{array}{ccc}{q}_0^2+q_1^2-q_2^2-q_3^2& 2\left(q_1{q}_2-q_0{q}_3\right)& 2\left(q_1{q}_3+q_0{q}_2\right)\\ 2\left(q_1{q}_2+q_0{q}_3\right)& q_0^2+q_2^2-q_1^2-q_3^2& 2\left(q_2{q}_3-q_0{q}_1\right)\\ 2\left(q_1{q}_3-q_0{q}_2\right)& 2\left(q_2{q}_3+q_0{q}_1\right)& q_0^2+q_3^2-q_1^2-q_2^2\end{array}\right]$

4

A translation vector is defined by $\stackrel{\rightharpoonup }{q_{\mathit{T}}}={\left[q_4 q_5 q_6\right]}^{\mathit{T}}$ .

The complete registration state vector is denoted by $\stackrel{\rightharpoonup }{q}={\left[ \stackrel{\rightharpoonup }{q_{\mathit{R}}} | \stackrel{\rightharpoonup }{q_{\mathit{T}}} \right]}^{\mathit{T}}$ .

Let $\left\{\stackrel{\rightharpoonup }{p_i}\right\}$ and $\left\{\stackrel{\rightharpoonup }{x_i}\right\}$ represent the adjacent point set of ${\mathit{P}}^{\text{state}\left(n\right)}$ and ${\mathit{P}}^{\text{state}\left(n-1\right)}$ , respectively. The mean square objective function to be minimized is

$f\left(\stackrel{\rightharpoonup }{q}\right)=\frac{1}{m}\sum _{i=1}^m\stackrel{\rightharpoonup }{x_i}-\mathit{R}\left(\stackrel{\rightharpoonup }{q_{\mathit{R}}}\right)\stackrel{\rightharpoonup }{p_i}-{\stackrel{\rightharpoonup }{q_{\mathit{T}}}}^2$

5

where m is the number of points to represent the object.

(2) Calculate the center of point set and form cross-covariance matrix.

The center of { $\stackrel{\rightharpoonup }{p_i}$ } and { $\stackrel{\rightharpoonup }{x_i}$ } is given by

$\stackrel{\rightharpoonup }{{\mu }_p}=\frac{1}{m}\sum _{i=1}^m\stackrel{\rightharpoonup }{p_i}$

6

$\stackrel{\rightharpoonup }{{\mu }_x}=\frac{1}{m}\sum _{i=1}^m\stackrel{\rightharpoonup }{x_i}$

7

The cross-covariance matrix of ${\mathit{P}}^{\text{state}\left(n\right)}$ and ${\mathit{P}}^{\text{state}\left(n-1\right)}$ is given by

$\sum _{px}=\frac{1}{m}\sum _{i=1}^m\left[\left(\stackrel{\rightharpoonup }{p_i}-\stackrel{\rightharpoonup }{{\mu }_p}\right){\left(\stackrel{\rightharpoonup }{x_i}-\stackrel{\rightharpoonup }{{\mu }_x}\right)}^{\mathit{T}}\right]=\frac{1}{m}\sum _{i=1}^m\left[\stackrel{\rightharpoonup }{p_i}{\stackrel{\rightharpoonup }{x_i}}^{\mathit{T}}\right]-\stackrel{\rightharpoonup }{{\mu }_p}{\stackrel{\rightharpoonup }{{\mu }_x}}^{\mathit{T}}$

8

Compute the optimal rotation vector and translation vector.

The cyclic components of the anti-symmetric matrix $A_{ij}={\left({\text{Σ}}_{px}-{\text{Σ}}_{px}{}^{\mathit{T}}\right)}_{ij}$ is used to form the column vector $\Delta ={\left[A_{23} A_{31} A_{12}\right]}^{\mathit{T}}$ , which is then used to form the symmetric 4 × 4 matrix

$\text{Q}\left({\text{Σ}}_{px}\right)=\left[\begin{array}{cc}tr\left({\text{Σ}}_{px}\right)& {\Delta }^{\mathit{T}}\\ \Delta & {\text{Σ}}_{px}+{\text{Σ}}_{px}{}^{\mathit{T}}-tr\left({\text{Σ}}_{px}\right){\mathit{I}}_3\end{array}\right]$

9

where I ₃ is the 3 × 3 identity matrix. In the iteration, the unit eigenvector corresponding to the maximum eigenvalue of the matrix $\text{Q}\left({\text{Σ}}_{px}\right)$ is selected as the optimal rotation vector. Applying the optimal rotation vector to equation (4), the optimal rotation matrix R between ${\mathit{P}}^{\text{state}\left(n\right)}$ and ${\mathit{P}}^{\text{state}\left(n-1\right)}$ is obtained.

The optimal translation vector is given by

$\stackrel{\rightharpoonup }{q_{\mathit{T}}}=\stackrel{\rightharpoonup }{{\mu }_x}-\mathit{R}\left(\stackrel{\rightharpoonup }{q_{\mathit{R}}}\right)\stackrel{\rightharpoonup }{{\mu }_p}$

10

Thus, the translation matrix $\stackrel{\rightharpoonup }{q_{\mathit{T}}}={\left[q_4 q_5 q_6\right]}^{\mathit{T}}$ is generated.

Motion detection scheme

According to the section “Analysis model of motion changes,” to detect the motion of in-hand objects, the relative coordinate changes of the points on the object surface from state(n − 1) to state(n) should be measured. Theoretically, the more points selected to represent the object’s surface, the more accurate estimation of the motion changes we can achieve. However, in practical application, the contact areas between the object and the fingers are limited. It’s well known that at least three noncollinear points on the object surface are needed to represent the object. As shown in Figure 5, a triangle composed of three points, whose relative locations are restrained by the cup, is used to represent the whole cup.

Figure 5.

Three noncollinear points on the cup.

Therefore, we design a detection scheme with three monitoring points (m = 3), as shown in Figure 6(a). At every detection point, a slip sensor is mounted in the fingertip. When a relative movement between the object and gripper fingers happens, the sensors can detect coordinate changes at the object–finger interface. Figure 6(b) shows the outputs of sensors 0, 1, and 2 at their respective position. The slip displacements L ₀, L ₁, L ₂ and slip directions ${\varphi }_0\mathrm{, }{\varphi }_1\mathrm{, }{\varphi }_2$ shown in the figure are detected by the sensors when the object slips relative to the robot fingers. Based on these slip data, the relative pose changes in 3-D space of in-hand objects can be computed using the principle presented above. It should be noted that a parallel gripper is adopted here. Therefore, the rotational angle around Y axis and linear displacement along Z axis will be pretty small no matter which direction the object slips relative to the fingers.

Figure 6.

Detection system design.

Experiment setup and calibration

Experiment setup

To verify the validity of the proposed method, we set up a gripper system with three slip sensors in the fingers according to the motion detection scheme presented in the section “Motion detection scheme.” As shown in Figure 7, the parallel gripper is manufactured by SCHUNK, whose model is WSG50. Its main parameters are (1) clamping range: 0–110 mm, (2) repeat positioning accuracy: ±0.013 mm, (3) clamping force range: 5–120 N, and (4) the parallel fingers are made of aluminum alloy.

Figure 7.

Experiment setup.

In order to easily measure the actual linear displacement and rotation angle of the object, a cylindrical bottle is adopted as the in-hand object. The coordinate paper which covers on the object is designed to indicate the actual displacement and rotation angle, which is not necessary in the practical application.

According to the detection principle presented in the section “Motion detection scheme,” the slip sensor used in this scheme should possess the ability to detect the slip displacement and the slip direction. Nowadays, many slip/tactile sensors can meet this requirement, such as the widely used piezoresistive tactile sensor arrays ^25,26 or the optical slip sensors. ^27,28 In this article, a video-based slip sensor ²⁹ is used to sense the multidimensional slip information. The slip sensor applies a micro-camera to capture the dynamic slip images at the surface of in-hand objects and the image processing algorithm is used to analyze the tracks of the slippage. The video-based slip sensor performs well on most of objects except those with transparent and reflective surface. ²⁹

Figure 8 shows the accessory equipment for measuring the actual pose changes of the in-hand object, by which the object’s actual linear displacement along X/Y axis and angles ${\theta }_x,{\theta }_z$ around X/Z axis can be measured. The holding frame, with the object fixed on, can rotate within the range of 0–360°. The inclinometer can measure rotation angles ${\theta }_x,{\theta }_z$ in the range of −90° to 90°, with the resolution of 0.01°; the linear displacements along X/Y axis are measured by a Vernier caliper (not shown in the figure).

Figure 8.

Experiment setup with measurement equipment.

Calibration of slip sensor

As shown in Figure 8, the test object is controlled to move alone X and Y axes with displacement $\Delta d_x$ and $\Delta d_y$ , which can be precisely measured using a Vernier caliper. During the moving process, each sensor will monitor the motion of the object and compute the number of pixels q_x and q_y , which covers the displacement $\Delta d_x$ and $\Delta d_y$ , respectively. Scale relations between the number of pixels and the displacement in the directions of X and Y can be calculated by

$d_{x\_\text{scale}}=q_x/\Delta d_x$

11

$d_{y\_\text{scale}}=q_y/\Delta d_y$

12

The calibration test is repeated five times and the mean of $d_{x\_\text{scale}}$ and $d_{y\_\text{scale}}$ will be used to correct the detection data in the following process. This calibration process will be carried out on every sensor. Here, the distortion of imaging system caused by the lens and CCD sensor is not calibrated, mainly due to two reasons: (1) Since relative displacement is measured here, the detection errors in a single image caused by the distortions of the lens and CCD will be offset in the subtraction operation between two adjacent images; and (2) only a subimage with 128 × 128 pixels located in the center of the captured image is used for slip detection, the distortion is pretty small compared to that in the boundary region.

Continuous track detection experiment

Continuous track detection procedures

The continuous track detection test is implemented as followed:

Translate the object 60 mm along X axis; meanwhile, add an extra rotation component to the translation by slowly rotating the object 60° around X axis.

Analyze the data from three sensors and calculate the rotation matrix R and translation matrix T by using the method in the section “Analysis model of motion changes.”

The continuous motion track of the object can be acquired by using the method in the section “Detection of object’s continuous track.”

Continuous track detection results

The result of continuous motion detection is shown in Figure 9. In order to show the object under continuous changing states, a triangle composed of three noncollinear points on the object surface in YOZ plane (as shown in Figure 9) is selected to represent the whole object. It can be seen that the object is moving along X axis with a continuous rotation around X axis. The translation displacement along X direction is approximate 60 mm and calculated angle of the rotation around X axis is close to 60°, which are consistent with the actual conditions.

Figure 9.

Continuous tracks of the in-hand object.

In the practical grasp manipulation, because of some limits of testing conditions, the actual tracks of the continuous motion state can’t be precisely measured; therefore, the continuous track detection test is designed and carried out qualitatively.

Verification experiments

To evaluate the effectiveness of the proposed method in quantitatively detecting the motion of in-hand objects, many verification experiments are carried out. Verification experiments focus on evaluating the measurement accuracy in linear displacements and rotational angles.

As mentioned in the section “Motion detection scheme,” the rotational angle around Y axis and the translational displacement along Z axis are pretty small because a parallel gripper is adopted in the experiment system. Therefore, only rotation angles around X/Z axis and linear displacements along X/Y axis are tested. The actual value of rotational angle and the linear displacement are measured by the universal instrument, such as an inclinometer and a Vernier caliper, respectively. The measurement accuracy is evaluated by comparing the test results acquired by the proposed method with the actual results measured by the universal instrument.

Rotation angle detection procedures

Rotate the object around X axis

As shown in Figure 8, the tested object with a diameter of 68 mm is clamped in the parallel gripper, while both ends are fixed on the holding frame. The tested object’s long axis is parallel to X axis and can rotate around X axis on the holding frame. The inclinometer is fixed on the test object, perpendicular to the object’s long axis. In the test, the object is rotated with a fixed angle. And the angle value can be recorded accurately by the inclinometer. It is considered as actual value and is denoted as ${\theta }_{xm}$ . The rotation angle detected by the proposed method is denoted as ${\theta }_{xc}$ .

The detected value ${\theta }_{xc}$ will be compared with the actual value ${\theta }_{xm}$ , and the absolute error is calculated by

${\theta }_{\text{xerrori}}=|{\theta }_{\text{xmi}}|-|{\theta }_{\text{xci}}|,i=1\sim 9$

13

Repeated nine times, the average of the absolute error will be

${\overline{\theta }}_{\text{xerror}}=\frac{{\sum }_{i=1}^9{\theta }_{\text{xerrori}}}{9}$

14

And the relative error is denoted by

${\delta }_{\text{xerrori}}=\frac{|{\theta }_{\text{xerrori}}|}{|{\theta }_{xmi}|}\times 100\text{\%},i=1\sim 9$

15

The average of relative detection error is given by

${\overline{\delta }}_{\text{xerror}}=\frac{{\sum }_{i=1}^9{\delta }_{\text{xerrori}}}{9}\times 100\text{\%}$

16

Rotate the object around Z axis

As shown in Figure 8, similar to test (“Rotate the object around X axis”), the object will be rotated around Z axis by nine times and the actual value measured by the inclinometer ${\theta }_{zm}$ and the detected value ${\theta }_{zc}$ detected by the proposed method will be recorded. The accuracy analysis procedures are similar to “Rotate the object around X axis”; angle error ${\theta }_{\text{zerrori}}$ , average of error ${\overline{\theta }}_{\text{zerror}}$ , relative error rate ${\delta }_{\text{zerrori}}$ , and average of the relative error rate ${\overline{\delta }}_{\text{zerror}}$ will be computed.

Linear displacement detection procedures

Translate the object along X axis

As shown in Figure 8, the object will be translated along X axis. The actual value of linear displacement $D_{xm}$ will be measured by a Vernier caliper and the linear displacement detected by the proposed method is denoted as $D_{xc}$ . Similar tests will be repeated by nine times. Similar to the section “Rotation angle detection procedures,” the displacement error $D_{\text{xerrori}}$ , average of error ${\overline{D}}_{\text{xerror}}$ , relative detection error rate ${\gamma }_{\text{xerrori}}$ , and the average of relative error rate ${\overline{\gamma }}_{\text{xerror}}$ will be computed.

Translate the object along Y axis

The test setup is shown in Figure 8. The object will be translated along Y axis. Similarly to “Rotate the object around X axis,” the actual displacement $D_{ym}$ , detected value of displacement $D_{yc}$ , detection error $D_{\text{yerrori}}$ , average of the detection error ${\overline{D}}_{\text{yerror}}$ , relative error rate ${\gamma }_{\text{yerrori}}$ , and the average of relative error rate ${\overline{\gamma }}_{\text{yerror}}$ are computed.

Experiment results and analysis

Results and analysis of rotation angle

Rotate around X axis

Tests were carried out following the procedures in the section “Rotation angle detection procedures.” The tested object was carefully controlled to rotate an angle of ${\theta }_{xm}=2.5°$ . The experimental results of nine tests are listed in Table 1, in which the first row is the actual rotation angle ${\theta }_{xm}$ , the second row is the detected value ${\theta }_{xc}$ measured by the proposed method, and the third row is the absolute error ${\theta }_{\text{xerrori}}$ calculated by equation (13).

Table 1.

Result of rotating around X axis.

$\hspace{0.17em}{\theta }_{xm}\hspace{0.17em}$ (°)	2.50	2.50	2.50	2.50	2.50	2.50	2.50	2.50	2.50
$\hspace{0.17em}{\theta }_{xc}$ (°)	−2.434	−2.441	−2.532	−2.537	−2.589	−2.498	−2.541	−2.522	−2.482
$\hspace{0.17em}{\theta }_{\text{xerrori}}\hspace{0.17em}$ (°)	0.066	0.059	−0.032	−0.037	−0.089	0.002	−0.041	−0.022	0.018

The average of the absolute error ${\overline{\theta }}_{\text{xerror}}$ computed by equation (14) is 0.04° and the average relative error rate ${\overline{\delta }}_{\text{xerror}}$ calculated by equation (16) is 1.6%. The relative errors around X axis ${\delta }_{\text{xerrori}}$ are shown in Figure 10(a). It can be seen that the relative errors in most of the tests are below 3%. And the maximum test relative error is 3.6% in test 5. After analysis, the source of the measurement error is caused by a number of factors, such as the error of the experiment setup, the calibration error as well as the resolution of the camera. One of the dominant sources is that the object will be deformed during rotation and translation, because the object is deformable and the coordinate paper has a little elasticity which will be stretched in every rotation operation, and these factors will lead to some errors.

Figure 10.

Rotation errors around X and Z axes. (a) Errors around X axis. (b) Errors around Z axis.

Rotate around Z axis

Following the process in the section “Rotate the object around Z axis,” the tested object was carefully controlled to rotate ${\theta }_{zm}=2.0°$ . The test was repeated by nine times. The experimental results of nine tests are listed in Table 2, in which the first row is the actual rotation angle ${\theta }_{zm}$ around Z axis, the second row is the detected value ${\theta }_{zc}$ measured by the proposed method, and the third row is the absolute error ${\theta }_{\text{zerrori}}$ .

Table 2.

Result of rotating around Z axis.

$\hspace{0.17em}{\theta }_{zm}$ (°)	2.0	2.0	2.0	2.0	2.0	2.0	2.0	2.0	2.0
$\hspace{0.17em}{\theta }_{zc}$ (°)	1.911	1.954	1.927	1.955	1.988	2.000	2.029	2.030	1.960
$\hspace{0.17em}{\theta }_{z\text{errori}}$ (°)	−0.089	−0.046	−0.073	−0.045	−0.012	0	0.029	0.030	−0.040

The average of the test absolute error ${\overline{\theta }}_{\text{zerror}}$ is 0.04°. The relative errors ${\delta }_{\text{zerrori}}$ of nine tests are shown in Figure 10(b) and the average of the relative error ${\overline{\delta }}_{\text{zerror}}$ is 2%. Similar to the section “Rotate the object around X axis,” the elasticity of the object and the coordinate paper are analyzed as the main error source. It is clear that the error of rotating around Z axis is larger than rotating around X axis by 0.4%. The reason may be that the distance from Z axis to the gripper is longer than that from X axis to the gripper and the error may be amplified by the longer distance.

Results and analysis of linear displacement detection tests

Translate along X axis

The test results are shown in Table 3, in which the first row is the actual value. The second row shows the detected value measured through the proposed method, and the third row is the absolute error. According to the section “Translate the object along X axis,” the relative errors are shown in Figure 11(a). The average of absolute error ${\overline{D}}_{\text{xerror}}$ is 0.02407 mm, and the average of the relative error ${\overline{\gamma }}_{\text{xerror}}$ is 0.12%.

Table 3.

Result of translating along X axis.

$\hspace{0.17em}{D}_{xm}\hspace{0.17em}$ (mm)	20.00	20.00	20.00	20.00	20.00	20.00	20.00	20.00	20.00
$\hspace{0.17em}{D}_{xc}$ (mm)	19.967	19.983	19.960	20.033	20.000	19.950	20.033	19.990	20.000
$\hspace{0.17em}{D}_{\text{xerrori}}$ (mm)	0.033	0.017	0.040	−0.033	0	0.050	−0.033	0.01	0

Figure 11.

Translation errors along X and Y axes. (a) Errors along X axis. (b) Errors along Y axis.

Similarly, the error is mainly caused by the deformation of the object and the stretch of the coordinate paper which covers the in-hand object.

Translate along Z axis

The test results are shown in Table 4, in which the first row is the actual value. The second row shows the detected value measure through the proposed method, and the third row is the relative error.

Table 4.

Result of translating along Y axis.

$\hspace{0.17em}{D}_{ym}$ (mm)	20.00	20.00	20.00	20.00	20.00	20.00	20.00	20.00	20.00
$\hspace{0.17em}{D}_{yc}$ (mm)	20.017	19.983	19.973	19.967	19.980	19.967	20.050	19.993	20.013
$\hspace{0.17em}{D}_{\text{yerrori}}$ (mm)	−0.017	0.017	0.027	0.033	0.020	0.033	−0.050	0.007	−0.013

According to the section “Translate the object along Y axis,” the relative errors are shown in Figure 11(b). The average of absolute error ${\overline{D}}_{\text{yerror}}$ is 0.02407 mm, and the average of the relative error ${\overline{\gamma }}_{\text{yerror}}$ is 0.12%. Similarly, the error is mainly caused by the deformation of the object and the stretch of the coordinate paper.

Comparative experiments and analysis

As discussed in the section “Introduction,” there are few comparable literatures that study the quantitative motion detection/slip detection of multiple motion information of in-hand objects. Therefore, we compare our experimental results with what have been presented in literatures that study the pose estimation of in-hand object. In Bimbo et al.’s research, ¹² the minimum errors of angle and linear displacement between the measured values and the ground truth values are 1.19°and 1 mm, respectively, in which the ground truth pose was displaced by a random rotation between 0° and 20° and a displacement between 0 mm and 10 mm around/on each directions [x, y, z]. The relative error of angle is consequently computed as 1.19°/20° = 5.95%, and the relative error of displacement is 1 mm/20 mm = 5%. In other literatures, only the mean absolute errors for pose estimation are reported, such as 5.2 mm in position and 1.51° in orientation ¹⁶ , 4 mm for position and 5° for angle ¹⁴ , and 5.2 mm and 3° for the average localization errors. ³⁰ As shown in the section “Experiment results and analysis,” the relative errors for rotation and translation presented in this article are both below 4.5%, and the absolute errors for both rotation and translation are also far less than that reported in the comparative literatures.

Discussion

Experiment results verify that the proposed method is effective in quantitative motion detection of in-hand objects and also has the ability in rebuilding the motion tracks of an in-hand object.

From the experiment results and analysis, it can be found that the proposed method is only suitable for objects which are non-deformable or with low elastic deformation under the grasp force. For objects with high elasticity or that are prone to deformation at the gripper–object interface, there will be large detection errors.

In the experiment system, the video-based slip sensor is used to sense the slip information of the object. The proposed measurement scheme is equally applicable if the video-based slip sensors are replaced by other omnidirectional slip sensors. Tactile sensors equipped on a gripper can be used for the slip information detection, no extra slip sensors are needed. Because the data acquisition speed is commonly limited by the frame rate of the camera, the response speed in tracking the continuous changing motion of the object will prominently improve if the video-based slip sensors are replaced by other slip sensors with better dynamic property, for example, the piezoresistive slip sensor.

Although a parallel gripper is adopted in the experiment system due to the limitations of experimental conditions, the motion detection approaches proposed in this article can be easily transplanted to more complicated hands, such as Barrett hands, and other dexterous hands.

Conclusions

In this article, we present a novel method to quantitatively measure the motions of in-hand objects. In the proposed scheme, three omnidirectional slip sensors are used to detect the multidimensional slip information of an in-hand object. The SVD method is applied to compute the rotation matrix and the translation matrix on the sensed slip information between adjacent states. The linear displacements along X/Y/Z direction and rotation angles around X/Y/Z axis can be quantitatively measured from the translation matrix and the rotation matrix and the continuous motion track in 3-D space can be further established. Experiment results show that this method can accurately measure the motions of in-hand objects and rebuild the continuous changing tracks of the motion. The proposed method makes robots possible to prevent or alleviate the occurrence of the grasp accident and to optimize the grasp planning, which has a significant impact on improving the stability and flexibility of the grasp manipulation.

The future work is to explore the specific information hidden in the motions of in-hand objects to optimize the grasp planning and to help the grasp operation more stable and flexible.