PAM: Research on posture alignment method for camera robot system

Abstract

By acquiring the posture information of the rigid body with markers, the camera motion on the robot end-effector can be controlled remotely. The essential step of posture alignment in teleoperation is to calibrate the transformation matrix of the world coordinate system of the optical motion capture system and the robot base coordinate system. It was necessary to change the position and attitude of the camera robot in the studio frequently. At the same time, the studio was complex with strong personnel mobility, thus the world coordinate system of the optical tracking system in the studio was needed to be re-established frequently. These two points asked the pose calibration scheme of the camera robot in the optical tracking system to be convenient and universal. In this paper, a novel posture alignment method (PAM) was proposed, which was an automatic, accurate and quick method to complete the posture alignment for related coordinate systems without non-systematic errors. PAM was applied to teleoperation for camera robot by inverse movement matrix. The comparative experiments prove that the proposed method is better than manual method, with higher precision and stability, more flexible physical setting for reference coordinate system, and shorter operation time consuming. Meanwhile, PAM is more suitable for camera robot teleoperation than existed automatic method, because it does not require paying attention to the calibration postures.

Introduction

The optical motion capture system (OMCS), image enhancement methods ¹ and network energy consumption analysis^2,3 are important components of virtual studio system. ⁴ OMCS captures the position of marker by optical sensor. A rigid body consists of three or four markers which form a Coordinate System (CS) in OMCS software. ⁵ Robot teleoperation can integrate human experience and mechanical power effectively.^6–8 It has many applications, such as medical treatment and avoiding working in dangerous environment.^9–12 For the camera robot application, a camera is treated as the end-effector. The posture (position and attitude) of the end-effector is assigned to align with the gripper frame, which is mounted with a general rigid body with markers (GRBM). When moving the gripper frame, the camera at the end of the manipulator is driven to reproduce the trajectory in real time,^13,14 as shown in Figure 1. The GRBM is shown in a blue box (the rectangle with the dashed line); the gripper frame is shown in a green box (the rectangle with the solid line); the mobile manipulator is shown in a yellow box (the rectangle with the dotted line); and the camera is shown in a red box (the rectangle with the double solid lines).

Figure 1.

The mobile arm-shaped camera robot teleoperation system.

Current scheme of posture acquisition in OMCS

The posture of GRBM obtained by the OMCS is described relative to either the CS defined in optical hardware or another GRBM. In certain OMCS, GRBM as reference CS scheme is more accuracy than optical hardware as reference CS, ¹⁵ for example, in the multi-camera OMCS, there is a physical marking ruler as the world CS, as shown in Figure 1(c). In this paper, the world CS or the reference CS was named as WoR CS.

Current alignment method

The essential step of posture alignment between the CS of the robot end-effector and the CS of GRBM in OMCS is to find the transformation relationship between the robot base CS and the WoR CS of OMCS.

A calibration scheme was proposed in Ref. ¹⁶ The GRBM representing the WoR CS was placed on the corner of table, and the manipulator was placed on the table. By these actions, the attitude of WoR CS was set to align with the robot base CS manually. Then by length measurement, the homogeneous transformation from the robot base CS to WoR CS was achieved.

In practice, there are a few problems. First, the robot base CS is usually not at the centre of the robot base, so the posture of robot base relative to GRBM on the desktop cannot be accurately determined. Second, due to the large weight of robot, it is difficult to fine-adjust its posture in practice, and the system contains random errors for installation each time. Third, in the system installation stage, the attitude of WoR CS and robot base CS shall be consistent, which is hardly achieved manually.

The disadvantages of the manual scheme are as follows. First, the transformation matrix (position and attitude) between WoR CS and robot base CS includes non-systematic error, and there are not appropriate measuring tools and physical measuring points. Second, when transformation matrix is determined, the OMCS devices cannot be moved. If robot or camera is moved accidentally, it is required to be measured again. Otherwise, an accident may occur.

The existed eye/hand calibration methods include Tsai-Lenz, ¹⁷ NAVY, ¹⁸ INRIA ¹⁹ and Dual Quaternion. ²⁰ The common goal of them was to solve linear function AX = XB . Here Tsai-Lenz method was selected to compare with PAM.

Tsai-Lenz had higher requirements for calibration attitude ¹⁸ due to the influence of error. In practise, we found the calibration posture of robot were not arbitrary. For example, the axes of camera CS should not be parallel to robot base CS. Otherwise the calibration result was going to distorted. Since calibration required at least 2 postures, and it was eye-to-hand for camera robot system in which the visual field of camera was fixed, the effect calibration postures should be selected carefully. Thus, the Tsai-Lenz method was not suitable for camera teleoperation application; since it was often necessary to change the position and attitude of the camera robot in the studio. At the same time, the studio was complex with strong personnel mobility, and the world coordinate system of the optical tracking system in the studio was needed to be re-established frequently. These two points asked the pose calibration scheme of the camera robot in the optical tracking system to be convenient and universal. Here Tsai-Lenz method was selected to compare with PAM. The performance would be discussed in the experiment part.

In this paper, an automatic, fast and accurate calibration method for teleoperation system was proposed, named as posture alignment method (PAM). PAM performed a set of simple standard actions. Then the attitude of the transformation matrix, from WoR CS to robot base CS, was calibrated. By inverse movement matrix the fixed initial position deviation was compensation. There was no non-systematic error.

In the first part, the structure of the camera robot teleoperation system (CRTS) was introduced. In the second part, the mathematical derivation of PAM was discussed in detail. Inverse movement matrix was used to complete the calibration task and the PAM method effectiveness was verified. In the third part, the manual method, PAM and Tsai-Lenz were used to achieve calibration task. By comparing the coefficient of variation, attitude error and operation time, the performance of these schemes in terms of stability, accuracy and time consuming was analysed. PAM was universal for any arm-shape robot, so different manipulators had the same experiment results in comparative experiment part.

Posture alignment method

CSs in CRTS

The CSs of the CRTS were described in Table 1. The origin of {T} was on the centre point of the lens, as shown in Figure 2. {Cc} was show in Figure 4.

Figure 2.

Css setup in details. (a) CSs in CRTS. (b) Gripper frame. (c) Marking ruler. (d) Robot base. (e) Flange. (f) End-effector.

Table 1.

CSs of the CRTS.

CS symbol	CS name
{W}	WoR CS of OMCS
{Cg}	CS of gripper frame, GRBM
{Cc}	CS of camera frame, GRBM
{B}	CS of robot base
{F}	CS of the flange at the end of robot
{T}	CS of robot end-effector

Mathematical derivation of PAM

${\mathit{A}}_x$ , ${\mathit{B}}_x$ and ${\mathit{C}}_x$ were $4\times 4$ homogeneous transformation matrices, consisted by a $3\times 3$ $\mathit{R}$ matrix which represented attitude, and a $3\times 1$ $\mathit{P}$ matrix which represented position. For example,

$${\mathit{A}}_1=\left[\begin{array}{cc}{\mathit{R}}_{a1}& {\mathit{P}}_{a1}\\ 0& 1\end{array}\right]$$

(1)

Where,

$${\mathit{R}}_{a1}=\left[\begin{array}{cc}{r}_{a11}& \begin{array}{cc}{r}_{a12}& r_{a13}\end{array}\\ \begin{array}{c}{r}_{a21}\\ r_{a31}\end{array}& \begin{array}{cc}\begin{array}{c}{r}_{a22}\\ r_{a32}\end{array}& \begin{array}{c}{r}_{a23}\\ r_{a33}\end{array}\end{array}\end{array}\right],{\mathit{P}}_{a1}=\left[\begin{array}{c}\begin{array}{c}{p}_{a1x}\\ p_{a1y}\end{array}\\ p_{a1z}\end{array}\right]$$

(2)

Our target was to realise posture alignment for {T} and {Cg}. The essential step was to determine ${\mathit{B}}_1$ , which represented joint state of robot. Taking ${\mathit{B}}_1$ as the target, by the inverse kinematic solution, the command values of each joint were calculated. The transformation chain in CRTS was shown in Figure 3., we had,

$${\mathit{A}}_2={\mathit{A}}_1\cdot {\mathit{B}}_1\cdot {\mathit{B}}_2\cdot {\mathit{C}}_1$$

(3)

Figure 3.

Transformations of CRTS.

In teleoperation procedure, the movement of {Cg} was mapping to {T} while the photographer changing the posture of GRBM. In other words, the attitude of {T} and {Cg} should be aligned at any time, and a fixed position deviation existed for the origin of two CSs in real world. When {T} and {C} were aligned, the rotation submatrix in ${\mathit{C}}_1$ was identity matrix,

$${\mathit{C}}_1=\left[\begin{array}{cccc}\begin{array}{c}1\\ 0\end{array}& \begin{array}{c}0\\ 1\end{array}& \begin{array}{c}0\\ 0\end{array}& \begin{array}{c}{p}_{c1x}\\ p_{c1y}\end{array}\\ 0& 0& 1& p_{c1z}\\ 0& 0& 0& 1\end{array}\right]$$

(4)

Then ${\mathit{B}}_1$ was calculated as follows,

$${\mathit{B}}_1={\mathit{A}}_1^{-1}\cdot {\mathit{A}}_2\cdot ({\mathit{B}}_2\cdot {\mathit{C}}_1{)}^{-1}$$

(5)

The posture of {W} was determined by the physical marking ruler. The posture of {Cg} was read directly from OMCS, and the transformation ${\mathit{A}}_2$ was described relative to {W}. The postures of {B} and {F} were set by the robot manufacturer.

There were two ways to determine the transformation ${\mathit{B}}_2$ of {T} relative to {F}. The first was based on precise mechanism dimension of the end-effector. The second was based on the universal calibration algorithm built in the robot system. ²¹ After {T} was determined, {F} and {T} was read from robot system at any time, and the transformations $\mathbf{(}{\mathit{B}}_1\cdot {\mathit{B}}_2)$ were described relative to {B}. In robot system, {F}/{T} was determined by robot joint value.

The core of PAM was to calibrate ${\mathit{A}}_1$ on the transfer chain automatically, accurately and quickly.

1. The Difficult of Calibration for ${\mathit{A}}_1$ :

First, it was hard to find a physical feature representing {B} posture on robot. What was worse, the origin of {B} was inside the robot hardware structure, or in the hollow of the robot base. Then, the origin and the orthogonal axis direction of {B} relation to {W} could not be measured by common tools.

Second, there was not a physical feature representing {Cg} posture on GRBM.

Third, the precise transformation ${\mathit{C}}_1$ between {F} and {Cg} was also hard to obtain.

2. Attitude Calibration of PAM:

The goal of attitude calibration was to make the attitude of {Cg} and {T} aligned, and the origins of the CSs kept an arbitrary position deviation.

A GRBM, {Cc}, was attached to the camera or camera frame with any posture, as shown in Figure 4. Placed physical marking ruler, {W}, in the workspace of OMCS.

Figure 4.

Attitude calibration.

For any robot control system, there were movement functions for {B}. With these functions, commanded robot end-effector move a certain distance in three principal axes of {B} separately. The origin position of {Cc} was recorded at the start and finish of simple action. Then the principal axis vectors of {B} relative to {W} was determined. The concrete steps were as follows.

The robot end-effector was driven to move any distance along the X axis of {B} with program (more than 10cm is recommended), and the origin position of {Cc} at the start and final state were recorded from OMCS software, denoting as ${\mathit{o}}_1$ and ${\mathit{o}}_2$ respectively.

Then, the origin position of the final state was subtracted from the origin position of the start state to obtain the positive direction vector of X axis of {B} corresponding to {W}.

$${\mathit{v}}_x=({\mathit{o}}_2-{\mathit{o}}_1)/|{\mathit{o}}_2-{\mathit{o}}_1|$$

(6)

In the same way, the positive direction vector of Y and Z axis of {B} corresponding to {W} were obtained, denoted as ${\mathit{v}}_y$ and ${\mathit{v}}_z$ .

Then, the attitude matrix ${\mathit{R}}_1$ was,

$${\mathit{R}}_1=\left[\begin{array}{ccc}{\mathit{v}}_x& {\mathit{v}}_y& {\mathit{v}}_z\end{array}\right]$$

(7)

In this scheme, GRBM installation was rather easy. Attitudes of two CSs were aligned.

As for mobile manipulator, for example, when the manipulator was installed on a mobile platform with straight track, the {B} was aligned with the CS of the track through mechanical positioning design, such as cylindrical surface matching with positioning pin. This method only contained manufacturing error and assembly error. The accuracy was under control. Therefore, the attitude calibration proposed above was also suitable to mobile robot.

3. Virtual Zero Position Deviation for {T} and {Cg} by Inverse Movement Matrix

In most commercial 3D modeling software for film production, the CS of camera was set as follows. Positive direction of Z axis was pointed to outside from lens. Positive direction of Y axis was along the normal vector of top plane of camera, as shown in Figure 2(f).

In multi-camera OMCS, GRBM had no CS in OMCS software before system initialisation. When {W} was determined, no matter GRBM was placed in any posture, the attitude of {Cg} would be aligned with {W} during GRBM register procedure. ⁵ Figure 5(a) shows that the gripper frame was hard for human control, because Z axis was not towards to the forward direction of gripper frame.

Figure 5.

Initialization for GRBM. (a) Unreasonable register posture. (b) Standard register posture.

Therefore, the gripper frame, mounted with GRBM representing the virtual camera, should be taken into consider in GRBM register procedure. Specifically, during register, the direction of the virtual camera set by the gripper frame should be aligned with the Z axis of {W}. The direction of normal vector of the top plane of the virtual camera, set by the gripper frame, should be aligned with Y axis of the {W}, as shown in Figure 5(b).

The desired work PAM procedure was that, first the gripper frame was placed in the start position. The camera robot mobile platform was placed in the workspace of OMCS where the camera could finish the entire shooting trajectory. Second the camera kept current position, usually a suitable point in dexterous workspace of manipulator. Third the attitude of {T} was modified to be aligned with {Cg}, as shown in Figure 6(b). There was a fixed position deviation, ${\mathit{P}}_{c1}$ , for two CSs. When {Cg} was moved by photographer, {T} reproduced the motion of {Cg}, as shown in Figure 6.

Figure 6.

Desired work procedure. (a) Start position. (b) Alignment. (c) Following.

According to Formula (4) and (5), the position vectors, ${\mathit{P}}_{a1}$ in ${\mathit{A}}_1$ and ${\mathit{P}}_{c1}$ in ${\mathit{C}}_1$ , were unknown. These position vectors were hard to be determined by general measure tools.

By analysis application requirement of camera teleoperation, it found that the origin of {Cg} and {T} could be treated as they were both at the origin of {W}, when the attitude of gripper frame and camera had been aligned. After this, the gripper frame could be moved by photographer.

In another word, it was unnecessary to determine the position deviations in ${\mathit{A}}_1$ and ${\mathit{C}}_1$ . Keeping the position deviation as virtual zero was fine. Then two inverse movement were added to Formula (5) as follows,

$${\mathit{V}}_{2i}\cdot {\mathit{A}}_2={\mathit{A}}_1\cdot {\mathit{V}}_{1i}\cdot {\mathit{B}}_1\cdot {\mathit{B}}_2\cdot {\mathit{C}}_1$$

(8)

Where,

$${\mathit{V}}_{2i}=\left[\begin{array}{cc}\mathit{I}& -{\mathit{v}}_{2i}\\ 0& 1\end{array}\right],{\mathit{V}}_{1i}=\left[\begin{array}{cc}\mathit{I}& -{\mathit{v}}_{1i}\\ 0& 1\end{array}\right]$$

(9)

$${\mathit{V}}_{1i}\cdot {\mathit{A}}_1=\left[\begin{array}{cc}{\mathit{R}}_{a2}& 0\\ 0& 1\end{array}\right]$$

(10)

$${\mathit{v}}_{1i}\cdot {\mathit{B}}_1\cdot {\mathit{B}}_2=\left[\begin{array}{cc}{\mathit{R}}_{b1b2}& 0\\ 0& 1\end{array}\right]$$

(11)

$${\mathit{v}}_{2i}=\left[\begin{array}{cc}\mathit{I}& -{\mathit{P}}_{a2}\\ 0& 1\end{array}\right],{\mathit{v}}_{1i}=\left[\begin{array}{cc}\mathit{I}& -{\mathit{P}}_{b1b2}\\ 0& 1\end{array}\right]$$

(12)

In Formula (8), $v_{2i}$ was position vector of {B} relative to {W}. Thus ${\mathit{V}}_{2i}$ was the inverse displacement relative to ${\mathit{A}}_2$ . The transformation $({\mathit{V}}_{2i}\cdot {\mathit{A}}_2)$ got the CS whose origin was coincident with {W} and the attitude was aligned with {Cg}, as shown in Figure 7(a).

Figure 7.

Inverse displacement. (a) ${\mathit{V}}_{2i}$ . (b) ${\mathit{V}}_{1i}$ .

${\mathit{v}}_{1i}$ was the position vector of {T} relative to {B}. Thus ${\mathit{V}}_{1i}$ was the inverse displacement relative to $({\mathit{B}}_1\cdot {\mathit{B}}_2)$ . The transformation $({\mathit{V}}_{1i}\cdot {\mathit{B}}_1\cdot {\mathit{B}}_2)$ got the CS which orgin was coincident with {B} and the attitude was aligned with {T}, as shown in Figure 7 (b).

Then we set,

$${\mathit{P}}_{a1}=0,{\mathit{P}}_{c1}=0$$

(13)

That is,

$${\mathit{A}}_1=\left[\begin{array}{cc}{\mathit{R}}_{a1}& {\mathit{P}}_{a1}\\ 0& 1\end{array}\right]=\left[\begin{array}{cc}{\mathit{A}}_{a1}& 0\\ 0& 1\end{array}\right]$$

(14)

$${\mathit{C}}_1=\left[\begin{array}{cc}\mathit{I}& {\mathit{P}}_{c1}\\ 0& 1\end{array}\right]=\mathit{I}$$

(15)

By Formula (15), the origins of {Cg} and {T} were both coincident with origin of {W}, which could be perceived from Figures 3 and 7. At the same time, Formula (8) still hold. That meant no matter where the gripper frame and robot base were placed, the attitude alignment could be processed.

• 4. Camera Teleoperation Instance

A registered GRBM was installed on the camera with arbitrary posture, as shown in Figure 6(a). The ${\mathit{B}}_1$ and ${\mathit{B}}_2$ at the start state were recorded as ${\mathit{B}}_{1\mathrm{s}}$ and ${\mathit{B}}_{2\mathrm{s}}$ . The transformation ${\mathit{B}}_{1\mathrm{s}}$ was read directly from robot system. The transformation ${\mathit{B}}_{2\mathrm{s}}$ could be obtained through calibration program embedded in robot system.

$${\mathit{B}}_{1s}=\left[\begin{array}{cccc}0& -1& 0& -800\\ -1& 0& 0& 10\\ 0& 0& -1& -200\\ 0& 0& 0& 1\end{array}\right]$$

(16)

$${\mathit{B}}_{2s}=\left[\begin{array}{cccc}1& 0& 0& 2\\ 0& 0& 1& 100\\ 0& -1& 0& 180\\ 0& 0& 0& 1\end{array}\right]$$

(17)

The end-effector was moved along the positive directions of three principal axes of {B} separately according to program. The origin position of GRBM in OMCS software was recorded at the start and finish of each movement, as in Table 2.

Table 2.

Records of simple standard actions.

Robot base CS	Origin position at start (mm)			Origin position at finish (mm)
X axis	−1300	1500	−2100	−1300	1500	−2300
Y axis	−1300	1500	−2100	−1560	1500	−2100
Z axis	−1300	1500	−2100	−1300	1800	−2100

By the Formula (6) and (7), the attitude of {B} was obtained, recorded as ${\mathit{R}}_{a1}$ , which was a submatrix of ${\mathit{A}}_1$ .

$${\mathit{R}}_{a1}=\left[\begin{array}{cc}\begin{array}{c}0\\ 0\end{array}& \begin{array}{cc}-1& 0\\ 0& 1\end{array}\\ -1& \begin{array}{cc}0& 0\end{array}\end{array}\right]$$

(18)

$${\mathit{A}}_1=\left[\begin{array}{cccc}0& -1& 0& p_{a1x}\\ 0& 0& 1& p_{a1y}\\ -1& 0& 0& p_{a1z}\\ 0& 0& 0& 1\end{array}\right]$$

(19)

The posture of registered GRBM on gripper frame was read directly from OMCS. The transformation was ${\mathit{A}}_2$ .

$${\mathit{A}}_2=\left[\begin{array}{cccc}0& -1& 0& -1000\\ 0& 0& 1& 100\\ -1& 0& 0& 80\\ 0& 0& 0& 1\end{array}\right]$$

(20)

According to Formula (12),

$${\mathit{v}}_{2i}=\left[\begin{array}{c}{\mathit{v}}_{2ix}\\ {\mathit{v}}_{2iy}\\ {\mathit{v}}_{2iz}\end{array}\right]=\left[\begin{array}{c}-1000\\ \begin{array}{c}100\\ 80\end{array}\end{array}\right]$$

(21)

$${\mathit{v}}_{1i}=\left[\begin{array}{c}\begin{array}{c}{\mathit{v}}_{1ix}\\ {\mathit{v}}_{1iy}\end{array}\\ {\mathit{v}}_{1iz}\end{array}\right]=\left[\begin{array}{c}\begin{array}{c}-900\\ 8\end{array}\\ -380\end{array}\right]$$

(22)

After inverse movement, the origins of {Cg} and {T} should be coincident to origin of {W}. By Formula (14) and (15),

$${\mathit{A}}_1=\left[\begin{array}{cc}{\mathit{R}}_{a1}& {\mathit{P}}_{a1}\\ 0& 1\end{array}\right]=\left[\begin{array}{cccc}0& -1& 0& 0\\ 0& 0& 1& 0\\ -1& 0& 0& 0\\ 0& 0& 0& 1\end{array}\right]$$

(23)

$${\mathit{C}}_1=\left[\begin{array}{cc}\mathit{I}& {\mathit{P}}_{c1}\\ 0& 1\end{array}\right]=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]$$

(24)

Now the desired teleoperation procedure was achieved. For instance, the gripper frame was shifted up 500 mm, and rotated $45\circ$ around the Y axis of {W}, denoted as ${\mathit{A}}_2$ .

$${\mathit{A}}_2=\left[\begin{array}{cccc}0.7071& 0& 0.7071& -1000\\ 0& 1& 0& 600\\ -0.7071& 0& 0.7071& 80\\ 0& 0& 0& 1\end{array}\right]$$

(25)

According to Formula (8), we had,

$${\mathit{B}}_1=({\mathit{A}}_1\cdot {\mathit{V}}_{1i}{)}^{-1}\cdot {\mathit{V}}_{2i}\cdot {\mathit{A}}_2\cdot ({\mathit{B}}_2\cdot {\mathit{C}}_1{)}^{-1}=\left[\begin{array}{cccc}0.7071& -0.7071& 0& -830.7\\ -0.7071& -0.7071& 0& 80.1\\ 0& 0& -1& 300\\ 0& 0& 0& 1\end{array}\right]$$

(26)

Compared with ${\mathit{B}}_{1s}$ in Formula (16), it could be seen that the origin of {F} moved 500 mm in the positive direction of the Z axis of {B}. At the same time, since the origin of {T} was not on the Z axis of {F}, the x and y coordinates of position vector in ${\mathit{B}}_1$ had a certain displacement for rotation. This result was conformed to vision perception in real world.

Calculation verification was as follows.

Substituted ${\mathit{B}}_1$ in the left of Formula (3),

$${\mathit{A}}_1\cdot {\mathit{B}}_1\cdot {\mathit{B}}_2=\left[\begin{array}{cccc}0.7071& 0& 0.7071& -8\\ 0& 1& 0& 120\\ -0.7071& 0& 0.7071& 900\\ 0& 0& 0& 1\end{array}\right]={\mathit{A}}_2\cdot {\mathit{V}}_f$$

(27)

Where,

$${\mathit{V}}_f=\left[\begin{array}{cccc}1& 0& 0& 121.6\\ 0& 1& 0& -480\\ 0& 0& 1& 1281.3\\ 0& 0& 0& 1\end{array}\right]$$

(28)

${\mathit{V}}_f$ indicated the camera was aligned with the attitude of gripper frame at any time and there was a fixed displacement between the origins of {T} and {Cg}. CRTS could reproduce the motion of gripper frame in real-time.

The flow chart of the PAM process was shown in Figure 8.

Figure 8.

PAM work flow.

Throughout the entire PAM process, the motion of the robot's end-effector along the three principal axes had utilised the basic linear motion functions within the robot's firmware. When the end-effector reached the specified pose, the pose reading function within the robot system firmware had been employed, which read the position of the tool CS relative to the base CS, these two functional functions were commonly found in the firmware systems of various robot manufacturers. Therefore, the PAM was applicable to the most of robot systems.

Comparative experiment

Considering the requirements of camera teleoperation application, the experiment focused on attitude calibration. Franka Emika manipulator and FusionTrack500 OMCS were used, as shown in Figure 9. {B} was defined in the underside of robot link 1 in Figure 9(a), without special physical identification. Thus, the exact position for origin of {B} could not be obtained.

Figure 9.

Calibration system description. (a) Calibration scene. (b) Reference GRBM - {W}.

Manual Vs PAM

The target of experiment was to calibrate the attitude transformation between the robot base CS, and reference GRBM that is {B} and {W}. Two methods were used for calibration. The first was the manual calibration scheme. The second was PAM.

1. Manual Calibration Scheme

In manual calibration scheme, it was hard to measure the attitude deviation of two CSs by general measurement tools. Here, by adjusting the height of the footmaster casters on robot base, the parallelism between the robot base plane and the workbench plane was got as shown in Figure 10(a). By adjusting the orientation of workbench, the parallelism between the vertical plane of the robot base platform and the workbench side was got. Next, the reference GRBM was adjusted to align with the attitude of the workbench, as shown in Figure 10(c). Then the attitude calibration task was completed manually.

Figure 10.

Manual attitude calibration scheme. (a) Adjustable height caster. (b) Parallelism judgment with visual. (c) Adjust reference GRBM.

The ground truth value of attitude in the manual calibration scheme was calculated through the GRBM, {Cc}, attached to the end of the manipulator, as shown in Figure 11.

Figure 11.

The relative attitude relation between {F} and {Cc}.

We described the specific algorithm as follows. Ideally, {W} had the same attitude as {B}. In the actual operation, only the coordinate axes of {W} and {B} were paralleled. For example, the positive direction of Z axis of {W} could be same as the positive direction of Y axis of {B}. The directions of corresponding axes were not necessary to match.

Next, we would discuss how to handle the deviation on the attitude of two CSs. Constructed a virtual CS, {D}, whose origin was coincident with {W} and directions of axes were aligned with {B}, as shown in Figure 9(a).

$${}_D^B{\mathit{R}}_{perfect}=\left[\begin{array}{cc}\begin{array}{c}1\\ 0\end{array}& \begin{array}{c}\begin{array}{cc}0& 0\end{array}\\ \begin{array}{cc}1& 0\end{array}\end{array}\\ 0& \begin{array}{cc}0& 1\end{array}\end{array}\right]$$

(29)

By rotation matrix ${}_D^W\mathit{R}$ , the attitude alignment of two CSs could be adjusted. According to the relative relation between {D} and {W}, there was,

$${}_D^W\mathit{R}=\left[\begin{array}{ccc}1& 0& 0\\ 0& 0& 1\\ 0& -1& 0\end{array}\right]$$

(30)

By fusionTrack500 system software, {Cc}, could be read relative to {W}, denoted as ${}_{Cc}^W\mathit{R}$ . We had,

$${}_{Cc}^W\mathit{R}=\left[\begin{array}{ccc}-0.018076& 0.98765& 0.15559\\ -0.38913& -0.15030& 0.90883\\ 0.921& -0.044119& 0.38704\end{array}\right]$$

(31)

The transformation of {W} relative to {Cc} was,

$${}_W^{Cc}\mathit{R}={}_{Cc}^W{\mathit{R}}^T$$

(32)

The transformation of {Cc} relative to {F} was denoted as ${}_{Cc}^F\mathit{R}$ , as shown in Figure 11.

We had,

$${}_{Cc}^F\mathit{R}=\left[\begin{array}{ccc}1& 0& 0\\ 0& -1& 0\\ 0& 0& -1\end{array}\right]$$

(33)

The transformation of {F} relative to {B} could be read directly from the robot system software, denoted as ${}_F^B\mathit{R}$ .

The rotation rule was as follow. The initial reference was {B}. It rotated around the $\mathrm{X}$ , $\mathrm{Y}$ and $\mathrm{Z}$ axes in turn corresponding to the updated CSs. We had,

$${}_F^B\mathit{R}={}_a^B{\mathit{R}}_x\cdot {}_b^a{\mathit{R}}_y\cdot {}_F^b{\mathit{R}}_z$$

(34)

Then we substituted the values of rotation into Formula (34).

The attitude ground truth value during manual calibration could be obtained by Formula (35),

$${}_D^B{\mathit{R}}_{manual}={}_F^B\mathit{R}\cdot {}_{Cc}^F\mathit{R}\cdot {}_W^{Cc}\mathit{R}\cdot {}_D^W\mathit{R}$$

(35)

The attitude deviation between ${}_D^B{\mathit{R}}_{perfect}$ and ${}_D^B{\mathit{R}}_{manual}$ could be obtained by calculating the sum angle deviation of each principal axis direction.

Twenty persons were selected for independent calibration procedure. The major non-systematic error in the calibration process was the parallelism judgement visually.

2. PAM Scheme

Before calibration, reference GRBM could be placed arbitrarily without any restriction, as shown in Figure 12.

Figure 12.

PAM scheme.

Through the robot control interface, the movements along the positive direction of standard axis were implemented. The positions of start and finish points were recorded. The unit vectors in three principal directions were calculated, denoted as ${\mathit{v}}_x$ , ${\mathit{v}}_y$ , ${\mathit{v}}_z$ . Then we had,

$${}_B^W\mathit{R}=\left[\begin{array}{ccc}{\mathit{v}}_x& {\mathit{v}}_y& {\mathit{v}}_z\end{array}\right]$$

(36)

The measurement accuracy of FusionTrack500 optical track system was $\pm 0.09$ mm. Therefore, the mean value of each measured data was taken as the ground truth value to calculate the nominal CS. Based on this, the attitude deviation of each measurement was calculated, and then the mean and standard deviation were calculated.

By data analysis, the results were shown in Figure 13. The following conclusions can be achieved.

The data measurement stability (coefficient of variation) of PAM is significantly higher than that of the manual calibration scheme, whose magnitude order is 3–4 of it. The high stability of data measurement indicates the high reliability of PAM.

The attitude error of PAM is within 0.2 degrees, and that of the manual calibration scheme is within 13 degrees. The accuracy of PAM is improved for 2 orders.

The time-consuming fluctuation (standard deviation) of manual calibration scheme is large, which is 6.2s. And the average time is 59s. The time-consuming fluctuation of PAM is very small, which is 0.03s. The average time is 14.1s.

Figure 13.

Performance comparison.

To sum up, PAM is faster and more efficient.

Tsai-Lenz vs PAM

Tsai-Lenz and PAM were both automated. For eye-to-hand system, the matrices in $\mathit{A}\mathit{X}=\mathit{X}\mathit{B}$ by Tsai-Lenz method were as follows. $\mathit{A}$ was ${}_B^T\mathit{H}$ , representing robot base CS relative to end-effector CS. It was read from robot control system. $\mathit{X}$ was ${}_A^B\mathit{H}$ , representing camera CS relative to robot base CS. $\mathit{B}$ was ${}_W^A\mathit{H}$ , representing GRBM CS relative to camera CS. It was read from OMCS. $\mathit{X}$ was calibration target.

Note that {A} was defined as camera CS in OMCS.

On the other hand, PAM calibration result was ${}_B^W\mathit{R}$ . By reading ${}_W^A\mathit{R}$ from fusionTrack OMCS software, ${}_A^B\mathit{R}$ was calculated as follows,

$${}_A^B{\mathit{R}}_{PAM}=({}_W^B\mathit{R}\cdot {}_A^W\mathit{R}{)}^{-1}$$

(37)

In processing of two methods, the posture of {W} was fixed.

• 1. Calibration Process

PAM calibration was executed 50 times for the same key points.

Tsai-Lenz calibration was executed 50 times. Each one had 3 stations, which included rotation matrix and displacement vector.

2. Performance Comparison

The rotation matrix of ${}_A^B\mathit{H}$ for two methods were compared.

• Both methods were automatic. The uniformity of calibrations result is good.

• In Figure 14, the attitude error is quite small for each calibration, less than 2 degrees. It shows two calibration results are almost same.

• The time consuming of both automatic calibration methods are almost same.

The usage condition comparison is as follows,

Since PAM is not based on $\mathit{A}\mathit{X}=\mathit{X}\mathit{B}$ , the error of PAM is independent of the calibration attitudes. On the other hand, Tsai-Lenz is depended on the calibration attitude. Thus, PAM is more convenient for camera robot teleoperation.

Since PAM relies on the relative position of two GRBMs, the measurement accuracy is greater than Tsai-Lenz which read the posture of GRBM relative to camera CS directly.

Tsai-Lenz can calibrate the posture information. PAM is good at attitude calibration. For camera robot teleoperation application, PAM is enough by inverse movement matrix.

Figure 14.

PAM VS Tsai-Lenz.

Conclusions

In this paper, we proposed PAM and discussed an implementation in teleoperation of camera robot based on OMCS. The installation of the GRBM used for calibration is arbitrary which is easy to implement. The calibration procedure of PAM is automatic, accurate and fast, which does not contain non-systematic error. PAM is suitable for almost all kinds of robots.

The comparative experiments show that PAM is better than the manual scheme in many fields, including stability, accuracy and time consuming. Tsai-Lenz has almost same performance with PAM. But PAM is more suitable for camera robot teleoperation application scenario, for which has no limitation on calibration pose.

Since the OMCS is rather expensive, it is better to look for local motion capture system for reducing costs.