PiROB: Vision-based pipe-climbing robot for spray-pipe inspection in nuclear plants

Abstract

Among many pipes in a nuclear power plant, the spray pipes in the reactor building are one of the most important pipes in view of their function and safety aspects. However, it is very difficult to manually reach and inspect the pipes for defects or damage, because these pipes are installed in very high places. To carry out this kind of inspection more easily, we developed a mobile robot to climb up and down and to cross over such pipes. A mobile robot should be small and light enough that it can be practically and safely operated in a nuclear power plant. Our robot is able to overcome obstacles such as valves, pipe flanges, and T-shaped branches, and it also meets the requirements of fail-safe, autonomous grasping, and self-power without the cables to the remote control station. The robot has a five-degree-of-freedom manipulator and two grippers and moves along the cylindrical pipes bypassing the obstacles. The robot should be able to grasp the next pipe autonomously, because the robot works in places high off the ground where the remote operator cannot see the next pipe for the robot to grasp. This article proposes a vision-based scheme for grasping a cylindrical pipe semi-autonomously and describes its solution along with the forward kinematics and inverse kinematics of the mobile robot. The configuration of the pipe-climbing robot, including its hardware and software, is described, and the robot control with visual grasping is explained. The robot can be used practically for spray-pipe inspection as well as many potential other applications, such as inspection of the roof frame of a stadium consisting of pipes.

Introduction

Mobile robotic technology has been widely explored within the inspection fields of various industries. Specifically, we look at a pipe-climbing robot to inspect the integrity of pipe structures. Such structures might include the containment spray system in a nuclear power plant or the roof frames of a stadium consisting of multiple pipes. In various light-water reactors worldwide, a containment spray system is installed as part of the accident management system on the ceiling of the containment interior, as shown in Figure 1.

Figure 1.

Containment spray-piping system in nuclear power plants.

When an accident such as a loss of coolant happens at a nuclear power plant, the increase in heat and pressure within the containment vessel is extreme. To handle such atmospheric condition, the containment spray system works to spray coolants from the ceiling of the containment building. The spray system is constructed by connecting various pipes together. These pipes must be inspected periodically to determine whether they have defects or faults. However, the pipes are installed in very high places, and it is difficult for a human to climb the pipes and inspect them. To carry out (mainly) visual inspections, we developed a mobile robot to climb up and down, and to cross over, such pipes. We designed the robot to meet the following requirements.

The mobile robot should be light enough so that it can be practically and safely operated in a nuclear power plant. In the field, for example, humans should avoid injury, even if the robot fails to grip the pipe and falls down. The robot should also not fall, even after a loss of power.

The robot should be able to climb spray pipes up to 50 m, work at the ceiling of the containment building, and then climb down again.

The robot should be able to bypass obstacles: T-shaped branches, pipe flanges, and valves.

The robot must be able to work with no power cables and no communication cables to the ground, thus avoiding problems involving the weight or twisting of the cable.

The robot should be able to autonomously grip nearby pipes because the remote operator often cannot see the target pipe well.

Thus, we designed and optimized a very light five-degree-of-freedom (DOF) manipulator to reduce unnecessary DOFs, whereas the conventional system uses a general six-DOF manipulator. The robot has five links and two grippers at both its ends, and moves along cylindrical pipes, bypassing obstacles such as flanges and valves, in motions such as an inchworm or a gibbon.

There is a body of research about climbing robots to be used for inspections and maintenance of vertical structures. ¹ Many forms of locomotion have been employed, including arms and legs (limbs), ^{2 –4} wheels and chains, ⁵ a sliding frame, ⁶ and wires and rails. ⁷ As with locomotion, a number of types of adhesion have also been used, depending on the given task, including magnetic adhesion, ^{8 –10} pneumatic adhesion, ¹¹ mechanical adhesion, ^{12 –15} electrostatic adhesion, ¹⁶ and chemical adhesion. ¹⁷

Among these options, we adopted arms and legs for locomotion and mechanical grip for adhesion. Limb-climbing robots are highly adaptable to surface structure and can overcome obstacles and steps. ¹⁸ Such grip-based adhesion has a key safety advantage because even loss of power does not necessarily lead to a drop-off of the system, even though admittedly such systems are not very fast. Similar climbing robots have also been developed by researchers. ^19,20

ROMA ¹⁹ was designed to climb on beam-based structures with its two grippers. InspiRat ²⁰ is a robot built from biological inspiration and is able to climb cables and tubes like a rat. Treebot ²¹ was designed to climb trees or poles and could be helpful in terms of inspection and maintenance applications involving wooden structures.

Three-dimensional (3-D) CLIMBER was developed by the University of Coimbra. ^22,23 That robot has a four-DOF manipulator and two grippers. It is a very useful mechanism which is able to climb arbitrarily bent poles. Climbot ^{24 –26} was also developed at South China University of Technology for pipe climbing using a five-DOF mechanism. It can climb cylindrical pipes as well as rectangular truss structures with high mobility and good manipulation function. However, none of these aforementioned robots are suitable for practical use for a containment spray system in a nuclear power plant in terms of the required combination of weight, automatic gripping function, and autonomy.

For these reasons, we developed a very practical robot for the inspection of containment spray-piping systems in nuclear power plants. Our mobile robot is very small and light enough that it is practical for operation in a nuclear power plant. It also has a fail-safe feature especially for use in cases of blackout. The robot has no power cables and no communication cables from the ground. It communicates with the ground control station via wireless communications. The robot autonomously grips nearby pipes because the remote operator often cannot see the objective pipe well. We designed and optimized a very light five-DOF manipulator to reduce unnecessary DOFs. The robot has five links and two grippers at each end and is able to move along cylindrical pipes, bypassing obstacles such as flanges and valves, like an inchworm or a gibbon.

The most essential technology for a pipe-climbing robot is how to grasp pipes firmly, quickly, and precisely. There have been several studies to try and solve this problem, ^{27 –29} which is a very difficult aim because the robot climbs so far up and away from where the human operates the robot. The human operator cannot see the robot’s hands or the object pipe from the ground. Equipping the robot with several stereo cameras is problematic owing to their weight and volume, and because remote visual operation requires the skills of highly trained people. For these reasons, we herein propose a practical scheme for a pipe-climbing robot to grasp pipes autonomously, and the scheme can be used in real time.

The remainder of this article is organized as follows. The configuration of the PiROB as well as its hardware and software is described in the next section. The third section includes the forward and inverse kinematics of the robot, as well as its Jacobian for master control. A vision-based grasping scheme is proposed and its achievement is described in the fourth section. Finally, we summarize the results and address items for future research.

Pipe-climbing robot and its kinematics

Our pipe-climbing robot, called PiROB, is shown in Figure 2. The robot was originally designed to climb up a spray-piping system located in the containment building of a nuclear power plant, as shown in Figure 1. The piping system has obstacles such as flanges and valves along the pipe, and the ceiling is up to 40 m high. The pipes have branches to feed spray water into horizontally circumferential pipes to reach the entire containment.

Figure 2.

General view of the vision-based pipe-climbing robot.

Robot design

The PiROB is a robot with five-DOF and two grippers at both ends and moves along the cylindrical pipes, bypassing pipe obstacles such as flanges and valves like an inchworm or gibbon. It has a symmetric shape in view of the determination of the base link. The two grippers work as a base by turn. Figure 3 shows a kinematic model. Suppose that gripper A is closed, gripping the pipe, and that gripper B is open. In this case, gripper A works as the base of the manipulator and gripper B is considered a hand of the five-DOF manipulator. The robot has five joints, which are all rotational joints. Frame {0} is placed at the palm of the hand for convenience.

Figure 3.

Kinematic modeling of the pipe-climbing robot and its coordinate systems.

DOF analysis

Generally, six-DOF is needed for a manipulator hand to be placed at any specified position and orientation (three-DOF for position and three-DOF for orientation). Because PiROB has five-DOF, it cannot place its hand at any desired position and orientation. However, since the pipe is cylindrical and symmetric about its center line, its orientation can be considered as two-DOF just like a line. Therefore, five-DOF PiROB can theoretically grasp the objective pipe, even though it cannot grasp the objective pipe at a specified three-DOF orientation. These are general phenomena of a five-DOF robot manipulator. ³³

If we consider the nature of the subspace of the five-DOF PiROB manipulator, we quickly realize that this subspace creates one constraint on the attainable orientation, as shown in Figure 4. The pointing direction of the gripper, the z ₅ axis, must lie in the “plane of the arm.” This plane is the vertical plane which contains the axis of joint 1, z ₁, and the point where axes 4 and 5 intersect. When the target point to grasp, P, is specified, the robot has to rotate about the first joint so that the plane of the arm passes the target point, P. The pointing direction, z_p , is determined so that z_p is perpendicular to the center line of pipe vector, u, and z_p lies in the plane of the arm, that is, $z_p\perp u$ and $z_p\in S$ .

Figure 4.

PiROB manipulator and its pointing direction to the pipe.

It can be summarized as follows: Since the robot has five DOFs, the robot cannot grasp the pipe at a specified position and arbitrary-specified orientation. When the target point of the pipe is specified, the robot grasps the pipe along the pointing direction which is passively determined. The robot uses three DOFs for positioning, one DOF to mate passively, determining pointing direction in the plane of the arm, and the other one DOF for the gripper to mate the pipe.

PiROB weighs 3.8 kg and its total length is 90 cm. Following is a description of the experimental setup as experienced. To reduce weight, the frame of the link between the joints was optimally designed, even though the frame deflected slightly. The motor casing was used as part of the link frame to reduce the weight of the robot. The number of motors was minimized based on the DOF analysis, thus the number of links and joints and their weights was reduced. The motors we used were light compared with its torque. It partially used plastic to reduce the weight, even though it resulted in some gear backlash. The gear backlash and frame deflection were compensated when we controlled the robot motion, which is something not described as well as gripper misalignment in this article. ^{29 –32} The gripper was also optimally designed to reduce weight and remove all the unnecessary parts. Even cameras and sensors were minimized in consideration of weight. Electronic devices and a battery were neatly attached to the link frame.

Robot gripper

As shown in Figure 5, the gripper was designed to grasp pipes of 3–5 inches in diameter, considering that the diameter of a spray pipe is normally 4 inches. Because of the limitation of the robot weight and battery power, the gripper was very much optimized and was implemented in a series of trials. Finally, the reduction gear train was a combination of worm and partial worm gear after a bevel gear, in consideration of a number of factors in its use, fail-safe against blackout, reduction ratio, gripping speed, motor torque, and repeatability of motion. By the irreversible worm and worm gear, the gripper would not be open when the electric power was off. The gripping motor was a small DC motor (0.6 W), and the gripper opened in 3 s. The fingers were covered with rubber to improve their grip. Six proximity sensors were embedded in the palm of each hand. The sensors provided a gap between the palm and the objective pipe surface. With regard to weight, we chose the proximity sensor OSG-105LF (Opto Sensor), which is super miniature and a thin, reflective sensor. The emitter and the detector were arranged in the same direction to the object, the emitter being a high output gallium arsenide infrared light-emitting diode (LED) and the detector a high-sensitivity silicon transistor. It had small dimensions (2.7 × 3.2 × 1.4 mm³), an optimum detection distance of 0.8–1.2 mm, and a wavelength of 940 nm.

Figure 5.

Illustration of gripper design and power transmission.

A camera and light were mounted on the robot hand to help see the object to grasp. The camera had a charge-coupled device with 525 lines interlaced and transmitted the image to the remote station wirelessly. A diode laser with a slit beam was also mounted there to assist during visually assisted grasping. The image captured by the camera was transmitted to a remote control station via wireless communication. The overall specification of PiROB is listed in Table 1.

Table 1.

Overall specifications of PiROB.

Quantity	Values
Weight	3.8 kgf
Length	900 mm
Climbing speed (max)	18 mm/s
Climbing height (with one charge)	69 m
Operating time without recharging	75 min (average)
Grasping force	6.5 kgf
Payload	4.5 kg
Joint servomotor	Mass: 0.154 kg Internal gear ratio: 184:1 Motor stall torque: 0.857 kgf·m
Sensor 1 (proximity sensor)	Proximity sensors, inclinometers
Sensor 2 (inclinometer)	Not installed yet
Sensor 3 (visual camera)	Color with WiFi
Battery	14.8 V, 5Ah

Forward kinematics and inverse kinematics

To analyze the kinematics of our robotic system, let us define the following coordinate system used to develop the modeling of the whole robotic system:

{W} is the world coordinate system $\left(x_w,y_w,z_w\right)$ .

{M_i } is the coordinate system of the link i $\left(x_i,y_i,z_i\right)$ .

{C} is the camera coordinate system $\left(x_c,y_c,z_c\right)$ .

In Figure 3, the robot is shown as grasping a horizontal pipe with gripper A, which, in this position, works as the base of the robot. For ease of calculation, the world coordinate system, {W}, is placed at the center line of the pipe grasped by gripper A, and the robot base coordinate system, {M ₀}, is placed at the palm of gripper A. We can think of this robot as a general five-DOF manipulator. Joints 1 and 5 implement roll movements, allowing the robot to rotate about its own axis. Joints 2, 3, and 4 achieve rotational movement like a human shoulder, elbow (or knee), and wrist (pitch), respectively, with its revolute axis perpendicular to the link axes. The coordinate system is placed at each joint according to the general convention that the rotating axis of each joint is presented as z_i -axis. ³³

The pipes were bent along the center line with a unit vector, u, with respect to the world coordinate system. To grasp the bent pipe, the robot should mate the coordinate vector, x ₅, with the pipe orientation vector, u, and the axis vector, z ₅, should direct to the center line of the pipe. The palm of gripper B should contact the external surface of the objective pipe.

In this section, we determine the position of gripper B with respect to the {Rob-Base} frame. Thus, we will derive the kinematics of the robot manipulator. From the basic equation, the position, P, with respect to {Rob-Base}, ^Rob P, is obtained by multiplication of the homogeneous transform from {Rob-Base} to {M5}, and the position with respect to {M5}, ^M5P. Thus, the positional vectors have the following relationship

${}_{}{}^{\text{ROB}}P={}_{ M5}{}^{\text{ROB}}T{}_{}{}^{M5}P$

1

and the kinematics of the robot can be described as

${}_{ M5}{}^{\text{ROB}}T={}_{ M1}{}^{\text{ROB}}T{}_{M2}{}^{M1}T{}_{M3}{}^{M2}T{}_{M4}{}^{M3}T{}_{M5}{}^{M4}T$

2

where ${}_B{}^{A}T$ denotes the homogeneous coordinate transformation matrix from coordinate system {A} to coordinate system {B} as a notational convention. ³³

After deriving each transformation in equation (2) in Appendix 1, we can calculate the entire transformation, ${}_{ M5}{}^{\text{ROB}}T$ , by multiplying each individual transformation, and rearrange the result through a series of calculations, as follows:

$\begin{array}{l}{}_{ M5}{}^{\text{ROB}}T={}_{ M1}{}^{\text{ROB}}T{}_{M2}{}^{M1}T{}_{M3}{}^{M2}T{}_{M4}{}^{M3}T{}_{M5}{}^{M4}T\\ =\left[\begin{array}{cccc}-c_1{c}_{234}{c}_5+s_1{s}_5& c_1{c}_{234}{s}_5+s_1{c}_5& c_1{s}_{234}& c_1{s}_2{L}_2+c_1{s}_{23}{L}_3+c_1{s}_{234}{L}_4\\ -s_1{c}_{234}{c}_5-c_1{s}_5& s_1{c}_{234}{s}_5-c_1{c}_5& s_1{s}_{234}& s_1{s}_2{L}_2+s_1{s}_{23}{L}_3+s_1{s}_{234}{L}_4\\ s_{234}{c}_5& -s_{234}{s}_5& c_{234}& L_0+L_1+c_2{L}_2+c_{23}{L}_3+c_{234}{L}_4\\ 0& 0& 0& 1\end{array}\right]\end{array}$

3

where c_ijk is shorthand for cos(i + j + k) and s_ijk for sin(i + j + k), and L ₀, L ₁, L ₂, and L ₃ are the length of each link, as shown in Figure 3.

Additionally, the transformation ${}_{\mathrm{ G}}{}^{M5}T$ and ^{M 5} P _contact becomes

${}_{\mathrm{ G}}{}^{M5}T=\text{Trans}\left(0,0,L_5\right)=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& L_5\\ 0& 0& 0& 1\end{array}\right]$

4

${}_{}{}^{M5}P{}_{\text{contact}}=\left[\begin{array}{c}0\\ 0\\ L_5+R_{\text{pipe}}\\ 1\end{array}\right]$

5

Inverse kinematics

The inverse kinematics is obtained by serious effort. When the position and orientation of the object are given by the matrix

${}_{\mathrm{ M}5}{}^{\text{ROB}}T={}_{M1}{}^{R0}T{}_{M2}{}^{M1}T{}_{M3}{}^{M2}T{}_{M4}{}^{M3}T{}_{M5}{}^{M4}T=\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& p_x\\ r_{21}& r_{22}& r_{23}& p_y\\ r_{31}& r_{32}& r_{33}& p_z\\ 0& 0& 0& 1\end{array}\right]$

6

we succeed in obtaining the final solution of the inverse kinematics: θ₁, θ₂, θ₃, θ₄, and θ₅

$\begin{array}{l}{\theta }_1=a\text{tan}2\left(p_x,p_y\right)\hspace{0.17em}\text{or} {\theta }_1=a\text{tan}2\left(p_x,p_y\right)+\pi \\ {\theta }_5=a\text{tan}2\left(s_1{r}_{11}-c_1{r}_{21},s_1{r}_{12}-c_1{r}_{22}\right)\\ {\theta }_3=a\text{tan}2\left(s_3,c_3\right)\hspace{0.17em}=a\text{tan}2\left(\pm \sqrt{1-{\left(\frac{A^2+B^2-L_2{}^2-L_3{}^2}{2L_2{L}_3}\right)}^2},\hspace{0.17em}\frac{A^2+B^2-L_2{}^2-L_3{}^2}{2L_2{L}_3}\right)\\ {\theta }_2={\text{sin}}^{-1}\left[\frac{F}{\sqrt{D^2+E^2}}\right]-a\text{tan}2\left(\frac{E}{\sqrt{D^2+E^2}},\frac{D}{\sqrt{D^2+E^2}}\right)\\ {\theta }_4=a\text{tan}2\left(c_2\left(c_1{r}_{13}+s_1{r}_{23}\right)-s_2{r}_{33},\hspace{0.17em}{s}_2\left(c_1{r}_{13}+s_1{r}_{23}\right)+c_2{r}_{33}\right)-{\theta }_3\end{array}$

7

where

$\begin{array}{c}\begin{array}{l}A=c_1{p}_x+s_1{p}_y-L_4\left(c_1{r}_{13}+s_1{r}_{23}\right)\\ B=p_z-L_0-L_1-L_4{r}_{33}\\ D=c_1{p}_x+s_1{p}_y-c_1{r}_{13}{L}_4-s_1{r}_{23}{L}_4\\ E=p_z-L_0-L_1-r_{33}{L}_4\\ F=c_3{L}_3+L_2\end{array}\end{array}$

To solve the inverse kinematics, various kinds of backward transforming techniques are used. For deriving θ₁, θ₅, and θ₃, we use the relationship

${\left[{}_{M1}{}^{R0}T\right]}^{-1}{}_{M5}{}^{\text{ROB}}T= {}_{M2}{}^{M1}T {}_{M3}{}^{M2}T {}_{M4}{}^{M3}T {}_{M5}{}^{M4}T= {}_{M5}{}^{M1}T$

and for deriving θ₂ and θ₄, we use

${\left[{}_{M2}{}^{R1}T\right]}^{-1}{\left[{}_{M1}{}^{R0}T\right]}^{-1}{}_{\mathrm{ M}5}{}^{\text{ROB}}T={}_{M3}{}^{M2}T {}_{M4}{}^{M3}T {}_{M5}{}^{M4}T= {}_{M5}{}^{M2}T$

Vision-based grasping scheme

For the robot to grasp a pipe, we have to know the position and orientation of the pipe in the form of a homogeneous transformation

$\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& p_x\\ r_{21}& r_{22}& r_{23}& p_y\\ r_{31}& r_{32}& r_{33}& p_z\\ 0& 0& 0& 1\end{array}\right]$

8

This can be calculated through a transformation of the robot-to-pipe coordinate system

${}_{\text{Pipe}}{}^{\text{ROB}}T={}_{M1}{}^{R0}T\left({\theta }_1\right){}_{M2}{}^{M1}T\left({\theta }_2\right){}_{M3}{}^{M2}T\left({\theta }_3\right){}_{M4}{}^{M3}T\left({\theta }_4\right){}_{M5}{}^{M4}T\left({\theta }_5\right){}_{\mathrm{ C}}{}^{M5}T{}_{\text{Pipe}}{}^{\mathrm{ C}}T\left(D_x,D_y,D_z,{\varphi }_x,{\varphi }_y,{\varphi }_z\right)$

9

where $D_x,D_y,D_z,{\varphi }_x,{\varphi }_y,{\varphi }_z$ are transformation variables of camera frame to pipe frame, as shown in Figure 6. We can calculate the first five transformations easily, because ${\theta }_1,\hspace{0.17em}{\theta }_2\cdot \cdot \cdot {\theta }_5$ is the current joint angle of the manipulator. The problem is how to determine the transformation of camera to pipe

${}_{\text{Pipe}}{}^{\mathrm{ C}}T=\text{Trans}\left(D_x,D_y,D_z\right)\hspace{0.17em}\text{ROT}\left(z,{\varphi }_z\right)\hspace{0.17em}\text{ROT}\left(y,{\varphi }_y\right)\text{ROT}\left(x,{\varphi }_x\right)$

10

Figure 6.

Perspective transformation and the variables of the coordinate transformation.

Here, we have the following three assumptions:

A 3-D cylinder is assumed to be a two-dimensional planar strip on a marked plane, as shown in Figure 7. In a real situation, the 3-D cylinder is shown a little bit larger than the planar strip. When the camera lens is closer to the pipe, for example, from the position (a) to (b), the difference becomes larger.

The pipe is cylindrical to be symmetric about the z _pipe-axis. Thus, we set the pipe coordinate system so that the transformation variable, ϕ_z , is zero. Therefore, ROT(z, ϕ_z ) in equation (10) is neglected.

The robot may grasp the pipe at an arbitrary point along the z-axis of the pipe. It does not matter where along z _pipe-axis the robot grasps. Thus, we set the origin of the pipe coordinate system as D_z = 0.

Figure 7.

Difference between 3-D cylinder and planar strip (top view). 3-D: three-dimensional.

From these three assumptions, ${}_{\text{Pipe}}{}^{\mathrm{ C}}T$ can be simplified as

${}_{\text{Pipe}}{}^{\mathrm{ C}}T\left(D_x,D_y,{\varphi }_x,{\varphi }_y\right)=\left[\begin{array}{cccc}\text{c}{\varphi }_y& \text{s}{\varphi }_y\text{s}{\varphi }_x& \text{s}{\varphi }_y\text{c}{\varphi }_x& D_x\\ 0& \text{c}{\varphi }_x& -\text{s}{\varphi }_x& D_y\\ -\text{s}{\varphi }_y& \text{c}{\varphi }_y\text{s}{\varphi }_x& \text{c}{\varphi }_y\text{c}{\varphi }_x& 0\\ 0& 0& 0& 1\end{array}\right]$

11

considering that we have four unknowns: $D_x,D_y,{\varphi }_x,{\varphi }_y$ .

By calculating the geometric relationship between the pipe and image plane, point Q on the pipe mark plane will be projected as P on the image plane of the camera. The relationship is expressed as

${}_{}{}^{i}P=G {}_{}{}^{i}Q=G {}_M{}^{i}T {}_{}{}^{M}Q$

12

or

$\left[\begin{array}{c}{x}_i\\ y_i\\ z_i\\ 1\end{array}\right]=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\\ 0& \frac{1}{f}& 0& 1\end{array}\right]{}_M{}^{i}T\left[\begin{array}{c}{x}_q\\ y_q\\ z_q\\ 1\end{array}\right]$

13

where the matrix, G, is the perspective transformation matrix and f is the focal length of the camera lens.

As shown in Figure 8, the planar pattern of the pipe is projected onto the image plane as a trapezoidal shape. The straight vertical line on the pipe-mark plane will be projected onto an image plane as an oblique line with slope, A ₀, and x-intercept, C ₀. Thus, the projected line is expressed as

$z_i=A_o\left(x_i-C_o\right)$

14

where

$\begin{array}{l}{A}_o=\frac{\left({\text{c}}_{\phi x}{\text{s}}_{\phi y}{\text{s}}_{\phi z}-{\text{s}}_{\phi x}{\text{c}}_{\phi z}\right)\left(-{\text{s}}_{\phi \mathrm{y }}{x}_{mi}+D_z\right)-\left({\text{c}}_{\phi y}{\text{s}}_{\phi \mathrm{z }}{x}_{mi}+D_y\right){\text{c}}_{\phi x}{\text{c}}_{\phi y}}{\left({\text{c}}_{\phi x}{\text{s}}_{\phi y}{\text{s}}_{\phi z}-{\text{s}}_{\phi x}{\text{c}}_{\phi z}\right)\left({\text{c}}_{\phi y}{\text{c}}_{\phi z}{x}_{mi}+D_x\right)-\left({\text{c}}_{\phi y}{\text{s}}_{\phi \mathrm{z }}{x}_{mi}+D_y\right)\left({\text{c}}_{\phi x}{\text{s}}_{\phi y}{\text{c}}_{\phi z}+{\text{s}}_{\phi x}{\text{s}}_{\phi z}\right)}\\ C_o^{}=-f\frac{{\text{c}}_{\phi x}{\text{c}}_{\phi y}{\text{s}}_{\phi y}{x}_m+\left({\text{c}}_{\phi y}{x}_m+D_x\right){\text{c}}_{\phi x}{\text{c}}_{\phi y}}{-{\text{s}}_{\phi x}{\text{s}}_{\phi y}{x}_m+{\text{c}}_{\phi x}{\text{c}}_{\phi y}{D}_y}\end{array}$

15

Figure 8.

Planar pattern of the pipe and its projected image angles. (a) Planar pattern of the pipe. (b) Projected image of the pipe pattern.

We can obtain two slopes, A ₀ ¹ and A ₀ ², and two x-intercepts, C ₀ ¹ and C ₀ ², through image processing. In other words, we have

four unknown values, $D_x,D_y,{\varphi }_x,{\varphi }_y$ , and

four equations with four known values (A ₀ ¹, A ₀ ², C ₀ ¹, C ₀ ²).

Solving the four nonlinear simultaneous equations (15) (i = 1, 2), we can obtain $D_x,D_y,{\varphi }_x,{\varphi }_y$ . Thus, we finally obtain the transformation of the pipe coordinate system, ${}_{\text{Pipe}}{}^{\text{ROB}}T$ , using equation (9). This means that we come to know the position and posture of the pipe as well as the vector u with respect to {Rob-base} coordinate system.

Next, we have to obtain the joint angles of the robot, ${\theta }_1,{\theta }_2,{\theta }_3,{\theta }_4,{\theta }_5$ , to mate the gripper frame {Grip} to the pipe frame {Pipe}. We can obtain the joint angles ${\theta }_1,{\theta }_2,{\theta }_3,{\theta }_4,{\theta }_5$ by solving the inverse kinematics (16), considering the effect of ${}_{\text{Grip}}{}^{\mathrm{ M}5}T$

${}_{\mathrm{ M}5}{}^{\text{ROB}}T\left({\theta }_1,{\theta }_2,{\theta }_3,{\theta }_4,{\theta }_5\right)={}_{\text{Pipe}}{}^{\text{ROB}}T{\left[{}_{\text{Grip}}{}^{\mathrm{ M}5}T\right]}^{-1}$

16

The solution can be obtained using the previous results of equations (6) and (7). Finally, we control the manipulator using the desired joint angles, ${\theta }_1,{\theta }_2,{\theta }_3,{\theta }_4,{\theta }_5$ , which were just derived.

Practical scheme

One issue is that we have to use numerical computation to solve the four nonlinear simultaneous equations (15), and it takes a lot of time to control the manipulator properly. Thus, we propose a more practical scheme to solve this problem for use in real environments.

We turned our attention to the point that the four unknowns, $D_x,D_y,{\varphi }_x,{\varphi }_y$ , have a strong relationship with the four known variables, A ₀ ¹, A ₀ ², C ₀ ¹, and C ₀ ² .

$\begin{array}{l}{D}_x\propto \frac{C_0^1+C_0^2}{2}\\ D_y\propto |C_0^2-C_0^1|\\ {\varphi }_x\propto \frac{w_1}{w_2}=1+\frac{H}{C_0^2-C_0^1}\left\{\frac{1}{A_0^2}-\frac{1}{A_0^1}\right\}\\ {\varphi }_y\propto \frac{1}{2}\left\{A_0^1+A_0^2\right\}\end{array}$

17

Thus, we can calculate the inverse kinematics (16) by supplying four variables for all visual feedback

$\begin{array}{l}{D}_x\left(k\right)=D_x\left(k-1\right)+K_{Dx}\frac{C_0^1+C_0^2}{2}\\ D_y\left(k\right)=D_y\left(k-1\right)+K_{Dy}|C_0^2-C_0^1|\\ {\varphi }_x\left(k\right)={\varphi }_x\left(k-1\right)+K_{\varphi x}\left[1+\frac{H}{C_0^2-C_0^1}\left\{\frac{1}{A_0^2}-\frac{1}{A_0^1}\right\}\right]\\ {\varphi }_y\left(k\right)={\varphi }_y\left(k-1\right)+K_{\varphi y}\left\{A_0^1+A_0^2\right\}\end{array}$

18

where k is the sampling time and K_Dx is the gain, which is determined experimentally.

Thus, we can command the robot manipulator using the joint θ₁, θ₂, θ₃, θ₄, and θ₅.

Robot control with visual grasping

Robot controller

Five servomotors were used for the five joints of the robot manipulator, and two DC motors were used to open and close the gripper. The servomotor, with reduction gears and absolute encoder inside, weighed 154 g, with dimensions of 40 × 65 × 46 mm. Its input voltage was 12 V, stall torque 8.4 Nm, stall current 5.2 A, and no-load speed 45 r/min. We used the same motors at all five joints because the robot works with various kinds of configurations; thus there were no typical joints which required less torque than the others.

The electronic hardware of the robot controller was mounted on the robot link frame. It consisted of three circuit boards: a power board, a CPU board, and a control board (Figure 9). The power board provided four voltages from the 14.8 V DC battery input, as the various components and sensors required different input voltages. The CPU board had a main processor, communicated with the remote computer system wirelessly, and controlled 12 proximity sensors in the palms and inclinometers. The microcontroller was a PIC32MX440F256 H 80 MHz with 256 kB flash and 32 kB RAM. The control board managed the five servomotors at the manipulator joints and the two DC motors at each gripper. It also managed the video camera, line beam laser, and illumination LED.

Figure 9.

Configuration of the electronic hardware of the robot controller.

The communication between the remote control station and the robot was through WiFi, with a frequency of 2.4 GHz, whereas the camera image was separately transmitted to the main control station through WiFi with a frequency of 5.8 GHz. That was because the amount of camera image was very large, and we had to prepare against noise of the transmission. They could communicate 50 m apart from each other, especially when there were no obstacles between them.

We used a lithium-ion rechargeable battery pack to supply power, which was assembled with four high-quality cylindrical cells (3.7 V) in series and built-in printed-circuit-board protection, thus its voltage became 14.8 V, capacity 2.6 Ah, weight 183 g, dimensions 68 × 37 × 38 mm, and maximum discharge current 5 A. The robot could continuously climb up for 64 min with the battery pack and go down for 86 min.

Figure 10 presents our reactor spray-pipe inspection system along with the pipe-climbing robot and the remote control station. A test bed of reactor spray-pipe structures was built in our laboratory. The mockup had typical obstacles (e.g. valves, flanges, T-branches). The remote control station is shown near the robot but would be located outside the containment building in a real situation. The robot communicated with the control station wirelessly. The robot was operated manually but also had some autonomous options.

Figure 10.

The robot-climbing pipe structures and remote control station.

The operator commanded the robot while watching a monitor. A haptic master, Phantom Omni, was used to command the manipulator motion, and send the velocity commands for the hand ( $\dot{x}$ , $\dot{y}$ , $\dot{z}$ ), as well as the angular velocity command of the hand, its pitch, and its roll. Wireless cameras mounted on both robot hands sent images to the remote computer. The images were processed using our algorithm (mentioned in the previous section) and were used for semi-autonomous grasping of the pipe.

Experimental procedure of visual grasping

The graphic user interface of the main control system included a motor control module, a sensor data monitoring module, a master manipulator status module, and a camera image display module among other parts, as shown in Figure 11. The camera image display module shows the gripper grasping the pipe (it is switchable display of two cameras). The camera image was used to calculate the transformation variables of the gripper from the pipe position. Twenty-five frames of the images were processed every second. By processing the captured image, we could obtain two slopes, A ₁ and A ₂, and two x-intercepts, C ₁ and C ₂, as defined in Figure 8. Figure 12 shows the image processing, deriving two slopes and two intercepts. Using the algorithm in the previous section, we could calculate the variables, $D_x,D_y,{\varphi }_x,{\varphi }_y$ , and obtain the transformation, ${}_{\text{Pipe}}{}^{\text{ROB}}T$ ; thus we could command the robot manipulator with joint angles θ₁, θ₂, θ₃, θ₄, and θ₅ after solving the inverse kinematics of equation (16) with equations (6) and (7).

Figure 11.

Graphic user interface of the remote control computer.

Figure 12.

The captured image of the pipe and its processing procedures.

Figure 12 shows the image processing procedure to obtain the line equation of the objective pipe edge. This is to handle lens-distortion compensation of the original image from the on-hand cameras of the robot, edge detection, selection of straight lines, and so on. Detection of the edge line of the pipe is not easy, especially when illumination is poor.

We also had to intelligently choose the correct line among several straight lines behind the pipe. For this, we used a laser line beam to detect straight lines (Figure 13) and determine the correct line from many background lines. The laser scanning device emits the laser line beam to the pipe, varying the incident angle to the pipe. We obtained the differential image between the image captured currently and the image captured just before. The differential image is shown as contours (Figure 12(a1)). After calculating the left and right endpoint of the contours (a2), we fit the line using the endpoints (a3).

Figure 13.

Structural line beam laser and vision system.

However, the obtained pipe edge lines are not the same as real pipe edge because of many uncertain environmental conditions such as illuminations. Thus, we also process a line detection of the original image using Hough transform and obtain many candidates for pipe edge line (b2). Among the candidates, we choose the most similar one (c) with the line obtained by laser scanning (b2).

Figure 14 presents the details of these processing methods. In order to grasp the objective pipe, we moved the gripper near the objective pipe manually with a master or joystick, until the pipe image was displayed on the monitor, or more specifically speaking, until both the left and right boundaries of the pipe were showing. We emitted the laser line beam to the pipe and captured the pipe image. By image processing explained above, we obtained the values of the pipe edge line, A ₁, A ₂, C ₁, C ₂. By solving the perspective equation (15), we obtained the perspective transformation variables, $D_x,D_y,{\varphi }_x,{\varphi }_y$ , and finally obtained ${}_{\text{Pipe}}{}^{\text{Cam}}T$ using equation (10). By supplying current joint angles, ${\theta }_1,{\theta }_2,{\theta }_3,{\theta }_4,{\theta }_5$ , we obtained pipe posture ${}_{\text{Pipe}}{}^{\text{ROB}}T$ using forward kinematics (4). To mate the gripper with the pipe, we solved inverse kinematics (16) considering the effect of ${}_{\text{Grip}}{}^{\mathrm{ M}5}T$ . Thus, we commanded joint angles obtained by solving inverse kinematics on the robot. Then, the robot gripper approached the pipe. During approach, the proximity sensor in the palm of the gripper checked the distance between the palm and pipe surface. If the distance was very close (less than 2 mm), the robot stopped and grasped the pipe. If the distance was more than 2 mm, the robot went to the first step and repeated the grasping procedure, as presented in Figure 14.

Figure 14.

Experimental procedures of visual grasping.

Experiments and results

We carried out a series of experiments under various conditions. We classify the pipe structures for the robot to move as (a) a linear pipe, (b) a curved pipe including T-shaped branch, and (c) parallel-separated pipes. Arbitrary separate pipes are not experimented in this article. When moving along the linear pipe, the robot control and its performance shows differently along the vertical pipe and horizontal pipe. Climbing upward and downward also shows different characteristics. The curved pipe such as T-branch is also classified just like the linear pipe. The next one involves the parallel pipes, which are divided into vertical and horizontal pipes.

PiROB has three gaits to move with five DOFs. ²⁵ The first one is inchworm mode. The robot raises and stretches the forearm and grasps the pipe, then releases the rear-hand gripper and moves close to the forearm. Turning over mode is to release the rear-hand gripper and turn over the body and then grasp the pipe. Turning around mode is to release the rear-hand gripper and then rotate about the forehand pivot and then grasp the pipe. Brachiation is to swing from the current pipe to the objective pipe similar to a gibbon. The latter is not studied in this article and is left for future study.

Table 2 shows the failure rate of visual grasping in each case. Experiments are conducted 50 times for each case at arbitrary varying experimental conditions, such as illumination, background, battery residual amount, wireless communication strength, and so on. The movement on the horizontally linear pipe is easiest, especially in inchworm mode. That is because the pose of the robot need not be changed much, and the gripper is already so close to the objective pipe that the captured pipe image is large enough to obtain the projected line equation. When there are no bumps on the pipe surface, inchworm movement can even be accomplished with the proximity sensor at the palm without visual grasping.

Table 2.

Experimental condition and failure rate of visual grasping for each case (%).^a

	(a) Linear pipe			(b) Curved pipe (T-branch)			(c) Parallel pipes		(d) Arbitrary- separated pipes
	Vertical		Horizontal	Vertical		Horizontal	Vertical	Horizontal
	Climb upward	Climb downward	Left or right	Climb upward	Climb downward	Left or right
Inchworm mode	6	4	0	14	10	4	X	X	X
Turning over mode	10	10	6	18	14	10	6	4	X
Turning around mode	6	4	4	18	14	6	6	4	X
Brachiation mode	X	X	X	X	X	X	X	X	X

^a X denotes that we did not experiment in this case. Failure rate is expressed with %.

Climbing upward is generally more difficult than climbing downward. When going downward, motor backlash and link deflection are compensated more easily, and vibrations are not so severe. In curve pipes, such as T-branches, our proposed algorithm is actually verified. If there is any mistake in the algorithm, the robot cannot grasp the arbitrary bended pipe any more. The common cause of the grasping failure is failure to obtain the projected line equation of the objective pipe edge. This is mainly because of the illumination change and confusing background lines. More generally, arbitrary-separated pipes and brachiation motion are not experimented here. The dynamic images of the experiments can be shown in YouTube. ³⁴ In this experiments, the motor backlash was roughly compensated using the information of the torque imposed on the motor joint, and the difference of the motor position command and the corresponding measured motor position.

Figure 15 shows the experiments of climbing up the vertical pipe at the hydraulic facility in our institute, which is a little bit similar to the spray-piping system in a nuclear power plant. The robot finally climbs the pipe up to 23 m high in 21.3 min with inchworm mode. The pipe is vertically linear without big obstacles such as valves and flanges. It was wirelessly controlled by the remote control station on the ground.

Figure 15.

Experiment of climbing at the hydraulic facility.

Summary

To inspect the reactor spray-piping system, we developed a very practical pipe-climbing robot. Our climbing robot has a five-DOF manipulator and two grippers and moves along cylindrical pipes like an inchworm, bypassing obstacles such as flanges and valves. The robot is able to work in very high places, grasping the pipes it is following semi-autonomously. The authors propose a vision-based scheme to allow a pipe-climbing robot to grasp a cylindrical pipe autonomously. This scheme could also be used in a practical pipe-climbing robot in real time. The configuration of the PiROB was described, including its hardware and software, and the forward and inverse kinematics of the mobile robot was derived. The robot control with visual grasping is also explained. The robot is practical for use in spray-pipe inspections, as well as for many other applications, such as the inspection of the roof frames of a stadium when they consist of pipes.

In a future study, a series of experiments will be performed, including pipe identification under the worst possible illumination conditions. In addition, error analysis based on our three assumptions and gain tuning of the visual feedback are required for better application of our scheme.