Sage Journals: Discover world-class research

Abstract

To solve the visual servoing tasks in complex environment, a path planning method based on improved rapidly exploring random trees algorithm is proposed. First, the improved rapidly exploring random trees planning method is adopted, which keeps the observed feature points in the field of view. The start node and the desired node are initialized as roots of multi-trees which grow harmoniously to plan path of the robot. Then, the planned path is used to project the three-dimensional target feature points into the image space and obtain the feature trajectory for the image-based visual servoing controller. Finally, the feature trajectory is tracked by the image-based visual servoing controller. The proposed visual servoing design method takes field of view constraints, camera retreat problem, and obstacle avoidance into consideration, which can significantly improve the ability of the robotic manipulator, especially in the narrow space. Simulation and experiment on 6-degree-of-freedom robot are conducted. The results present the effectiveness of the proposed algorithm.

Keywords

Path planning improved rapidly exploring random trees algorithm image-based visual servoing robotic manipulator narrow space

Introduction

The visual servoing control system refers to a control system using visual feedback. The control goal is to adjust the task function $e (s^{*} - s)$ to a minimum, where $s$ and $s^{*}$ are the current state and expected state of the system, respectively. Unlike a conventional control system, $s$ is based on the image information observed from the camera fixed to the robot’s work space (eye-to-hand) or mounted on the robot’s end-effector (eye-in-hand) in visual servoing system. This visual servoing system has higher dimension and greater information than using a traditional sensor, thus it can improve the flexibility of the robot system. In some visual system, the spatio-temporal context (STC) learning algorithm is introduced into a visual tracking issue.¹ According to the definition of system error, the current visual servoing can be divided into position-based visual servoing (PBVS), image-based visual servoing (IBVS), and hybrid visual servoing. The essential problem of the position-based visual servoing control² is the use of image features to estimate the current pose of the target. Therefore, the accuracy of the control system depends on the target pose estimation, which depends on the three-dimensional (3D) spatial model of the object. It is difficult to ensure accuracy in this system if the target is out of the visual field. In the IBVS, the error signal^3,4 is directly defined by the image feature, and the visual information feedback is directly used to change the image feature. The control quantity is calculated by the image characteristic error signal which turned into the robot motion space to drive the robot to the target position. This method does not need to estimate the position of the target in the Cartesian coordinate system. The camera and calibration error, as well as the target model error, has a strong robustness that overcomes the shortcomings of the PBVS.

In IBVS, there may be infeasible camera motion for the image trajectory. Moreover, there is no direct control over the image/camera/robot trajectories induced by the servoing loop in the image and in the physical space. Therefore, these trajectories might violate the image or physical constraints. Such approaches are ineffective in complex visual servoing tasks. Chaumette⁵ summarized the potential problems of IBVS method. Singularities in the image Jacobian matrix and local minima resulting from large displacements in the depth direction may cause instability and non-convergence. In Nadi and Derhami,⁶ artificial neural networks were used to estimate inverse Jacobian matrix. For the traditional IBVS system, due to the lack of depth information, the camera sometimes performs an unnecessary translation along the z-axis away from the target and back again. The resulting motion is not time optimal and can require large and unachievable camera motions, which may cause the visual task failure. This phenomenon is termed camera retreat. The spherical model can not only avoid the camera retreat problem⁷ but also facilitate the design of translation and rotation decoupling error vector.^8–12 To solve this issue and recognize the features, some system can use the depth-sensing technology combining depth and color information for foreground segmentation based on an advanced color-based algorithm.¹³ Relevant literature on partitioning and switching strategies shows that only a small fraction of the above constraints can be incorporated.¹⁴ Wang et al.¹⁵ presented a quasi-min–max model predictive control-based IBVS controller with tensor-product (TP) model transformation method. The control signals can be solved through numerical method, which avoid the singularities of the image Jacobian matrix and hence can solve the constraint problems for the classical IBVS controller. However, the existence of obstruction and other obstacles’ avoidance problem were not considered.

The path planning can program a feasible image feature trajectory between the initial and desired positions of the robot and guide it to move along the trajectory. In order to solve the above-mentioned problems of IBVS comprehensively and also for complex scenes with obstacles, it is a very effective method to incorporate global and general path planning into the control loop. The optimization of path planning in the robotic field has been widely studied and applied in visual servoing tasks, where the optimization indicators include the distance from the image boundary, the robot path length, and energy consumption. However, the IBVS principle is to continuously reduce the error of the current image and the desired image. The position obtained by path planning may not be consistent with the motion range of the camera, exceeding the joint limits of the robot or colliding with the obstacles. One of the ideas is to parameterize the trajectory and convert it into an optimization problem. The commonly used trajectory description methods are polynomials,¹⁶ Rodriguez formulas,¹⁷ and so on. The optimization objectives include the length of the trajectory, the curvature, and so on. The constraints include the field of view (FOV) constraints, robot joint angle limitations, and so on.¹⁸ When there is an error in the calibration parameters of the camera, the trajectory of the actual execution is different from the planned trajectory, and the constraint may not be satisfied.

Another idea is to use the search method to optimize. Based on prior knowledge, the boundary between the target visible region and the invisible area can be calculated, and then the search algorithm is used in the robot path planning to calculate a path that satisfies the constraint and project it into the image space. The commonly used methods are based on rapidly exploring random trees (RRT)¹⁹ and probabilistic maps.²⁰ The RRT is a single-query (SQ) algorithm in Wang et al.,²¹ which is less practical than the multi-query (MQ) method and is suitable for dynamic environments. In order to solve the problem of the movement path of a manipulator in the road, Biyun et al.²² proposed a modified double tree random tree. However, it also suffers from an excessive computational load in the narrow space. Yunfeng et al.¹⁹ and Wang et al.²³ proposed a multi-RRT method. The experiments show that this method has a good effect on solving the path planning under narrow space. The problem with some end-effector constraints has not been addressed in previous works. Based on the IBVS, an improved multi-RRT is introduced into the control. Unlike the traditional RRT, when a collision occurs in the multi-RRT, a new extension tree is generated and the path is planned in a multi-tree coordinated manner which smooths the trajectories. The advantages of the proposed algorithm are that both the camera FOV and “camera retreat”⁶ are considered in the path planning phase.

The article is organized as follows: in section “Preliminary concepts and system frame,” the preliminary concepts and system frame are explained. In section “Path planning algorithm for IBVS controller,” the multi-RRT algorithm is revealed. Section “Results” is dedicated to the simulation and experiment analysis of the results.

Preliminary concepts and system frame

Camera model

In computer vision, it is common to use the central perspective imaging model shown in Figure 1. The rays converge on the origin of the camera frame {C} and a non-inverted image is projected onto the image plane located at $z = f$ . Similarly, a 3D point $P$ at the camera frame can be shown with homogeneous coordinates [X, Y, Z, 1]^T. The intrinsic parameters of the camera are

K = [\begin{matrix} f p_{u} & 0 & u_{0} \\ 0 & f p_{v} & v_{0} \\ 0 & 0 & 1 \end{matrix}]

(1)

{\begin{matrix} u = u_{0} + f p_{u} \frac{X}{Z} \\ v = v_{0} + f p_{v} \frac{Y}{Z} \end{matrix}

(2)

where $f$ denotes the focal length, and $u_{0}$ and $v_{0}$ are the pixel coordinates of camera principal point. $p_{u}$ and $p_{v}$ are the pixel sizes in both u and v directions, respectively. So the corresponding pixel coordinates can be mapped to the camera imaging plane $s = [u, v, 1]^{T}$ by the following relationship.²⁴

Figure 1.

A perspective camera model.

Planning feasible camera trajectories

First, a feasible camera trajectory $Γ (k)$ is planned between the initial and desired camera vectors $x_{i}$ and $x_{d}$ , where $k$ is the sampling time. The definition of the camera planning vector $x (k)$ depends on the topology of the space in which the camera trajectories are planned

x (k) = {[\begin{matrix} T (k) & R (k) \end{matrix}]}^{T}

(3)

where $x$ is a seven-dimensional (7D) vector, $T (k) = [x_{c}, y_{c}, z_{c}]^{T}$ is the camera position vector at the sampling time k, and $R (k)$ denotes the rotational component of the camera which is expressed by the unit quaternion at the sampling time k.²⁵

The proposed planner explores the camera planning space for permissible trajectories by iteratively extending a search tree in this space. Simultaneously it tracks these trajectories in the robot configuration space.

As shown in Figure 2, both the image constraints and the camera work space are taken into consideration. A planning trajectory $Γ (k)$ obtained by path planning algorithm is then used to project a feature trajectory $s^{*} (k)$ in the image space, which is followed by the IBVS controller as desired features.

Figure 2.

System framework.

Path planning algorithm for IBVS controller

Compared with the traditional RRT algorithm, this article proposes a multi-RRT algorithm for visual servoing tasks. The planning trajectories can effectively guide the movements of the robot in the work space.

Basic principles of RRT algorithm

RRT is a fast search algorithm based on random sampling, which stores the collected random samples in a tree structure and continues to search for unknowns till the target is reached and the obstacles are known as a priori.^21,26 As shown in Figure 3, the search tree is first generated from the initial point $x_{start}$ and used as the root node of the search tree. Homogeneous and random sampling $x_{rand}$ is performed at a certain probability in the work space to guide the search tree extension. Finally, for the computational efficiency, only the nearest node of the existing tree is considered for the next extension step. If the node with the nearest vector $x_{near}$ in the tree is found, then $x_{near}$ is extended toward $x_{rand}$ . If a collision happened during the expansion process, this extension will stop and proceed onto the next random sample. It will repeat the process until it is connected to the target point $x_{goal}$ .

Figure 3.

RRT algorithm extension process.

In order to successfully implement the extension process, the path must be connected. Thus, path $c_{0}$ needs to be abandoned, which is non-connected due to conditional restrictions, such as joint limits or obstacles.²⁷ However, this traditional RRT method cannot meet the constraint requirements such as FOV and camera retreat in the IBVS system. Therefore, an improved multi-RRT approach is proposed.

Improved multi-RRT for camera path planning

The planned path will be served for the IBVS controller, as shown in Figure 2. Before the planning stage, the limit of the camera work space in Figure 4 (shown in point cloud) should be the input for the planning stage. The first three joints determine the position of the robot end. We choose step 20 for saving the memory resources.

Figure 4.

Camera work space.

In the proposed algorithm (see Table 1), the exploring tree $Γ$ is iterated over the camera work space until the desired position or set time is reached. The node with a random camera vector $x_{rand}$ is generated, and the node with the nearest vector $x_{near}$ to $x_{rand}$ is found in each interaction. Then, $x_{near}$ is extended toward $x_{rand}$ . The local path $Δ x$ is projected into the image space, and image constraints are checked through sampling synthetic images along the local path.²⁸

Table 1.

Proposed new multi-RRT algorithm.

1	Input: Camera initial and desired vectors $x_{i}$ and $x_{d}$ , Camera work space
2	Output: Camera trajectory $Γ$
3	do while path not found
4	$x_{rand}$ ←GenerateRandomVector()
5	for each tree
6	$x_{near}$ ←FindNearestNode $(x_{rand})$
7	$(x_{new}, Δ x)$ ←ExtendCheckConstrains $(x_{near}, x_{rand})$
8	CollisionCheck $(Δ x)$
9	if no collision
10	Add $x_{new}, Δ x$ to the tree
11	Else
12	Add new tree
13	end for
14	return $Γ$

In order to improve the computational efficiency, a simple but effective strategy is implemented. If a point fails to be connected to existing tree, the new trees are created when collisions occur in environments exhibiting high obstacle densities or when regions are isolated by narrow passageways. If search spaces have zero or few obstacles, a lower number of trees will be created during the search.

In the case of planning in the camera configuration space, a random camera configuration is built by generating a uniform random camera position $T_{rand}$ and a uniform random unit quaternion $R_{rand}$ in the camera work space. The random camera position is generated as

T_{r a n d} = [\begin{matrix} x_{c \min} + (x_{c \max} - x_{c \min}) r_{x} \\ y_{c \min} + (y_{c \max} - y_{c \min}) r_{y} \\ z_{c \min} + (z_{c \max} - z_{c \min}) r_{z} \end{matrix}]

(4)

where $r_{x}, r_{y}, r_{z}$ is the scalar value generated uniformly in the range [0, 1], and $x_{c min}, x_{c max}, y_{c min}, y_{c max}, z_{c min}, z_{c max}$ denote the limits of the camera works space along XYZ axes. A unit quaternion is generated uniformly as²⁸

R_{r a n d} = [\begin{matrix} \cos (2 π r_{y}) \sqrt{r_{z}} \\ \sin (2 π r_{x}) \sqrt{1 - r_{z}} \\ \cos (2 π r_{x}) \sqrt{1 - r_{z}} \\ \sin (2 π r_{y}) \sqrt{r_{z}} \end{matrix}]

(5)

Once a $x_{rand}$ is selected, the respective in each existing tree will be determined. And the tree node $x_{near}$ with the closest distance to $x_{rand}$ is found based on a metric $d (X_{1}, X_{2})$ , which is a measure of the relative positional relationship between two camera positions, that is, $X_{1}$ and $X_{2}$ in the camera planning space. The metric $d (X_{1}, X_{2})$ in camera configuration space is defined as

d (X_{1}, X_{2}) = \sqrt{{(x_{1} - x_{2})}^{2} + {(y_{1} - y_{2})}^{2} + {(z_{1} - z_{2})}^{2}}

(6)

where $(x_{i}, y_{i}, z_{i})$ is the position component of pose $X_{i} (i = 1, 2)$ , and the orientation component of pose is simply ignored in this metric. As a supplementary note, the $x_{near}$ can also be the node on the nearest edge of each tree. By comparing the distance of node and edge, the minimum value is determined.

Once the tree node with the nearest camera vector $x_{near}$ is found, the camera vector $x_{near}$ is extended toward $x_{rand}$ by applying the control input $u (k)$ and integrating equation (7) over a sampling time step. For the extension strategies in this thesis, a fixed-step Euler method for numerical integration is used with $x (k + 1) = x (k) + Δ tf (x (k), u (k))$ , where $Δ t$ is the integration time step

\dot{x} (k) = f (x (k), u (k)) = [\begin{matrix} \dot{T} (k) \\ \dot{R} (k) \end{matrix}] = [\begin{matrix} ν (k) \\ \frac{1}{2} ω (k) \cdot R (k) \end{matrix}]

(7)

where $ν (k)$ and $ω (k)$ are the linear and angular velocity control inputs, respectively; $ω (k) \cdot R (k)$ is the dot product between $[\begin{matrix} 0 & ω_{x} & ω_{y} & ω_{z} \end{matrix}]^{T}$ and $R (k)$ . The values of the linear and angular velocity are chosen randomly from their limits, and the directions of these two velocities are the same as the current state.

Then the “CollisionCheck” function returns if a collision occurs between an edge and a set of obstacles. To judging whether the point of intersection inside the obstacle uses “SameSide” technology which is usually used for checking a point in a mesh (triangle).²⁹ However, here we do the same for a four-point planner which generates the obstacles in the environment.

In this search phase, each new point can be connected to the existing trees and there is no need for any additional trees. The trees are merged when a connection contains two or more trees, which controls the total amount of trees not exceeding the specified value. Finally, the returned $Γ$ containing the discrete feature trajectories will be turned into the robot joint angle, which is the motion with the large rotation along the z-axis. In other words, the end-effector with large range of changes will be removed.

Tracking trajectories using IBVS

During the tracking process, the camera path (as edges of the camera tree) is determined as the camera FOV constraints, to ensure that the target features remain within the FOV and adjust to the planned path dynamically. For a 3D point with the image coordinates, $s (k) = [\begin{matrix} u & v \end{matrix}]^{T}$ . FOV limitations can simply be expressed by the following inequalities

{\begin{matrix} u_{\min} < u < u_{\max} \\ v_{\min} < v < v_{\max} \end{matrix}

(8)

where u_min, u_max, v_min, and v_max denote the FOV constraints.

Suppose that $[R_{0} (k), t_{0} (k)]$ is a given pose of the camera in the planned path $Γ (k)$ , the image feature $s (k) = [\begin{matrix} u & v \end{matrix}]^{T}$ of the feature point $P = [X, Y, Z, 1]^{T}$ with homogeneous coordinates in the object frame is computed as

Z {(u, v, 1)}^{T} = K [R_{o} (k), t_{o} (k)] P

(9)

where $Z$ denotes the distance between the camera and the feature point. All the feature points are turned into their pixel coordinate $s$ .³⁰ Then, the feature trajectories $s^{*} (k)$ are obtained.

In IBVS controller, the feature trajectories $s^{*} (k)$ will be followed as the desired features. The $s^{*} (k)$ and error signal are expressed in the image plane

e (k) = s (k) - s^{*} (k)

(10)

where $s (k)$ is the current image features. Taking the derivative of equation (10) with respect to time, we obtain

\dot{e} (k) = J (k) u (k) - {\dot{s}}^{*} (k)

(11)

where $J (k)$ denotes the image Jacobian matrix as

J (k) = [\begin{matrix} - \frac{k_{u}}{Z} & 0 & \frac{u}{Z} & \frac{u v}{k_{u}} & - (k_{u} + \frac{u^{2}}{k_{u}}) & v \\ 0 & - \frac{k_{v}}{Z} & \frac{v}{Z} & k_{v} + \frac{v^{2}}{k_{v}} & - \frac{u v}{k_{v}} & - u \end{matrix}]

where $k_{u} = f p_{u}$ and $k_{v} = f p_{v}$ are the amplification factors in u and v directions, respectively. Then the interaction matrix for n $(n \geq 3)$ feature points can be represented by simple stacking as the following expression

J (k) = {[J_{1}^{T} (k) … J_{n}^{T} (k)]}^{T}

(12)

Note that in the path plan period, the desired feature trajectories are discrete. So at time $k$ , the item ${\overset{\cdot}{s}}^{*} (k) = 0$ . The classical feedback control law is as follows

u (k) = J^{+} (- λ e (k) + {\dot{s}}^{*})

(13)

where $J^{+}$ is the pseudo-inverse of $J$ and $λ$ is a positive gain.

Results

In order to verify the performance of the proposed method, the framework has been both implemented and simulated by a 6-degree-of-freedom (DOF) ABB IRB120 manipulator equipped with an eye-in-hand camera and a robotic toolbox for MATLAB, as shown in Figure 5. Four target feature points are selected as the coordinates of the center of four co-planar markers on the target object.

Figure 5.

Simulation model and eye-in-hand system.

The intrinsic parameter of the camera is

k = [\begin{matrix} 952 & 0 & 290 \\ 0 & 952 & 240 \\ 0 & 0 & 1 \end{matrix}]

(14)

The image feature constraint is $0 < u < 640, 0 < v < 480$ . Because IBVS is a remarkably robust approach to vision-based control, it is very tolerant to errors in the depth of points. $Z = 0.5 m$ is chosen in our experiment.

Simulation and analysis

First, an environment with obstacles is generated. The goal is located behind the gray wall (on the same side of the wall with the feature points) and small windows exist on the wall. The planned trajectory must pass the wall to connect the start and goal points. The coordinates of the points are [−0.45 +0.15 +0.60] and [+0.25 −0.20 +0.55] in the robot coordinate frame. The camera work space [x_min, x_max; y_min, y_max; z_min, z_max] is equal to [−0.35, 0.35; −0.35, 0.35; 0, 0.63]. For the sake of clarity, only the end position of the manipulator is shown on the screen. The link mechanism is ignored.

The process of the multi-RRT extension is shown in Figure 6. In Figure 6(a), two trees explore the environment. When the “CollisionCheck” is true, a new tree is added. Then, five trees are extended in Figure 6(b). Finally, the planned path is shown in Figure 6(c). Besides, a path planned by traditional RRT is depicted in Figure 6(c) as well. Compared to the traditional RRT, the result of proposed multi-RRT algorithm is smoother.

Figure 6.

The multi-RRT extension: (a) two trees explore the environment, (b) five trees explore the environment, and (c) the final planned path.

As mentioned in section “Tracking trajectories using IBVS,” the simulation time and the planned path’s smoothness are related to $Δ t$ . The higher values of $Δ t$ can result in faster extension, but the trajectory is more curved, while the lower value will lead to the opposite result. As shown in Figure 7, the planned path is smoother from 2 to 0.5. Typically, lower $Δ t$ is recommended in complex environments. So $Δ t = 0.5$ is chosen in the following experiment.

Figure 7.

Planned path.

The initial pose of 6-DOF robot’s joint angle is from [−1.95 −0.24 0.05 −0.18 0.94 0.13] to desired [−1.58 0.15 −0.11 −0.53 1.07 1.35]. The results of feature points’ trajectories are shown in Figure 8, where the cross represents the initial position of the visual feature points and the circles are the desired features obtained at the goal position.

Figure 8.

Image feature trajectories: (a) result of the traditional IBVS and (b) result of the multi-RRT.

Because $Z$ is a constant, the resulting motion causes camera retreat and the feature points out of view as shown in Figure 8(a). In the proposed method, the planned path compensates this problem, for some motions with rotation along the z-axis are removed during the path planning period. The simulation results in Figure 8(b) show the proposed approach keeping the image features within the FOV effectively.

In the next section, experiments will be presented, which demonstrate the validity of the proposed scheme in image and camera work space constraints for the visual servoing task.

Experiments without obstacles

In the experiments, a square of 7 cm formed by four colored points is considered to be the target object. The initial joints of robot q₀ = [−1.35, 0.56, −0.89, −0.52, 1.78, 0.12] in radians. The centers of each point are selected as the image features, and the initial and desired image features are as shown in Figure 9.

Figure 9.

Initial and desired image features.

First, the traditional IBVS is implemented. As shown in Figure 10(a), camera retreat phenomenon causes the pose of joints 2 and 3 beyond their limits. Meanwhile, the image feature points are out of the FOV in Figure 10(b), and the visual servoing task is a failure.

Figure 10.

Experimental results with traditional IBVS: (a) robot pose and (b) image plane.

Then the proposed multi-RRT-based algorithm is implemented as shown in Figure 11. As the figures demonstrate, the proposed multi-RRT algorithm ensures that the feature points are within the FOV. Compared with Figure 10(a), the pose in Figure 11(a) avoids the singular position.

Figure 11.

Experimental results with the proposed algorithm: (a) robot pose and (b) image plane.

Experiments with obstacles

Some boxes are put in the manipulator work space which make the obstacles in the following experiment as shown in Figure 12, and in some regions the feature points are not visible in the FOV. The width of narrow channel is 0.21 m. The center of four target points is at [−0.1, −0.2, 0.3] in the robot-based coordinate frame. The initial pose of robot’s joints angle is q₀ = [−1.30, 0.13, −0.46, −0.66, 1.76, 1.95], and the desired pose of robot’s joints angle is q_d = [−1.69, 0.43, −0.45, −0.18, 1.34, 1.95].

Figure 12.

Simulation environment and planned path using multi-RRT.

In the previous experiment, the traditional IBVS technique is first used to carry out the task of servoing. The traditional IBVS method results in the collision between manipulator and obstacles which also causes some parts of points out of view in Figure 13. Finally, the task was a failure.

Figure 13.

Experimental results by traditional IBVS.

However, the proposed multi-RRT-based algorithm can take these constraints into consideration. As shown in Figure 14, this approach keeps the feature points in the FOV and completes the task without collision as planned.

Figure 14.

Experimental results with the proposed approach: (a) robot pose avoiding the obstacles and (b) image feature trajectories.

Conclusion

In this article, a new visual servoing design method based on improved multi-RRT algorithm is proposed, which can drive the manipulator from any given position within the working range to the position where the desired visual feature is located. The proposed improved multi-RRT method is introduced into the image visual servoing control which makes the trajectories smoother than traditional RRT algorithm, and the camera FOV is considered at the path planning phase. Compared with the traditional IBVS system, the approach proposed in this article can exhibit a better performance in the narrow space. In the future work, this method will be extended to an online IBVS controller for visual servoing containing joint limits’ avoidance and also experimental testing will be carried out.

Footnotes

Handling Editor: Zhaojie Ju

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Natural Science Foundation of China (nos. 61403122 and 51605137), Changzhou Sci&Tech Program (no. CJ20160013), and Fundamental Research Funds for the Central Universities (no. 2017B15114).

References

Zhou

Liu

Jiang

et al . Improving video segmentation by fusing depth cues and the visual background extractor (ViBe) algorithm. Sensors 2017; 17: 1177.

Yahya

Arshad

. Position-based visual servoing for underwater docking of an autonomous underwater vehicle. In: Proceedings of the IEEE international conference on underwater system technology: theory and applications (USYS), Penang, Malaysia, 13–14 December 2016, pp.121–126. New York: IEEE.

Hwang

K-S

Chung

M-H

Jiang

W-C

. Image based visual servoing using proportional controller with compensator. In: Proceedings of the IEEE international conference on systems, man, and cybernetics, Kowloon, China, 9–12 October 2015, pp.347–352. New York: IEEE.

Heshmati-Alamdari

Karavas

Eqtami

et al . Robustness analysis of model predictive control for constrained image-based visual servoing. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), Hong Kong, China, 31 May–7 June 2014, pp.4469–4474. New York: IEEE.

Chaumette

Potential problems of stability and convergence in image-based and position-based visual servoing. In: Kriegman

Hager

Morse

(eds) Confluence of vision and control. Berlin; Heidelberg: Springer, 1998, pp.66–78.

Nadi

Derhami

. Visual servoing control of robot manipulator with Jacobian matrix estimation. In: Proceedings of the international conference on robotics and mechatronics, Tehran, Iran, 15–17 October 2014, pp.405–409. New York: IEEE.

Corke

. Spherical image-based visual servo and structure estimation. In: Proceedings of the 2010 IEEE international conference on robotics and automation, Anchorage, AK, 3–7 May 2010, pp.5550−5555. New York: IEEE.

Wang

et al . A new image-based visual servoing method with rotational compensation. In: Proceedings of the IEEE international conference on robotics and biomimetics (ROBIO), Qingdao, China, 3–7 December 2016, pp.1099−1104. New York: IEEE.

Wang

et al . Partially decoupled image-based visual servoing using different sensitive features. IEEE T Syst Man Cy: S 2017; 47: 2233−2243.

10.

Wan

et al . Decoupled control for visual servoing with SVM-based virtual moments. In: Proceedings of the IEEE international conference on information and automation, Lijiang, China, 8–10 August 2015, pp.2121–2126. New York: IEEE.

11.

Ding

. Second-order sliding mode controller design subject to mismatched term. Automatica 2017; 77: 388–392.

12.

Ding

Levant

. Simple homogeneous sliding-mode controller. Automatica 2016; 67: 22–32.

13.

Zhou

Liu

Yang

et al . Multi-channel features spatio-temporal context learning for visual tracking. IEEE Access 2017; 5: 12856–12864.

14.

Kazemi

Gupta

Mehrandezh

. Path planning for visual servoing: a review and issues. In: Chesi

Hashimoto

(eds) Visual servoing via advanced numerical methods. Berlin: Springer, 2010, pp.189–208.

15.

Wang

Xie

Liu

et al . Quasi-min-max model predictive control for image-based visual servoing with tensor product model transformation. Asian J Control 2015; 17: 402–416.

16.

Srinivasan

Chakraborty

. Path planning with user route preference—a reward surface approximation approach using orthogonal Legendre polynomials. In: Proceedings of the IEEE international conference on automation science and engineering (CASE), Fort Worth, TX, 21–25 August 2016, pp.1100−1105. New York: IEEE.

17.

Chesi

. Visual servoing path planning via homogeneous forms and LMI optimizations. IEEE T Robot 2009; 25: 281−291.

18.

Chesi

Shen

. Conferring robustness to path-planning for image-based control. IEEE T Contr Syst T 2012; 20: 950–959.

19.

Yunfeng

Zhenshu

Huagang

. Multi-RRT path planner for EOD manipulators in narrow passage. J Comp Measure Control 2015; 23: 3123–3126.

20.

Padgett

Campbell

. Probabilistic qualitative mapping for robots. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), Stockholm, 16–21 May 2016, pp.5388–5394. New York: IEEE.

21.

Wang

et al . An adaptive roadmap guided multi-RRTs strategy for single query path planning. In: Proceedings of the IEEE international conference on robotics and automation, Anchorage, AK, 3–7 May 2010, pp.2871–2876. New York: IEEE.

22.

Biyun

Jing

. Motion planning of 7R manipulator based on rapidly expanding random tree. J Mech Eng 2012; 48: 63–69.

23.

Wang

et al . Triple RRTs: an effective method for path planning in narrow passage. J Adv Robot 2010; 24: 943–962.

24.

Chaumette

Hutchinsh

. Visual servo control, part I: basic approaches. IEEE T Robotic Autom 2006; 13: 82–90.

25.

Kazemi

Gupta

Mehrandezh

. Kinodynamic planning for visual servoing. In: Proceedings of the IEEE international conference on robotics and automation, Shanghai, China, 9–13 May 2011, pp.2478–2484. New York: IEEE.

26.

Shuang

Weiwei

Kun

. Bidirectional RRT algorithm based grasping manipulation of humanoid robot. J Univ Sci Technol China 2016; 23: 13–20.

27.

Hammouda

Mekki

Kaaniche

et al . Image-based visual servoing dealing with constraints. In: Proceedings of the international image processing, applications and systems (IPAS), Hammamet, Tunisia, 5–7 November 2016, pp.1–6. New York: IEEE.

28.

Arslan

Tsiotras

. Use of relaxation methods in sampling-based algorithms for optimal motion planning. In: Proceedings of the IEEE international conference on robotics and automation (ICRA), Karlsruhe, 6–10 May 2013. New York: IEEE.

29.

Ericson

. Point in triangle test [DB/OL], http://blackpawn.com/texts/pointinpoly/default.html

30.

Kazemi

Gupta

Mehrandezh

. Randomized kinodynamic planning for robust visual servoing. IEEE T Robot 2013; 29: 1197–1211.

Path planning for visual servoing with search algorithm

Abstract

Keywords

Introduction

Preliminary concepts and system frame

Camera model

Planning feasible camera trajectories

Path planning algorithm for IBVS controller

Basic principles of RRT algorithm

Improved multi-RRT for camera path planning

Tracking trajectories using IBVS

Results

Simulation and analysis

Experiments without obstacles

Experiments with obstacles

Conclusion

Footnotes

Declaration of conflicting interests

Funding

References