A geometric approach for homography-based visual servo control of underactuated UAVs

Abstract

This paper proposes a new geometric control method for homography-based visual servo control of Underactuated UAVs. In order to solve the application difficulties of geometric control in HBVS and explore a visual servo control technology that can be applied to aerial detection operations, this paper integrates the geometric control into the visual servoing framework and design a new homography-based geometric visual servoing controller. The outer loop is used as feedback information using the virtual homography matrix between the two images. The inner loop controls the orientation of the UAVs through geometric control. The stability of the proposed controller is proved based on Lyapunov’s theory. The proposed method has better transient performance and dynamic performance than the conventional visual servo method. The excellent performance of the controller has been proven by a large number of experiments. In addition, the application of the controller on an unmanned aerial manipulator is demonstrated.

Keywords

Visual servoing homography geometric controller underactuated UAVs

Introduction

Unmanned aerial vehicles (UAVs) are widely used in both civilian and military applications because UAVs can perform some dangerous tasks instead of humans.¹ In many applications, a degree of autonomy for the UAV is a guarantee of mission accomplishment. Normally, UAVs can accomplish most of their missions through GPS and IMU,² but relying solely on these sensors cannot obtain position information relative to local targets, making it impossible to complete tasks that require local information, such as inspection and manipulation. UAVs can perform many tasks through visual servo control, such as target tracking,^3,4 target search,⁵ autonomous landing,⁶ etc.

In the current study, most researchers tend to use the image-based visual servo (IBVS) to design controllers.^7,8 IBVS uses only image information to extract feature points for feedback control, which is not very demanding on the camera and easy to implement in practice. In,⁹ the visual dynamic equation of point features is constructed under the spherical projection model, and the nonlinear controller is constructed and designed by using the backstepping method. A controller design method based on circular features is proposed to estimate the relative position and attitude of the target by analyzing the model of spherical features for control design.¹⁰ Dynamical with point features is considered and subsequent controller design is performed.¹¹ In the above method, points, lines and other features in the image are used in the design and only part of the information in the image is used, which requires that the features should always remain in the field of view during the motion.

There is another type of visual servo control that uses the Homography matrix as the image feature for controller design, called Homography-based visual serving (HBVS).^12,13 When there is an overlap between two images, the Homography matrix can be calculated in several ways. The homography matrix contains parameters such as rotation matrix, displacement, and normal vector, and decoupling these geometric parameters is difficult. Currently, HBVS is often used in industrial manipulator or full actuated system. A new homography-based visual tracking method is proposed and global stability was demonstrated.¹⁴ The underwater robot applied only vision sensors, calculated the homology matrix and used the homography matrix as a feedback feature.¹⁵ Motivated by the Backstepping method and Lyapunov theory, an adaptive controller of unknown depth was constructed using a homography matrix.¹⁶ Using the homography matrix to control underactuated UAVs is a challenging task. Author defined a new virtual homography matrix based on the backstepping method to design the controller,¹⁷ but the yaw angle control was independent of the position in the control design, which meant that the yaw angle needed to be stabilized quickly before the whole could be stabilized. Huang et al. proposed a servo control method based on homography for tracking a six degree of freedom moving ship using a rotary wing drone. The feature points of the moving target were measured using a monocular camera and the projection mapping transformation between the target and the drone was calculated. Next, based on these transformations, the tracking error is converted into pose error, and a controller is designed to control the rotor drone.¹⁸ Zheng et al. proposed an image-based dynamic visual servo control method for the positioning and control of unmanned aerial vehicles. By solving the homography matrix to reconstruct the pose and integrating it into the controller design, it has good real-time performance and robustness, and can be applied to unmanned aerial vehicle control applications in complex environments. This method usually requires operations such as singular value decomposition of the homography matrix to calculate the position and attitude, which increases the computational workload of the onboard processor.⁸ In recent years, a new type of controller has emerged for use in underactuated UAVs, called the geometry controller. The application of geometric control in UAVs attitude control has good effects, as it can overcome modeling errors and external disturbances, improve the stability and robustness of the control system, and have fast response control capabilities. It was first proposed for the problem of tracking the attitude of a fully actuated rigid body at a point.¹⁹ Later, the geometric controller continued to be developed and was used in underactuated UAVs,²⁰ using position information as a feedback feature and proposing a control method with global stability. A geometric L1 adaptation method to estimate uncertainty was demonstrated and enabled better robustness of UAVs.²¹ The above is mainly position-based servo control, but since homography-based dynamics is different from position-based dynamics, extending the geometric controller to HBVS is a challenging task. For example, in homography-based visual servo control, it is a challenge to associate visual features and homography estimation results with the controller, and design effective controllers to accurately respond to changes in homography matrix and stably control attitude. However, by extending geometric control to visual servo control based on homography matrix, we can not only leverage the advantages of geometric control in attitude control in HBVS, but also expand the field of visual servo.

Influence of unmodeled dynamic factors

In Ref.,²² the author designed a hybrid framework of opportunity reinforcement learning and deep learning to avoid the impact of frequent changes in the autopilot system, and replaced traditional electronic flight controllers with 3D gestures. This method is different from most methods and has strong novelty. The author proposes a linear extended observer and a sliding mode control method based on dual loop one-dimensional fuzzy control to ensure stability of the quadcopter unmanned aerial vehicle, which belongs to nonlinear and underactuated systems and introduces dynamic factors and uncertainties for modeling. This method uses an observer to ensure the stability of the system.²³ In Ref.,²⁴ the author proposes an adaptive robust control algorithm to address the issues of instability and low convergence rate caused by unmodeled dynamic factors in quadcopter unmanned aerial vehicles. The author did not use an observer, but it can also ensure the stability of the system. In Ref.,²⁵ the author proposes a dual loop one-dimensional fuzzy sliding mode control (DLSDF-SMC) to improve the performance of spiral trajectory tracking, and proposes a state feedback controller composed of a position estimator and a disturbance observer design for unmodeled dynamic factors. The author proposed various observer and control design methods, evaluated the performance of underactuated quadcopter unmanned aerial vehicle dynamics models based on Newton Euler equations and quaternions, and verified that using quaternion mathematical models can be applicable to various observer designs and adapt to unmodeled dynamic factors.²⁶ In Ref.,²⁷ the author proposes a model-free one-dimensional fuzzy sliding mode control (MFSDF-SMC) to control the attitude and position of underactuated quadcopter drones, addressing the problem of difficult control caused by unmodeled dynamic factors and uncertainties. In Ref.,²⁸ the author extensively reviewed several hybrid observer design methods and new control strategies to address the difficulty of addressing unmodeled dynamic factors in various flight modes of underactuated quadcopter aircraft. In Ref.,²⁹ the author described the hybrid control design of underactuated four rotor systems without modeling dynamic factors. The author conducted performance analysis on fuzzy based sliding mode control and one-dimensional fuzzy sliding mode control methods in underactuated four rotor unmanned aerial vehicles, and designed a single input fuzzy sliding mode control method.³⁰

From the above literature, it can be seen that underactuated multirotor UAVs have the influence of unmodeled dynamic factors on control stability. Therefore, many researchers have designed algorithms or observers based on fuzzy sliding mode control to address the stability challenges brought by unmodeled dynamic factors. Some researchers have solved the problems caused by unmodeled dynamic factors by designing adaptive control methods, robust control methods, and reinforcement learning. In addition, inversion control can also solve the problems caused by unmodeled dynamic factors. In this paper, to address this issue, this paper introduces backstepping control method to enhance the robustness and adaptability of underactuated multi rotor UAVs to unmodeled dynamic factors, without using methods such as designing observers. Specifically, this paper defines a new velocity error based on inversion control technology to design visual dynamics equations to address the impact of unmodeled dynamics factors, as detailed in the following text.

Compared with the existing HBVS solutions, our contributions of this paper are summarized as follows: (1) The proposed method establishes the visual dynamics equations based on the homography matrix between two images. Different from the conventional IBVS,^9–11 the homography-based architecture allows the UAV to work with a small field of view. (2) A new geometric control method for homography-based visual servo control of Underactuated UAVs is proposed. Compared to other controllers,^15,16 geometric controller has better results for UAV attitude control. In addition, geometric control method can overcome modeling errors and external disturbances, improve the stability and robustness of the control system, and have fast response control capabilities. Therefore, extending geometric control to visual servo control based on homography matrix not only ensures the stability and accuracy of posture in HBVS, but also extends geometric control in the field of visual servo control based on homography matrix. A stability analysis of the closed-loop system is performed and proved using Lyapunov. The final experimental results also demonstrate the excellent dynamic performance of the proposed method. (3) To demonstrate the broad applicability and robustness of the proposed method, this paper applied it to the unmanned aerial manipulator.

The paper is organized into six sections. The notation and the dynamics of UAV are mentioned in Section II. A definition of virtual homography matrix and problem formulation is given in Section III. A new geometric approach for homography-based visual servo Control of underactuated UAVs is present in Section IV. This paper shows the experimental results in Section V and draw conclusions in Section VI.

Preliminary background

Notation

Let us consider the UAV equipped with the vision suite. Assuming that the camera frame overlaps with the UAV frame,¹¹ it can usually be solved by using an external method to obtain the transformation between the body frame and the camera frame. The following notation are used throughout the article:

Let $m$ , $J = diag (J_{x}, J_{y}, J_{z}) \in R^{3 \times 3}$ denote the mass of the total UAV and the inertia matrix around the center of mass, respectively. The origin of the body frame is located at the center of mass of this vehicle. $I \in R^{3 \times 3}$ denote $3 \times 3$ identity matrix.

As shown in Figure 1, let $A = {O; {\vec{e}}_{1}^{a}, {\vec{e}}_{2}^{a}, {\vec{e}}_{3}^{a}}$ denote the reference frame (the final desired pose), let $C = {O; {\vec{e}}_{1}^{c}, {\vec{e}}_{2}^{c}, {\vec{e}}_{3}^{c}}$ denote the body frame, let $V = {O; {\vec{e}}_{1}^{v}, {\vec{e}}_{2}^{v}, {\vec{e}}_{3}^{v}}$ denote the virtual frame, whose pitch and roll angle is zero with respect to the reference frame.

The Euler angles relative to the reference frame are expressed as $η = [φ, θ, ψ]^{T} \in R^{3}$ . The rotation matrix $R \in SO (3)$ represents the orientation of the body frame $C$ relative to the reference frame $A$ , where $R = R_{ψ} R_{θ} R_{φ}$ , with $R_{ψ}, R_{θ}, R_{φ} \in SO (3)$ represent yaw, pitch, roll rotation matrix, respectively.

The operator $sk (\cdot) : R^{3} \to SO (3)$ is defined by the condition that $sk (x) y = x \times y$ . The operator $vex (\cdot) : SO (3) \to R^{3}$ denotes the inverse of $sk (\cdot)$ .

Figure 1.

Geometric relationship of each frame.

Dynamics of UAVs

UAV is assumed as a symmetric quadrotor model. Based on the previous literature,³¹ the dynamics are

\overset{\cdot}{x} = RV

(1)

\overset{\cdot}{R} = R sk (Ω)

(2)

m \overset{\cdot}{V} + msk (Ω) V = F + F_{ext}

(3)

F = - T E_{z} + mg R^{T} e_{z}

(4)

J \overset{\cdot}{Ω} + sk (Ω) J Ω = M + M_{ext}

(5)

where $x = [x_{1}, x_{2}, x_{3}]^{T} \in R^{3}$ denotes Euclidean coordinates in the reference frame $A$ . $Ω \in R^{3}$ is the angular velocity and $V \in R^{3}$ is the linear velocity both expressed in $C$ . $M = [M_{1}, M_{2}, M_{3}]^{T} \in R^{3}$ denotes the control torque expressed in the body frame $C$ . The force $F \in R^{3}$ mainly consists of gravity components and thrust. $M_{ext}$ and $F_{ext}$ denote the moments and forces acting on the UAV due to external disturbance terms. $T \in R$ denotes the thrust force which is applied in the direction of the third axis ${\vec{e}}_{3}^{c}$ of the body frame $C$ . The vector $E_{z} = e_{z} = [0, 0, 1]^{T} \in R^{3}$ are unit vectors in the body frame and reference frame, respectively.

Visual dynamics

Image features and dynamics

The control objective is to stabilize the UAV’s pose to the desired pose based on the current image and the reference image. This paper assumes that there is a good homography matrix estimate for the control design. This paper selects the homography matrix $H$ between the body frame $C$ and the reference frame $A$ as the visual feature of visual servo control.

H = R^{T} - \frac{1}{z^{*}} R^{T} x {n^{*}}^{T}

(6)

where $n^{*} \in R^{3}$ denotes the unit normal. $z^{*}$ denotes the distance from the target plane to the UAV’s optical center. The time derivatives of $H$ is derived from the following equation.

\overset{\cdot}{H} = - sk (Ω) H - \frac{1}{z^{*}} V {n^{*}}^{T}

(7)

The pitch angle $θ$ and roll angle $φ$ can be obtained through the onboard Sensors(Inertial Measurement Unit). As shown in Figure 1, the virtual camera³² has the same optical center and the same intrinsic parameters as the body camera. Also, the virtual frame eliminates the dependence on roll and pitch rates. This paper defines a virtual homography matrix $H_{v}$ which represents isomorphism between the reference frame and the virtual frame. The virtual homography matrix $H_{v} = R_{φ θ} H$ is obtained as

H_{v} = R_{ψ}^{T} - \frac{1}{z^{*}} R_{ψ}^{T} x {n^{*}}^{T}

(8)

This paper solves for the derivative of equation (8) as follows

{\overset{\cdot}{H}}_{v} = - sk (\overset{\cdot}{ψ} e_{z}) H_{v} - \frac{1}{z^{*}} R_{φ θ} V {n^{*}}^{T}

(9)

where $R_{φ θ} = R_{θ} R_{φ}$ represents Rotation matrix between the body frame $C$ and the virtual frame $V$ .

Under assumption 1, it can be known $n^{*} = [0, 0, 1]^{T}$ . The yaw rotation can be constructed based on the virtual homography matrix $H_{v}$ , and the relative yaw angle $ψ$ is constructed as $ψ = \arcsin (\frac{1}{2} vex {(H_{v}^{T} - H_{v})}^{T} e_{z})$ .

Assumption 1. The third axis of the reference frame $A$ is parallel to the normal vector of the target $n^{*}$ .

Proof. Considering the virtual homography matrix (8) and $n^{*} = [0, 0, 1]^{T}$ , the formula can be further written as

H_{v} = [\begin{matrix} \cos ψ & \sin ψ & - t_{x} \\ - \sin ψ & \cos ψ & - t_{y} \\ 0 & 0 & 1 - t_{z} \end{matrix}]

(10)

where $t = \frac{1}{z^{*}} R_{ψ}^{T} = [t_{x}, t_{y}, t_{z}]^{T}$ .

vex (H_{v}^{T} - H_{v}) = [- t_{y}, t_{x}, 2 \sin ψ]^{T}

(11)

The yaw angle can be calculated by equation (11).

Problem formulation

According to the reference image in desired pose (the reference frame) and the current image captured using a down-looking monocular camera, the control goal is to stabilize the body pose of the UAV at the desired pose. Also, the control goal is to ensure that $H_{v} = I$ . An important property of the virtual homography matrix is that it is independent of roll and pitch angle. According to previous work,¹⁴ the error $e_{1}$ is designed as

e_{1} = (I - H_{v}) m^{*}

(12)

where $m^{*} \in S^{2}$ is a unit vector and is chosen under the circumstance that ${n^{*}}^{T} m^{*} > 0$ . Combining (8) and (9), the time derivative of (12) is

{\overset{\cdot}{e}}_{1} = - sk (\overset{\cdot}{ψ} e_{z}) (e_{1} - m^{*}) + a^{*} V_{v}

(13)

where $a^{*} = \frac{{n^{*}}^{T} m^{*}}{z^{*}} > 0$ . $V_{v} = R_{φ θ} V$ denotes the virtual linear velocity. Let us choose $m^{*} = [0, 0, 1]^{T}$ to bring into the equation (13). In this way, $sk (\overset{\cdot}{ψ} e_{z}) m^{*} = 0$ , then the formula becomes concise, which is beneficial to the subsequent controller design. Designing control modules without considering the influence of external factors $Γ_{ext} = 0$ and $F_{ext} = 0$ . Then, the dynamics based on the image features can be obtained

{\overset{\cdot}{e}}_{1} = - sk (\overset{\cdot}{ψ} e_{z}) e_{1} + a^{*} V_{v}

(14)

{\overset{\cdot}{V}}_{v} = - sk (\overset{\cdot}{ψ} e_{z}) V_{v} + \frac{F_{v}}{m}

(15)

where $F_{v} = R_{φ θ} F = - T R_{φ θ} E_{z} + mg R_{ψ}^{T} e_{z}$ .

From (2), (4), (5), (14), (15), the control objective is to stabilize $e_{1} \to 0_{3 \times 1}$ by designing the thrust $T$ and the control torque $M$ .

HBVS control design

The structure of HBVS scheme is shown in Figure 2. The image-based outer loop generates the control thrust and the desired attitude, acts on the geometric attitude controller. The geometric attitude controller calculates the required torque.

Figure 2.

HBVS control structure.

Outer loop control

According to the backstepping technique, this paper defines a new velocity error $e_{2} = \frac{1}{c} V_{v} + e_{1} \in R^{3}$ . The visual dynamic equation can be written as

{\overset{\cdot}{e}}_{1} = - sk (\overset{\cdot}{ψ} e_{z}) e_{1} + a^{*} c (e_{2} - e_{1})

(16)

{\overset{\cdot}{e}}_{2} = - sk (\overset{\cdot}{ψ} e_{z}) e_{2} + \frac{F_{v}}{mc} + a^{*} c (e_{2} - e_{1})

(17)

The following control law guarantees the visual errors $e_{1}$ and $e_{2}$ converging to zero. Considering $F_{v}$ as control input, its design is

F_{v} = - K_{v} e_{2}

(18)

where $K_{v}$ is a constant diagonal positive definite matrix.

Proof. Considering the candidate Lyapunov function $L_{1}$ :

L_{1} = \frac{1}{2} e_{1}^{T} e_{1} + \frac{1}{2} e_{2}^{T} e_{2}

(19)

By using (16), (17), the time derivative of (19) is obtained as

{\overset{\cdot}{L}}_{1} = - a^{*} c e_{1}^{2} - (\frac{λ_{\min} (K_{v})}{mc} - a^{*} c) e_{2}^{2}

(20)

When the parameters $c$ , $K_{v}$ satisfy the following conditions (21), ${\overset{\cdot}{L}}_{1}$ is equal to zero only at equilibrium and less than zero in other cases.

{\begin{matrix} a^{*} c > 0 \\ \frac{λ_{\min} (K_{v})}{mc} - a^{*} c > 0 \end{matrix}

(21)

Accordingly, the visual errors $e_{1}$ , $e_{2}$ are stable and converge to zero.

Then, according to the defined virtual control input (18) and model equation (4), the thrust $T$ is obtained as

T = - {[F_{v} - mg e_{z}]}^{T} (R_{θ φ} e_{z})

(22)

Geometric attitude controller

Given that the rotation matrix is continuously changing in the Lie group $SO (3)$ , then, to make the error converge faster, this paper designs a geometric tracking controller to track the desired attitude $R_{d} = [b_{1 d}, b_{2 d}, b_{3 d}] \in SO (3)$ . $R_{d}$ is constructed by the $F_{v}$ . In order to get a smooth tracking process, this paper defines

b_{3 d} = \frac{ξ}{‖ ξ ‖}

(23)

where $ξ = R_{ψ} (- \frac{F_{v}}{mc} + g e_{z})$ .

Based on practical experience and to avoid the attitude singularity problem, this paper assumes $‖ R_{ψ} (- \frac{F_{v}}{mc} + g e_{z}) ‖ \neq 0$ . Given a specific yaw $ψ_{c}$ , $b_{1 c} = {[\cos (ψ_{c}) \sin (ψ_{c}) 0]}^{T} \in R^{3}$ can be constructed and is assumed not parallel to $b_{3 d}$ . According to geometric relationship, it can be known that

b_{2 d} = \frac{b_{3 d} \times b_{1 c}}{‖ b_{3 d} \times b_{1 c} ‖}

(24)

b_{1 d} = b_{2 d} \times b_{3 d}

(25)

Then, the desired attitude $R_{d}$ is generated. The geometric attitude controller is designed to ensure that the attitude of the UAV quickly reaches the desired attitude. Inspired by Lee et al.,²⁰ the function $ϒ (R, R_{d})$ denotes the error of the current attitude relative to the desired attitude, which is defined as follows

ϒ (R, R_{d}) = \frac{1}{2} tr [I - R_{d}^{T} R]

(26)

The time derivatives of $ϒ (R, R_{d})$ are calculated as

\overset{\cdot}{ϒ} (R, R_{d}) = e_{R}^{T} e_{Ω}

(27)

According to the proposed function $ϒ (R, R_{d})$ , the attitude error $e_{R} = \frac{1}{2} vex (R_{d}^{T} R - R^{T} R_{d})$ and the angular velocity error $e_{Ω} = Ω - R^{T} R_{d} Ω_{d}$ . The time derivatives of $e_{R}$ and $e_{Ω}$ are as follows respectively

{\overset{\cdot}{e}}_{R} = B (R_{d}, R) e_{Ω}

(28)

{\overset{\cdot}{e}}_{Ω} = \overset{\cdot}{Ω} + sk (Ω) R^{T} R_{d} Ω_{d} - R^{T} R_{d} {\overset{\cdot}{Ω}}_{d}

(29)

where $B (R_{d}, R) = \frac{1}{2} (tr [R^{T} R_{d}] I - R^{T} R_{d})$ is a function about $R, R_{d}$ . The $Ω_{d}$ is constructed as $Ω_{d} = vex (R_{d}^{T} {\overset{\cdot}{R}}_{d})$ . ${\overset{\cdot}{R}}_{d} = [{\overset{\cdot}{b}}_{1 d}, {\overset{\cdot}{b}}_{2 d}, {\overset{\cdot}{b}}_{3 d}]$ denotes the derivative of $R_{d}$ , which is calculated as

{\overset{\cdot}{b}}_{3 d} = b_{3 d} \times \frac{\overset{\cdot}{ξ}}{‖ ξ ‖} \times b_{3 d}

(30)

{\overset{\cdot}{b}}_{2 d} = b_{2 d} \times \frac{{\overset{\cdot}{b}}_{3 d} \times b_{1 c} + b_{3 d} \times {\overset{\cdot}{b}}_{1 c}}{‖ b_{3 d} \times b_{1 c} ‖} \times b_{2 d}

(31)

{\overset{\cdot}{b}}_{1 d} = {\overset{\cdot}{b}}_{2 d} \times b_{3 d} + b_{2 d} \times {\overset{\cdot}{b}}_{3 d}

(32)

Considering the dynamic (5), (28) and (29), this paper designs a control input $M$ for a given attitude command $R_{d}$ as follows

M = - K_{R} e_{R} - K_{Ω} e_{Ω} + sk (Ω) J Ω + Δ

(33)

where $Δ = - J (sk (Ω) R^{T} R_{d} Ω_{d} - R^{T} R_{d} {\overset{\cdot}{Ω}}_{d})$ . $K_{R} \in R^{3 \times 3}$ , $K_{Ω} \in R^{3 \times 3}$ is a constant positive definite gain matrix. Under the control of this input torque, the attitude error $e_{R}$ , $e_{Ω}$ are converging to zero.

Proof. Considering the candidate Lyapunov function $L_{2}$ :

L_{2} = \frac{1}{2} e_{Ω}^{T} J e_{Ω} + K_{R} ϒ (R, R_{d})

(34)

By using (5), (27), (29), the derivative of (34) is obtained as

\begin{matrix} {\overset{\cdot}{L}}_{2} = e_{Ω}^{T} J {\overset{\cdot}{e}}_{Ω} - K_{R} \overset{\cdot}{ϒ} (R, R_{d}) \\ = - K_{R} e_{Ω}^{T} e_{R} - K_{Ω} e_{Ω}^{2} + K_{R} e_{R}^{T} e_{Ω} \\ = - K_{Ω} e_{Ω}^{2} \end{matrix}

(35)

From equation (35), it can be known that ${\overset{\cdot}{L}}_{2} \leq 0$ . The errors $e_{R}$ and $e_{Ω}$ are stable.

Experimental results

The proposed control method has been tested in kind with a quadrotor shown in Figure 3. The quadrotor we used was developed based on an existing quadrotor model (P-600) with a wheelbase of 600 mm. The total mass of the quadrotor is 3.3 kg. The quadrotor is equipped with a camera and an onboard computer. The camera is a monocular grayscale camera that provides image data in 640 × 480 size and is captured at 60 Hz. The size of this camera is 3 × 3 × 2 cm. The PIXHAWK autopilot is mounted on the quadrotor and obtains pitch, roll and angular velocity from the built-in inertial measurement unit (IMU) and exchanges data with the onboard computer via UART. An external motion capture system is used to obtain the true speed of the quadrotor. The brand of this external motion capture system is Optitrack, with a total of 12 cameras and a space size of 9 × 6 × 4 m. All code is programed in C++ and runs on a quadrotor onboard computer. The onboard computer runs the Robot Operating System (ROS) and sends control commands to the PIXHAWK autopilot. The outer loop control frequency depends on the image acquisition frequency and is 60 Hz. The frequency of the geometric attitude controller is set to 250 Hz. The diagonal axis distance of the quadrotor is 60 cm and other physical parameters are as follows: $m = 2.48 kg$ , $J = diag (7.56, 7.89, 12.77) \times 10^{2} kg \cdot m^{2}$ . The physical parameters are given in Table 1.

Figure 3.

Experiment platform.

Table 1.

Control gains and parameters.

Controller	Parameters
Outer loop control	$n^{*} = [0, 0, 1]^{T}$
	$z^{*} = 1.0$
	$g = 9.8091$
	$c = 0.7$
	$K_{v} = diag ([6.35, 6.35, 3.56])$
Attitude control	$K_{R} = diag ([0.365, 0.365, 0.325])$
Attitude control	$K_{Ω} = diag ([0.068, 0.068, 0.035])$

Performance testing

This experiment aims to demonstrate the superior performance of the above algorithm on a real UAV. The purpose of the experiment is to stabilize the UAV from the current position to the reference position. This paper sets up two common initial states, one with errors in position and the other with errors in attitude angle, and 15 experiments were conducted for both cases. Initial states are as follows: (1) $x_{1} = {(- 0.35, - 0.05, 1.25)}^{T} m$ , $η_{1} = {(0, 0, 0)}^{T} \deg$ , (2) $x_{2} = {(0, 0, 1.43)}^{T} m$ , $η_{2} = {(0, 0, 21)}^{T} \deg$ , respectively. For the convenience of description, the two experiments are denoted as exp1, exp2 respectively. The desired position is $x_{d} = {(0, 0, 1)}^{T} m$ . The desired yaw angle $ψ_{c}$ is set to zero, $η_{d} = {(0, 0, 0)}^{T} \deg$ .

The whole experiment lasted for 10 s, and the results of the experiment are shown in Figures 4 to 7.

Remark 1. Manually fly the UAV to the desired location $(x_{d} = {(0, 0, 1)}^{T} m)$ and take a picture at the current frame. The current frame is the reference frame and the picture is the reference picture. To be able to compute the homography matrix of the current image and the reference image faster, this paper uses a checkerboard to speed up the extraction of feature points for matching computation. Homography is calculated faster and matches more accurately from this approach than using other features, such as ORB or SIFT features. This paper calculates the Homography matrix using openCV’s “findHomography” function.

Figure 4.

Experimental results of position and attitude of exp1 and exp2: (a) position of exp(1), (b) attitude of exp(1), (c) trajectory of exp(1), (d) position of exp(2), (e) attitude of exp(2), and (f) trajectory of exp(2).

Figure 5.

Attitude error function $ϒ (R, R_{d})$ : (a) exp1 and (b) exp2.

Figure 6.

Experimental results with large position offsets (exp1): (a and b) visual error $e_{1}, e_{2}$ , (c and d) Attitude tracking error $e_{R}, e_{Ω}$ , (e and f) command torque $M$ and thrust $T$ . (a) $e_{1}$ , (b) $e_{2}$ , (c) $e_{R}$ , (d) $e_{Ω}$ , (e) $M$ , and (f) $T$ .

Figure 7.

Experimental results with large yaw offsets (exp2): (a and b) visual error $e_{1}, e_{2}$ , (c and d) Attitude tracking error $e_{R}, e_{Ω}$ , (e and f) command torque $M$ and thrust $T$ , (g) $e_{1}$ , (h) $e_{2}$ , (i) $e_{R}$ , (j) $e_{Ω}$ , (k) $M$ , and (l) $T$ .

The UAV initiates offboard control mode flight at t = 1 s, after which it smoothly and quickly reaches the desired position at about t = 3 s shown in Figure 4. The entire process takes about 2 s. This indicates that the proposed method has excellent transient performance. It remains in a steady state for the time afterward. To show the excellent steady-state performance of the proposed method, this paper shows the mean, standard deviation of the position and attitude angles for all experiments in Table 2.

Table 2.

Pose mean and standard deviations.

	x (cm)	y (cm)	z (cm)	$φ$ (deg)	$θ$ (deg)	$ψ$ (deg)
Mean	−0.117	0.134	0.051	0.156	0.094	−0.063
Std	0.833	1.014	0.732	0.314	0.292	0.497

In addition, this paper also provides the dynamics of all variables in the proposed method during the experiment shown in Figures 6 and 7. The graphs showing the evolution of the various errors metrics are very useful to understand how inner and outer loops of the system have different dynamics with different convergence rates and steady-state values.

The visual errors $e_{1}, e_{2}$ converge to zero shown in Figures 6(a), (b), 7(a) and (b). Compared to the outer loop, the inner loop of the attitude converges significantly more rapidly. Attitude error $ϒ (R, R_{d})$ converges to zero is shown in Figure 5. The attitude tracking errors $e_{R}, e_{Ω}$ converge to zero shown in Figures 6(c), (d), 7(c) and (d). The control thrust $T$ and moments $M$ calculated by the proposed method are shown in Figures 6(e), (f), 7(e) and (f).

After several experiments, it is proved that our proposed attitude control apparatus based on homography features has good control performance.

Remark 2. Because this paper uses PIXHAWK Autopilot as the controller, its torque and thrust control inputs are a normalized value ( $T \in (0, 1), M (0) \in (- 1, 1), M (1) \in (- 1, 1), M (2) \in (- 1, 1)$ ). The thrust generated by the UAV is almost exactly balanced with the gravity of the UAV at $T = 0.435$ .

Application to a unmanned aerial manipulator

In terms of application, the controller we have designed can be widely used in the visual servo application control of various multirotor UAVs, as dynamics and geometric control are not limited. In application scenarios, our designed controller can be used for visual servo based contact detection tasks, such as bridge internal corrosion detection, fan blade crack detection, and oil and gas pipeline internal corrosion detection. In some complex outdoor environments, RTK can be used to provide more accurate location information, ensuring the robustness and adaptability of the system. Here, we apply our designed controller to detect the thickness of the target steel plate, a task similar to detecting the thickness of oil and gas pipelines. The system configuration is shown in Figure 8. This paper equips the UAV with a device to detect the thickness of steel plates and moved the camera to the front of the UAV. The desired position is $x_{d} = {(1, 0.5, 1)}^{T} m$ .

Remark 3. Manually maneuver the drone to fly until the contact probe is right on the steel plate, at which point a reference photo is taken. Usually, there is no checkerboard grid to speed up corner point detection and matching, and a generic photo can be taken at the target to compute Homography by detecting and matching features such as ORB and SIFT. Since the detection and matching of ORB features takes a lot of time, it may make the controller unstable because the outer loop rate is limited by the computational rate of Homography.

Figure 8.

System configuration.

The corresponding results are shown in Figure 9(a) to (d). Under the action of the control torque and thrust, the UAV gradually reaches the desired position shown in Figure 9(a) and (b). Because this paper only verifies the effect of the controller mentioned in this paper, this paper does not consider the effect of the contact force between the UAV and the steel plate. When external contact forces exist $(F_{ext} \neq 0)$ , the system state of the UAV is affected to some extent. This work demonstrates that the controller has broad applicability and may be used in more scenarios in the future.

Figure 9.

Experimental results of contact detection: (a) visual error $e_{1}$ , (b) attitude tracking error $e_{R}$ , (c and d) Command torque $M$ and thrust $T$ .

Conclusions

This paper proposed a new visual servo geometric control method for underactuated UAVs based on homography. The paper proposed the use of a virtual homography matrix for the backstepping-control of the visual position error and geometric-control of orientation. Among them, this paper used virtual homography to define visual position error, used visual position error to control the position and yaw angle of the drone, and then integrated the visual position error into the control law of the geometric method to achieve control of the UAV’s attitude. This paper used theoretical analysis and extensive experiments to verify the performance of our proposed control method. The experimental results showed that this method had good transient performance, dynamic performance, and robustness. This work had solved the challenges of geometric control methods in visual servo control based on homography, and expanded the field of visual servo, which was a new direction. The field of visual servo control has great prospects, whether in academic fields, industrial applications, or civilian consumer applications. Therefore, the field of visual servo control is worth further research, and we will continue to study this field, improve the application of geometric control in this field, and explore more efficient visual servo control methods.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Natural Science Foundation of China under Grant No. (62103140, 62173132, 62133005), Natural Science Foundation of Hunan Province under Grant No. (2021JJ40124).

ORCID iD

Hang Zhong

Data availability statement

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

References

Baca

Petrlik

Vrba

, et al. The MRS UAV system: pushing the frontiers of reproducible research, real-world deployment, and education with autonomous unmanned aerial vehicles. J Intell Robot Syst 2021; 102(1): 1–28.

Cho

Kang

Y-S

Park

B-J

, et al. Altitude integration of radar altimeter and GPS/INS for automatic takeoff and landing of a UAV. In: 2011 11th International conference on control, automation and systems, 2011, pp.1429–1432. New York: IEEE.

Wang

Jiang

Zhang

, et al. Development of UAV-based target tracking and recognition systems. IEEE Trans Intell Transp Syst 2020; 21(8): 3409–3422.

Liu

Wang

, et al. A novel real-time moving target tracking and path planning system for a quadrotor UAV in unknown unstructured outdoor scenes. IEEE Trans Syst Man Cybern Syst 2019; 49(11): 2362–2372.

Kashino

Nejat

Benhabib

. Multi-UAV based autonomous wilderness search and rescue using target iso-probability curves. In 2019 international conference on unmanned aircraft systems (ICUAS), 2019, pp.636–643. New York: IEEE.

Baca

Stepan

Spurny

, et al. Autonomous landing on a moving vehicle with an unmanned aerial vehicle. J Field Robot 2019; 36(5): 874–891.

Jabbari

Oriolo

Bolandi

. Dynamic IBVS control of an underactuated UAV. In: 2012 IEEE international conference on robotics and biomimetics (ROBIO), 2012, pp.1158–1163. New York: IEEE.

Zheng

H-N

Liu

Z-Y

. Image-based visual servoing of quadrotor using homography reconstruction. In: 2019 Chinese control conference (CCC), 2019, pp.4089–4094. New York: IEEE.

Mebarki

Lippiello

Siciliano

. Nonlinear visual control of unmanned aerial vehicles in GPS-denied environments. IEEE Trans Robot 2015; 31(4): 1004–1017.

10.

Thomas

Welde

Loianno

, et al. Autonomous flight for detection, localization, and tracking of moving targets with a small quadrotor. IEEE Robot Autom Lett 2017; 2(3): 1762–1769.

11.

Zhang

Fang

Zhang

, et al. A novel geometric hierarchical approach for dynamic visual servoing of quadrotors. IEEE Trans Ind Electron 2019; 67(5): 3840–3849.

12.

Krupinski

Allibert

Hua

, et al. An inertial-aided homography-based visual servo control approach for (almost) fully actuated autonomous underwater vehicles. IEEE Trans Robot 2017; 33(5): 1041–1060.

13.

de Plinval

Morin

Mouyon

, et al. Visual servoing for underactuated VTOL UAVs: a linear, homography-based framework. Int J Robust Nonlinear Control 2014; 24(16): 2285–2308.

14.

Benhimane

Malis

. Homography-based 2D visual tracking and servoing. Int J Rob Res 2007; 26(7): 661–676.

15.

Nguyen

Hua

Allibert

, et al. A homography-based dynamic control approach applied to station keeping of autonomous underwater vehicles without linear velocity measurements. IEEE Transactions on Control Systems Technology. 2020; 29(5): 2065–2078.

16.

Wang

. Adaptive homography-based visual servo for micro unmanned surface vehicles. Int J Adv Manuf Technol 2019; 105(12): 4875–4882.

17.

Zhong

Wang

Miao

, et al. A homography-based visual servo control approach for an underactuated unmanned aerial vehicle in GPS-denied environments. IEEE Trans Intell Vehicles 2023; 8: 1119–1129.

18.

Huang

Zhu

Zheng

, et al. Homography-based visual servoing for underactuated VTOL UAVs tracking a 6-dof moving ship. IEEE Trans Vehicular Technol 2022; 71(3): 2385–2398.

19.

Bullo

Lewis

. Geometric control of mechanical systems: modeling, analysis, and design for simple mechanical control systems. New York, NY: Springer, 2019, Vol. 49.

20.

Lee

Leok

McClamroch

. Geometric tracking control of a quadrotor UAV on SE (3). In: 49th IEEE conference on decision and control (CDC), 2010, pp.5420–5425. New York: IEEE.

21.

Kotaru

Edmonson

Sreenath

. Geometric l1 adaptive attitude control for a quadrotor unmanned aerial vehicle. J Dyn Syst Meas Control 2020; 142(3): 031003.

22.

Khan

Mohd

MNH

Zulkifli

SABM

, et al. Deep reinforcement learning based unmanned aerial vehicle (UAV) control using 3d hand gestures. Comput Mater Contin 2022; 72(3): 5741–5759.

23.

Abro

GEM

Bin Mohd Zulkifli

Asirvadam

. Dual-loop single dimension fuzzy-based sliding mode control design for robust tracking of an underactuated quadrotor craft. Asian J Control 2023; 25(1): 144–169.

24.

Abro

GEM

Zulkifli

SABM

Ali

, et al. Fuzzy based backstepping control design for stabilizing an underactuated quadrotor craft under unmodelled dynamic factors. Electronics 2022; 11(7): 999.

25.

Abro

GEM

Ali

Zulkifli

, et al. Performance evaluation of different control methods for an underactuated quadrotor unmanned aerial vehicle (QUAV) with position estimator and disturbance observer. Math Probl Eng 2021; 2021: 1–22.

26.

Abro

GEM

Zulkifli

SABM

Asirvadam

. Performance evaluation of newton euler & quaternion mathematics-based dynamic models for an underactuated quadrotor UAV. In: 2021 11th IEEE international conference on control system, computing and engineering (ICCSCE), 2021, pp.142–147. New York: IEEE.

27.

Abro

GEM

Zulkifli

SABM

Asirvadam

, et al. Modelfree-based single-dimension fuzzy smc design for underactuated quadrotor UAV. Actuators 2021; 10: 191.

28.

Abro

GEM

Asirvadam

Bin Mohd Zulkifli

, et al. Effects of unmodelled dynamic factors on an under-actuated quadrotor: A review of hybrid observer design methods. Meas Control 2020; 53(9–10): 1978–1987.

29.

Mustafa Abro

Asirvadam

Zulkifli

SAB

, et al. Review of hybrid control designs for underactuated quadrotor with unmodelled dynamic factors. In: Emerging technologies in computing: Third EAI international conference, iCETiC 2020, London, UK, 19–20 August 2020, Proceedings 3, 2020, pp.62–85. Springer.

30.

Mustafa Abro

Asirvadam

Zulkifli

. Single-input fuzzy-sliding mode control for an underactuated quadrotor craft. In: 2020 IEEE 2nd international conference on artificial intelligence in engineering and technology (IICAIET), 2020, pp.1–6. New York: IEEE.

31.

Guenard

Hamel

Mahony

. A practical visual servo control for an unmanned aerial vehicle. IEEE Trans Robot 2008; 24(2): 331–340.

32.

Fink

Xie

Lynch

, et al. Dynamic visual servoing for a quadrotor using a virtual camera. Unmanned Syst 2017; 5(1): 1–17.