3D target localization based on multi–unmanned aerial vehicle cooperation

Abstract

Ground target three-dimensional positions measured from optical remote-sensing images taken by an unmanned aerial vehicle play an important role in related military and civil applications. The weakness of this system lies in its localization accuracy being unstable and its efficiency being low when using a single unmanned aerial vehicle. In this paper, a novel multi–unmanned aerial vehicle cooperative target localization measurement method is proposed to overcome these issues. In the target localization measurement stage, three or more unmanned aerial vehicles simultaneously observe the same ground target and acquire multiple remote-sensing images. According to the principle of perspective projection, the target point, its image point, and the camera’s optic center are collinear, and nonlinear observation equations are established. These equations are then converted to linear equations using a Taylor expansion. Robust weighted least-squares estimation is used to solve the equations with the objective function of minimizing the weighted square sum of re-projection errors from target points to multiple pairs of images, which can make the best use of the effective information and avoid interference from the observation data. An automatic calculation strategy using a weight matrix is designed, and the weight matrix and target-position coordinate value are updated in each iteration until the iteration stopping condition is satisfied. Compared with the stereo-image-pair cross-target localization method, the multi–unmanned aerial vehicle cooperative target localization method can use more observation information, which results in higher rendezvous accuracy and improved performance. Finally, the effectiveness and robustness of this method is verified by numerical simulation and flight testing. The results show that the proposed method can effectively improve the precision of the target’s localization and demonstrates great potential for providing more accurate target localization in engineering applications.

Introduction

In recent years, unmanned aerial vehicles (UAVs) have been successfully applied in various fields. Their value as surveillance platforms has been proven repeatedly in both military and civilian domains.^1–5 As an essential function of UAVs, target localization aims to solve the three-dimensional (3D) coordinate of the point on the ground by combining the data obtained by the measurement equipment carried on the UAV with the UAV’s flight data.^6–10

Vision sensors have been widely used in UAV target localization due to their passive and non-cooperative characteristics. Vision sensors do not emit any energy to measure the target and do not need to communicate with the targets.^11–13 However, the monocular vision sensor imaging projects 3D objects onto the two-dimensional (2D) image plane, and therefore it can only provide the bearing information of the target and not the range information. ¹⁴ In order to solve this problem, many schemes have been proposed. In their study, ¹⁵ Hosseinpoor et al. used a UAV with real-time kinematic (RTK) global positioning system (GPS) for estimation and localization, and processed the localization results through extended Kalman filtering. Meanwhile, the scale-invariant feature transform (SIFT) was used to extract the feature points of the same ground target in different frames in Han and de Souza’s study. ¹⁶ The relative height of the target and the UAV was calculated through 3D reconstruction.

Quintero et al. ¹⁷ in their work studied the optimal routing of two camera-equipped fixed-wing aircraft cruising at fixed altitude and cooperatively tracking a single ground target. In Sharma et al.’s study, ¹⁸ the extended Kalman filter was used to improve the accuracy of target localization. However, this method needs to assume that the altitude of the ground is zero, and it only works on the flat ground. In Barber et al.’s work, ¹⁹ a vision-based target localization system for micro fixed-wing aircraft was introduced. In the flight test, the UAV located the stationary target while flying around the target.

In the above studies, one single UAV was used to achieve the target localization. However, a single UAV cannot be equipped with many sensors for target localization since it suffers drawbacks such as the payload weight limit. In recent years, with the development of control, communication, electronics, and other technologies, the cooperative work of multiple UAVs has been realized.^20–25 For this reason, a multi-UAV cooperative target localization scheme is proposed in this paper. The main contributions of this paper are summarized as follows:

1. We propose a novel multi-UAV cooperative target localization framework. The optoelectronic platform is allocated to each UAV, and each camera observes the same target point simultaneously and acquires multiple remote sensing images.

2. We establish a mathematical model of the multi-UAV cooperative target localization system, which is a set of linear equations derived using the constraints of three-point collinearity.

3. We formulate the objective in terms of minimizing the weighted square sum of re-projection errors from target points to multiple pairs of images and design an automatic calculation strategy for the weight matrix.

Furthermore, the effectiveness and robustness of our method is verified by numerical simulation and flight testing. The results show that our method can effectively improve the precision and enhance the anti-interference of the target localization system.

System composition and coordinate definition

The overall structure of the system

In this study, multiple UAVs locate the same ground target cooperatively. Each UAV is equipped with a GPS, inertial measurement unit (IMU), and optoelectronic platform, as shown in Figure 1. The optoelectronic platform includes a camera and a camera pan-and-tilt system placed on the plane in a pod structure. During target localization, the video and telemetry information of each UAV are transmitted through a data link to the ground station. The manipulator can control the camera pan-and-tilt system to search for targets using a joystick, and when targets of interest appear on the screen, the optoelectronic platform locks onto the image and simultaneously measures the azimuth and elevation angles of the camera. Meanwhile, the UAV position and attitude are measured by GPS/IMU devices. The measured data of each UAV are unified in the reference coordinate system, and the target coordinates are solved by a multi-UAV cooperative target localization algorithm.

Figure 1.

Multi-UAV target localization system.

Coordinate system

Seven coordinate systems are used in this study. They are defined as follows:^26–28

1. Geodetic coordinate system: based on international terrestrial reference system WGS-84. Geodetic coordinate system describes spatial position by latitude, longitude, and geodetic elevation (λ, ϕ, h).

2. Terrestrial rectangular coordinate system G(O_g–X_gY_gZ_g): the origin is located in the center of the reference ellipsoid. The axis Z_g points to the north pole of the reference ellipsoid, and the axis X_g is directed to the intersection point of the Greenwich meridian plane and equator. The axis Y_g is perpendicular to the X_g and Z_g axes, following the right-hand rule.

3. UAV geographic coordinate system N(O_n–X_nY_nZ_n): the origin is the position (λ, ϕ, h) of a UAV, the X_n points to the east, and the Y_n points to the north. The axis Z_n is perpendicular to the Y_n and X_n axes, following the right-hand rule.

4. UAV body coordinate system B(O_b–X_bY_bZ_b): the coordinate origin is at the center of mass of the UAV, X_b points to the direction right above the UAV, Y_b points to the UAV nose, and Z_b, X_b, and Y_b form a right-handed coordinate system.

5. Camera coordinate system C(O_c–X_cY_cZ_c): the origin is the optical center of the camera, the X axis and Y axis are parallel to the X and Y axes of the image, and the Z axis is the optical axis of the camera, which is perpendicular to the image plane.

6. Image coordinate system: the origin is the intersection point of the camera’s optical axis and the image plane, and the image coordinate system is a 2D rectangular coordinate system. The actual physical scale (mm, um, etc.) is the unit.

7. Pixel coordinate system: the origin is the upper left corner of the image, the X_p and Y_p represent the number of rows and columns of the pixel in the digital image. Figure 2 shows the relationship between camera coordinate system, image coordinate system, and pixel coordinate system.

Figure 2.

Correlation diagram of camera coordinates, image coordinates, and pixel coordinates.

Key technology of multi-UAV target localization

Principle of stereo-image-pair cross-target localization

According to the perspective projection imaging principle, the three points of target point A, the corresponding image point a, and the camera’s optic center S are collinear when the camera observes the ground target, as shown in Figure 3. Therefore, the following equation can be formulated according to the aforementioned geometric relation^29,30

$$\{\begin{array}{c}\multicolumn{1}{c}{x=-f\frac{a_1(X_A-X_S)+b_1(Y_A-Y_S)+c_1(Z_A-Z_S)+T_x}{a_3(X_A-X_S)+b_3(Y_A-Y_S)+c_3(Z_A-Z_S)+T_z}}\\ \multicolumn{1}{c}{y=-f\frac{a_2(X_A-X_S)+b_2(Y_A-Y_S)+c_2(Z_A-Z_S)+T_y}{a_3(X_A-X_S)+b_3(Y_A-Y_S)+c_3(Z_A-Z_S)+T_z}}\end{array}$$

(1)

where (x, y) is the image coordinate of the image point a, (X_A, Y_A, Z_A) is the coordinate of the target points in the reference coordinate system, (X_S, Y_S, Z_S) is the coordinate of the camera’s optic center in the reference coordinate system, and f is the focal length of the camera.

Figure 3.

Perspective projection imaging principle.

The rotation matrix of the coordinate-system conversion from the reference coordinate system to the camera coordinate system can be represented as R, and the translation matrix is T

$$\mathbf{R}=\left[\begin{array}{ccc}{a}_1\hfill & a_2\hfill & a_3\hfill \\ \multicolumn{1}{c}{b_1}& b_2\hfill & b_3\hfill \\ \multicolumn{1}{c}{c_1}& c_2\hfill & c_3\hfill \end{array}\right]\mathbf{T}=\left[\begin{array}{c}{T}_x\hfill \\ \multicolumn{1}{c}{T_y}\\ \multicolumn{1}{c}{T_z}\end{array}\right]$$

(2)

The terrestrial rectangular coordinate system is used as the reference coordinate system in this paper, and the conversion process from the reference coordinate system to the camera coordinate system is shown in Figure 4.

Figure 4.

Coordinate system transformation relations.

The coordinates of the target in the camera coordinate system can be calculated as

$$\begin{array}{c}\multicolumn{1}{c}{{\mathbf{X}}_C={\mathbf{R}}_{\mathrm{B}}^C(\alpha ,\beta )·{\mathbf{R}}_{\mathrm{N}}^{\mathrm{B}}(\varphi ,\gamma ,\theta )·{\mathbf{R}}_{\mathrm{G}}^{\mathrm{N}}(\lambda ,\phi )·({\mathbf{X}}_G-{\mathbf{T}}_{\mathrm{G}}^{\mathrm{N}}(\lambda ,\phi ,h))}\\ ={\mathbf{R}}_{\mathrm{B}}^C(\alpha ,\beta )·{\mathbf{R}}_{\mathrm{N}}^{\mathrm{B}}(\varphi ,\gamma ,\theta )·{\mathbf{R}}_{\mathrm{G}}^{\mathrm{N}}(\lambda ,\phi )·{\mathbf{X}}_G\hfill \\ -{\mathbf{R}}_{\mathrm{B}}^C(\alpha ,\beta )·{\mathbf{R}}_{\mathrm{N}}^{\mathrm{B}}(\varphi ,\gamma ,\theta )·{\mathbf{R}}_{\mathrm{G}}^{\mathrm{N}}(\lambda ,\phi )·{\mathbf{T}}_{\mathrm{G}}^{\mathrm{N}}(\lambda ,\phi ,h)\hfill \\ =\mathbf{R}·{\mathbf{X}}_G+\mathbf{T}\hfill \end{array}$$

(3)

where ${\mathbf{X}}_C={\left[\begin{array}{ccc}{X}_{\mathrm{c}}\hfill & Y_c\hfill & Z_{\mathrm{c}}\hfill \end{array}\right]}^{\mathrm{T}}$ and ${\mathbf{X}}_G={\left[\begin{array}{ccc}{X}_{\mathrm{g}}\hfill & Y_{\mathrm{g}}\hfill & Z_{\mathrm{g}}\hfill \end{array}\right]}^{\mathrm{T}}$ are the coordinates of the target in the camera coordinate system and terrestrial rectangular coordinate system, respectively; ${\mathbf{T}}_{\mathrm{G}}^{\mathrm{N}}(\lambda ,\phi ,h)$ represents the origin of the UAV geographic coordinate system in the terrestrial rectangular coordinate system, and ${\mathit{R}}_G^N(\lambda ,\phi )$ , ${\mathit{R}}_N^B(\varphi ,\gamma ,\theta )$ , and ${\mathit{R}}_{\mathrm{B}}^C(\alpha ,\beta )$ are the rotation matrices for coordinate conversion.

In the target localization process, the UAV position (λ, φ, h) and attitude (ϕ, γ, θ) can be provided by GPS and IMU, respectively, and the azimuth and elevation angle (α, β) can be obtained by the optoelectronic platform; thus, R and T can be obtained by equation (3).

We let two UAVs observe the same ground target point A at S₁ and S₂, and obtain image points a₁ and a₂, respectively, as shown in Figure 5. In the absence of error conditions, the rays S₁a₁ and S₂a₂ intersect at ground target point A. By combining equations (1) and (3), and taking the target point coordinate (X_A, Y_A, Z_A) as the unknown quantity, four equations can be formulated, and therefore (X_A, Y_A, Z_A) can be solved.

Figure 5.

Stereo-image-pair cross-target localization.

Multi-UAV cooperative target localization based on robust weighted least-squares estimation

The above target localization method is very sensitive to all kinds of noise. This is because for aerial surveys the object distance of the imaging system is much greater than the focal length. In this paper, based on the principle of stereo-image-pair cross-target localization, the same target point is measured multiple times by multiple UAVs, and the optimization algorithm is used to improve the precision and robustness of the target localization method.

Multiple UAVs acquire n (n > 2) images for the same target point from different angles. From equation (1) and the transformation between the camera coordinate system and reference coordinate system given by equation (3), and with the target coordinates (X_A, Y_A, Z_A) unknown, the following equation can be derived

$$\begin{array}{c}\mathrm{Z}=\mathrm{H}\left(\mathbf{X}\right)=\left[\begin{array}{c}{\mathrm{h}}_1(X_A,Y_A,Z_A)\hfill \\ \multicolumn{1}{c}{{\mathrm{h}}_2(X_A,Y_A,Z_A)}\\ \multicolumn{1}{c}{⋮}\\ \multicolumn{1}{c}{{\mathrm{h}}_{2n-1}(X_A,Y_A,Z_A)}\\ \multicolumn{1}{c}{{\mathrm{h}}_{2n}(X_A,Y_A,Z_A)}\end{array}\right]\hfill \\ \{\begin{array}{c}{\mathrm{h}}_i=-f_i\frac{{a_i}_1(X_A-X_{\mathrm{iS}})+b_{i1}(Y_A-Y_{\mathrm{iS}})+c_{i1}(Z_A-Z_{\mathrm{iS}})+T_{\mathrm{ix}}}{a_{i3}(X_A-X_{\mathrm{iS}})+b_{i3}(Y_A-Y_{\mathrm{iS}})+c_{i3}(Z_A-Z_{\mathrm{iS}})+T_{\mathrm{iz}}}\hfill \\ {\mathrm{h}}_{i+1}=-f_i\frac{a_{i2}(X_A-X_{\mathrm{iS}})+b_{i2}(Y_A-Y_{\mathrm{iS}})+c_{i2}(Z_A-Z_{\mathrm{iS}})+T_{\mathrm{iy}}}{a_{i3}(X_A-X_{\mathrm{iS}})+b_{i3}(Y_A-Y_{\mathrm{iS}})+c_{i3}(Z_A-Z_{\mathrm{iS}})+T_{\mathrm{iz}}}\hfill \end{array}\end{array}$$

(4)

where $\mathit{Z}={\left[\begin{array}{ccccc}{x}_1\hfill & y_1\hfill & \dots \hfill & x_n\hfill & y_n\hfill \end{array}\right]}^{\mathrm{T}}$ and $\mathbf{X}={\left[\begin{array}{ccc}{X}_A\hfill & Y_A\hfill & Z_A\hfill \end{array}\right]}^{\mathrm{T}}$ .

Equation (4) is a nonlinear equation that is transformed into a linear equation using a Taylor expansion. The first-order Taylor expansion of equation (4) at an initial value ${\mathit{S}}^0$ is

$$\mathbf{Z}=\mathrm{H}\left({\mathbf{X}}^0\right)+\mathbf{B}·(\mathbf{X}-{\mathbf{X}}^0)+\mathrm{\Delta }\mathbf{n}$$

(5)

where

$$\mathbf{B}={\left[\begin{array}{ccc}{\frac{\partial {\mathrm{h}}_1}{\partial X_A}|}_0\hfill & {\frac{\partial {\mathrm{h}}_1}{\partial Y_A}|}_0\hfill & {\frac{\partial {\mathrm{h}}_1}{\partial Z_A}|}_0\hfill \\ \multicolumn{1}{c}{{\frac{\partial {\mathrm{h}}_2}{\partial X_A}|}_0}& {\frac{\partial {\mathrm{h}}_2}{\partial Y_A}|}_0\hfill & {\frac{\partial {\mathrm{h}}_2}{\partial Z_A}|}_0\hfill \\ \multicolumn{1}{c}{⋮}& ⋮\hfill & ⋮\hfill \\ \multicolumn{1}{c}{{\frac{\partial {\mathrm{h}}_{2n}}{\partial X_A}|}_0}& {\frac{\partial {\mathrm{h}}_{2n}}{\partial Y_A}|}_0\hfill & {\frac{\partial {\mathrm{h}}_{2n}}{\partial Z_A}|}_0\hfill \end{array}\right]}_{2n\times 3}$$

Defining the vectors $\mathbf{U}=\mathbf{Z}-\mathrm{H}({\mathbf{X}}^0)$ and $\mathbf{V}=\mathbf{X}-{\mathbf{X}}^0$ , equation (5) can be rewritten as the following linear system

$$\mathbf{U}=\mathbf{BV}+\mathrm{\Delta }\mathbf{n}$$

(6)

The objective function minimizes the sum of squares of errors

$${(\mathbf{U}-\mathbf{BV})}^{\mathrm{T}}(\mathbf{U}-\mathbf{BV})=min$$

(7)

According to least-squares estimation, the vector V can be estimated as follows

$$\mathbf{V}={\left({\mathbf{B}}^{\mathrm{T}}\mathbf{B}\right)}^{-1}{\mathbf{B}}^{\mathrm{T}}\mathbf{U}$$

(8)

During the process of UAV target localization, the attitude and height of the UAV and the azimuth and elevation angle of the camera are all different at each measurement point. In this case, even if the same type of camera is used, due to the camera’s different external parameters, the localization accuracy of each measurement point is different as is the contribution to the error. Therefore, the weight matrix $\mathbf{P}$ is introduced, and the weights for the X and Y axes of the same image are set to the same value

$$\mathbf{P}=\left[\begin{array}{cccccc}{p}_1\hfill & 0\hfill & 0\hfill & 0\hfill & \cdots \hfill & 0\hfill \\ \multicolumn{1}{c}{0}& p_1\hfill & 0\hfill & 0\hfill & \cdots \hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & p_2\hfill & 0\hfill & \cdots \hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & p_2\hfill & \cdots \hfill & 0\hfill \\ \multicolumn{1}{c}{⋮}& ⋮\hfill & ⋮\hfill & ⋮\hfill & ⋮\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & 0\hfill & 0\hfill & p_n\hfill \end{array}\right]$$

(9)

The objective function to be solved can be expressed as

$${(\mathbf{U}-\mathbf{BV})}^{\mathrm{T}}\mathbf{P}(\mathbf{U}-\mathbf{BV})=min$$

(10)

According to weighted least-squares estimation, the vector can be estimated as follows

$$\mathbf{V}={\left({\mathbf{B}}^{\mathrm{T}}\mathbf{PB}\right)}^{-1}{\mathbf{B}}^{\mathrm{T}}\mathbf{PU}$$

(11)

Then, the vectors $\mathbf{U}=\mathbf{Z}-\mathrm{H}({\mathbf{X}}^0)$ and $\mathbf{V}=\mathbf{X}-{\mathbf{X}}^0$ , and the estimated value of the variable $\mathbf{X}$ , can be obtained as

$$\hat{\mathit{X}}={\left({\mathbf{B}}^{\mathrm{T}}\mathbf{PB}\right)}^{-1}{\mathbf{B}}^{\mathrm{T}}\mathbf{P}(\mathbf{Z}-\mathrm{H}({\mathbf{X}}^0))+{\mathbf{X}}^0$$

(12)

The initial value ${\mathbf{X}}^0$ can be obtained based on the stereo-image-pair cross-target localization method.

When there is an increasing number of images, a robust estimation method is often introduced to improve target localization accuracy. In this paper, an iteration method is used to assign weights for images. The main steps are as follows

• 1. Set an initial value of the weight matrix

$$\mathbf{P}={\left[\begin{array}{ccccc}1\hfill & 0\hfill & 0\hfill & \cdots \hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 1\hfill & 0\hfill & \cdots \hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 1\hfill & \cdots \hfill & 0\hfill \\ \multicolumn{1}{c}{⋮}& ⋮\hfill & ⋮\hfill & ⋮\hfill & ⋮\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 0\hfill & \cdots \hfill & 1\hfill \end{array}\right]}_{2n\times 2n}$$

(13)

The first estimate of target point coordinates can be derived from equation (12).

• 2. Set

$$\mathbf{U}-\mathbf{BV}={\left[\begin{array}{cccccc}\mathrm{\Delta }{x}_1\hfill & \mathrm{\Delta }{y}_1\hfill & \mathrm{\Delta }{x}_2\hfill & \mathrm{\Delta }{y}_2\hfill & \cdots \hfill & \mathrm{\Delta }{y}_n\hfill \end{array}\right]}_{1\times 2n}$$

(14)

Equation (10) can then be written as

$$\begin{array}{c}\sum _{i=1}^n{p}_i(\mathrm{\Delta }{x_i}^2+\mathrm{\Delta }{y_i}^2)=\sum _{i=1}^n{p}_i\left(\mathrm{\Delta }{d_i}^2\right)=min\hfill \end{array}$$

(15)

Calculate the standard deviation as follows

$$\sigma =\sqrt{\frac{\sum _{i=1}^n{p}_i\left(\mathrm{\Delta }{d_i}^2\right)}{n-2}}$$

(16)

• 3. The weight factor ${\omega }_i$ is defined as

$${\omega }_i=\{\begin{array}{cc}1\hfill & \left|u_i\right|<k_0\hfill \\ \multicolumn{1}{c}{\frac{k_0}{\left|u_i\right|}}& k_0\le \left|u_i\right|<k_1\hfill \\ \multicolumn{1}{c}{0}& \left|u_i\right|\ge k_1\hfill \end{array}$$

(17)

where $u_i=\raisebox{1ex}{$\Delta d_i$}\!\left/ \!\raisebox{-1ex}{$\sigma $}\right.$ , and $k_0$ and $k_1$ are empirical constants. In our experiment, $k_0=0.5$ and $k_1=2.0$ .

• 4. Let ${\overline{p}}_i=p_i{\omega }_i$ replace $p_i$ in equation (12), and recalculate the target point coordinates.

• 5. Repeat steps (2)–(4) until the difference between the two estimated results is less than the threshold value.

Experimental results and analysis

It is assumed that the truth values of the target point coordinates are ${\mathbf{X}}_A=[\begin{array}{ccc}{\lambda }_A=39.569151°\hfill & {\phi }_A=105.364066°\hfill & h_A=100m\hfill \end{array}{]}^{\mathrm{T}}$ , and three groups of images with known interior and exterior elements (see Tables 1–3) are given, where f is the focal length of the camera; ${\mathbf{X}}_s={\left[\begin{array}{ccc}{\lambda }_s\hfill & {\phi }_s\hfill & h_s\hfill \end{array}\right]}^{\mathrm{T}}$ are the coordinates of the UAV position; $\varphi ,\gamma ,\theta$ are the attitude angles of the UAV; and $\alpha ,\beta$ are the azimuth angle and the elevation angle of the camera, respectively.

Table 1.

Both interior and exterior orientation elements of known images at small heights.

Elements	Image 1	Image 2	Image 3	Image 4	Image 5	Image 6
f/mm	100	120	100	120	100	120
${\lambda }_S/°$	39.560854	39.566871	39.575170	39.577451	39.571429	39.563132
${\phi }_S/°$	105.35944	105.34795	105.35257	105.36869	105.38018	105.37555
hS/m	800	800	800	800	900	700
$\varphi /°$	330	310	200	190	120	50
$\gamma /°$	−62	−55	−51	−60	−66	−65
$\theta /°$	−1	1	2	−2	−1	0
$\alpha /°$	20	−8	37	−12	3	13
$\beta /°$	1	−6	−10	−1	5	6

Table 2.

Both interior and exterior orientation elements of known images at large heights.

Elements	Image 1	Image 2	Image 3	Image 4	Image 5	Image 6
f/mm	100	120	100	120	100	120
${\lambda }_S/°$	39.560854	39.566871	39.575170	39.577451	39.571429	39.563132
${\phi }_S/°$	105.35944	105.34795	105.35257	105.36869	105.38018	105.37555
hS/m	8000	9000	8000	8500	9000	7500
$\varphi /°$	330	310	200	190	120	50
$\gamma /°$	−12	−9	−8	−16	−16	−11
$\theta /°$	−1	1	2	−1	−2	1
$\alpha /°$	24	−8	33	−10	1	10
$\beta /°$	1	−4	−1	2	5	1

Table 3.

Both interior and exterior orientation elements of known images at mixed heights.

Elements	Image 1	Image 2	Image 3	Image 4	Image 5	Image 6
f/mm	100	120	100	120	100	120
${\lambda }_S/°$	39.560854	39.566871	39.575170	39.577451	39.571429	39.563132
${\phi }_S/°$	105.35944	105.34795	105.35257	105.36869	105.38018	105.37555
hS/m	8000	9000	8000	800	800	7500
$\varphi /°$	330	310	200	190	120	50
$\gamma /°$	−11	−13	−10	−61	−62	−55
$\theta /°$	−3	−0.5	1	−1	−2	1
$\alpha /°$	18	−7	40	−9	2	11
$\beta /°$	1	−1	−3	−1	3	−8

Verification of algorithm correctness

To verify the correctness of the algorithm, equation (1) can be used to calculate the pixel coordinates ${\mathbf{x}}_i$ of the target point in different images. Gauss errors of 2 m and 0.1° are added to the UAV position and attitude angle, respectively. Then, the target localization algorithm presented in this paper is used to calculate the new coordinates ${\mathbf{X}}_i$ of the ground point A in three groups of images, and the results are finally compared with the truth values. In this example, all images are set with the same precision, which means the weight of all of them is 1.

We use the Monte-Carlo method to conduct 1000 target localization experiments for these three groups and statistically analyze the localization errors. The results show that the target localization errors (root mean squared error, RMSE) of the three groups are 3.81, 7.74, and 4.95 m. This test verifies the correctness of the algorithm.

Verification of algorithm robustness

Taking the first group of data in Table 1 as an example to verify the robustness of this algorithm, 1 pixel (4 μm in size) of Gaussian noise is added to the coordinates of each image point, and a 2-mm gross error is added to the horizontal coordinate of the image point of image 1. Then, the method proposed in this paper is used to calculate the multi-UAV cooperative target localization. To display the precision of the localization intuitively, $\mathrm{\Delta }d$ is used to represent the displacement deviation of the target localization result from the truth value. Each image is assigned a weight of 1 in the initial calculation. Table 4 shows the weight of each calculation and the estimated result for the ground point.

Table 4.

Robust weighted least-squares estimation of ground target point.

Number of iterations	Weight 1	Weight 2	Weight 3	Weight 4	Weight 5	Weight 6	$\mathrm{\Delta }d/\mathrm{m}$
0	1	1	1	1	1	1	0.121
1	0.912	1	1	1	1	1	0.112
2	0.832	1	1	1	1	1	0.082
3	0.743	1	1	1	1	1	0.073
4	0	1	1	1	1	1	0.038
5	0	1	1	1	1	1	0.038

As shown in Table 4, after several iterations the weight of the image containing the gross error decreases to 0 gradually, thus nullifying its effects during target localization. This method can avoid gross errors by redetermining the weight matrix during the target localization process, thus demonstrating good robustness. On the contrary, the stereo-image-pair cross-target localization method has poor robustness since the method does not make use of the redundant observation information and is greatly affected by the observation precision of single images.

Fifteen ( $C_6^2$ ) localization results were generated by the stereo-image-pair cross-target localization method and are shown in Table 5. Owing to the presence of a gross error, at least five estimated values among the stereo-image-pair cross-target localization results deviated largely from the truth value of the ground point.

Table 5.

Results of stereo-image-pair cross-target localization method.

Image pairings	$\mathrm{\Delta }d/\mathrm{m}$
Image 1 & image 2	4.810
Image 1 & image 3	4.897
Image 1 & image 4	5.164
Image 1 & image 5	6.331
Image 1 & image 6	4.312
Image 2 & image 3	0.041
Image 2 & image 4	0.039
Image 2 & image 5	0.045
Image 2 & image 6	0.054
Image 3 & image 4	0.067
Image 3 & image 5	0.052
Image 3 & image 6	0.043
Image 4 & image 5	0.059
Image 4 & image 6	0.049
Image 5 & image 6	0.038

It can be seen from Table 4 that after iteration, weight 1 is equal to 0, and weights 2–6 are equal to 1, which means that the final target localization value is determined by images 2–4. Compared with the results in Table 5, this localization accuracy is significantly better than that of stereo-image-pair cross-target localization method, which shows that the random error can be effectively suppressed. To sum up, the iterative weighting is to eliminate gross errors, and multi-UAV cooperative observation is to suppress random errors. Both of them improve the robustness of the localization algorithm.

Comparison of target localization precision of two algorithms

Since the target localization method proposed in this paper can effectively utilize the information of multiple images, it is not sensitive to measurement errors. To verify this feature, the proposed method was compared with the stereo-image-pair cross-target localization method. A curve of its localization precision changing with errors (0–100 pixels, tested at an interval of 1 pixel) of image point coordinates for the first group of images is plotted in Figure 6(a). Similarly, curves of the target localization precision changing with errors of the image point coordinate are plotted in Figure 6(b) and (c) for the other two groups of images (the group of images at large heights (errors of 0–10 pixels, tested at an interval of 0.1 pixel) and the group of images at mixed heights (errors of 0–20 pixels, tested at an interval of 0.2 pixel)). As shown in Figure 6, the localization precision of both the stereo-image-pair cross-target localization method and the method proposed in this paper decreases approximately linearly with increasing error of the image point coordinates. Meanwhile, for each group of images at the same error level, a significantly greater localization precision is obtained when the proposed method is used. Upon adoption of the proposed target localization method, for the images at small heights, a localization precision of approximately 9 cm can be achieved when the image point coordinates are at the 10-pixel error level and a localization precision of 80 cm can still be achieved when the error is 100 pixels. For the images at large heights, a localization precision of approximately 1 m can be achieved when the image point coordinates are at the 10-pixel error level; meanwhile, for the images at mixed heights, a localization precision of approximately 80 cm can be achieved when the image point coordinates are at the 10-pixel error level. Similar to the case in which errors are added to the image point coordinates, target localization precision also shows a linearly decreasing trend with the errors in the exterior line element (UAV position) and angle elements (UAV attitude angle and camera angles), which will not be discussed in more detail here. As shown in Figure 6, it is concluded that at the same error level the target localization precision of the proposed method is much greater than that of the stereo-image-pair cross-target localization method.

Figure 6.

Target localization precision of images at different heights with image coordinate errors. Target localization precision of images at (a) small heights; (b) large heights; and (c) mixed heights.

Error analysis of proposed target localization algorithm

To analyze how the algorithm is affected by errors, the following simulation test was designed:

1. Truth values of ground point coordinates: 108 ground points with known coordinates uniformly distributed within a certain area were selected, as shown in Figure 7.

2. Error size: Gaussian errors of 20 pixels, 40″, and 100 m were added to image point coordinates, exterior line element, and angle elements, respectively.

3. Calculated values of ground point coordinates: upon the introduction of errors in step 2, the method proposed in this paper was used for target localization, and the calculated values of the ground point coordinates were obtained.

4. Analysis of localization results: The calculated values of the ground points were compared with their truth values, and the deviations of the components of the calculated values plotted in Figures 8–10. Finally, the impacts of these errors on the localization results as well as their combined impacts were analyzed.

Figure 7.

Target points distribution.

Figure 8.

Deviation distribution of ground points brought by different errors on images at small heights. (a) Error in image point coordinates; (b) error in angular elements; and (c) error in line elements.

Figure 9.

Deviation distribution of ground points brought by different errors on images at large heights. (a) Error in image point coordinates; (b) error in angular elements; and (c) error in line elements.

Figure 10.

Deviation distribution of ground points brought by different errors on images at mixed heights. (a) Error in image point coordinates; (b) error in angular elements; and (c) error in line elements.

As shown in Figure 8, for the images at small heights, after the errors in image point coordinates, exterior line elements, and angle elements were added individually, the components on the X, Y, and Z axes exhibit a similar deviation curve amplitude. However, the 10-m error of the line element led to the greatest deviation of the localization results from the truth values. As shown in Figure 9, for the images at large heights with an error being added to exterior line elements individually, the components on the X and Y axes exhibit a similar deviation amplitude and are much more stable than the component on the Z axis and slightly more affected by the error. The 10-m error of the line element led to the greatest deviation of the localization result from the truth value. As shown in Figure 10, compared with the images at large heights, the component on the Z axis in the images at mixed heights is much less sensitive to the error in the exterior line element. Figures 8–10 show that the addition of the same level of noise into images at different heights will cause differences in the deviation degree of the localization results. Among the deviations, images at small heights have the smallest localization deviations, followed by the images at mixed heights, and then by the localization results of the images at large heights. When multi-UAV cooperative target localization is used at large heights, the component on the Z axis is more sensitive to errors, and thus the deviation of localization results is worse. To address this problem, information of the images at small heights can be used to effectively reduce the sensitivity of the component on the Z axis to errors, thus achieving greater localization precision.

Flight test

In this section, the proposed approach is validated using flight test data. In order to verify the feasibility of the algorithm, the flight test is performed in an outside field environment. One known target point is arranged on the ground, and six UAVs are launched to observe the target point simultaneously. Two of the medium UAVs are equipped with an optoelectronic platform (resolution ratio 1920×1080, a pixel size of 4.2 µm), while the other four small UAVs have vertical down-looking cameras (resolution ratio 1280×720, a pixel size of 5.5 µm). Each UAV is equipped with a high-precision RTK GPS module, and its positioning error is less than 10 cm (circular error probable, CEP). The UAVs are shown in Figure 11. A ground receiving station receives the measurement data and video sequences from the six UAVs.

Figure 11.

Multiple UAVs.

Four indices are used to evaluate the target localization accuracy: sB represents the longitude error, sL represents the latitude error, sH represents the altitude error, and ssum represents the error radius.

• Test 1: Six UAVs locate the ground target cooperatively at small heights (50–80 m), as shown in Figure 12. In this case, the localization error is as follows: sB = 8.8110 × 10⁻⁶ (°), sL= 1.9633 × 10⁻⁵ (°), sH = 2.79 m, and ssum = 3.51 m.

• Test 2: Six UAVs locate the ground target cooperatively at large heights (200–220 m), as shown in Figure 13. In this case, the localization error is as follows: sB = 2.8419 × 10⁻⁵ (°), sL = 6.0389 × 10⁻⁵ (°), sH = 6.54 m, and ssum = 8.92 m.

• Test 3: Six UAVs locate the ground target cooperatively at mixed heights, as shown in Figure 14. Among them, four UAVs are at small heights (50–80 m), and the other two UAVs are at large heights (200–220 m). In this case, the localization error is as follows: sB = 1.4790 × 10⁻⁵ (°), sL = 3.1557 × 10⁻⁵ (°), sH = 2.41 m, and ssum = 3.98 m.

Figure 12.

Remote sensing images from six UAVs at small heights (50–80 m).

Figure 13.

Remote sensing images from six UAVs at large heights (200–220 m).

Figure 14.

Remote sensing images from six UAVs at mixed heights (50–220 m).

The flight test results show that the proposed target localization algorithm can achieve high precision. In the actual environment, the localization accuracy of low-flying UAVs is significantly higher than that of high-flying UAV, which is consistent with the simulation results. With the increase in UAV height, the localization result becomes very sensitive to all kinds of noise. This is because for aviation remote sensing systems the object distance is much greater than the focal length, and hence a slight error, such as from the GPS, IMU, or camera, will significantly enlarge the localization error.

Conclusion

To address the limitations of the existing UAV target localization method, a novel multi-UAV cooperative target localization method is proposed in this paper. A mathematical model for multi-UAV cooperative target localization based on collinear equations is established. In order to make the best use of the effective information and avoid interference from the observation data, a robust weighted least-squares estimation is used to solve the model. Finally, the effectiveness and robustness of this method is verified by numerical simulation and flight testing. The results show that the proposed method can effectively improve the precision of target localization, and it demonstrates great potential for providing more accurate target localization in engineering applications.