Sage Journals: Discover world-class research

Abstract

A multi-camera network is proposed to estimate an aircraft’s motion parameters relative to the reference platform in large outdoor fields. Multiple cameras are arranged to cover the aircraft’s large-scale motion spaces by field stitching. A camera calibration method using dynamic control points created by a multirotor unmanned aerial vehicle is presented under the conditions that the field of view of the cameras is void. The relative deformation of the camera network caused by external environmental factors is measured and compensated using a combination of cameras and laser rangefinders. A series of field experiments have been carried out using a fixed-wing aircraft without artificial makers, and its accuracy is evaluated using an onboard Differential Global Positioning System. The experimental results show that the multi-camera network is precise, robust, and highly dynamic and can improve the aircraft’s landing accuracy.

Keywords

Multi-camera network camera calibration deformation measurement motion parameter estimation

Introduction

Estimating an aircraft’s motion parameters is an important application of multi-camera networks,¹ such as vision-based aircraft landing.^2,3 It is a basic requirement of flight tests to measure the aircraft’s distance, azimuth, velocity, and pose relative to the reference platform on which it will land, so that the aircraft can land universally and dynamically with high precision, even when the platform is moving.^4,5

There are a number of motion parameter measurement systems for aircrafts currently in service. GPS can provide an accurate aircraft position, velocity, and pose estimate with an onboard hardware, and they are incapable of determining distance, azimuth, velocity, and pose estimates relative to a moving reference platform without additional ground infrastructure.⁶ The GPS is not reliable in certain conditions, such as under electronic jamming, near trees, or near tall buildings.⁷ In addition, GPS have been ruled out in certain applications, for example, some platforms like warships cannot send their position to the aircraft for fear of jeopardizing their security.⁴ Therefore, this consideration places emphasis on passive sensors.

The passive sensors, such as radar and microphone arrays, have been widely applied for motion parameter estimation of aircrafts. Thales Company in France has developed a series of advanced radar systems that meet a wide range of needs for surveillance, like SMART-L, or VARIANT. Due to its larger power budget, SMART-L is dedicated for the early detection and tracking of small aircraft in very long-range performance.⁸ VARIANT is a multipurpose, short-/medium-range 2-D radar for air and surface surveillance. Its principal role is automatic target detection and tracking for both aircraft and surface targets.⁹ Meanwhile, Thales has developed an advanced calibration system for radar system, those calibration systems perform well even under heavy weather condition, to satisfy the requirements of weapon systems. However, the radar is quite expensive, for example, the cost of SMART-L radar is around 12 million dollars. The arrays of microphones can also be used for detecting and tracking aircrafts as an alternative for radars. The main advantages of microphone arrays in comparison with radar systems are comparatively the lower cost, and it is electromagnetic and pollution free.¹⁰ The radar and microphone array systems are particularly noted for their excellent all-weather and automatic aircraft detection and track capabilities at long distance. Nevertheless, these systems are highly sophisticated and expensive. Moreover, since the motion parameters of aircraft in flight tests are expected to measure relative to a reference platform in the range of 20–1000 m only under normal weather conditions, other measurement systems that are inexpensive, simple, and reliable are highly desired.

Compared to other measuring means, such as GPS, radar, or microphone arrays, a multi-camera network has distinct advantages regarding the complicacy of the facilities and the expense of the costs.^11,12 Therefore, multi-camera networks serve as an effective way to estimate an aircraft’s motion parameters. Measuring volume and accuracy are important performance characteristics of vision systems. Both are dependent on the field of view (FOV) and resolution of the cameras.¹³ With the high-speed and long motion range capabilities in aircraft landing,⁴ the FOV of cameras should cover the aircraft’s motion spaces (20–1000 m), and the motion parameter measurements of the aircraft have the requirement of measuring volume and accuracy at the same time, that is, the large FOV and high spatial resolution, respectively. It is almost impossible for conventional binocular vision to perform this large field measurement tasks with an acceptable degree of continuity or reasonable accuracy. To cope with this problem, using multiple cameras by arranging them as a sensor network to observe the whole motion and measure the whole range has an important application value.^14,15

To calibrate the cameras’ parameters precisely is the core of motion parameter estimation of an aircraft. Because the cameras often need to adjust the depth of field and installation angle, etc., according to the actual measurement task, calibrating the camera parameters in the laboratory in advance is not suitable.¹⁶ It is necessary to calibrate the camera parameters directly in the field, even if the FOV of the camera is void. For example, cameras that are pointing to sky cannot find any calibration objects in the FOV. In addition, the reference platforms where the cameras are installed are subject to elastic deformation, due to the external environmental factors in field experiments, such as high wind,¹⁷ subsidence,¹⁸ and deformation of installed towers.¹⁹ The inevitable deformation will affect the measurement accuracy.²⁰ For example, as the baseline distance of two cameras is about 70 m, 1 arc minute of angle deformation and 1 mm of displacement deformation between the left and the right camera coordinate system would result in at least 6 m of positioning error in the measuring distance of 1000 m. Therefore, the relative deformation between cameras is a crucial error source for measuring data from a multi-camera network, since an accurate measurement and compensation of relative deformation in real time are of great significance.

In order to obtain the motion parameters of an aircraft, we have built a multi-camera network whose FOV is combined by multiple groups of cameras and developed a method to compensate the relative deformation. The proposed multi-camera network has been tested when a fixed-wing aircraft is landing in large outdoor fields, and the results of the tests are shown in this article. The rest of this article is organized as follows: the second section presents the architecture of our multi-camera network; the third section describes the correlative algorithms in the multi-camera network, including camera calibration, relative deformation measurement, and motion parameter measurements. Experimental results and discussions are given in the fourth section, followed by a brief conclusion in the last section.

Architecture of our multi-camera network

Figure 1 shows a multi-camera network that is designed to measure the aircraft’s motion parameters relative to a reference platform in the range of 20–1000 m. The multi-camera network is composed of two sets of measurement units that are installed on the left side and the right side of the reference platform, respectively. Each measurement unit mainly consists of three parts.

Figure 1.

Architecture of our multi-camera network.

(1) Three high-speed cameras (C₁, C₃, C₅ and C₂, C₄, C₆). These cameras are used to measure far-field (500–1000 m), middle-field (100–500 m), and near-field (20–100 m), respectively. The pixel resolution of the cameras is 1280 × 1024 pixels at 50 frames per second (fps), and the pixel patch is 14 × 14 μm. The corresponding focal lengths of the camera lens are 80 mm with the FOV 12.78° × 10.24°, 50 mm with the FOV $2 0. 32 ° \times 16.32 °$ , and 28 mm with the FOV $35.49 ° \times 28.72 °$ .

(2) Relative deformation measurement equipment (C₇, LRF₁ and C₈, LRF₂), including a high-speed camera and a laser rangefinder (LRF). C₇ and C₈ are set to see each other for measuring the platform’s relative deformation between the left side and the right side. The pixel resolution of the cameras is 1024 × 1024 at 50 fps, and the pixel patch is 2.45 × 2.45 μm. The focal length of the camera lens is 100 mm with the FOV $0.96 ° \times 0.96 °$ . Considering the precision of translation measurements along the camera’s optical axis is generally much lower than that of which is perpendicular to the optical axis in monocular vision measurement due to the inherent limitation, especially in the case that the FOV of a camera is small and the shooting distance is long. An LRF is installed near the camera, which will improve the precision of translation measurements along the camera’s optical axis with laser ranging. The measurement accuracy of the LRF is 1.0 mm. The output light spot of the LRF is consistent with the camera’s optical center, and the laser direction of the LRF is approximately parallel with the camera’s optical axis by adjusting in advance.

(3) Other equipment, such as an image transmission and storage system, a GPS time synchronization system, and a temperature control system. All the high-speed cameras and the LRFs will receive a common synchronized time stamp provided by the GPS time synchronization system during the measurement process.

To elaborate the measuring principles of our multi-camera network in this article, some symbols are introduced as follows.

Platform coordinate system $O_{P} - X_{P} Y_{P} Z_{P}$ : Using the camera coordinate system of camera C₁ on the left side as platform coordinate system. The motion parameters of the aircraft are obtained in this coordinate system $O_{C 1} - X_{C 1} Y_{C 1} Z_{C 1}$ .

Calibration coordinate system $O_{W} - X_{W} Y_{W} Z_{W}$ : The calibration coordinate system is built in the reference platform according to the demands of the calibration task. A total station or other surveying instruments can be used in this process. X_W represents the coordinates of the control points in the calibration coordinate system.

Body coordinate system $O_{B} - X_{B} Y_{B} Z_{B}$ : The body coordinate system is defined in the aircraft. It can be defined in advance or according to the demands of on-site measurement task.

Parameters of camera C_i: K_i represents the camera’s intrinsic matrix, and $D i s_{i}$ represents the lens distortion coefficients. The camera’s extrinsic parameters H_i define the homogeneous transformation between the calibration coordinate system $O_{W} - X_{W} Y_{W} Z_{W}$ and the camera coordinate system $O_{C i} - X_{C i} Y_{C i} Z_{C i}$ , where R_i denotes a rotation matrix and T_i denotes a translation vector.

The workflow of our multi-camera network roughly divides into three phases: (1) Calibrate the camera parameters ( $C_{1} - C_{8}$ ) in the calibration coordinate system $O_{W} - X_{W} Y_{W} Z_{W}$ ; (2) measure and compensate the relative deformation between the left and the right measurement unit in the platform; (3) calculate the motion parameters of the aircraft relative to the platform coordinate system $O_{P} - X_{P} Y_{P} Z_{P}$ .

Principle of the multi-camera network

Camera calibration

Measuring the motion parameters of the aircraft requires a previous calibration of the multi-camera network in order to find the intrinsic parameters K_i, the lens distortion $D i s_{i}$ , and the extrinsic parameters H_i of the cameras. According to the FOV of the cameras, two camera calibration methods are adopted, respectively: (1) If the sufficient control points exist in the FOV, like four cameras $C_{5} - C_{8}$ , camera calibration methods including Tsai’s (at least six noncoplanar control points)²¹ and Shang’s (at least four noncollinear control points and approximate position of the camera)¹⁶ methods, etc., can be directly used in this situation. (2) If the FOV is void, like four cameras $C_{1} - C_{4}$ , a new camera calibration method using the dynamic control points created by a multirotor unmanned aerial vehicle (UAV) is presented. This article introduces this calibration method extensively.

The position of the UAV is measured by the designed positioning measurement system, which is composed of an azimuth camera C_A, a wide-beam LRF, and a GPS synchronization controller, as shown in Figure 2(a). In order to cover a wide area for measuring the UAV’s position, the pose of the azimuth camera is usually adjusted to the state of a large pitch angle, so the projections of the control points which are close to the ground platform are in the lower half of the image, and the control points and the azimuth camera are approximately coplanar. In this condition, Shang et al.’s method¹⁶ is adopted to calibrate the parameters of the azimuth camera, including the intrinsic parameters K_A, lens distortion coefficients $D i s_{A}$ , and extrinsic parameters $(R_{A}, T_{A})$ in the calibration coordinate system. In Shang et al.’s method, the simplified camera model is adapted by setting the longitudinal equivalent focal length equal to the transverse equivalent focal length and neglecting the tangency distortion. First, the image’s center is set as the principle point’s initial position and 0 is set as the distortion coefficients’ initial values. Then, the equivalent focal lengths and the rotation angles’ initial values are calculated by solving linear equations. Finally, based on the initial values of the camera parameters, their precise values are calculated by using bundle adjustment optimization. More details can be found in the work by Shang et al.¹⁶

Figure 2.

Constitute of the positioning measurement system. (a) The positioning measurement system for a UAV. (b) Multirotor UAV and prism. UAV: unmanned aerial vehicle.

A multirotor UAV and a prism are shown in Figure 2(b). The multirotor UAV can make a flexible motion, and the prism is mounted underneath the UAV, which is used to reflect the laser light. The azimuth camera and wide-beam LRF controlled by the GPS synchronization controller are used to obtain the azimuth $(θ_{h}^{k}, θ_{v}^{k})$ and the distance D_k of the UAV at time $k = 1, 2, \dots, m$ , respectively. Considering that the distance between the UAV and the azimuth camera is much greater than the installation distance between the wide-beam LRF and the azimuth camera, the installation distance is negligible. The coordinates of the UAV in the azimuth camera coordinate system $X_{A}^{k}$ can be obtained by

X_{A}^{k} = (\begin{matrix} X_{A}^{k} \\ Y_{A}^{k} \\ Z_{A}^{k} \end{matrix}) = (\begin{matrix} D_{k} \cos θ_{v}^{k} sin θ_{h}^{k} \\ D_{k} sin θ_{v}^{k} \\ D_{k} \cos θ_{v}^{k} \cos θ_{h}^{k} \end{matrix})

According to the extrinsic parameters $(R_{A}, T_{A})$ of the azimuth camera, the coordinates of the UAV in the calibration coordinate system $X_{W}^{k}$ can be expressed as follows

X_{W}^{k} = R_{A}^{- 1} (X_{A}^{k} - T_{A})

Figure 3 shows the idea of the proposed calibration method for the cameras, for which the FOV is void. The method roughly divides into two steps. First, the cameras $C_{1} - C_{4}$ continuously capture images when the UAV is flying at the FOV, the position of the UAV is measured by the positioning measurement system synchronously, thus all the UAV’s positions in the entire flight can be used as the dynamic control points for camera calibration, and the control points are easily distributed in space or in the calibration image rationally. Second, the camera parameters are solved linearly by using the sufficient number of control points and its corresponding image coordinates²² and optimized subsequently by a Levenberg–Marquardt algorithm,²³ which minimizes the reprojection errors of the UAV’s prism. The minimized cost function F is described in the following. Under the assumption of Gaussian pixel noise, the proposed algorithm is a maximum-likelihood estimator

F = \min \sum_{k = 1}^{m} ‖ {\tilde{x}}_{i}^{k} - \tilde{x} (K_{i}, D i s_{i}, R_{i}, T_{i}, X_{W}^{k}) ‖^{2}

Figure 3.

Schematic diagram of the proposed camera calibration method.

where ${\tilde{x}}_{i}^{k}$ is the UAV’s image coordinate and $\tilde{x} (K_{i}, D i s_{i}, R_{i}, T_{i}, X_{W}^{k})$ is the projection of the UAV coordinate $X_{W}^{k}$ in the camera at time k. An initial guess of ${K_{i}, R_{i}, T_{i} | i = 1, 2, 3, 4}$ can be computed by using linear initialization and an initial guess of $D i s_{i}$ can be obtained simply by setting them to 0. The Ceres-Solver library,²⁴ which is an open-source C++ library for modeling and solving large, complicated optimization problems, is used as a framework for the implementation of the minimized cost function F.

Thus, the intrinsic parameters K_i, lens distortion $D i s_{i}$ , and extrinsic parameters H_i of cameras $C_{1} - C_{8}$ are known due to a calibration process.

Relative deformation measurement

The principle of relative deformation measurement is shown in Figure 4. Cameras C₇ and C₈ combined with LRF1 and LRF2 are used to measure the relative deformation between the measurement unit in the left side and the other one in the right side. Each measurement unit is fixedly connected with the corresponding control points, and these control points are set to be seen by the camera across form themselves. Since the platform coordinate system is built on the camera coordinate system of camera C₁ on the left side of the platform, it is assumed that the measurement unit on the left side remains stationary, and the measurement unit on the right side occurs deformation relative to the platform coordinate system. $C_{8}^{'}$ represents the camera coordinate system of C₈ after deformation.

Figure 4.

Schematic of our relative deformation measurement method.

Using the extrinsic parameters H₇, H₈ of cameras C₇, C₈ calculated in the previous camera calibration, the coordinates of the control points that are fixedly connected with the measurement unit can be expressed in the corresponding camera coordinate system

X_{C 8}^{i} = H_{8} X_{W}^{i} (i = 1 \dots n_{7})

X_{C 7}^{j} = H_{7} X_{W}^{j} (j = 1 \dots n_{8})

where n₇ and n₈ are the number of the control points captured by the cameras C₇ and C₈, respectively. $X_{C 8}^{i}$ is the coordinate of the control points that are fixedly connected with the right measurement unit in the camera coordinate system of C₈, and $X_{C 7}^{j}$ is the coordinate of the control points that are fixedly connected with the left measurement unit in the camera coordinate system of C₇. Since the distribution of these control points is close in the corresponding camera coordinate system, $X_{C 8}^{i}$ and $X_{C 7}^{j}$ are approximately constant, even if the reference platform was deformed. The homogeneous transformation H₇₈ between the camera coordinate system of C₇ and C₈ after calibration can be expressed as follows

H_{78} = H_{8} H_{7}^{- 1}

Due to the deformation, the homogeneous transformation $H_{78}^{'}$ between the camera coordinate systems of C₇ and $C_{8}^{'}$ is calculated using the minimum of the sum of the reprojection errors of the control points in two measurement units as objective function, under the distance constraint provided by two LRFs, as follows

{\begin{cases} \begin{array}{l} \min (\sum_{i = 1}^{n_{7}} ‖ x_{7}^{i} - {\hat{x}}_{7}^{i} (K_{7}, D i s_{7}, {(H_{78}')}^{- 1}, X_{C 8}^{i}) ‖^{2} \\ + \sum_{j = 1}^{n_{8}} ‖ x_{8}^{j} - {\hat{x}}_{8}^{j} (K_{8}, D i s_{8}, H_{78}', X_{C 7}^{j}) ‖^{2}) \\ Subject to H_{78}' (3, 4) = d \end{array} \end{cases}

where ${\hat{x}}_{7}^{i} (K_{7}, D i s_{7}, {(H_{78}')}^{- 1}, X_{C 8}^{i})$ is the projection of the control point $X_{C 8}^{i} (i = 1 \dots n_{7})$ in the camera C₇, and $x_{7}^{i}$ is the image coordinate of the corresponding control point in the camera C₇. ${\hat{x}}_{8}^{j} (K_{8}, D i s_{8}, H_{78}', X_{C 7}^{j})$ is the projection of the control point $X_{C 7}^{j} (j = 1 \dots n_{8})$ in the camera $C_{8}^{'}, and x_{8}^{j}$ is the image coordinate of corresponding control point in the camera $C_{8}^{'}$ . d is the average distance measured by two LRFs. An initial guess of $H_{78}^{'}$ is set to H₇₈.

The deformation of the right measurement unit $Δ H_{88}^{'}$ can be expressed as follows

Δ H_{88}' = H_{78}' H_{78}^{- 1}

Then, the angle deformations $(A_{Δ X}, A_{Δ Y}, A_{Δ Z})$ are obtained by matrix decomposition of $Δ R_{88}^{'}$ in the order of $' X Y Z'$ . Similarly, the displacement deformations $(T_{Δ X}, T_{Δ Y}, T_{Δ Z})$ are evaluated by $Δ T_{88}^{'}$ .

Motion parameters measurement

As mentioned, the motion parameters of the aircraft are obtained in the platform coordinate system $O_{C 1} - X_{C 1} Y_{C 1} Z_{C 1}$ , so the extrinsic parameters of the cameras $C_{1} - C_{8}$ in the calibration coordinate system $O_{W} - X_{W} Y_{W} Z_{W}$ should be expressed in the platform coordinate system $O_{C 1} - X_{C 1} Y_{C 1} Z_{C 1}$ , considering the relative deformation as follows

H_{1 i} = H_{i} H_{1}^{- 1} (i = 1, 3, 5, 7)

H_{1 i} = H_{8 i} H_{78}' H_{17} (i = 2, 4, 6, 8)

Based on the matched feature points on the stereo vision images, the position $(X_{P}, Y_{P}, Z_{P})$ and pose $(A_{X}, A_{Y}, A_{Z})$ of the aircraft relative to the platform coordinate system $O_{C 1} - X_{C 1} Y_{C 1} Z_{C 1}$ can be estimated, whether the aircraft is a cooperative target or an uncooperative target.²⁵ In particular, the position and pose of the aircraft is estimated using more than two cameras at the junction of the far-field and the middle-field, and the junction of the middle-field and the near-field. $(A_{X}, A_{Y}, A_{Z})$ denotes the vector of three Euler angles in the order of $' X Y Z'$ that defines the orientation of the aircraft. The velocity V of the aircraft is calculated by the optical polynomial filtering algorithm.²⁶ Since the other motion parameters, distance D and azimuth $(A, E)$ are in the platform coordinate system $O_{C 1} - X_{C 1} Y_{C 1} Z_{C 1}$ , we have that

D = \sqrt{{(X_{P})}^{2} + {(Y_{P})}^{2} + {(Z_{P})}^{2}}

A = \arctan (X_{P} / Z_{P})

E = \arcsin (- Y_{P} / D)

Experiments and results

To verify the performance of our proposed method, the multi-camera network has been set up in a large outdoor field, which is used to estimate the motion parameters of a fixed-wing aircraft during actual landing under normal weather conditions. Since there are not any other existing systems, such as radar and microphone arrays, in the reference platform, we have not compared the performance of our multi-camera network with other passive sensors. Considering the onboard Differential Global Positioning System (DGPS), which is installed in the aircraft, the comparison of the multi-camera network with an onboard DGPS is presented.

Camera calibration experiments

Due to limitations of the outdoor field, the measurement unit in the left side is installed at about 0.5 m from the ground, but the measurement unit in the right side is installed at about 10 m on specially constructed towers. The baseline distance of the two measurement units is about 70 m. The control points’ position coordinates in the calibration coordinate system are measured by a total station (Leica NTS312) first, then the cameras can be calibrated by the presented calibration method. The azimuth camera is calibrated by Shang’s method. The calibration results of azimuth camera C_A in the calibration coordinate system $O_{W} - X_{W} Y_{W} Z_{W}$ are listed in Table 1.

Table 1.

Calibration results of camera C_A.

Calibration parameters	Camera C_A
Principal point (C_x , C_y )/pixels	(1299.2519, 968.3283)
Equivalent focal length (F_x , F_y )/pixels	(3730.8892, 3730.8892)
Lens distortion (Weng’s)	(0.120890, 0.0329, 0.0000,−0.00241, 0.0000)
Euler angle (R_x , R_y , R_z )/(°)	(87.4656, 6.8806, 85.6619)
Translation vector (T_x , T_y , T_z )/m	(−1.2355, −3.5430, 4.0936)
RMS of reprojection errors (du, dv)/pixels	(0.1375, 0.1268)

RMS: root mean square.

All the UAV’s positions in the calibration coordinate system $O_{W} - X_{W} Y_{W} Z_{W}$ are measured by the positioning measurement system. Take the calibration experiments of the cameras C₁ and C₂, for example. The flight range of the UAV is roughly from 600 m to 400 m relative to the reference platform. The flight trajectory of the UAV captured by the camera C₁ is shown in Figure 5. The dynamic control points are distributed in space and the calibration image rationally. The prism is detected using least-squares template matching to a precision of 0.5 pixel, and the measurement accuracy of the wide-beam LRF in the designed positioning measurement system is 0.25 m.

Figure 5.

The flight trajectory of UAV captured by the camera C₁. UAV: unmanned aerial vehicle.

The calibration results of the cameras C₁ and C₂ in the calibration coordinate system $O_{W} - X_{W} Y_{W} Z_{W}$ are listed in Table 2. The root mean square (RMS) of the reprojection errors is about 0.5 pixel. The accuracy of the calibration is mainly affected by the extraction precision of the prism and the range measurement accuracy of the UAV. The range measurement error of the UAV is comparably small for its flight range, so the reprojection errors of the proposed calibration method are basically in accordance with the extraction precision of the prism.

Table 2.

Calibration results of cameras C₁ and C₂.

Calibration parameters	Camera C₁	Camera C₂
Principal point (C_x , C_y )/pixels	(641.6610, 509.2363)	(638.7424, 509.9384)
Equivalent focal length (F_x , F_y )/pixels	(9639.6680, 9640.2181)	(9618.3686,9620.0151)
Lens distortion (Weng’s)	(−0.1505, 0.0035, −0.0055, −0.0039, 0.0010)	(−0.0810, 0.0055, −0.0020, −0.0018, −0.0031)
Euler angle (R_x , R_y , R_z )/(°)	(175.5839, 0.8547, 88.2005)	(172.0519, 0.2391, 87.7839)
Translation vector (T_x , T_y , T_z )/m	(20.7242, 12.2785, 24.8652)	(−26.0245, 3.7762, 75.2521)
RMS of reprojection errors (du, dv)/pixels	(0.4232, 0.3154)	(0.3901, 0.3309)

RMS: root mean square.

Motion parameter measurement experiments

The motion parameter estimation of the aircraft is divided into two steps: the relative deformation between the two measurement units caused by external environmental factors is first measured, then we compensate the deformation and calculate the motion parameters of the aircraft relative to the platform coordinate system. A series of field experiments have been conducted to evaluate the multi-camera network. The landing experiments for a fixed-wing aircraft without artificial makers are detailed in the following. The motion region of the fixed-wing aircraft to be measured ranges from 1000 m to 20 m relative to the platform coordinate system. The aircraft gradually descends from a height of 200 m with a glide-slope angle of 3.0°. The speed of the aircraft is about 30 m/s, and the time of landing is about 35 s.

The measurement results of the angle deformation $(A_{Δ X}, A_{Δ Y}, A_{Δ Z})$ and displacement deformation $(T_{Δ X}, T_{Δ Y}, T_{Δ Z})$ between the left and the right measurement units have been shown in Figures 6 and 7, respectively. Clearly, the deformation of the platform is relatively continuous and smooth. The RMS error of the angle deformation is $(3.15', 3.97', 1.42')$ and the RMS error of the displacement deformation is $(0.46 mm, 1.58 mm, 2.00 mm)$ . The experimental testing results show that the relative deformation of the platform is inevitable and cannot be ignored in field experiments.

Figure 6.

Measurement results of angle deformation.

Figure 7.

Measurement results of displacement deformation.

In the landing experiments, it is necessary to determine the distance, azimuth, velocity, and pose of the aircraft with respect to the reference platform. The accuracy of the proposed multi-camera network is evaluated using an onboard DGPS system. Distance measurement errors of the fixed-wing aircraft between the DGPS and the multi-camera network while landing are shown in Figure 8. According to the results, the multi-camera network performs as expected with the distance error decreasing when the aircraft approaches, and the distance error is significantly decreased in the process of entering the near-field from middle-field, because the calibration accuracy of the cameras in near-field ( $C_{5} - C_{6}$ ) is higher than the cameras in middle-field and far-field ( $C_{1} - C_{4}$ ), in view of adopting different calibration methods with the control points measured by a total station and dynamic control points created by a UAV, respectively. The RMS error of the distance without correction of the platform’s relative deformation is 5.02 m, and the RMS error of the distance after correction of the platform’s relative deformation is 1.32 m. The comparison of distance errors shows that the measurement results after correction of the platform’s relative deformation are much better than without correction. Therefore, the platform deformation is a crucial error source for measuring data of the multi-camera network, and the proposed solution provided by this article is essential and effective.

Figure 8.

Distance measurement errors between the DGPS and the multi-camera network. DGPS: Differential Global Positioning System.

The measurement results of the aircraft’s azimuth, velocity, and pose relative to the platform coordinate system during the landing have been shown in Figures 9 and 10, respectively. Since the pose of the aircraft has not been measured by an onboard DGPS, the accuracy of the pose cannot be assessed, but the measurement results are quite smooth and consistent with the IMU information recorded in the aircraft. The results of our experiments show that the multi-camera network can be used to obtain the motion parameters of the aircraft over a large range with a fairly high accuracy.

Figure 9.

The azimuth and velocity of the aircraft relative to the platform coordinate system.

Figure 10.

The pose of the aircraft relative to the platform coordinate system.

Error analysis and discussion

Although promising results have been obtained under normal weather conditions, there are some factors that could influence the measurement accuracy of the proposed multi-camera network. The main factors are divided into three parts: camera calibration accuracy, relative deformation measurement accuracy, and environmental factors.

The accuracy of camera calibration is mainly influenced by the measurement accuracy of the control points. According to the view field of the cameras, the coordinates of the control points are measured by a total station and the designed positioning measurement system, respectively. The main factors that could influence the designed positioning measurement system are the installation distance between the wide-beam LRF and the azimuth camera, the ranging error of wide-beam LRF, and the calibration error of the azimuth camera. Considering that the distance between the UAV and the azimuth camera is much greater than the installation distance between the wide-beam LRF and the azimuth camera, the installation distance is negligible. The ranging error of the wide-beam LRF is 0.25 m. The azimuth camera C_A is calibrated by Shang’s method using the control points measured by total station. Compared with the accuracy of the total station, the accuracy of the control points measured by the designed positioning measurement system is lower. Therefore, the calibration accuracy of cameras in middle-field and far-field ( $C_{1} - C_{4}$ ) is lower than the cameras in near-field ( $C_{5} - C_{6}$ ). The designed positioning measurement system for the multirotor UAV is desirable to be further improved. In addition, other calibration means, such as launching fireworks arrows, or emitting visible lasers into the air, can be attempted.

The accuracy of relative deformation measurement is mainly influenced by the installation error between the LRF and the camera, the ranging error of LRF, and the calibration accuracy of the cameras $C_{7} - C_{8}$ . The LRF and the camera have been adjusted in the laboratory in advance. The measurement accuracy of the LRF is 1.0 mm. The z-component of translation measurements, which is along the camera’s optical axis, can be directly determined by the laser ranging of LRF. The calibration accuracy of cameras $C_{7} - C_{8}$ is consistent with the cameras in near-field ( $C_{5} - C_{6}$ ), because all those control points used for calibration are manually measured by total station. The measurement results show that cameras combined with LRFs are useful to measure the relative deformation of the platform.

Due to the visible light camera, the performance of the proposed multi-camera network is inevitability influenced by environmental restrictions, such as fog, heavy storms, temperature, air turbulence, and ambient lighting.²⁷ These factors are all we need to solve or reduce the impact. For temperature variations, we have installed the temperature control system which ensures that the cameras operate under relatively constant temperatures. The temperature control system is helpful for reducing the influence of temperature variations in outdoor fields. The performance of the proposed multi-camera network is poor, and even cannot work under heavy weather conditions, for example, fog at sea, heavy storms, or ambient lighting, because the image features of the aircraft cannot be correctly detected and tracked. In this condition, some light-emitting devices have to be installed on the aircraft in advance, which highlight the key positions of an aircraft, like nose and wings of a plane.¹⁴ But the artificial makers will increase the load of aircraft. The proposed multi-camera network has not considered the influence of air turbulence under normal weather conditions. But the compensation of air turbulence is necessary in condition of high temperature and strong wind in large outdoor fields. The implementation of air turbulence compensation methods is the focus of our future research.

Conclusion

In this article, a multi-camera network is proposed for measuring an aircraft’s motion parameters relative to a reference platform. A calibration method for the camera for which the FOV is void is presented, by using the dynamic control points created by a multirotor UAV, and the cameras combined with LRFs are adopted to measure the relative deformation between the measurement unit on the left side and the other one on the right side. To verify the proposed multi-camera network, its performance has been tested when a fixed-wing aircraft without artificial makers is landing in a large outdoor field, and its accuracy is evaluated using an onboard DGPS system. The experimental results show that the multi-camera network is precise, robust, and highly dynamic under normal weather conditions. Compared to a current setup for measurement systems like radar and GPS, it can meet the requirements of accuracy, but has the character of measuring large sets of points and has no special requirements on the structure of the aircraft. We can get the conclusion that the proposed multi-camera network can effectively establish a new type of measurement means for an aircraft without artificial markers. It can implement the accurate landing of an aircraft without increasing its load. Certainly, for the cases that the aircrafts are taking off or are launching from the reference platform, the proposed multi-camera network can also be used to measure the aircraft’s motion parameters without modifications in the range of 20–1000 m, as long as the aircraft appears in the FOV of the multi-camera network.

Our future work will focus on the implementation of other calibration means, such as launching fireworks arrows, or emitting visible lasers into the air. Furthermore, the implementation of air turbulence compensation methods in large outdoor fields is also the focus of our research.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research was supported by the National Natural Science Foundation of China (No.11172323 and No.51509251) and National Key Scientific Instrument and Equipment Development Project (No.2013YQ140517).

References

Aghajan

Cavallaro

. Multi-camera networks: principles and applications. Pittsburgh: Academic Press, 2009, pp. 3–4.

Kurnaz

Çetin

. Autonomous navigation and landing tasks for fixed wing small unmanned aerial vehicles. Acta Polym Hung 2010; 7(1): 87–102.

Shakernia

Vidal

Sharp

. Multiple view motion estimation and control for landing an unmanned aerial vehicle. In: IEEE International Conference on Robotics and Automation (eds William

Anthony

George

), Washington, DC, USA, 11–15 May 2002, Paper no. Vol. 3, pp. 2793–2798. Washington: IEEE.

Yakimenko

Kaminer

Lentz

. Unmanned aircraft navigation for shipboard landing using infrared vision. IEEE Trans Aerospace Electron Syst 2002; 38(4): 1181–1200.

Wenzel

Masselli

Zell

. Automatic take off, tracking and landing of a miniature UAV on a moving carrier vehicle. J Intell Robot Syst 2011; 61(1–4): 221–238.

Ghyzel

. Vision-based navigation for autonomous landing of unmanned aerial vehicles. PhD Thesis, Naval Postgraduate School, USA, 2000.

Altuğ

Ostrowski

Taylor

. Control of a quadrotor helicopter using dual camera visual feedback. Int J Robot Res 2005; 24(5): 329–341.

THALES. SMART-L EWC, https://www.thalesgroup.com/en/worldwide/defence/smart-l-ewc (accessed 10 September 2016).

THALES. VARIANT – Lightweight 2D short to medium-range surveillance radar, https://www.thalesgroup.com/en/variant-lightweight-2d-short-medium-range-surveillance-radar (accessed 10 September 2016).

10.

Caronna

Roselli

Testa

. Acoustic system for aircraft detection and tracking based on passive microphones arrays. J Acoust Soc Am 2004; 116(4): 2633–2646.

11.

ZhuXin

. Research on technology for motion parameters measurement using linear array optical image and its applications. PhD Thesis, National University of Defense Technology, China, 2012.

12.

Pebrianti

Kendoul

Azrad

. Autonomous hovering and landing of a quad-rotor micro aerial vehicle by means of on ground stereo vision system. J Syst Des Dynam 2010; 4:269–284.

13.

Zhang

Zhu

Yuan

. A universal and flexible theodolite-camera system for making accurate measurements over large volumes. Opt Lasers Eng 2012; 50(11): 1611–1620.

14.

Zhou

Lei

. Ground-based videometric method and system for the real-time measurement of aircraft landing track. Appl Mech Mater Trans Tech Publ 2014; 635: 824–831.

15.

Mason

. Expert system-based design of close-range photogrammetric networks. ISPRS J Photogr Remote Sens 1995; 50(5): 13–24.

16.

Shang

Sun

Yang

. A camera calibration method for large field optical measurement. Optik-Int J Light Elec Opt 2013; 124(24): 6553–6558.

17.

Tamura

Matsui

Pagnini

. Measurement of wind-induced response of buildings using RTK-GPS. J Wind Eng Industr Aerodynam 2002; 90(12): 1783–1793.

18.

White

Take

Bolton

. Soil deformation measurement using particle image velocimetry (PIV) and photogrammetry. Geotechnique 2003; 53(7): 619–632.

19.

Lee

McClure

. Elastoplastic large deformation analysis of a lattice steel tower structure and comparison with full-scale tests. J Constr Steel Res 2007; 63(5): 709–717.

20.

Liu

Sun

Zhang

. Hull deformation measurement for spacecraft TT&C ship by Photogrammetry. Sci China Tech Sci 2015; 58(8): 1339–1347.

21.

Tsai

. A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE J Robot Auto 1987; 3(4): 323–344.

22.

Abdel-Aziz

Karara

Hauck

. Direct linear transformation from comparator coordinates into object space coordinates in close-range photogrammetry. Photogr Eng Remote Sens 2015; 81(2): 103–107.

23.

Ranganathan

. The Levenberg–Marquardt algorithm. Tutorial LM Algorithm 2004; 11(1): 101–110.

24.

Google. Ceres Solver, http://ceres-solver.org/ (accessed 10 September 2016).

25.

Shang

Yang

. Researches on vision-based pose measurements for space targets. PhD Thesis, National University of Defense Technology, China, 2006.

26.

Qifeng

Yang

. Videometrics: Principles and Researches. Beijing: Science Process, 2009, pp. 183–185.

27.

Cuypers

Van Gestel

Voet

. Optical measurement techniques for mobile and large-scale dimensional metrology. Opt Lasers Eng 2009; 47(3): 292–300.

Multi-camera networks for motion parameter estimation of an aircraft

Abstract

Keywords

Introduction

Architecture of our multi-camera network

Principle of the multi-camera network

Camera calibration

Relative deformation measurement

Motion parameters measurement

Experiments and results

Camera calibration experiments

Motion parameter measurement experiments

Error analysis and discussion

Conclusion

Footnotes

Declaration of conflicting interests

Funding

References