Quadcopter localization and health monitoring method based on multiple virtual silhouette sensor integration

Abstract

With the widespread deployment of quadcopters, the flight safety issue attracts increasingly public and academic attentions. This article presents a quadcopter flight regime extraction algorithm for quadcopter localization and health monitoring using imageries captured by general purpose monocular cameras. First, contour information is extracted from quadcopter shadows on the ground. In order to better illustrate the three-dimensional silhouette information contained in shadow contour on the ground, a virtual sensor named Shadow Projection Tunnel is designed. Then, multiple Shadow Projection Tunnels are generated according to the extracted silhouette information and corresponding light source positions. Finally, three-dimensional quadcopter positions and flight regimes are extracted based on the aggregation between multiple Shadow Projection Tunnels. The proposed method is validated to be accurate and efficient in monitoring quadcopter position and flight regimes based on the comparative analyses. In comparison with traditional quadcopter health monitoring methods, the proposed method has advantages on deployment convenience, system robustness, precision expandability, and scenario adaptability, making it an ideal solution for quadcopter monitoring in outdoor scenarios.

Introduction

The flourishing development of quadcopter platform contributes enormously to promising fields including drone light shows, intelligent surveillance, and virtual reality. Additionally, the portability and continence of quadcopters reshape traditional industries including aerial photography, surveying, and mapping. However, reports on accidents caused by quadcopter crashes regularly appear on major media.

The health of quadcopters effects not only the task fulfillment but also influences the security of ground personals. Thus, worldwide researchers attach great importance to the real-time health monitoring technology for quadcopter, including quadcopter localization and flight regime extraction. Signals from multiple sensors can be adopted in quadcopter health monitoring mechanism, including vibration,^1–3 acceleration,^4,5 and acoustical signals. Specifically, various methods have been proposed to enhance the monitoring technology for quadcopters based on quadcopter onboard sensors or external sensor networks.

For airborne sensor information–based methods, onboard sensors including inertial measurement unit (IMU), onboard optical flow camera, acoustic proximity sensor, and Global Positioning System (GPS) receiver provide position information for timely correction. The advantage of airborne sensor information–based methods is the low latency of quadcopter localization. However, the efficiency and reliability of airborne sensor information–based methods are affected by the extra power consumption and potential parameter drift from the applied onboard sensors.

In the second category of methods, external sensors including Carnegie Mellon University camera (CMU-cam), infrared (IR) camera, and ultra-wideband (UWB) transceivers are deployed in the form of sensor network to fulfill the real-time quadcopter monitoring tasks. The advantage of external sensor network–based methods is the high accuracy of spatial localization. Nevertheless, the complexity of the external sensor system and the unavailability of flight regimes raise the bar of deploying external sensors in the quadcopter localization method.

Instead of deploying algorithms on current onboard or external sensor-based platform, this work pays attention to the commonest projection of quadcopter on the ground. Shadow is the projection of an opaque object on a certain surface, containing single-view silhouette information of the opaque object. Various methods have been developed to extract three-dimensional (3D) information from shadow, focusing on the recovery of mesh model^6,7 or point clouds. ⁸ In this article, a novel quadcopter monitoring method based on shadow is proposed. The proposed method is capable of localizing the quadcopter position and acquiring the flight regimes simultaneously, offering a high-performance quadcopter health monitoring system at a low deployment complexity.

The contributions of the proposed method are listed below:

1. Simplified monitoring instrument. Flight regimes including Euler angles and quaternion of a quadcopter are extracted from each single frame captured by a monocular camera.

2. Improved deployment convenience. Only one monocular camera and an embedded system are required to fulfill the quadcopter monitoring task.

3. Expandable monitoring precision. The precision of quadcopter localization and flight regime extraction can be elevated with future camera models offering higher image resolutions.

4. Interference-resistant quadcopter monitoring. Without the wireless interconnection between multiple sensors and the ground station, the robustness of quadcopter health monitoring system is enhanced.

The remaining sections of this article are organized as follows: in section “Background,” the previous quadcopter monitoring methods are analyzed and compared. In section “The proposed method,” the basic theory for shadow-based single-frame quadcopter localization is introduced first, followed by the quaternion extraction method. In section “Quadcopter flight status monitoring,” the precision of quadcopter localization and flight regime extraction results based on the proposed method is evaluated in comparison with the onboard IMU statistics.

Background

Inspired by the increasing concerns of quadcopter flight safety, the real-time quadcopter monitoring technology is developed synchronously with new quadcopter hardware designs. Two prevalent streams of quadcopter monitoring algorithms are widely used in the academic research and industrial application, including airborne sensor information–based methods and external sensor information–based methods.

Airborne sensor information–based methods

In order to take full advantage of quadcopter onboard sensors, airborne sensor information–based methods are the earliest developed quadcopter monitoring technologies. According to the output of different onboard sensors, airborne sensor information can be acquired from the GPS receiver, IMU parameter, and optical flow camera, which provides underlying data of quadcopter location and flight regimes.

GPS-based geo-fencing method

Dill et al. ⁹ designed the safeguard system to monitor and predict position of unmanned aerial vehicle (UAV) based on the GPS signal. Additionally, the safeguard system is designed to predict nonconformance against the preset operating constraints. The geo-fencing setup ¹⁰ in safeguard system not only establishes stay-in and stay-out regions but also set up mission-oriented constraint including path deviation, speed, and altitude limitations. The effectiveness and functionality of a geo-fencing system heavily rely on stable GPS signal or Global Navigation Satellite System (GNSS) signal. However, these satellite signals are broadcasting through space-based radio frequency, which can be frequently affected by signal attenuation and shadowing incidents. Restricted by the accuracy of civil grade GPS signal, the positioning accuracy of geo-fencing methods stays above 10 m.

Optical flow method

Grabe et al. ¹¹ deployed the onboard optical flow camera and onboard computation resources in a closed-loop control system for a quadcopter. The optical flow method recovers continuous holography constraint from the optical flow information, providing effective fall back routine for the controlled quadcopter. The advantage of this method is the simplification of deployed equipment resources. The optical flow method relies on only a single monocular onboard camera and onboard computer system, providing a low-cost indoor quadcopter positioning solution. The positioning accuracy of the optical flow method largely depends on the texture of ground surface. In an ideal experimental environment, the positioning accuracy remains at the sub-meter level.

Acoustic proximity sensor

The acoustic proximity sensor is applied in the UAV sensing and avoiding method proposed by Finn and Franklin. ¹² Based on the onboard acoustic sensors fulfilled by the microphone array, a 360-degree field of view sense and avoid capability are acquired by the quadcopter in the experiment. Additionally, based on the view sense, the proposed method empowers the quadcopter to take quick reaction measures to avoid predicted threads in the view sense. The accuracy of the acoustic proximity sensor–based method can be frequently affected by noise interference, restricting the average positioning accuracy above 5 m.

External sensor information–based methods

The airborne sensor information–based methods provide real-time access to quadcopter position and flight regimes. However, extra power consumption and potential parameter drift affect the robustness of quadcopter monitoring. In order to enhance the monitoring robustness and accuracy, external sensors are introduced in the quadcopter monitoring system. Based on advantages including direct spatial position localization, multiple view angles, and independent power supplement, external sensors including IR cameras, CMU-cam, and UWB transceivers are deployed in the form of sensor network for real-time quadcopter monitoring.

IR camera network

IR tracking camera networks have unparalleled advantage in space coordinate system localization accuracy. Thus, multiple IR camera network platforms have been developed, including OptiTrack motion capture cameras and Intel Real Sense cameras. Based on the IR tracking platform, Sadeghzadeh et al. ¹³ proposed an active fault-tolerant control (AFTC) algorithm for a quadcopter. The AFTC algorithm is deployed based on a fault detection and diagnosis (FDD) block receiving visual feedback from IR camera networks. The high localization accuracy provided by IR camera networks contributes to both real-time fault-tolerant control and proportional–integral–derivative (PID) controller training. Based on the highly efficient IR optical sensors, the IR camera network can provide the state-of-the-art positioning accuracy below 10 cm. However, the high price threshold of deploying IR tracking camera networks depresses the application in small research institutions. Additionally, the limited detection distance restricts the application of IR camera network in compact indoor scenarios.

CMU-cam network

In contrast to the high price threshold of IR tracking camera networks, CMU-cam network is a cost-efficient solution for quadcopter localization tasks. Based on the compact integration of monocular camera and microcontroller, the CMU-cam is capable of tracking dedicated color block in the real-time video stream. Equipped with two orthogonal CMU-cams, a 3D spatial localization system can be established. Yang and Wang ¹⁴ proposed a quadcopter vision control system based on a CMU-cam network, providing sufficiently quick and accurate quadcopter localization information. Noticeably, the accuracy of CMU-cam-based quadcopter localization can be frequently affected by occlusion or illumination variations due to the orthogonal angles of two CMU-cams. The highest positioning accuracy of the CMU-cam network–based method can reach 20 cm.

UWB network

In order to overcome the potential interference brought by occlusion in localization methods powered by IR camera network or CMU-cam network, an UWB transceiver network is designed to support 3D quadcopter localization. The UWB network adopts short signal pulses to establish datalinks between different transceivers. ¹⁵ The datalinks operate at a wideband of frequencies covering bandwidth ranging from 3.1 to 10.6 GHz, distinguishing the reflected signals from submitted signals. Based on the UWB networks, the accurate indoor 3D localization for mini quadcopters is achieved^16–18 with the equivalence accuracy of IR camera network–based methods. However, more than four UWB transceivers are required to establish an effective UWB network, resulting in the complex and expensive wiring requirement.

Analyses of current quadcopter localization methods

For current quadcopter localization methods belonging to two different streams, a perfect balance between localization accuracy, system robustness, and deployment convenience is hard to reach. As shown in Table 1, airborne sensor information–based methods achieve high deployment convenience at the expense of system robustness, positioning accuracy, and extra onboard power load. Due to the limitation of positioning reference, the positioning accuracy of airborne sensor information–based methods is limited to sub-meter level.

Table 1.

Comparison between airborne sensor information–based quadcopter monitoring methods.

Methods	Deployment convenience	Flight regimes	Positioning accuracy
Geo-fencing	High	Euler angles	> 10 m
Optical flow	Middle	NA	> 50 cm
Acoustic sensor	Middle	NA	> 5 m

On the contrary, external sensor information–based methods achieve high monitoring accuracy and system robustness at the expense of system deployment convenience as shown in Table 2. The complicate building procedure of the network between multiple sensors and the computing terminal obstructs the outdoor deployment of external sensor information–based methods in open fields.

Table 2.

Comparison between external sensor information–based quadcopter monitoring methods.

Methods	Deployment convenience	Flight regimes	Positioning accuracy
IR camera	Low	> 6	< 10 cm
CMU-cam	Middle	> 2	> 20 cm
UWB network	Low	> 4	< 10 cm

IR: infrared; UWB: ultra-wideband.

In order to achieve the balance between monitoring accuracy, system robustness, and deployment convenience, a highly reliable and accessible feature is required to design the refined method. More specifically, a favorable feature must provide multiple view angle quadcopter information using the least number of external sensors.

The proposed method

Instead of refining current quadcopter monitoring platforms based on traditional features, a new feature is analyzed, transformed, and applied in quadcopter monitoring. The quadcopter shadow projected on the ground is concerned in this work because of its high reliability and accessibility.

The basic theory of quadcopter shadow information–based quadcopter monitoring method is presented in this section, laying the foundation for 3D quadcopter localization and flight regime extraction. The overall procedure of the proposed method is presented in Figure 1. In order to illustrate the proposed methods clearly, the concept of Shadow Projection Tunnel ( $\mathrm{SPT}$ ) is introduced first. Then, the basic theory of 3D quadcopter localization based on integration of $\mathrm{SPT}$ s is illustrated. Finally, the advanced algorithm designed to extract the quadcopter flight regimes based on the aggregation of multiple $\mathrm{SPT}$ s is illustrated.

Figure 1.

Flowchart of the proposed method.

This work is motivated by the procedure of taking silhouette photo. During this procedure, the target object blocks a part of light from reaching film or sensor, leaving a body sketch on the silhouette photo. Quadcopter shadow on the ground, from the aspect of silhouette imaging, can be regarded as a silhouette photo of the quadcopter on a special giant film. The ground surface plays the role of image sensor. For captured frames containing quadcopter shadows, each shadow on the ground can provide an extra quadcopter contour information from an unique observation angle view, which plays the role of virtual monochrome sensor. Multiple shadows generated by different light sources can effectively contribute to a virtual monochrome sensor network.

This article mainly focuses on the extraction and aggregation of the extra silhouette information from spatial discrete quadcopter shadows on the ground, aiming to extract 3D quadcopter location and flight regimes from shadow information captured by a single monocular camera in both indoor and outdoor scenarios.

Virtual silhouette sensor: SPT

Shadow is an area where light beams emitted from a certain light source cannot reach. When light beams are blocked by an opaque object, the 3D space behind the opaque object is occupied by shadow. In order to describe the shadow occupied 3D space, a new virtual sensor named $\mathrm{SPT}$ is designed. In a single light source scenario, an $\mathrm{SPT}$ contains two 3D spatial areas: the 3D space occupied by light beams projected on the object and the corresponding 3D space of shadow area behind the object. In correspondence to the concept of light path, $\mathrm{SPT}$ is defined as the 3D space through which shadows are projected.

As shown in Figure 2(a), the 3D space included in $\mathrm{SPT}$ contains all light beams and their corresponding extension cords connecting the light source and the cross section of shadow on a surface. Object shadow on the ground is the ground cross section of $\mathrm{SPT}$ . From the aspect of silhouette photography, it can be regarded as a silhouette photo imaging on the ground surface.^19,20 In other words, each shadow on the ground contains the contour information of an object captured from the unique imaginative position of a light source. ⁸

Figure 2.

Example of SPT: (a) a scene generating SPT and (b) a simulated SPT in point clouds.

Each $\mathrm{SPT}$ can be divided into three sections based on different illumination conditions, including the light section, the opaque section, and the dark section as is shown in Figure 2(b):

1. The light section $\mathrm{Se}{c}_{\mathrm{light}}$ contains the 3D path of all light beams $l_{<S,O>}$ initiated from the light source $S$ and blocked by the target object $O$ .

2. The opaque section $\mathrm{Se}{c}_{\mathrm{opa}}$ refers to the 3D space occupied by the opaque target object $O$ , blocking light beams $l_{<S,O>}$ .

3. The dark section $\mathrm{Se}{c}_{\mathrm{dark}}$ contains all the line extensions of light beams $l_{<s,o>}$ blocked by the target object. $\mathrm{Se}{c}_{\mathrm{dark}}$ occupies the area behind the targeted object where light beams cannot reach. All the line extensions in dark section are terminated at the cross section of shadow on an opaque surface.

It is the shape of cross section $C_{\mathrm{op}}$ and the position of light source $S$ contribute to the shape of shadow area in the dark section $C_{\mathrm{dark}}$ . As shown in Figure 3(a), while the cross section of object shadow is included in an $\mathrm{SPT}$ , the object itself must be included in the opaque section of $\mathrm{SPT}$ as well. $\mathrm{SPT}$ not only silhouettes the outline of an object but also indicates the potential 3D position of the targeted object. Acknowledged with potential position zone presented by $\mathrm{SPT}$ , it is possible to calculate the position of the object based on auxiliary information.

Figure 3.

Quadcopter localization method based on $\mathrm{SPT}$ intersection.

In order to illustrate the optical feature of $\mathrm{SPT}$ , we analyzed $\mathrm{SPT}$ from a microscopic view. Based on the introduction above, three basic features of $\mathrm{SPT}$ can be proved as is shown in equation (1).

1. Every point $p_{\mathrm{sh}}$ in the cross-sectional area ${\Pi }_{\mathrm{sh}}$ is the projection of its corresponding point $p_{\mathrm{op}}$ . Point $p_{\mathrm{op}}$ belongs to the opaque section.

2. Function $f_O(p_{\mathrm{op}})$ is defined as the correspondence from $p_{\mathrm{op}}$ to $p_{\mathrm{sh}}$ , which is a one-to-one correspondence. Thus, every illuminated point $p_{\mathrm{opi}}$ in opaque section must have its corresponding point $p_{\mathrm{sh}}$ in shadow area

$$\begin{array}{c}\begin{array}{c}{p}_{\mathrm{op}}\to p_{\mathrm{sh}},p_{\mathrm{sh}}\in {\Pi }_{\mathrm{sh}}\hfill \\ p_{\mathrm{sh}}=f_O(p_{\mathrm{op}})=\overrightarrow{O{p}_{\mathrm{op}}}•\frac{d(p_{\mathrm{op}},{\Pi }_{\mathrm{sh}})}{|\overrightarrow{O{p}_{\mathrm{op}}}|}+p_{\mathrm{op}}\hfill \end{array}\hfill \end{array}$$

(1)

• 3. The physical intention of function $f_O(p_{\mathrm{op}})$ is a light beam $L$ blocked by the opaque section at $p_{\mathrm{opi}}$ . $L$ must have its extension cord behind the opaque section, pointing at the relative point $p_{\mathrm{sh}}$ in ${\Pi }_{\mathrm{sh}}$

$$\begin{array}{c}{p}_{\mathrm{op}1}\to p_{\mathrm{sh}1},p_{\mathrm{op}2}\to p_{\mathrm{sh}2}\hfill \\ \multicolumn{1}{c}{p_{\mathrm{op}1}\ne p_{\mathrm{op}2},p_{\mathrm{sh}1}\ne p_{\mathrm{sh}2}}\end{array}$$

(2)

Quadcopter localization method based on multiple SPT integration

In a multi-light-source scenario, multiple $\mathrm{SPT}$ s of a quadcopter are generated. Each $\mathrm{SPT}$ of a quadcopter is decided by two factors:

1. The shape of quadcopter contour.

2. 3D spatial relationship between the light source and the quadcopter.

Reproducing 3D information from any single shadow information on the ground is impossible because each single quadcopter shadow on the ground is restricted in a two-dimensional (2D) plate. However, based on multiple shadows generated from different light sources, multiple quadcopter contour information captured from different 3D spatial view angles is available. The extracted multi-angle quadcopter contour information allows the reproduction of 3D information.

3D voxel model of an object can be recovered from shadows generated by an annular set of light sources. ⁸ However, out of the laboratory environment, accurate manual arrangement of light source positions is elusive. Thus, the proposed quadcopter localization method is designed to be adaptive to the posterior combination of random light source positions. With two or more $\mathrm{SPT}$ s generated by different light sources, our method is capable of simulating 3D quadcopter position based on the intersection of multiple $\mathrm{SPT}$ s.

In a multi-light-source scenario shown in Figure 3, $S{h}_a$ and $S{h}_b$ are shadows of a quadcopter fuselage $M$ , generated by light sources $S_a$ and $S_b$ , respectively. The whole scenario is captured by camera $C$ . For quadcopters hovering over 1m above the ground surface, the overlapping section between shadows generated by different light sources can be avoided. In the captured frame, quadcopter fuselage $M$ and shadows $S{h}_a$ and $S{h}_b$ are captured as $M_c$ , $S{h}_{a}c$ , and $S{h}_{b}c$ , respectively. With the captured frame, 3D position of the fuselage $M$ can be extracted based on the following three steps.

Projection transformation reproduction

3D scenario reproduction

Based on affine transformation, ²⁰ original silhouette information $S{h}_a$ and $S{h}_b$ are reproduced from $S{h}_{a}c$ and $S{h}_{b}c$ . The positions of light sources $S_a$ and $S_b$ are simulated in the simulated 3D scenario.

The shadow of quadcopter is projected on the ground, containing precise quadcopter silhouette information. In the 3D scenario reproduction procedure, the original 2D quadcopter shadow silhouette is established based on the corresponding quadcopter shadow image $S{h}_c$ captured by a top-view camera $C$ .

Since meticulous transform between quadcopter shadow coordinates on ground plane and corresponding pixels on the camera sensor plane is required in the 3D reproduction procedure, a perspective transformation is introduced first. For normal open field ground, the surface levels of different areas are normally discontinuous because of different reasons including patching engineering and function segmentation. Traditional perspective transformation method deploys plane to plane global transformation. In this work, a block-matrix-based perspective transformation set is designed to overcome the inaccuracy brought by the surface level discontinuity.

For each projection transformation block, checkerboard markers are set up at the four corners of each unit block on the ground. Based on the matrix established by placed checkerboard markers, a 2D coordinate system $P{c}_S$ is available on the ground surface. For each unit block on the ground, the projection transformation between ground plane coordinates $[u,v]$ and camera sensor pixel coordinates $[x\prime ,y\prime ]$ can be presented as

$$\begin{array}{c}[x\prime ,y\prime ,w\prime ]=[u,v,w]·P\hfill \\ \multicolumn{1}{c}{P=\left[\begin{array}{ccc}{p}_{11}\hfill & p_{12}\hfill & p_{13}\hfill \\ \multicolumn{1}{c}{p_{21}}& p_{22}\hfill & p_{23}\hfill \\ \multicolumn{1}{c}{p_{31}}& p_{32}\hfill & p_{33}\hfill \end{array}\right]}\end{array}$$

(3)

The projection transformation parameter matrix $P$ is made up of three major parameter groups, including the linear transformation parameter group, the translation transformation group, and perspective transformation group. Linear transformation parameter group is composed of $p_{11}$ , $p_{12}$ , $p_{21}$ , and $p_{22}$ . Translation transformation parameter group is composed of $p_{31}$ and $p_{32}$ . Perspective transformation parameter group is composed of $p_{13}$ and $p_{23}$ .

For each square unit in the checkerboard, four corner markers are marked as $\{(u_i,v_i),i=1,2,3,4\}$ , and their corresponding sensor plane coordinates are $\{(x{\prime }_i,y{\prime }_i),i=1,2,3,4\}$ . Based on four pairs of coordinates, the projection transformation parameter matrix can be extracted through solving the equation below

$$\begin{array}{c}\begin{array}{c}\frac{\begin{array}{c}x\prime \hfill \end{array}}{\begin{array}{c}w\prime \hfill \end{array}}=\frac{\begin{array}{c}{p}_{11}u+p_{21}v+p_{31}\hfill \end{array}}{\begin{array}{c}{p}_{13}u+p_{23}v+p_{33}\hfill \end{array}}\hfill \\ \multicolumn{1}{c}{\frac{\begin{array}{c}y\prime \hfill \end{array}}{\begin{array}{c}w\prime \hfill \end{array}}=\frac{\begin{array}{c}{p}_{12}u+p_{22}v+p_{32}\hfill \end{array}}{\begin{array}{c}{p}_{13}u+p_{23}v+p_{33}\hfill \end{array}}}\end{array}\hfill \end{array}$$

(4)

Six auxiliary variables are introduced in the following equations to simplify the extraction procedure

$$\begin{array}{c}\Delta x_1=x_2-x_3\hfill \\ \multicolumn{1}{c}{\Delta x_2=x_4-x_3}\\ \multicolumn{1}{c}{\Delta x_3=x_1-x_2+x_3-x_4}\\ \multicolumn{1}{c}{\Delta y_1=y_2-y_3}\\ \multicolumn{1}{c}{\Delta y_2=y_4-y_3}\\ \multicolumn{1}{c}{\Delta y_3=y_1-y_2+y_3-y_4}\end{array}$$

(5)

Each parameter in the projection transformation parameter matrix can be successfully extracted based on equation (4) and auxiliary variables

$$\begin{array}{c}{p}_{12}=y_2-y_1+p_{13}{y}_2;p_{11}=x_2-x_1+p_{12}\hfill \\ \multicolumn{1}{c}{p_{21}=x_4-x_1+p_{12}{x}_3;p_{22}=y_4-y_1+p_{23}{y}_4}\\ \multicolumn{1}{c}{p_{23}=y_1;p_{31}=x_1}\\ \multicolumn{1}{c}{p_{12}=\frac{\left|\begin{array}{cc}\Delta x_1\hfill & \Delta x_3\hfill \\ \multicolumn{1}{c}{\Delta y_1}& \Delta y_3\hfill \end{array}\right|}{\left|\begin{array}{cc}\Delta x_1\hfill & \Delta x_2\hfill \\ \multicolumn{1}{c}{\Delta y_1}& \Delta y_2\hfill \end{array}\right|};p_{13}=\frac{\left|\begin{array}{cc}\Delta x_3\hfill & \Delta x_2\hfill \\ \multicolumn{1}{c}{\Delta y_3}& \Delta y_2\hfill \end{array}\right|}{\left|\begin{array}{cc}\Delta x_1\hfill & \Delta x_2\hfill \\ \multicolumn{1}{c}{\Delta y_1}& \Delta y_2\hfill \end{array}\right|}}\end{array}$$

(6)

For each unit block area on the ground, the 2D real-world coordinates $(u,v)$ can be calculated based on the imaging coordinates $(x\prime ,y\prime )\in S{h}_{i}c$ and the projection transformation coordinate matrix $P$ .

Based on a set of precise projection transformations on multiple unit block areas, each real-world quadcopter shadow silhouette $S{h}_i$ on the ground plane can be extracted through the transformation based on the imaging pixel coordinates $S{h}_{i}c$ . The overall transformation can be defined as

$$\begin{array}{c}S{h}_i=F(S{h}_{i}c,P)\hfill \end{array}$$

(7)

SPT extraction

While any light beam from light source $S_a$ is blocked by the certain part $M_{\mathrm{part}}$ of fuselage $M$ , shadow is produced on the ground. Thus, the direction of blocked light beam $l_a$ leads to point $S{p}_a$ in shadow $S{h}_{a\_\mathrm{part}}$ , going through fuselage body part $M$ . $\mathrm{SP}{T}_a$ presents the $\mathrm{SPT}$ containing all possible $l_a$ connecting light source $S_a$ and all points $S{p}_a$ in the specific shadow area $S{h}_a$

$$\begin{array}{c}S{p}_a\in S{h}_a\hfill \\ \mathrm{SP}{T}_a=\{l_a|S{p}_a\in l_a,S_a\in l_a\}\hfill \\ \exists M\in \{M{p}_a|M{p}_a=L_a\cap M\}\hfill \end{array}$$

(8)

SPT intersection

Similarly, for light beam from another light source $S_b$ blocked by the same fuselage body $M_{\mathrm{part}}$ , it follows that

$$\begin{array}{c}S{p}_b\in S{h}_b\hfill \\ \mathrm{SP}{T}_b=\{l_b|S{p}_b\in l_b,S_b\in l_b\}\hfill \\ \exists M\in \{M{p}_b|M{p}_b=L_b\cap M\}\hfill \end{array}$$

(9)

Since $M{p}_a\cap M{p}_b\ne \varnothing$ , we have

$$\begin{array}{c}(L_a\cap M_{\mathrm{part}})\cap (L_b\cap M_{\mathrm{part}})=(L_a\cap L_b)\cap M_{\mathrm{part}}\ne \varnothing \hfill \\ \exists M{p}_k,M{p}_k\in L_a\cap L_b,M{p}_k\in M\hfill \end{array}$$

(10)

The intersection space between $\mathrm{SP}{T}_a$ and $\mathrm{SP}{T}_b$ contains points $M{p}_k$ belonging to $M$ , if $M$ is specific as the 3D space taken by a quadcopter fuselage. Figure 3 demonstrates the recovery of fuselage area $M$ based on two related shadow areas $S{h}_a$ and $S{h}_b$ . Based on the solution of multiple $\mathrm{SPT}$ integration, a 3D enclosure containing specific 3D position of quadcopter can be calculated from captured shadows. For scenarios containing multiple UAVs, the movement trajectories of UAV fuselages can be applied in distinguishing overlapped shadows from different UAVs.

Quadcopter flight status monitoring

Based on the basic quadcopter localization theory illustrated in the last section, this section presents an optimized algorithm for quadcopter flight regime monitoring. The optimized algorithm includes three steps. First, the 3D spatial position of each rotor on a quadcopter is extracted based on the $\mathrm{SPT}$ integration procedure. Then, the 3D quadcopter frame is simulated with auxiliary information. Finally, the quaternion is calculated based on the real-time frame Euler angle.

Rotor position extraction

2D rotor position extraction

As shown in Figure 4(d), rotors are four most significant convex points on each quadcopter shadow silhouette contour; extracting four rotor positions based on silhouette contours can provide specific shadow area for the generation of $\mathrm{SP}{T}_{\mathrm{rotor}}$ . The extraction of 2D rotor position based on quadcopter shadow silhouette contour is a two-step procedure.

Figure 4.

Quadcopter flight regime extraction procedure: (a) rotor labeling, (b) shadow information extraction, (c) refined silhouette information extraction, (d) rotor region recognition, (e) rotor region extraction, and (f) rotor-based $\mathrm{SPT}$ extraction.

First, the gravity center of quadcopter shadow is calculated

$$\begin{array}{c}\overline{x}=\frac{1}{N}\sum _{(x,y)\in \mathrm{Sh}}x\hfill \\ \multicolumn{1}{c}{\overline{y}=\frac{1}{N}\sum _{(x,y)\in \mathrm{Sh}}y}\end{array}$$

(11)

Then, the convex points on shadow contour are located based on the distance curve between contour points and the gravity center. The convex point appears at the maximum point on the distance curve. For each point $(x_i,y_i)$ on the shadow contour, corresponding denoised distance $\overline{D_i}$ can be calculated as below

$$\begin{array}{c}{D}_i=\sqrt{{(x_i-\overline{x})}^2+{(y_i-\overline{y})}^2}\hfill \\ \multicolumn{1}{c}{\overline{D_i}=\frac{1}{\mathrm{length}+1}\sum _{k=-\frac{\mathrm{length}}{2}}^{\frac{\mathrm{length}}{2}}{D}_{i+k}}\end{array}$$

(12)

Based on the denoised distance curve $\overline{D}$ , the convex points presenting four quadcopter rotors can be located through seeking local maximum points.

Theoretical feasibility proof

In order to extract the rotor 3D position, the application subject of the $\mathrm{SPT}$ intersection is narrowed down to each single rotor on a floating quadcopter. More specifically, instead of extracting $\mathrm{SPT}$ from the whole shadow area $S{h}_a$ , only a specific shadow area $S{h}_{a\_\mathrm{rotor}}$ is adopted in the $\mathrm{SPT}$ extraction procedure. $S{h}_{a\_\mathrm{rotor}}$ is projected on the ground when light beam from light source $S_a$ is blocked by one rotor part $M_{\mathrm{rotor}}$ on a quadcopter fuselage $M$ . Thus, the direction of blocked light beam $l_a$ leads to point $S{p}_a$ in shadow $S{h}_{a\_\mathrm{rotor}}$ , going through the rotor $M_{\mathrm{rotor}}$ . Shadow area $S{h}_{a\_\mathrm{rotor}}\in S{h}_a$ contains all $S{p}_a$ points on the ground. Merging all possible $l_a$ connecting light source $S_a$ and all points $S{p}_a$ , $\mathrm{SP}{T}_a$ can be extracted from the specific shadow area $S{h}_{a\_\mathrm{rotor}}$

$$\begin{array}{c}S{p}_a\in S{h}_{a\_\mathrm{rotor}},S{h}_{a\_\mathrm{rotor}}\subset S{h}_a\hfill \\ L_a=\{l_a|S{p}_a\in l_a,S_a\in l_a\}\hfill \\ \exists M{p}_{\mathrm{ai}}\in \{M{p}_a|M{p}_a=L_a\cap M_{\mathrm{rotor}}\}\hfill \end{array}$$

(13)

Similarly, for light beam from another light source $S_b$ blocked by the same rotor $M_{\mathrm{rotor}}$ , it follows that

$$\begin{array}{c}S{p}_b\in S{h}_{b\_\mathrm{rotor}},S{h}_{b\_\mathrm{rotor}}\subset S{h}_b\hfill \\ L_b=\{l_b|S{p}_b\in l_b,S_b\in l_b\}\hfill \\ \exists M{p}_{\mathrm{bi}}\in \{M{p}_b|M{p}_b=L_b\cap M_{\mathrm{rotor}}\}\hfill \end{array}$$

(14)

Since $M{p}_a\cap M{p}_b\ne \varnothing$ , we have

$$\begin{array}{c}(L_a\cap M_{\mathrm{rotor}})\cap (L_b\cap M_{\mathrm{rotor}})=(L_a\cap L_b)\cap M_{\mathrm{rotor}}\ne \varnothing \hfill \\ \exists M{p}_k,M{p}_k\in L_a\cap L_b,M{p}_k\in M_{\mathrm{rotor}}\hfill \end{array}$$

(15)

Point $M{p}_k\in M_{\mathrm{part}}$ is involved in the intersection enclosure between $\mathrm{SP}{T}_a$ and $\mathrm{SP}{T}_b$ . Since $M_{\mathrm{rotor}}$ is the specific 3D space containing a rotor, the corresponding $M{p}_k$ position can be figured out. Figure 4(b)–(f) demonstrates the recovery of rotor positions $M_{\mathrm{roto}{r}_i}$ based on two sets of related shadow areas $S{h}_{a\_\mathrm{roto}{r}_i}$ and $S{h}_{b\_\mathrm{roto}{r}_i}$ . Noticeably, for shadows projected on the wall or other planes, the shadow silhouette information will not be able to provide valid rotor shadow area position. For scenarios containing more than two light sources, the remaining shadow silhouettes are still sufficient for rotor position $M_{\mathrm{rotor}}$ extraction under this circumstance.

Quadcopter 3D flight information extraction

Based on the algorithm proposed in theoretical proof, the precise position $M_{\mathrm{roto}{r}_i}$ of each rotor $\mathrm{roto}{r}_i$ can be extracted from the intersection of $\mathrm{SPT}$ s. Then, the quadcopter flight regimes including quaternion information and flight speed are available through resolving the $M_{\mathrm{roto}{r}_i}$ array. Since rotors are the four significant convex points on each shadow silhouette from different projection angles, the extraction of rotor positions provides stable position extraction performance. Additionally, the rigid model of UAV can be efficiently established based on four extracted rotor positions. Four rotors on a quadcopter are numbered from 1 to 4 as shown in Figure 4(a). Rotor numbers 1–4 are marked in red, blue, green, and yellow, respectively. In order to extract quadcopter time-related flight regimes, the 3D position $M_{\mathrm{roto}{r}_i}$ of $\mathrm{roto}{r}_i$ is further defined as ${\mathrm{Rotor}}_t^i$ .

Since the fuselage of a quadcopter is a symmetrical rigid body, the barycentric coordinate $\mathrm{Fus}{e}_t$ of a quadcopter is the average of four ${\mathrm{Rotor}}_t^i$

$$\begin{array}{c}\mathrm{Fus}{e}_t=\frac{\sum _{i=1}^4{\mathrm{Rotor}}_t^i}{4}\hfill \end{array}$$

(16)

Additionally, the real-time normal vector of the quadcopter can be obtained based on the rotor position array $\{{\mathrm{Rotor}}_t^i|i=1,2,3,4\}$

$$\begin{array}{c}\overrightarrow{\mathrm{Fus}{e}_t}=\frac{(\sum _{i=1}^2{\mathrm{Rotor}}_i^t-\sum _{i=3}^4{\mathrm{Rotor}}_i^t)}{|\sum _{i=1}^2{\mathrm{Rotor}}_i^t-\sum _{i=3}^4{\mathrm{Rotor}}_i^t|}\hfill \end{array}$$

(17)

Assuming the original orientation of the quadcopter is $\overrightarrow{O}$ , the real-time rotation angle $\alpha$ and the rotation matrix $R_{\alpha }$ of the quadcopter are accessible as well

$$\begin{array}{c}\alpha =\arccos \left(\frac{\overrightarrow{\mathrm{Fus}{e}_i}•\overrightarrow{O}}{\left|\overrightarrow{\mathrm{Fus}{e}_i}\right|•\left|\overrightarrow{O}\right|}\right)\hfill \end{array}$$

(18)

$$\begin{array}{c}{R}_{z,\theta }\left(\begin{array}{ccc}1\hfill & 0\hfill & 0\hfill \\ \multicolumn{1}{c}{0}& \cos \theta \hfill & -\sin \theta \hfill \\ \multicolumn{1}{c}{0}& \sin \theta \hfill & \cos \theta \hfill \end{array}\right)\hfill \\ \multicolumn{1}{c}{R_{y,\beta }\left(\begin{array}{ccc}\cos \beta \hfill & 0\hfill & \sin \beta \hfill \\ \multicolumn{1}{c}{0}& 1\hfill & 0\hfill \\ \multicolumn{1}{c}{-\sin \beta }& 0\hfill & \cos \beta \hfill \end{array}\right)}\\ \multicolumn{1}{c}{R_{x,\varphi }\left(\begin{array}{ccc}\cos \varphi \hfill & -\sin \varphi \hfill & 0\hfill \\ \multicolumn{1}{c}{\sin \varphi }& \cos \varphi \hfill & 0\hfill \\ \multicolumn{1}{c}{0}& 0\hfill & 1\hfill \end{array}\right)}\\ \multicolumn{1}{c}{R_{\alpha }=R_{z,\theta }{R}_{y,\beta }{R}_{x,\varphi }}\end{array}$$

(19)

More specifically, the Eulerian pitch angle $\theta$ , yaw angle $\beta$ , and roll angle $\varphi$ are defined as follows

$$\begin{array}{c}\theta =a\tan 2(-R_{\alpha }(3,1),\sqrt{{(R_{\alpha }(3,2))}^2+{(R_{\alpha }(3,3))}^2})\hfill \\ \beta =a\tan 2(\frac{R_{\alpha }(3,2)}{\cos \beta },\frac{R_{\alpha }(3,3)}{\cos \beta })\hfill \\ \varphi =a\tan 2(\frac{R_{\alpha }(2,1)}{\cos \beta },\frac{R_{\alpha }(1,1)}{\cos \beta })\hfill \end{array}$$

(20)

Finally, the real-time quadcopter quaternion $q$ can be calculated based on the Eulerian angle parameters

$$q=\left[\begin{array}{c}w\hfill \\ \multicolumn{1}{c}{x}\\ \multicolumn{1}{c}{y}\\ \multicolumn{1}{c}{z}\end{array}\right]=\left[\begin{array}{c}\sin \frac{\psi }{2}\sin \frac{\theta }{2}\cos \frac{\varphi }{2}+\cos \frac{\psi }{2}\cos \frac{\theta }{2}\sin \frac{\varphi }{2}\hfill \\ \multicolumn{1}{c}{\sin \frac{\psi }{2}\cos \frac{\theta }{2}\cos \frac{\varphi }{2}+\cos \frac{\psi }{2}\sin \frac{\theta }{2}\sin \frac{\varphi }{2}}\\ \multicolumn{1}{c}{\cos \frac{\psi }{2}\sin \frac{\theta }{2}\cos \frac{\varphi }{2}-\sin \frac{\psi }{2}\cos \frac{\theta }{2}\sin \frac{\varphi }{2}}\\ \multicolumn{1}{c}{\cos \frac{\psi }{2}\cos \frac{\theta }{2}\cos \frac{\varphi }{2}-\sin \frac{\psi }{2}\sin \frac{\theta }{2}\sin \frac{\varphi }{2}}\end{array}\right]$$

(21)

To sum up, the procedure of quadcopter localization and flight regime monitoring method is presented in Algorithm 1.

Algorithm 1:

Quadcopter flight regime extraction algorithm.

Input: ${\mathrm{Sh}}_{S_j}^t$ : the shadow of fuselage $M$ projected from light source $S_j$ ; $t$ : time coordinate of the frame; $S_j$ : the position of light source $j$ ; Output: $\mathrm{Fus}{e}_t$ : the position of quadcopter barycentric coordinate; $q$ : the quaternion of quadcopter; 1 foreach ${\mathrm{Sh}}_{S_j}^t$ do 2  Get the specific shadow area ${\mathrm{Sh}}_{<j,i>}^t$ for each rotor $\mathrm{Roto}{r}_i$ in the shadow ${\mathrm{Sh}}_{S_j}^t$ ; 3  Generate the ${\mathrm{SPT}}_{<j,i>}^t$ for each rotor shadow under the light source $S_j$ . 4 end 5 foreach $\mathrm{Roto}{r}_i$ do 6  Get the intersection position ${\mathrm{Rotor}}_i^t$ between all ${\mathrm{SPT}}_{<j,i>}^t$ generated by different light source $S_j$ . 7 end 8 Calculate the quadcopter barycentric coordinate $\mathrm{Fus}{e}_t$ ; 9 Get the quadcopter normal vector $\overrightarrow{\mathrm{Fus}{e}_t}$ ; 10 Calculate the rotation angle $\alpha$ and rotation matrix $R_{\alpha }$ ; 11 Calculate the Eulerian angles $\theta$ , $\beta$ and $\varphi$ ; 12 Get the quadcopter quaternion $q=[w,x,y,z]$ .

Experimental results

In this section, the experimental scenario settings are illustrated first. Then, the experimental results of quadcopter position and flight regime tracking are presented. Finally, the experimental results are analyzed to evaluate the proposed quadcopter monitoring algorithm.

Scenario setup

As shown in Figure 5(a), the experiment equipment contains two major subsystems: the testing subsystem extracting quadcopter parameters based on proposed method and the onboard subsystem collecting airborne sensor information as the ground truth.

Figure 5.

Two subsystems for comparative analyses of quadcopter localization and flight regime extraction method: (a) the flowchart of the comparative experiment, (b) the testing subsystem for the proposed method, and (c) the onboard subsystem for ground truth data.

The testing subsystem shown in Figure 5(b) includes an embedded platform powered by NVIDIA TX1 embedded system and a normal monocular full-HD (high-definition) resolution camera capturing 120 frames per second. The testing subsystem is set up at an open hardening playground, where traditional sensor network–based methods are ineffective due to the inaccessibility of cables. In order to capture clear quadcopter shadow information, the test field is set up with two spotlights at 3 m high with the depression angle of ${45}^{\circ }$ . The full-HD camera is set up on the top of a spotlight pole at a height of 15 m. In comparison with the airborne camera, the fixed camera provides much more stable view of field which can contribute to the high precise 3D position estimation. Based on the shadow information, the TX1 embedded system extracts quadcopter parameters frame by frame, outputting the quadcopter location and flight regimes to the embedded flash memory. For the onboard subsystem shown in Figure 5(c), a F450 quadcopter frame is chosen to as the quadcopter platform, equipped with a PX4 flight control module and a Raspberry Pi 3 microcomputer. The PX4 flight control module transmits quadcopter position and flight regimes to Raspberry Pi 3 through the general-purpose input/output port at the frequency of 100 Hz.

Noticeably, the parameter transmitting procedures are launched simultaneously on the two subsystems.

Result analyses

Quadcopter localization precision analyses

The quadcopter localization precision analyses are launched based on the deviation between two sets of parameters, including the quadcopter position based on the proposed algorithm and the recorded position based on the onboard PX4 module output.

As shown in Figure 6(a), the black curve, red curve, and blue curve present the deviation on the $X$ axle, $Y$ axle, and $Z$ axle, respectively. Noticeably, extreme high deviations on $X$ and $Y$ axles are obvious during the initial and terminal stages, corresponding to take-off and landing procedures of the quadcopter. This phenomenon is caused by partially occluded quadcopter shadows at low altitude, where the overlapping between quadcopter fuselage and quadcopter shadows can be observed from the camera view.

Figure 6.

Experimental results: (a) deviations between 3D quadcopter locations based on proposed method and onboard system output, (b) a comparative analysis of pitch angles extracted by the proposed method and onboard output, (c) a comparison between roll angles extracted by the proposed method and onboard output, and (d) a comparative analysis of yaw angles extracted by the proposed method and onboard output.

In the temporal interval from 300 to 1600 ms, stable quadcopter localization performance can be observed. The quadcopter location deviation on $Z$ axle maintains below 100 mm. Additionally, the quadcopter location deviations are below 30 mm on $X$ and $Y$ axles, resulting in a horizontal deviation below 100 mm. Based on the analyses result above, the proposed method can provide reliable quadcopter localization with a horizontal deviation below 100 mm and vertical deviation below 100 mm.

Quadcopter flight regime precision analyses

For the evaluation of extracted quadcopter flight regime precision, the three Euler angles extracted by the proposed method are evaluated in comparison with the corresponding onboard subsystem output. The Euler angles include three angles: the pitch angle indicating the quadcopter balance status on front direction, the roll angle indicating the quadcopter balance status on the horizontal direction, and the yaw angle indicating the navigation direction:

1. Pitch angle extraction precision.Figure 6(b) shows the pitch angle parameters collected from two different subsystems. The red curve presents the pitch angle from testing subsystem, which is consistent with the black line based on onboard subsystem output. Additionally, the blue curve presents the deviation between two sets of pitch angles, remaining under $5^{\circ }$ . The pitch angle monitoring precision of the proposed method is sufficient for quadcopter flight regime monitoring.

2. Roll angle extraction precision. Figure 6(c) illustrates the comparison between roll angle parameters from two subsystems. The roll angle from the testing subsystem is presented in the red curve, with the corresponding output from the onboard subsystem is illustrated in the black line. The deviation between two output roll angles from different subsystems is depicted in the blue line, which peaks at $3^{\circ }$ . A good continuity condition can be observed between testing subsystem roll angle output and onboard subsystem roll angel output.

3. Yaw angle extraction precision. Figure 6(d) presents the yaw angle output from two different subsystems. The yaw angles from testing subsystem output and onboard subsystem output are plotted in red curve and black curve, respectively. Their deviations are depicted in the blue curve with a maximum value under $3^{\circ }$ , indicating a reliable yaw angle output from the testing subsystem based on the proposed method.

To sum up, the proposed method is capable of extracting quadcopter flight regimes precisely according the reliable output from the testing subsystem. Based on the precise quadcopter flight regime extraction, the proposed method is superior to the previous external sensor–based methods restricted in quadcopter localization.

Conclusion

In this article, we proposed a shadow-based quadcopter localization and flight regime extraction method. The proposed method extracts 3D quadcopter position and quadcopter quaternion parameters based on the quadcopter shadow information on the ground. In comparison with current quadcopter location monitoring methods, the proposed method has obvious advantages on deployment convenience, system robustness, precision expandability, and scenario adaptability.

Deployment convenience

1. Instead of heavily depending on special equipments like IR cameras or UWB transceivers, the proposed 3D spatial quadcopter monitoring method is based on a general purpose monocular camera.

2. The proposed method requires only one monocular camera and an embedded system. The network topology of sensor monitoring network is largely simplified due to the fine manipulation of spatial discrete quadcopter shadow information.

System robustness

For scenarios containing multi-UAVs and scenarios under complex electromagnetic environment, the positioning signals could encounter serious mutual interference, leading to potential security risks. Through deploying the proposed method on UAV testing fields, operators can overcome the signal interference problem and archive secured flight circumstance in complex signal environment.

Precision expandability

Based on the foreseeable dramatic enhancement of image sensor resolution, the precision of the proposed quadcopter localization and monitoring method can be elevated.

Scenario adaptability

1. Based on the deployment of virtual silhouette sensors, an SPT-based 3D UAV flight regime acquisition network can be set up at open field where signal cables are inaccessible.

2. Thanks to the low power consumption feature of embedded system, and the testing subsystem is convenient for mobile deployment based on battery power supply or vehicle power supply. Especially, for open field areas where power cables are unavailable, traditional sensor networks are nonfunctional, while the testing subsystem can provide stable quadcopter health monitoring.

In conclusion, the proposed method offers a quadcopter localization and flight regime monitoring method. Its unique features, such as simple hardware requirements, deployment convenience, precision expandability, and scenario adaptability, make the proposed method a superior choice for quadcopter status motoring especially in outdoor open fields. Based on the unique advantages provided by the proposed method, our future works will be focused on the enhancement of quadcopter quaternion extraction precision based on high resolution monocular cameras.