Human-tracking system using quadrotors and multiple environmental cameras for face-tracking application

Abstract

In this article, a system for tracking human’s position and orientation in indoor environment was developed utilizing environmental cameras. The system consists of cameras installed in the environment at fixed locations and orientations, called environmental cameras, and a moving robot which mounts a camera, called moving camera. The environmental cameras detect the location and direction of each person in the space, as well as the position of the moving robot. The robot is then controlled to move and follow the person’s movement based on the person’s location and orientation, mimicking the act of moving camera tracking his/her face. The number of cameras needed to cover the area of the experiment, as well as each camera’s position and orientation, was obtained by using particle swarm optimization algorithm. Sensor fusion among multiple cameras is done by simple weighted averaging based on distance and knowledge of the number of robots being used. Xbox Kinect sensors and a miniature quadrotor were used to implement the system. The tracking experiment was done with one person walking and rotating in the area. The result shows that the proposed system can track the person and quadrotor within the degree of 10 cm , and the quadrotor can follow the person’s movement as desired. At least one camera was guaranteed to be tracking the person and the quadrotor at any time, with the minimum number of two for tracking the person and only a few moments that only one camera was tracking the quadrotor.

Introduction

Research background

Data of the position and orientation of a person moving inside an indoor environment can provide various information about the person, such as what he/she is looking at, prediction of his/her movement, or the pose of his/her face. The data can be used for a number of applications, for example, monitoring the behavior of people in the environment, human following behavior of service robots, or providing the museum or gallery visitors with corresponding information according to their current location. One of the interesting uses of the person’s position and orientation is about the pose of the face. With this information, we know where we should put our robot so that it can have interaction with the person as well as taking photos of his/her face, without the need of searching. Therefore, we can see that the system which can track person’s position and orientation is important.

Related works

Various works have been done on tracking. A system named W ⁴ was conducted using monocular gray-scale video or infrared video to detect and track multiple people for monitoring their activities in an outdoor environment. ¹ Multiple stereo cameras with slightly overlapped views were used to track motions of multiple people over a wide area and can perform well in crowded condition. ² Various types of 3-D range sensors were used to track the positions and body angles of people inside a shopping mall and analyzed the obtained data for people density, direction of movement, body angles when people stop. ³ Those data can give some insights about behavior of people and can be used for improving the layout for the customers.

There are also attempts of using robots to track human beings. Combination of RGB camera and thermal camera were used for tracking of multiple people with handling of occlusion, ⁴ without controlling the robot to follow people. Face tracking by on-board camera and leg pattern from the on-board laser range finder were used to control mobile robots to track people. ^5,6 Similarly, skin-color detection was used for face detection, with sonar and tactile sensors for distance information, and the robot was controlled to track and follow human. ⁷ Face detection by skin-color detection was also utilized in the robot for tracking a human, taking his/her picture and printing it out. ⁸ Data from RGB-D sensor were used to track the motion of people in the crowd, and some following activities were shown. ⁹ The mobile robots could track and follow human, but the process started after a person was found by the robots, so initial search was also required. The shapes of the mobile robots were also tall to raise the attached camera close to the eye level, which increased the risk of toppling, disrupting the tracking process.

Unmanned aerial vehicles are also becoming a popular platform for robotic researches, with some researches working on object tracking which can also be applied to human-tracking application. Color-based tracking with particle filter was applied to images from a camera underneath a quadrotor to control the quadrotor to chase a moving object. ¹⁰ However, the quadrotor needed to be above the object at the beginning of tracking. An AR Drone 2.0 quadrotor was used to perform visual-based object tracking in global positioning system (GPS)-denied environment. ¹¹ The user needed to select the object to be tracked, then the AR Drone used the selected area as the target for tracking and controlling its movement to follow the object. When the tracking was lost, the quadrotor could hover and wait for the recovery of tracking or re-selection of the target, but it could not recover tracking by itself.

Face recognition at a distance (FRAD) involves recognizing people from a distance without being noticed by the subject. One of the techniques used in FRAD is a multi-camera active vision system, in which wide-field-of-view cameras detect and locate people, while narrow-field-of-view cameras are actively controlled to capture high-resolution facial image by pan-tilt-zoom (PTZ) commands. Two static cameras and two PTZ cameras were used to track and recognize multiple people in a small space. ¹² The static cameras detected humans and triangulated for world coordinate of each person, which was used to provide coarse estimate of pan and zoom commands. Face detection was then used to improve tracking performance and lock the camera to the target. The facial images had improved resolution compared to the cropped images from the static cameras and could be used for face recognition. A similar system with a standard-resolution static camera and a high-resolution PTZ camera was created to track and recognize multiple people walking in a long corridor, mimicking conditions in airports or subway stations. ¹³ The PTZ camera was controlled to capture one face at a time and facial recognition was performed. The resulting identity was then matched with the subject detected by the static camera. The system achieved tracking and face capture at 25–50 m and recognition at 15–20 m. The researches provided face tracking in an area, but only in one direction, therefore the ability of face tracking was limited to the case of people walking in only one direction, that is, toward the camera.

From the previous works, there are some systems capable of tracking people in an indoor environment to some extent, with reasonable construction cost. However, there is still requirement in initialization for human following robot, as well as the optimization of the configuration of the system for better tracking coverage.

Objectives

The research in this article aims at constructing a prototype sensing system for measuring the position and orientation of a person inside an indoor environment and using the information for controlling a robot to follow the person. The situation was simulated as the robot was a moving camera and tracked the person’s face. The construction should be applicable to any environment, providing that the dimension of the environment is known.

In order to solve the problem, the following challenging points need to be considered:

System configuration—how to select proper system, and how to optimize sensor placement, in which we propose to use the combination of moving cameras and environmental cameras and optimize their placement by particle swarm optimization (PSO);

Fusion of data from multiple-camera system and noise handling—how to analyze the data from multiple cameras that they belong to the same object and merge them together: Instead of using Kalman filter, which requires sensor’s noise model, and acquisition process may not be easy, we propose to use the distance and simple averaging, together with the knowledge of the maximum number of the objects being tracked; thresholding is also applied to reduce unnecessary motion induced by measurement noise.

We have already published a paper about a system utilizing a Kinect sensor and a quadrotor to track a person in Srisamosorn et al., ¹⁴ which eliminates initial requirements that the quadrotor be in front of the person. The threshold algorithm was proposed for stabilization of the quadrotor in order to improve the tracking performance. This work is an extension of the system to cover larger area and more types of movements with multiple cameras. Optimization is done to find camera configuration to cover selected space and guarantee full coverage, and multiple cameras are set up into the real environment. Sensor fusion, tracking, and control are also required. An experiment of tracking a person in the designated area was conducted to prove the optimization result and the concept of human tracking. By our proposed optimization algorithm, the design of the system is not limited to a specific environment but can be adapted to use in other environment as well.

Problem statement and specifications

The problem in consideration of this article is about tracking of the position and orientation of a person moving inside a restricted area and controlling a robot to be in front of the person at a distance, mimicking tracking of the person’s face. The robot is not strictly required to be in front all the time, but should be able to recover when the person moves or turn, and get back to his/her frontal side. The evaluation was done in the control aspect, that is, the errors from the desired goal position and angle, and the error from the ideal position and angle based on the person’s ground truth position and angle. For initial experiment, the following assumptions are applied to the system to simplify the requirements:

The environment (room and environmental cameras’ positions) is not changing during the tracking process.

There is only one person in the area.

The person being tracked is moving smoothly and not too fast at standard speed (approximately 1 m/s).

The behavior of the person’s face looking up and down is not considered.

The person being tracked turns with the whole body, not by turning only his/her head.

Person standing near the edge is not looking into the edge (equivalent to looking into the wall, which means the robot needs to be in between the person and the wall, and can be dangerous).

System design with environmental cameras

Indoor position-tracking system using cameras

In indoor environment, the availability of the GPS signals is limited and cannot be used to provide positions of objects inside. There are various researches conducting on estimating the positions of objects in indoor environment, ^15,16 with different ranges, prices, and accuracies, varying from room-level to sub-millimeter level.

Criteria used in consideration are

Accuracy—Accuracy in the range of centimeters should be enough for person tracking and quadrotor tracking;

Cost—The cost should not be too high to be implemented in real environment outside laboratory;

Device or marker—It is more preferable without devices or markers because it is difficult to force people to carry the transmitter.

Among the methods compared in the papers, most of them require that the objects or people to be positioned carry a kind of transmitter or marker. The approach which requires no transmitters nor markers is the vision-based system. Although high processing power is required in performing image processing for position tracking, the cost is still less expensive compared to multiple other options. Therefore, position-tracking system using cameras was selected.

System configuration

The following terms are defined as follow:

Environmental cameras fixed in the environment with fixed position and orientation, possible of capturing face if the person is in the view and the face is looking toward the camera;

Moving cameras attached to moving structures or robots and can move around.

Considering using cameras for face tracking, there are a number of possible configurations, with pros and cons.

Utilizing only environmental cameras This method places and fixes the cameras at the designed positions and orientations, similar to the setup in. ^1,2 Facial images are taken directly from these cameras.

Utilizing only moving cameras In this method, movable robots, equipped with cameras as in literature, ^{4 –11} use images obtained from the cameras and other sensors to control their movement such that the cameras can keep track of the people’s faces in a better view.

Utilizing environmental cameras and moving cameras As a combination of the previous methods, environmental cameras and moving cameras are used together, similar to the use of PTZ cameras in earlier studies. ^12,13 The environmental cameras are used to locate people’s location and the moving cameras are moved there to capture the face.

In (a), the setup must be done such that for a person at any point in the area, looking in any direction, at least one environmental camera must be capturing his/her face at a suitable angle. This dramatically increases the number of cameras needed (quantitatively analyzed in the section “Optimization for camera configuration”). Moreover, the distance from a camera to a face can be quite large, reducing the resolution of the face, if not undetectable, and cameras with higher resolution are needed.

Using robots in (b) can reduce the number of cameras, as we ideally need one robot per one person. The requirement of the resolution can also be reduced, as the cameras can be closer to the face. Nevertheless, initialization or searching is required before tracking can start, and the person’s position and orientation cannot be estimated precisely. Increasing the number of robots can reduce the searching time but will be more complex.

In order to solve the problem of searching for people in (b), environmental cameras are proposed for providing each person’s position, as in (c). Comparing to the use of environmental cameras alone, the requirement is reduced to having cameras covering the area such that people standing at any point, regardless of direction, can be detected, which results in smaller number of cameras needed. This method of robot control would be similar to those found in Oh et al., ¹⁷ where multiple cameras detected colored markers to locate the quadrotor, or the control of a mobile robot in the Intelligent Space (iSpace). ¹⁸

The problems to be considered are how to place the cameras in the area to achieve face tracking, both environmental cameras and moving cameras. For the environmental cameras, it is required that the cameras can cover the whole area to keep track of people inside, which would be done by optimization. The moving camera needs to be in front of the person to capture the face, in which robot control based on the information of the person’s position and orientation would be implemented.

System overview

In the system, environmental cameras are implemented by depth sensors and moving cameras are controlled by quadrotors. Depth sensors are used to detect humans’ positions and direction in the space as well as 3-D positions of the quadrotors (Figure 1(a)). How to determine the configuration of each camera is described in the section “Optimization for camera configuration.” The data from each depth camera belonging to the same person or quadrotor are fused together to provide the position and orientation for tracking (“Sensor fusion and quadrotor’s tracking and control”). The quadrotor’s yaw angles are taken from the on-board inertial measurement unit (IMU) sensor. The pose of each person is used to set the desired position for each quadrotor such that its camera is facing toward the person’s face at a fixed distance and angle (Figure 1(b)), with some margin on the height to reduce the danger of crashes. The computer then sends command signals to navigate the quadrotor to the desired position (Figure 1(c)), where its camera can capture the face and transmit images back for further processing.

Figure 1.

Concept and steps of the system.

Optimization for camera configuration

In order to set up the environmental cameras for covering the desired area, the number of cameras and their arrangement are required. There are works on optimizing the configuration of cameras, such as optimizing the angles of randomly deployed cameras for maximum coverage in 2-D using PSO, ¹⁹ optimizing positions and orientations of cameras limited with possible locations for camera placement using 3-D model to cover a 2-D plane, ²⁰ and optimizing for deploying cameras in 3-D for surveillance of 2-D plane. ²¹ We proposed an optimization algorithm for 3-D coverage with limits on the possible locations of environmental cameras. With proper adjustment on area’s dimension and limits of locations, the proposed algorithm can be applied to any environment and cameras.

Assumptions

Camera’s field of view

To solve the coverage problem, the field of view (FOV) of the camera is required. This usually includes the horizontal and vertical angles of view, α_{c, h} and α_c,v , for a simplified model. Maximum distance from the camera can also be indicated based on the requirement on the size of detected object.

Placement error prevention

The simplified model can be used directly in the optimization if the cameras can be precisely placed, otherwise the placement error can result in incomplete coverage. Therefore, the model must be modified so that the optimization result is robust to those errors.

To improve the robustness, the FOV is reduced. In this model, the area within this FOV must still be in the FOV of the camera if the errors of the position and orientation of the camera are within the range specified. Thus, the model is approximated from maximum error in each dimension.

Figure 2 shows a cross section of the full FOV, the maximum angle error δ, and the reduced FOV after considering the error. The triangles can be used for both the boundary on the minimum depth and the maximum depth by changing D. Firstly, the viewing angle is reduced to α′ _c , where

${\alpha \prime }_c={\alpha }_c-\delta$

1

Figure 2.

Calculation of camera’s reduced span.

The rotation also has some effects on reducing the depth range of the FOV. By small-angle approximation, the reduction in this dimension can be approximated as

$\Delta D=D\text{tan}{\alpha }_c\text{tan}\delta \approx \delta D\text{tan}{\alpha }_c$

2

The larger α_c has larger effects on ΔD, therefore, the horizontal angle α_c,h is selected.

The error in the Kinect’s roll component is complex and therefore approximated as an additional reduction of the horizontal and vertical viewing angles. However, error in roll has no effects of reducing FOV’s depth.

In the final stage, the translational errors Δ are applied by additionally removing the FOV along x, y, z axes.

Possible placement locations

In reality, cameras cannot be placed without constraints. It is possible to place a camera 2 m above the floor in the center of the room, but some supporting structure, such as a tripod, would be necessary, which can obstruct people and therefore is undesirable.

Generally, the possible and suitable locations to place the cameras are usually along the edge of the room to increase the coverage as well as to avoid obstruction in the space. This can reduce the complexity of optimizing the location of the cameras. Instead of searching through 3-D space xyz, searching on the xy-plane can be reduced to searching along those edges, reducing one dimension. Moreover, if the height of the camera placement is fixed by the height of the existing structure, z is also restricted, so the 3-D search problem is now reduced to only 1-D.

Camera placement optimization algorithm

In order to obtain the optimal, or nearly optimal, solution, a camera is added one-by-one to minimize the uncovered area. After the addition, the configurations of all cameras are adjusted around the current configurations to also minimize the uncovered area. The two steps are repeated until the whole area is covered. After that, the final optimization is done to maximize the coverage. Each camera is readjusted one-by-one to maximize the sum of the number of cameras covering each point while keeping every point covered. All optimization processes are done by PSO ²² following previous works, ^19,21 using a Python library pyswarm. ²³ The process’s flowchart is illustrated in Figure 3.

Figure 3.

Flowchart of camera’s location design process.

Particle’s state

PSO requires definition of the state of each particle. Each variable in the state is the value which can be adjusted to optimize the objective function.

In this coverage optimization problem, a state of each camera is defined to represent camera’s location and orientation. The location depends on the limitation of location of cameras, and the orientation includes pitch and yaw angles around fixed y-axis and z-axis, respectively. Roll angle is set to zero. So, the state is (x_state, θ_state, ψ_state).

Coverage test

To evaluate the coverage of the system, we need to know if the point P (x, y, z) within the region of interest (ROI) is in the FOV of the camera C_i ( $FO{V}_{C_i}$ ), given the position $\hspace{0.17em}{O}_i=(O_{x_i},O_{y_i},O_{z_i})$ and orientation (ϕ_i, θ_i, ψ_i) of C_i, for all cameras in the system. This is done by transforming the point P (x, y, z) relative to world frame, into P _i (x_i, y_i, z_i) relative to the camera C_i‘s frame. By comparing x_i, y_i, z_i to the obtained reduced model, we can judge whether P is in $FO{V}_{C_i}$ .

Process of adding a new camera

The first step in each loop is adding a camera to the system. To cover the ROI using the least cameras, the placement is chosen by minimizing the number of points in the area which is not covered by any cameras. That is at point P (x, y, z), the index f( P ) is defined as 0 if P is in FOV of any camera, and 1 otherwise. The objective function f_opt to be minimized is

$f_{opt}={\displaystyle \sum _{P\in ROI}}f(\mathit{P}).$

3

PSO is implemented to minimize f_opt with the particle’s state representing the location and orientation of the new camera. The final result is then added to the camera list.

Process of readjustment

In this step, all configurations, including those of the newly added camera, are changed within small boundary around their current settings, that is, for C_i with $x_{\text{state},i},{\mathit{\theta }}_{\text{state},i},{\psi }_{\text{state},i}$ , C_i will be varied within the range of $[x_{\text{state},i}-x_b,x_{\text{state},i}+x_b]$ , $[{\mathit{\theta }}_{\text{state},i}-{\mathit{\theta }}_b,{\mathit{\theta }}_{\text{state},i}+{\mathit{\theta }}_b]$ , and $[{\psi }_{\text{state},i}-{\psi }_b,{\psi }_{\text{state},i}+{\psi }_b]$ , respectively, given that the ranges are truncated to the limit. $x_b,{\mathit{\theta }}_b,\hspace{0.17em}\text{and}\hspace{0.17em}{\psi }_b$ are the size of the boundary of $x_{\text{state}},{\mathit{\theta }}_{\text{state}},\hspace{0.17em}\text{and}\hspace{0.17em}{\psi }_{\text{state}}$ , respectively.

The objective function used in this process is the same as those used in the adding process, that is, f_opt . If the result provides lower number of remaining points, the result is kept; otherwise, the process continues with the previous result.

After this process, if all the points in ROI are covered, that is, f_opt = 0, the optimization process breaks the loop and gets to the final readjustment. If there are still some points left, the loop continues and proceeds to add a new camera again.

Process of final optimization

This final optimization aims at maximizing the coverage while maintaining the complete coverage without adding new camera. At each point P ∈ ROI, the number of cameras seeing the point is denoted as n( P ), and the objective function to be maximized is

$f_{op{t}_2}={\displaystyle \sum _{\mathit{P}\in ROI}}n(\mathit{P})$

4

subject to the constraint that $n(\mathit{P})>0,\forall \mathit{P}\in ROI$ .

First, $f_{op{t}_2}$ is calculated based on the camera configurations from the loop, stored as f_cur and f_loop . Next, configuration of each camera is adjusted one-by-one within the boundary, and $f_{op{t}_2}$ is calculated. If $f_{op{t}_2}>f_{cur}$ , the adjusted result is kept and f_cur is changed to $f_{op{t}_2}$ .

After adjusting all cameras, f_cur is then compared with f_loop at the beginning of the loop. If $f_{cur}>f_{loop}$ , there was some improvement in the previous loop, and the optimization continues. f_loop is then set to f_cur and the adjustment continues. On the other hand, f_cur = f_loop indicates that there was no improvement in the previous loop. Therefore, the process is stopped, and the final result is obtained.

Implementation

A space in our laboratory is a good candidate for testing the optimization algorithm, as there are possible places in complex shapes to put the cameras. By measuring the dimension of the space and the structures, a model of the area was obtained.

The area of the experiment is set as a 3-by-3.5-m area, sampled by 0.1-m step, as drawn in dots in Figure 4. In order to cover the area such that people’s position as well as quadrotor’s position can be obtained, the area must be covered from the height of 0.7–2.5 m, also 0.1-m step. The points become our set of ROI. The optimization is done to cover all of these sampled points.

Figure 4.

Area of interest for the experiment.

In the system, Kinect’s depth sensors were utilized. According to the specification of Xbox Kinect sensor, ²⁴ the horizontal angle of view (2α _c,h ) is 57° and the vertical angle of view (2α _c,v ) is 43°. The depth range of the Kinect is set at 0.45 m–4.5 m. By placing x-axis along the depth, and y-axis pointing to the left, the simplified model of Kinect’s FOV can be illustrated as dashed black lines in Figure 5.

Figure 5.

Reduced FOV compared to full FOV. FOV: field of view.

With reduced model discussed in the section “Placement error prevention”, for each degree of error δ with α_c = 28°, ΔD_min ≈ 0.005 m on the minimum depth side (D_min = 0.45 m), and ΔD_max ≈ 0.05 m on the maximum depth side (D_max = 4.5 m). Error in roll angle results in additional 1° reduction in both horizontal and vertical angles.

With the angular errors δ = 2° and the translational errors Δ = 0.03 m, the reduced FOV is shown in Figure 5. Point P_i(x_i, y_i, z_i) in Kinect K_i‘s frame is in its FOV if

$\begin{array}{c}0.49\le x_i\le 4.37,\\ -\text{tan}({25}^{\circ })x_i+0.03\le y_i\le \text{tan}({25}^{\circ })x_i-0.03,\\ -\text{tan}({18}^{\circ })x_i+0.03\le z_i\le \text{tan}({18}^{\circ })x_i-0.03\end{array}$

5

• Limitation of possible placement locations

Figure 6 shows the elevated structures possible for placing Kinect in thick red lines. As the Kinects are to be placed on those edges, and their heights are fixed, z_state cannot be adjusted but depends on x_state, and the possible locations are reduced to 1-D state coordinate x_state ∈ [0, 12.73] along the edge of those structures (thick blue axes in Figure 6).

• Conversion of states to position and orientation

Figure 6.

Possible locations of cameras and the angle ψ = 0 for each side and corner (blue dashed lines and dotted arrows are added in the environmental-camera-only case).

Roll angle ϕ of the camera is always set as zero, while pitch angle θ is used directly, with positive pitch equals to the Kinect looking down from the horizontal level, that is, θ = θ_state. The yaw angle ψ_state is defined such that yaw_state is perpendicular to the edge, or halving the corner in case of corners, as shown by thin black arrows in Figure 6.

The extreme limits of the states are set as x_state ∈ [0 m, 12.73 m], θ_state ∈ [10^∘, 30^∘], and ψ_state ∈ [−70^∘, 70^∘]. The boundary for readjustment is set as x_b = 0.5 m for x_state, and θ_b = ψ_b = 15^∘ for θ_state and ψ_state, subject to the extreme limits. Other PSO’s parameters used are provided in Table 1.

Table 1.

PSO’s parameters used in the optimization simulations.

Parameter’s name	Value
No. of particles	50
Maximum iterations	200
C1, C2, C3 (scaling factor)	0.5
Δgmin (min. step of position for convergence)	10−8
Δfmin (min. step of obj. value for convergence)	10−8

PSO: particle swarm optimization.

Optimization result

The optimization script was run for 20 times, with each run resulting in different number of cameras required, ranging from 5 to 7 cameras. Each run took around 1–3 days to complete, depending on the number of Kinects to use.

The run which resulted the lowest number of cameras, that is, 5, was selected. The final readjustment in the section “Process of final optimization” was applied to the result. This process took more than a week before converging to the final result. The plots showing the position of the cameras, with their reduced fields of view and the number of cameras covering each point are shown in Figure 7(a) and (b). The combination of all five Kinects is drawn in 3-D in Figure 7(c).

Figure 7.

Camera configuration of the final optimization result.

Optimization using only environmental cameras

The optimization problem utilizing only environmental cameras was also completed for comparison of the number of required cameras discussed in the section “System configuration.”

Using only environmental cameras to track the face, the requirement for coverage is increased. Instead of only covering all the points, we need to cover the points from all directions to obtain the face. The camera needs to look into the face, with the maximum deviation angle from the camera’s principal axis less than or equals to 30°.

As the optimization is for comparison with the proposed method, the simulation’s conditions and steps were maintained to be as similar as possible. In addition to covering all points in the ROI, the angles which can be covered by the quadrotor in the proposed system must also be covered. The subroutine of final readjustment is also removed to reduce time as only the number of cameras is needed.

Using the same possible locations for camera placement obviously cannot cover all angles as no cameras can observe the face in +y direction. Therefore, additional locations were added along the dashed blue lines in Figure 6, with black dotted arrows show the central angle.

Figure 8 shows the top view of the person relative to the camera. θ_p is the angle of the person’s location from the camera’s principal axis, which can be obtained by

${\mathit{\theta }}_p=\text{arctan2}(y_p-y_c,x_p-x_c)-{\psi }_c$

6

where (x_p, y_p) and (x_c, y_c) are the position of the camera and the person, respectively, and ψ_c is the rotation angle of the camera from x-axis. So, the person is inside the camera’s FOV if |θ_p| ≤ α_{c, horizontal}, camera’s horizontal viewing angle.

Figure 8.

Position and rotation of a person relative to a camera.

If (x_p, y_p) is not in the FOV, then all angles cannot be seen by this camera. If it is inside, the angle α can be obtained by

$\alpha ={\psi }_p-{\psi }_c-{\mathit{\theta }}_p-{180}^{\circ }$

7

where ψ_p is the rotation angle of the person from x-axis. So

$\left|\alpha \right|\hspace{0.17em}\le {30}^{\circ }\to \text{Face at }(x_p,y_p)\text{looking at }{\psi }_p\text{ is detected}.$

8

The condition is checked for (x_p, y_p) ∈ ROI and ψ_p that can be tracked by the quadrotor without getting out of area.

The simulation was run with the angle steps of 1°, 2°, and 3°. Some angles at positions on the top left corner cannot be covered by the possible camera positions, resulting in incomplete solutions. The simulations were stopped after there was no improvement, and the lowest number of cameras obtained was 44 cameras in 1° step simulation, as shown in Figure 9 (red areas show uncovered face angles and positions). There are a large number of cameras for real implementation compared to our proposed system, which requires only five environmental cameras and some moving cameras, basically one camera per one person.

Figure 9.

Result of optimization with only environmental cameras: 44 cameras.

Sensor fusion and quadrotor’s tracking and control

As multiple Kinect sensors are used, a quadrotor or a person is likely to be detected by more than one Kinect, so it is necessary to correctly correlate data from the same object together. Data fusion is implemented by simple weighted averaging, whose weights are determined empirically.

Person fusion and tracking

The positional data for a person are 3-D positions (x, y, z) relative to the world’s coordinate. For the orientation, only a rotation angle is required, as looking up or down cannot be detected by the human-tracking algorithm. Rotation angle is taken as the orientation of the whole body as looking left and right cannot be observed.

At the beginning, a tracking_list is initialized as an empty list. In each sample, if tracking_list is empty, all detected heads are grouped as the same head by Euclidean distance. The position and orientation of each head is averaged in the group. On the other hand, if tracking_list is not empty, the detected heads will be correlated to those in the tracking_list. It is considered as belonging to the same person if they are close enough. Old and new data are averaged to x_avg, y_avg, z_avg in the ratio of 3:1, in which the new data are equally averaged among the new data.

If there are any heads left after the matching process, the remaining heads are grouped, averaged, and added to the tracking_list. The frames which are not updated for too long are eliminated (considered as the person has left the area and the tracking is lost). At the end, tracking_list is published to Robot Operating System (ROS) in the form of tf frames. The fused head is published as a frame with origin at the averaged (x _avg, y _avg, z _avg), rotated by ψ_{z, avg} around the world’s z-axis so that the frame’s x-axis points into the person’s face.

Quadrotor fusion and tracking

Kinect sensors can only detect the quadrotor’s 3-D position, not the orientation, by the depth images. Positions of the quadrotor are fused to provide 3-D position of the quadrotor in the world’s coordinate. To handle noise and false detection, the information about the number of cameras seeing the same object and the number of quadrotor in use is utilized. The orientation of the quadrotor is obtained from the on-board IMU and later coordinated with the 3-D position.

The fusion process is generally similar to the fusion for humans, with some modifications. As the size of the quadrotor is quite small, false detection is possible after background subtraction. If there are more than one sensor detecting an object around the same position, there is higher possibility that the object really exists at that position. A boolean flag tracking is introduced: It is set to false if there is only one Kinect seeing this object at the first detection, and true if there are two or more Kinects. The number of detecting cameras is also used to arrange the order of the quadrotors in tracking_list during the initialization.

After tracking flag is set and the quadrotor is kept in tracking_list, it is used in the tracking process. If the flag is false and in the next time step more than one camera can see this object, it is considered as a quadrotor and tracking becomes true. On the other hand, if no camera can see this object in the next time step, the previous detection is considered as a noise, and it is removed from tracking_list.

The number of detected quadrotors is also limited to the number of quadrotors being used (n _quad), which is generally known beforehand. The length of the list is truncated to n _quad as it is impossible to have more quadrotor than that.

The new data are matched with the tracking_list in the same way as fusion of the head. As the quadrotor tends to move faster, the ratio of old data to new data is 1:1.

After the matching process, if there are any points left, they are grouped together by distance, averaged, and added to extra list. If there is a true flag in extra and the length of tracking_list is less than n _quad, the quadrotor in extra with true flag is added to tracking_list. If the length of tracking_list is n _quad, but there is a false flag in the list, then extra‘s quadrotor with true flag replaces that false quadrotor. The process is repeated until there is no more true flag in extra or false flag in tracking_list. Before publishing tracking_list to ROS’s tf topic, the outdated quadrotor is removed from tracking_list.

Quadrotor control

We proposed the threshold algorithm, ¹⁴ with the experiments using one Kinect for controlling a quadrotor to track a person. In this article, the same algorithm is used with multiple Kinect sensors controlling one quadrotor.

When a person enters the area, the head is detected and fused, and the goal ${({\mathit{x}}_d^T,{\psi }_d)}^T$ is calculated. Instead of directly using this goal for the control, it is averaged for 1 s, and then set as the goal for the quadrotor, that is, ${({\mathit{x}}_{d,\text{f}}^T,{\psi }_{d,\text{f}})}^T=({\overline{\mathit{x}}}_d^T,{\overline{\psi }}_d{)}^T$ . A threshold ${({\mathit{x}}_{th}^T,{\psi }_{th})}^T$ is applied around the fixed goal position.

From this point, new head’s position and orientation are constantly used to compute goal, while the quadrotor’s goal is kept unchanged. If the person is standing still, the new goal is only affected by the noise and should not go beyond the threshold consecutively for some time. If the new goal breaks the threshold consecutively for 1 s, it is recognized as the person is in movement and the quadrotor’s goal ${({\mathit{x}}_{d,\text{f}}^T,{\psi }_{d,\text{f}})}^T$ should be recalculated. The process of goal averaging then starts again. In this way, quadrotor’s goal is not frequently changed, resulting in less oscillating trajectory, but the quadrotor can still move correspondingly to track human face. Sudden change of goal position can also cause a problem to the proportional integral derivative (PID) controller, as the derivative value can be very high, resulting in large control command and the instability of the control system. To reduce the effect of this surge and provide smooth trajectory, a simple low-pass filter is applied to the goal position output from the threshold algorithm by

$\left(\begin{array}{c}{\mathit{x}}_{d,q}\\ {\psi }_{d,q}\end{array}\right)[i]=w\left(\begin{array}{c}{\mathit{x}}_{d,q}\\ {\psi }_{d,q}\end{array}\right)[i-1]+(1-w)\left(\begin{array}{c}{\mathit{x}}_{d,\text{f}}\\ {\psi }_{d,\text{f}}\end{array}\right)[i],$

9

where ${({\mathit{x}}_{d,\text{f}}^T,{\psi }_{d,\text{f}})}^T[i]$ is the output of the threshold algorithm at time step i, $({\mathit{x}}_{d,q}^T,{\psi }_{d,q})[i]$ is the filtered goal to be used by the PID controller to control the quadrotor at time step i, and 0 ≤ w ≤ 1 is the weight of the filter.

Experiment and results

For evaluating the optimization result and tracking performance, Kinects were set up following the optimization, and tracking experiment was conducted. Position-tracking accuracy and tracking success ratio were also investigated.

System and implementation

Hardware components

The system composes of: Xbox 360 Kinect sensors, Crazyflie Quadrotor, and controlling computers.

• Xbox 360 Kinect

Xbox 360 Kinect has four major components: RGB camera, 3-D depth sensor (IR projector and IR camera), microphones, and accelerometer for reading current tilt angle. ²⁴

Xbox 360 Kinect uses structured light pattern and triangulation in depth calculation. The IR emitter projects a constant pattern of IR light speckles onto the scene, where the IR camera captures those speckles and correlates them with the memorized reference pattern to give depth values. ²⁵ Depth images are transmitted at 30 Hz for 640 × 480 pixels resolution.

Interference between multiple Kinect sensors

Multiple Kinects project unmodulated IR patterns into the same area and therefore detect multiple patterns, resulting in confusion and loss of depth information. A mechanical modification using a vibrating unit, made of a simple DC motor and an unbalanced load, to vibrate each Kinect was proposed by Maimone and Fuchs ²⁶ and Butler et al. ²⁷ The motion synchronously moves the Kinect’s projector and IR camera and therefore its own pattern is clearly visible. On the other hand, patterns from other Kinects move in different direction and frequency and are blurred. Altogether, the depth sensing of the Kinect is recovered.

As the system is relatively easy to reproduce, the method is chosen to solve the interference problem. The vibration has a drawback on blurry RGB images, but as they are not used in our processing, this does not affect the system. The other drawback is that the vibration creates a disturbing sound, possibly due to multiple movable parts of the structures holding the cameras. Redesigning the structure by adding some cushion and reducing excessive vibration may help reducing the noise, but this will be ignored at this point.

• Small-sized quadrotor

Even though large quadrotors are more stable, controllable, and have more payloads, in an indoor environment with people, the size of the selected quadrotor matters. Concerning about the programmability and modifiability, Crazyflie 2.0 was chosen.

The Crazyflie 2.0 is a tiny quadrotor built using the PCB as the frame. ²⁸ With open-source firmwares and libraries, wiki page, and active communities of developers, it is one of the platforms suitable for developments. Its tiny size and light weight make it a good choice for indoor use with people. The specification is given in Table 2.

Table 2.

Specifications of Crazyflie 2.0.

Size	92 × 92 × 29 mm (motor-to-motor, including motor’s feet)
Weight	27 g
Flight time	7 min (no load)
CPU	32-bit, 168 MHz
Sensors	10-DOF (accelerometer, gyroscope, magnetometer, and pressure sensor)
Payload	max recommended 15 g

Software implementation

The codes were improved from the original code by Dunkley, ²⁹ which uses one Kinect to detect and control one quadrotor by setting goal position using graphical user interface or Playstation3 Joystick. The system is implemented on ROS. ³⁰ The programs were modified to use the goal position based on human tracking. The system was also expanded to use multiple Kinects across multiple computers. Final block diagram is illustrated in Figure 10, using five Kinects running on three computers with different specifications.

• Quadrotor detection

Figure 10.

Block diagram of the final system.

Each Kinect individually detects quadrotor by background subtraction on depth images. To increase the surface area, paper is attached on the quadrotor, as shown in Figure 11. Colors added were for visibility of the direction of the quadrotor in video recording and were not used in detection. The algorithm is adopted with little modification.

Figure 11.

Crazyflie 2.0 with paper for depth image detection.

Prior to the quadrotor detection, the background is created from the minimum value of each pixel in a period of time, which corresponds to the nearest depth. Then, new depth images are compared to the background for any additional objects, which are classified as a quadrotor based on the detected size so that small noise and human being are not detected as a quadrotor. The detected quadrotors’ positions are published into ROS in the form of tf frames, where they are processed further by the sensor fusion process to merge data of the same quadrotor into a single one.

• Human skeleton tracking

In ROS, openni_tracker ³¹ package is available for detection and tracking of humans using depth images based on the open-source OpenNI framework. ³² The package outputs the positions and orientations of 15 body joints in the form of ROS’s tf frames. From our observation, the orientation of the head joint is perpendicular to the line connecting two shoulders. openni_tracker initially supports human skeleton tracking using only one RGB-D sensor. Some modifications have been made so that it can be used with multiple Kinects.

• Sensor fusion and tracking

A quadrotor or a person is likely to be detected by more than one Kinect, so correct correlation and fusion are necessary, as discussed in the section “Sensor fusion and quadrotor’s tracking and control.”

If data from different Kinects are close enough, they are considered as the same person or quadrotor and are fused together by averaging. In case of quadrotor, if more than one Kinect can see the same object, that object is more likely to be the quadrotor. The number of detected quadrotors is limited by the number of quadrotors being used, predefined by the user. Tracking of both person and quadrotor is done based on the previous data.

• Goal position setup

The camera must be in front of the person, with the camera facing to his/her face to capture the face. The fused head has x-axis pointing into the face and z-axis pointing up to the ceiling as shown in Figure 12. The angle ${\widehat{\psi }}_{\mathit{\text{head}}}$ is therefore the frame’s yaw angle. The goal position $\hspace{0.17em}\mathit{x}{}_d$ is defined at a distance d_goal in front of the face and h_goal above. These values should be adjusted according to the camera’s specification and settings. For the current experiment, $d_{\mathit{\text{goal}}}=1.5\text{m}$ and $h_{\mathit{\text{goal}}}=0.6\text{m}$ . By assuming that the camera is installed along the quadrotor’s x-axis, rotating ${\widehat{\psi }}_{head}$ around the world’s z-axis will point the camera toward the face. So, we set ${\psi }_d={\widehat{\psi }}_{\mathit{\text{head}}}$ . $\hspace{0.17em}\mathit{x}{}_d$ and ψ_d are then used to set the goal frame for the quadrotor.

• Control system

Figure 12.

Goal position related to detected head (top view).

PID controllers are used to control the quadrotor. By setting the quadrotor’s x-axis to point in the forward direction, y-axis to the left, and z-axis upward, pitch angle controls x-axis motion and roll angle controls y-axis motion (Figure 13). Yaw angle ψ is the counterclockwise rotation angle around z-axis.

Figure 13.

From the current position to goal (top view).

The quadrotor receives the command set of roll, pitch, yaw speed, and thrust $(u_{\varphi },u_{\theta },u_{\psi },u_T)$ , and the on-board firmware calculates the power of each motor. The error is obtained by finding the transformation from the quadrotor’s frame to the goal frame, resulting in e_x, e_y, e_z, and e_ψ, which are used to compute u_θ, u_ϕ, u_T, and u_ψ, respectively. For x, y, and ψ, proportional derivative (PD) controllers are used, with the same coefficients for x and y. The integral part is included for z to compensate the power drop during the flight.

Camera placement and calibration

The method of calibrating relative position and orientation between two cameras was performed. Another camera was added as the reference for the calibration, with its FOV intersecting with other cameras’ FOVs. Its orientation was also set to be simple, that is, $\varphi ,\mathit{\theta },\psi \in \{0,\pm \frac{\pi }{2}\}$ to minimize possible error in setting up the reference, comparing to setting up at any specific non-zero angle of one of the five cameras.

In ROS, there is a package, camera_pose_calibration, ³³ for extrinsic camera calibration. To calibrate for depth sensor, the images from the IR camera are used, with the IR pattern projector covered. The calibration process involves holding a checkerboard in different locations, outputting each camera’s position and orientation.

The calibration was done one-by-one between each camera and the reference camera. For each camera, after the coarse placement, its relative position and orientation was calibrated. If the error from the optimization result exceeds the error limit assumed for the optimization (i.e. 2° for angles and 0.03 m for position), the adjustment is made and the calibration process is repeated until the error is within the error limit. Figure 14 shows the Kinects set in place.

Figure 14.

Kinects set up in position from optimization result: 5 Kinects above, with the reference camera on the tripod.

Experimental setup

An experiment with one moving person was set to test the constructed system. A path was created to cover the space as shown in Figure 15, where the blue dots are the approximate positions (marked by markers on the floor for the person to stand), the black arrows show the approximated facing direction, and the dashed red line shows the order of moving with the movement numbers. The path was designed such that the quadrotor’s goal will not be outside the area, with some margins to allow errors. The path ends in the center and the person rotates and stops at around $-\frac{\pi }{2}$ , −π, $\frac{\pi }{2}$ , and 0 radian, respectively, before finishing at $-\frac{\pi }{2}$ radian.

Figure 15.

Path for the experiment with the area’s boundary.

The quadrotor’s detection size was set to (10±3)×10⁻² m. PID controllers’ coefficients are given in Table 3. Threshold values of 0.15 m, 15° were applied, with the weight w in equation (9) set as 0.98. The displacement threshold for quadrotor fusion and head fusion was set to 0.5 m.

Table 3.

PID controllers for each dimension.

x, y			z			ψ
Pxy	Ixy	Dxy	Pz	Iz	Dz	Pψ	Iψ	Dψ
1.2	0.0	0.6	4.5	4.0	6.0	0.9	0.0	0.2

The concern about safety problem was not explicitly dealt with in this implementation. The goal position of the quadrotor was set to be higher than the height of the person for 0.6 m, and 1.5 m away in the horizontal direction to reduce the risk of collision, but there was no limit on the closest distance the quadrotor can be from the person.

For better evaluation, the motion capture system by motion analysis ³⁴ was used for more accurate positions and orientations. As two systems were run separately, time synchronization was manually done. Our motion analysis uses 12 units of “Kestrel Digital RealTime System,” with 3 units at each corner of the area. Each unit can record up to 2048 × 1088 pixels at 300 fps and can provide the position of each marker at millimeter-level accuracy. The output of the position of each marker was set at 120 Hz in the experiment.

For the head’s position and rotation, the person wore a cap with four markers. On the quadrotor, four markers were attached to the tip of each wing, with another marker in front.

Results

Tracking of person’s position

The person’s 2-D positions measured by the system are compared to those obtained by motion analysis system in Figure 16. Black dots indicate the markers’ positions, blue solid line shows the person’s path tracked by motion analysis, while dashed red line draws the path tracked by the proposed system. Black solid line draws the boundary of the experiment area. Plots of x, y, z, and ψ of the person’s head are shown in Figure 17.

Figure 16.

2-D projection of the person’s trajectory as tracked by motion analysis and the proposed system, with the markers.

Figure 17.

Comparison of position tracking of person’s head by the proposed system with motion analysis.

Tracking and controlling of quadrotor

The quadrotor’s 2-D positions tracked by motion analysis (blue solid line) and the proposed system (dashed red line) are plotted in Figure 18. This shows that the experiment involved controlling the quadrotor throughout the area. There were also some moments in which the quadrotor went beyond the limit, but the system could still manage to control it back into the region.

Figure 18.

Trajectory of the quadrotor tracked by motion analysis and the proposed system on xy-plane.

Human-tracking result

Figure 19 shows the snapshots of the tracking experiment. Graphs showing the quadrotor’s position and orientation compared to the goal position set up by the person’s position are plotted in Figure 20. Dotted black line indicates the thresholded goal position and orientation, while dashed red line indicates the quadrotor’s fused position and orientation. Blue solid line shows the position and orientation tracked by motion analysis. The average absolute errors, comparing motion analysis result to the goal position, were 0.23 m (SD 0.17 m) for 3-D position and 0.15 rad (SD 0.13 rad) for rotation angle. Video of the tracking experiment can be found in the supplementary file.

Figure 19.

Snapshots of quadrotor following the person.

Figure 20.

Comparison of position tracking of quadrotor by the proposed system with motion analysis.

Discussion

The error of the tracking result compared to the ground truth from motion analysis was calculated. As the data rate of the motion capture system is higher (120 Hz compared to approximately 30 Hz), motion analysis’s data were down-sampled and only the data with the closest time-stamp with the Kinect’s data were used in calculating error. Means and standard deviations of the errors, d: displacement obtained by $\sqrt{e_x^2+e_y^2+e_x^2}$ and ψ: rotation angle, are shown in Table 4.

Table 4.

SD of the system with reference to motion analysis.

Value	Errors for head		Errors for quadrotor
	Mean	SD	Mean	SD
d (m)	7.8 × 10−2	3.5 × 10−2	6.5 × 10−2	3.6 × 10−2
ψ (rad)	2.1 × 10−1	1.8 × 10−1	9.4 × 10−2	4.9 × 10−2

SD: standard deviation.

The accuracy of the system is within 10 cm for the position and about 0.2 rad (approximately 12°) for the direction. The result also shows that IMU can provide accurate orientation of the quadrotor in a short time. The error in the quadrotor’s rotation was partly due to the setup of the quadrotor’s initial orientation, that it might not be exactly zero. The effect of the gyroscope drift is also visible during the end of the experiment, where the errors of yaw angle tend to increase. Possible solutions to this can be the attachment of camera and using the face for quadrotor control, or the use of magnetometer to obtain global orientation relative to the Earth’s magnetic field. There can be some improvement to the precision of the person’s direction, which would help set the goal more precisely.

We analyzed the number of cameras tracking the person and the quadrotor by using the positions obtained from motion analysis system, the calibrated camera’s positions, and the full FOV model of the Kinects. Position and orientation tracking errors divided by the number of cameras, coded by colors, can be found in Figure 21. All time samples had at least one camera seeing the person and the quadrotor. For the person’s head, the smallest number of tracking cameras was 2, and there were only a few time samples in which only one camera was tracking the quadrotor. From this result, we can see that the optimization algorithm can deploy the cameras such that at least one camera is observing the object inside the predefined area.

Figure 21.

3-D Euclidean distance, angle α, and height difference between the person and the quadrotor (the numbers show the order of movement).

By analyzing further into the relationship between the number of tracking cameras and the tracking error, we can see that the error significantly changed, either increased or decreased, when the number of tracking camera changed. This corresponds to the object crossing one of the FOV borders and that the object was moving. We can also see that the moment in which only two cameras were tracking the person or the quadrotor, the tracking error got considerably high. Adding more cameras to cover the area with low coverage may help reduce the tracking errors, but is not guaranteed as seen in the end of the experiment. This, anyway, certainly increases the total number of cameras.

Based on the position of the quadrotor and the relative orientation with the person, our positioning system could guarantee full coverage, get good estimate of the positions and orientations, and use them to control the quadrotor to perform tracking of person based on the obtained position and orientation, keeping good control when the person was not moving. The quadrotor could also adjust to the movement of the person and recover the desired position, distance, and orientation.

Conclusion and future works

A position-tracking system was proposed in this article using multiple Kinect sensors fixed in the environment. Position and orientation of person was tracked, together with the position of a flying quadrotor. The quadrotor was controlled based on the positions and orientations to follow the movement of the person, mimicking the attempt of face tracking. The experiment inside a 3-by-3.5-m area showed that our optimization algorithm guarantees full camera coverage in the area, and our system can respond to the movement of the person properly, keeping good position and orientation for performing face tracking. With the addition of camera onto the quadrotor and the feedback of the images, the quadrotor’s position and orientation can be finely adjusted to obtain better images of the face.

Apart from addition of on-board camera, the expansion of the system to perform tracking of multiple people with multiple quadrotors is possible. Topics such as collision avoidance and matching and switching of human-quadrotor pairs when people are moving need to be concerned in future works. Safety issue should also be addressed, for example, minimum distance should be implemented such that the quadrotor quickly moves away from the person when it gets too close. The attitude of people toward flying quadrotors should also be studied to find the best distance which does not make people feel dangerous or being offended.