Intrusion Detection and Cooperative Tracking Using PTZ Network Thermal Imagers

Abstract

Nowadays more and more network thermal imaging cameras are working over distributed networks, offering the capability of online remote intelligent video surveillance. In this paper, we propose an original intrusion detection and cooperative tracking approach applied for PTZ (Pan/Tilt/Zoom) network thermal imagers. It consists of three modules. The key module is the real-time FOV (Field of View) matching module, which is realized in a parallel way. The intrusion detection module first eliminates the offset between current frame and prior frame through FOV matching and then handles intrusion detection by motion detection in the preset surveillance zone. The cooperative tracking module shifts the priority of tracking by imager pose estimation, which is also based on FOV matching, avoiding transferring the local features from one imager to another. Experiments are conducted to demonstrate that the proposed approach is of high accuracy for intrusion detection and cooperative tracking and keeps the frame rate over 20 fps.

1. Introduction

Nowadays more and more network cameras and thermal imagers are working over distributed networks, offering the capability of online remote intelligent video surveillance for antiterrorism and civil protection [1].

Much research has been done for object detection and tracking using a single camera [2–4]. Benfold and Reid designed a multitarget tracking system, which detects objects using a HOG based detector and guarantees fast tracking with the KLT method and MCMCDA [2]. Andriyenko et al. presented a model to reason about occlusions, especially the interobject occlusions [3]. Robust and reliable tracking in complex scene can be easily achieved through this method. Moreover, nonlinear models learned online are adopted for explaining nonlinear motions and appearance in the tracking framework [4]. Recently, the multicamera surveillance system becomes more and more popular for its sufficient coverage compared to single camera. Führ and Jung presented a robust pedestrian tracking approach with monocular calibrated static cameras using patch matching in object detection [5], and Liem and Gavrila realized a system with overlapping cameras for detecting and tracking multiple pedestrians [6].

Though intrusion detection and cooperative tracking using one or multiple static cameras have been already achieved with high accuracy, they are hardly applied for PTZ (Pan/Tilt/Zoom) cameras or thermal imagers when working under patrol mode. This is because existing solutions are generally based on motion detection that needs static background, but the FOV (Field of View) usually changed continuously when using PTZ cameras or thermal imagers. Thus, obtaining the FOV offset accurately and getting the thermal imager pose simultaneously are the key points to realize intrusion detection and cooperative tracking using PTZ network thermal imagers [7].

To address the problems mentioned above, we proposed an original intrusion detection and cooperative tracking approach. The proposed approach includes three major modules. They are the FOV matching module, the intrusion detection module, and the cooperative tracking module. The key module is real-time FOV matching, which is realized in a parallel way that can be substantially optimized for multicore processor. One thread tracks the key points employing an ultra-low-weight key point description and hybrid spill-tree, and the other thread handles three-dimensional mapping. The intrusion detection module first eliminates the offset between current frame and prior frame through FOV matching and then handles intrusion detection by motion detection in the preset surveillance zone. The cooperative tracking module shifts the priority of tracking by imager pose estimation, which is also based on FOV matching, avoiding transferring the local features from one imager to another.

The remainder of this paper is organized as follows. Section 2 gives the implementation details of the FOV matching module. Section 3 introduces the implementation of the other two modules. In Section 4, experimental results are presented. Lastly, in Section 5, conclusions are presented.

2. Real-Time FOV Matching

As mentioned above, the real-time FOV matching module is realized in a parallel way which is suitable for optimization by multicore processors. This module includes tracking thread and mapping thread, partially inspired from the framework of PTAM [8, 9]. However, FAST corner based registration [10] is substituted by our ultra-low-weight registration algorithm [11].

2.1. Tracking Thread

Modern detectors tend to locate key points by simply examining the intensities of certain pixels around the tentative locations in order to guarantee sufficient number of candidate key points [12–14]. Following the methodology similar to the algorithms proposed in [12], we consider only the intensities along a circle of 16 pixels around each candidate key point.

We locate a candidate key point at a pixel P if the intensities of at least 12 contiguous pixels are all above (negative) or all below (positive) the intensity of P over a certain threshold. Usually, a featureless key point can be rejected quickly without scanning the entire circle.

Once a key point has been located, the intensities of every three contiguous pixels on the discrete circle are weightedly summed, resulting in a total of 16 sums. The 16 sums are utilized to construct a compact descriptor D. In fact, the intensity is substituted by the gradient G from the point on the circle to its center P in order to resist the illumination change. Assuming that $R_i$ is a pixel on the circle R, $R_{i\text{cw}}$ and $R_{i\text{ccw}}$ are the neighboring points in the clockwise and counterclockwise directions of R, respectively. Each element in the descriptor D can be described as

$$\begin{array}{l}{D}_i=a·G_{R_{i\text{cw}}\to P}+b·G_{R_i\to P}+c·G_{R_{i\text{ccw}}\to P}\hfill \\ i\in \left[\mathrm{1,16}\right].\hfill \end{array}$$

(1)Here $\left[\begin{array}{ccc}a& b& c\end{array}\right]$ is the weight and is chosen as $\left[\begin{array}{ccc}0.5& 1.0& 0.5\end{array}\right]$ by trial and error.

The largest sum is chosen as the first element of the descriptor, and the vector is filled with the remaining sums of the circle in a clockwise direction. This simple process is sufficient for rectifying the detected key point with respect to in-plane rotation. Moreover, it considerably reduces the computational cost in calculating the orientation of the small image patch and sorting the descriptor elements using the mean squared deviation (MSD) [15]. Then, the descriptor vector is normalized in order to remove the variance of contrast. The polarity of every descriptor is also recorded such that the key points can be categorized as positive or negative. This partition is efficient for matching since positive features need not be compared with negative ones.

In the matching section of the tracking thread, the hybrid SP-Tree is employed for its high efficiency [16]. Finally, RANSAC, as the effective way of eliminating spurious matches, should not be omitted [17]. The authors do not intend to give a detailed description here, as similar work can be found in [11, 18].

2.2. Mapping Thread

In this thread, a map consists of a collection of M key points located in a world coordinate frame. Each key point represents a locally planar textured patch in the world. The map also contains N key frames. These key frames are the snapshots taken by a thermal imager at various points in time. A map usually contains 2000 to 6000 points and 40 to 120 key frames.

Once an initial map is built using a stereo method, the map is continually refined and expanded as new key frames are added by the tracking thread. This thread is partially realized by PTAM. For more details, please refer to [8, 9].

3. Intrusion Detection and Cooperative Tracking

3.1. Intrusion Detection

The intrusion detection module is used for real-time alerting in case of intrusion into preset surveillance zone. There are many approaches for intrusion detection in a continuous video stream. They are mainly based on motion detection, namely, comparing the current frame with one from the previous frames. We could get a binary image by comparing each pixel between current frame and previous frame. It is possible to count the number of pixels, and if the amount is greater than a predefined alarm level we can signal about a motion event.

It should be noted that moving objects, such as pedestrians and automobiles, usually have higher intensities compared with the background in grey-level pseudocolour mode of thermal imager. According to this phenomenon, the negative impact coming from the environmental change in FOV, such as the changing shadows and the trembling of leaves, can be eliminated. Figure 1(b) shows the reconstructed frame after foreground segmentation of Figure 1(a).

Figure 1

Background segmentation. (a) A frame acquired by the thermal imager and (b) the frame after background segmentation.

After compensating the FOV offset obtained from the real-time FOV matching, it can facilitate a reliable detection of unauthorised accesses under arbitrary weather conditions.

3.2. Cooperative Tracking

When an intrusion event is detected, the related PTZ thermal imager tracks it using an improved particle filtering method and estimates the pose of the imager simultaneously utilizing the information obtained from FOV matching module. Once the intrusion object enters an overlapping FOV of two imagers over the network, the handoff thermal imager is activated immediately according to the spatial relationship built by pose estimation and turns to tracking and pose estimation status. Meanwhile, the previous thermal imager returns to waiting status, waiting for reactivation from a new cooperative tracking event or a new intrusion event. Figure 2 shows the flowchart of the proposed cooperative tracking framework.

Figure 2

Flowchart of cooperative tracking module.

Particle filtering is an effective approach for object tracking [19]. The core of particle filtering is to approximate the relevant distributions with random measures composed of particles. For the reason that the particle swarm optimization algorithm can search the sample area around the last time object position depending on current observation, it can distribute the particles in a high likelihood area even though the dynamic model of the object cannot be obtained. Hence, we adopt an improved particle filtering approach based on particle swarm optimization (abbreviated as PSO-PF) in this module to handle the object tracking [20, 21]. PSO-PF algorithm does not distribute particles like sampling-importance resampling (SIR) for the reason that sampling-importance resampling usually distributes particles in the wrong likelihood area, particularly when tracking objects in a cluttered scene. PSO-PF first finds the sample area by PSO algorithm and then distributes the particles based on two different base points in order to achieve diversity and convergence in feature matching. For more details, please refer to [7].

4. Results

We have evaluated the proposed approach on an ordinary workstation with Intel Q9 series CPU and 4 GB RAM. It can handle 2 channels of standard D1 ( $704\times 576$ ) live video stream over 20 fps.

Now the approach has been integrated in the intelligent video surveillance system of Chongqing Rail Transit. Four thermal imagers and cameras are used to monitor an area of more than 3000 square meters. According to the test report, the false alarm rate (FAR) of the proposed approach is below 0.0021%, and the missing report rate (MRR) is below 0.0060%. Table 1 shows the false alarm rate and missing report rate of 10-week observation acquired from Chongqing Rail Transit.

Table 1

The false alarm rate and missing report rate of 10-week observation.

	Week										Mean	Standard deviation
	1	2	3	4	5	6	7	8	9	10
FAR	0.003	0.000	0.003	0.001	0.003	0.002	0.005	0.000	0.002	0.002	0.0021	0.0014
MRR	0.004	0.009	0.017	0.002	0.001	0.010	0.004	0.002	0.007	0.004	0.0060	0.0046

5. Conclusions

In conclusion, an effective intrusion detection and cooperative tracking approach which is suitable for PTZ thermal imagers is proposed. It has been proven in our tests that this approach not only runs fast, but also has the capability of tolerating the change of circumstances. Now this approach has been applied in Chongqing Rail Transit successfully.