Fast Object Tracking Employing Labelled Particle Filter for Thermal Infrared Imager

Abstract

More and more network cameras are now working over distributed networks, offering the capability of remote intelligent video surveillance. In this paper, we bring forward an original particle filter tracking algorithm named labelled particle filter which describes each image patch with a binary label. Based on the imaging theory of thermography, moving objects, such as pedestrians and automobiles, usually have higher intensities compared with the background in a gray-level pseudocolor mode. Thus an image patch can be classified into two categories according to its intensity distribution, and we can use a one-bit binary label, positive or negative, to describe the attribute of image patch. Therefore, the candidate target template is established only if the label of candidate target matches the label of reference target, and the computational complexity is reduced consequently. Experiments are conducted to show that the proposed algorithm can handle real-time object tracking with less time cost while maintaining high tracking accuracy.

1. Introduction

Visual tracking, which provides cohesive information about the target objects, has been extensively used in computer vision, especially in intelligent video surveillance for antiterrorism and civil protection [1]. With the tendency that thermal infrared imagers are being widely applied in distributed networks, robust object tracking can be achieved for the desirable property, as the background is relatively simple in thermal infrared video.

The main difficulties for the detection and tracking of infrared targets, which are lack of texture and color information, are summarized into two aspects, namely, the requirements for robustness and meanwhile real-time performance of the algorithm. The challenge becomes greater for achieving these goals, when tracking small blurred target, and the object contains rapid and arbitrary target motion. Numerous algorithms have been proposed for addressing these issues, including the Kalman filter techniques [2–4], the mean shift algorithms [5–7], and the particle filtering methods [8–10]. The approaches can be divided into two categories: the deterministic methods and the stochastic methods. Deterministic methods usually involve a gradient descent search to minimize a cost function. By contrast, stochastic methods introduce some stochastic factors into the searching process, having a higher probability of reaching the global optimum. Recently, the particle filter, as a stochastic method, is the main approach to handle the object tracking tasks, due to its desirable performance in nonlinear and non-Gaussian state estimation [11]. It generates a set of random samples, which are propagated and updated recursively in order to approximate the state probability density function of the system. However, large calculated amount is brought about by the abundant particles, despite of the high precision.

In this paper, we propose an original object tracking algorithm with the name of labelled particle filter (abbreviated as LPF), on the basis of the particle filter. Our approach improves the description of target template and accordingly shortens the search stage, maintaining high accuracy with low computational cost. First, we classify the intensity distribution space into two categories; thus an image patch can be described with a label of one bit by classifying pixels in certain intensity level. Then, we establish the candidate target template only if the label of candidate target matches the label of reference target and adopt an improved particle filtering approach based on particle swarm optimization to track the selected image patch [12].

The remainder of this paper is organized as follows. The related work is reviewed in Section 2. The implementation details of labelled particle filter are explained in Section 3. In Section 4, experimental results are presented. Lastly in Section 5, conclusions are presented.

2. Related Work

The Kalman filter was first described in [13]. It is a state estimate method based on linear dynamical systems. Namely, the Kalman filter is a recursive solution to the streams of noisy input data producing an optimal estimate of the system state. The algorithm is widely used for object tracking. Chan et al. [2] use the Kalman filter to predict the location of the objects at time, $t+1$ , based on its previous state, and the observation is then corrected using the measurements in the current image. Wang et al. [3] proposed a visual tracking approach, by applying a Kalman filter to velocity vectors in the tangent planes of Grassmann manifolds. Tyagi and Davis [4] proposed a Kalman filter on the manifold for visual object tracking. The weakness of the methods is that they cannot well handle nonlinear and non-Gaussian visual objects.

Mean shift is a nonparametric and mode-seeking method for locating the maxima of a density function, which involves an iterative procedure [14, 15]. The approach was introduced to computer vision by Comaniciu et al. [5], aiming at locating the position of the object with the color histogram by finding the peak of a confidence map. For the scale selection after estimating the position, Collins [6] exploited an additional mean shift procedure combined with image pyramids. Yilmaz [7] also introduced an asymmetric kernel mean shift, in which the scale and orientation of the kernel adaptively change depending on the observations at each iteration. Although the mean shift algorithms are fast and easy to implement, the tracking accuracy substantially drops when the presence of significant background clutter, and more parameters, such as translation and scale, need to be estimated [16].

Particle filters are popular models for estimating the state of a dynamical system. Tracking using the particle filtering framework is based on Monte Carlo sampling and recursive Bayesian estimation [17, 18]. Monte Carlo based particle filter adopts the sequential importance sampling theory, which does not lead to filter divergence. It allows a posterior distribution, estimated in the previous image frame, to be sampled with a set of particles, and these particles are propagated iteratively to successive frames, using continuously updated observations and a prediction model. The method is a de facto standard, which has been popularly used to handle various tracking problems. Gustafsson et al. [8] designed a framework for positioning, navigation, and tracking problems using particle filters. Li et al. [9] proposed an incremental self-tuning particle filtering framework for visual tracking on the affine group. Yet particle filters degrade in performance as the dimensionality of the state space increases and the support of the likelihood decreases; Vermaak et al. [10] hence introduced a variational approximation to the tracking recursion. The variational inference is intractable in itself and is combined with an efficient importance sampling procedure to obtain the required estimates. However, it is still computationally expensive and, therefore, cannot meet the real-time tracking demand.

3. Labelled Particle Filter

Labelled particle filter is based on the particle filter also known as condensation filter. Particle filter is deduced by a filtering method based on Monte Carlo to solve the Bayesian estimation problem according to the law of large numbers [19].

The Bayesian state estimation for nonlinear and non-Gaussian tracking problems can be described by system equation and measurement equation, which can be described as

$$\begin{array}{cc}& {\mathbf{X}}_k=\mathbf{F}\left({\mathbf{x}}_{k-\mathrm{1}},n_{k-\mathrm{1}}\right),\end{array}$$

(1)

$$\begin{array}{cc}& {\mathbf{Z}}_k=\mathbf{H}\left({\mathbf{x}}_k,v_k\right),\end{array}$$

(2)

where ${\mathbf{X}}_k$ related to ${\mathbf{x}}_{k-\mathrm{1}}$ by the function F is the state vector in frame k, ${\mathbf{Z}}_k$ computed by function H is the observation vector, and $n_{k-\mathrm{1}}$ , $v_k$ represent the independent white noises, respectively. The functions F and H both are nonlinear.

The standard particle filter includes four parts, including particles initialization, target template establishment, particles tracking, and particles resample. In labelled particle filter, label establishment and particles selection are introduced to reduce the computational complexity. The process of labelled particle filter is as follows.

Note that moving objects, such as pedestrians and automobiles, usually have higher intensities compared with the background in gray-level pseudocolor mode of thermal imager. According to this phenomenon, the negative impact coming from the environmental change in field-of-view, such as the changing shadows and the trembling of leaves, can be eliminated. In order to enhance the robustness of the algorithm, we should segment the foreground from the frame which was directly acquired by thermal infrared imager before tracking.

3.1. Particles Initialization

The object is obtained by manual selection in the first frame. The particle set ${\{{\mathbf{x}}_0^i,w_0^i\}}_{i=\mathrm{0}}^N$ is generated by the prior probability $p({\mathbf{x}}_0)$ , and the initial particle weight is $w_0^i=1/N$ , where N is the total number of particles.

3.2. Label Establishment and Particles Selection

It is known that high tracking accuracy depends on large numbers of particles. More particles usually bring more abundant information about the object we are tracking. Nevertheless, they also produce redundancy which is not essential to the label establishment and cause huge computational complexity [20]. Thus, it is important to discard the particles that cannot describe the feature of object well.

First, we divide the image patches into two categories by the intensity distribution. The image patch mainly containing the object is defined as positive category. On the contrary, the patch mainly containing the background is defined as negative category. As a result, we can use a one-bit binary label to describe the attribute of image patch. We define L as

$$\begin{array}{cc}& L=\left\{\begin{array}{cc}\mathrm{1},\hfill & \text{if}\hspace{0.17em}\hspace{0.17em}I\left(x,y\right)\ge T\hfill \\ \mathrm{0},\hfill & \text{else},\hfill \end{array}\right.\end{array}$$

(3)

where $I(x,y)$ is the pixel intensity located at $(x,y)$ of the current image patch and T is a predefined threshold which can be acquired through intensity distribution from region of interest in sequences.

Then, we establish the candidate target templates only if the label of candidate target matches the label of reference target. Hence the computational complexity is reduced consequently.

3.3. Target Template Establishment

Researchers often utilize the color distribution, contour, and texture of targets to describe the features extracted from video images in the area of visual tracking, and the descriptors generated from the step can be further used to establish the target template. However, there is insufficient texture and color information in infrared sequences. Therefore, we can only use intensity distribution information to generate the target template and must provide stable performance in the case of scale changes and partial occlusion.

For the desirable property of catching spatial information, kernel color histogram [21] is commonly applied for the establishment of the target template. We first endow the pixels with different distances to the target centre and then select the intensity distribution with kernel function as the target template. It should be noted that different distances have their own different weights. Thus, target template can be established as follows:

$$\begin{array}{cc}& q_{\mathbf{u}}=C{\displaystyle \sum }_{i=\mathrm{1}}^M\left(K\left|\frac{{\mathbf{x}}_{\mathrm{0}}-{\mathbf{x}}_i}{h}\right|\right)\delta \left(b\left({\mathbf{x}}_i\right)-\mathbf{u}\right).\end{array}$$

(4)

The normalization constant C is defined as

$$\begin{array}{cc}& C={\left[{\displaystyle \sum }_{i=\mathrm{1}}^{M}K\left|\frac{{\mathbf{x}}_{\mathrm{0}}-{\mathbf{x}}_i}{h}\right|\right]}^{-\mathrm{1}},\end{array}$$

(5)

where ${\mathbf{x}}_0$ is the coordinate of the image patch center and ${\mathbf{x}}_i$ is the coordinate of the pixel in the image patch, K is the kernel function (generally Gauss function) related to the distance between ${\mathbf{x}}_i$ and ${\mathbf{x}}_0$ , $\delta (·)$ is the Kronecker Delta function, M is the total number of pixels in the patch, u is the index of intensity condensed, and $b({\mathbf{x}}_i)$ is the intensity level of the pixel point in the space condensed.

3.4. Particles Tracking

The state transformation of particles is calculated by system equation shown in (1) to predict state prior probability distribution. In other words, the state transformation is to achieve the state of particles in the current frame using the state of previous frames. We choose the second order regression model shown in (6) to perform the state transformation:

$$\begin{array}{cc}& {\mathit{X}}_k-{\mathit{X}}_{k-\mathrm{1}}={\mathit{X}}_{k-\mathrm{1}}-{\mathit{X}}_{k-\mathrm{2}}+n_{k-\mathrm{1}}.\end{array}$$

(6)

The particle weight at time k is computed by

$$\begin{array}{cc}& w_k^i=\frac{\mathrm{1}}{\sqrt{\mathrm{2}\pi }\sigma }{e}^{-d^{\mathrm{2}}/\mathrm{2}{\sigma }_i^{\mathrm{2}}},\end{array}$$

(7)

where d is the similarity between candidate model P and reference model Q which are computed using the Bhattacharyya coefficient as follows:

$$\begin{array}{cc}& d=\sqrt{\mathrm{1}-\rho \left(\mathbf{Q},\mathbf{P}\left(y_j\right)\right)}.\end{array}$$

(8)

Then the particle weight can be normalized by

$$\begin{array}{cc}& w_k^i=\frac{w_k^i}{{\sum }_{i=\mathrm{1}}^N{w}_k^i}.\end{array}$$

(9)

The estimation of x based on weighted particles is defined as

$$\begin{array}{cc}& {\widehat{\mathbf{x}}}_k={\displaystyle \sum }_{i=\mathrm{1}}^N{w}_k^i{\mathbf{x}}_k^i,\end{array}$$

(10)

where ${\mathbf{x}}_k^i$ is computed by (6).

3.5. Particles Resample

In this stage, sampling importance resampling based on the particle weight is adopted for particles degeneration. Then, turn back to the label establishment and particles selection stage and repeat the following procedures.

4. Results

The experiments are conducted by Matlab on an ordinary workstation with 3.2 GHz Xeon CPU and 4 G RAM. We evaluate the performance of the proposed algorithm on two datasets: one is the well-known dataset OTCBVS and the other is the image sequences from surrounding scenes which is acquired through Sheenrun thermal infrared imager.

We compare the proposed LPF algorithm against the mean shift and the standard particle filter algorithms. The tracking results are shown in Figures 1 and 2, respectively. It can be seen that the three methods can obtain accurate locations of the target on the OTCBVS dataset. However, when tracking on the sequences acquired from Sheenrun imager, whose backgrounds are more complex, the mean shift algorithm cannot track the object at frames 120 and 150 exactly; by contrast, our LPF algorithm and the standard particle filter can still achieve accurate tracking. The comparison of time cost between the three approaches is shown in Tables 1 and 2. It shows that the time cost of tracking using our LPF method can be approximately 1.3 times faster than traditional particle filter.

Table 1

The comparison of time cost and average frame rate when tracking 150 frames on OTCBVS dataset.

Methods	Time cost (s)	Average frame rate (fps)
Mean shift	1.20	125.0
Standard particle filter	3.41	43.9
Labelled particle filter	2.90	51.7

Table 2

The comparison of time cost and average frame rate when tracking 150 frames on the sequences acquired through Sheenrun imager.

Methods	Time cost (s)	Average frame rate (fps)
Mean shift	1.15	127.1
Standard particle filter	2.30	65.3
Labelled particle filter	1.81	82.9

Figure 1

The snapshots of tracking on OTCBVS dataset. (a), (b), and (c) are the tracking results achieved by mean shift, standard particle filter, and the proposed LPF algorithm, respectively. The four frames of each algorithm are 1# frame, 60# frame, 120# frame, and 150# frame correspondingly.

Figure 2

The snapshots of tracking on the sequences acquired from Sheenrun imager. (a), (b), and (c) are the tracking results achieved by mean shift, standard particle filter, and the proposed LPF algorithm, respectively. The four frames of each algorithm are 1# frame, 40# frame, 120# frame, and 150# frame correspondingly.

In general, our LPF algorithm has the comparable tracking accuracy with the standard particle filter algorithm, but it is faster than the method. In spite of the slightly higher speed of the mean shift method, our algorithm is much more robust than the method, under complicated background clutter or noise.

5. Conclusions

In this paper, we bring forward an effective object tracking algorithm with the name of labelled particle filter which is especially suitable for thermal infrared imager. Employing label establishment and particles selection, the computational complexity of labelled particle filter is considerably reduced. Experiments are conducted to show that the proposed algorithm can handle real-time object tracking with less time cost while maintaining high tracking accuracy.