Abnormal Event Detection via Multikernel Learning for Distributed Camera Networks

Abstract

Distributed camera networks play an important role in public security surveillance. Analyzing video sequences from cameras set at different angles will provide enhanced performance for detecting abnormal events. In this paper, an abnormal detection algorithm is proposed to identify unusual events captured by multiple cameras. The visual event is summarized and represented by the histogram of the optical flow orientation descriptor, and then a multikernel strategy that takes the multiview scenes into account is proposed to improve the detection accuracy. A nonlinear one-class SVM algorithm with the constructed kernel is then trained to detect abnormal frames of video sequences. We validate and evaluate the proposed method on the video surveillance dataset PETS.

1. Introduction

Detecting abnormal events via video sequence analysis is crucial for public security management. Compared with the single camera setting, distributed camera networks with overlapping views are capable of obtaining additional information for surveilling the movement of the crowds. We illustrate a multicamera setting with the performance evaluation of tracking and surveillance (PETS) dataset [1]. This dataset was collected for testing existing or new systems for the detection of one or more of 3 types of crowd surveillance characteristics/events within a real-world environment. The scenarios are filmed from multiple cameras and involve up to approximately forty actors. The scene and the camera locations are shown in Figure 1. Several normal and abnormal scenes from different angles of view are shown in Figure 2. In Figures 2(a), 2(c), and 2(e), people were walking in random directions, which are considered to be normal. In Figures 2(b), 2(d), and 2(f), people were walking or running in the same direction. This group behavior implies that they were attracted by some particular events. Consequently, these scenes are considered to be abnormal. Note that the scenes were captured by different cameras. Figures 2(a) and 2(b) were captured by camera 1 which was set at the side of the road, and the movement of the people can be easily identified. Figures 2(c) and 2(d) were captured by camera 2 which was set towards the movement direction of the people, with occlusion of the individuals existing in this view. Figures 2(e) and 2(f) were captured by camera 3 which was also set at the side of the road, but with larger distance. The purpose of the distributed camera surveillance is to detect abnormal events by benefitting from information conveyed by the multiview video sequences. Distributed camera networks are currently widely used for security surveillance applications.

Figure 1

The plan of the multicamera localizations in PETS dataset. Three cameras are set in the campus as marked.

Figure 2

Examples of the normal and abnormal scenes captured by distributed cameras of PETS dataset. (a, c, e) The people are walking in different directions, the normal scenes of Time 14–55 sequence; (b, d, f) the people are walking in the same direction, the abnormal scenes of Time 14–17 sequence; (a, b) scenes captured by camera 1; (c, d) scenes captured by camera 2; (e, f) scenes captured by camera 3.

In the literature [2, 3], the framework for multiple pedestrian tracking by using overlapping cameras was presented. In [4], several major challenges in distributed video processing, including robust and computationally efficient inference and opportunistic and parsimonious sensing, were discussed. This progress is coupled with the fact that large-scale video networks start to play an important rule for video surveillance, object recognition, abnormal event detection, and people tracking in crowd environments. Modeling the movement feature of pixels is fundamental for detecting the abnormal event. In [5], a method that tracked the local spatiotemporal interest points was proposed, and the abnormal activity was indicated by uncommon energy-velocity of the feature points. In [6], a spatiotemporal descriptor was computed based on computing the histograms of optical flow in the neighborhood of detected points. In [7], dense points were sampled from each frame and were tracked based on displacement information from an optical flow field. But for the crowd event analysis, it is difficult to obtain the predetected pixels of the blob due to the occlusion of the individuals. In order to deal with the uncertainty of observations existing in video events, Bayesian modeling approaches such as hierarchical Dirichlet processes were used in [8], and probabilistic latent semantic analysis was used in [9]. A method based on the variable-duration hidden Markov model was proposed in [10], where the durations of states were modeled except for the transitions between states, and the temporal understanding of the structure of complex events was tackled. Latent Dirichlet allocation (LDA) was also a typical standard topic model which was used to model video clips as being derived from a bag of topics drawn from a fixed set of proportions [11]. In [12], the covariance matrix descriptor fusing the optical flow to encode moving information of a frame was presented. In the above literature, feature extraction methods and event models were intensively studied for abnormal detection; however, only single view video sequences were considered in the analysis. Abnormal event detection via distributed camera networks is an emerging problem and attracts researchers to investigate. In this paper, we propose an abnormal detection algorithm to identify unusual events captured by multiple cameras. The visual event is represented by the histogram of the optical flow orientation descriptor and then a nonlinear one-class SVM algorithm using a multikernel strategy that takes the multiview scenes into account is proposed to improve the detection accuracy. Simulation will illustrate the effectiveness of the proposed scheme.

The rest of the paper is organized as follows. In Section 2, the optical flow-based feature for video analysis is introduced. In Section 3, the abnormal detection framework based on one-class SVM classification method is presented, and then a multikernel strategy is proposed to deal with the abnormal event detection problem for distributed camera networks. In Section 4, the experimental results are illustrated and discussed. Finally, Section 5 concludes the paper and gives a perspective of future work.

2. Feature Selection for Abnormal Detection

The proposal by Horn-Schunck (HS) has been chosen to compute the optical flow for the representation of the movement information. The HS method formulates the optical flow as a global energy functional for the gray image sequence:

$$\begin{array}{cc}& E={{\displaystyle \iint }}_{}^{}\left[{\left(I_{x}u+I_{y}v+I_t\right)}^{\mathrm{2}}+\alpha \left({‖\nabla u‖}^2+{‖\nabla v‖}^2\right)\right]dx\hspace{0.17em}dy,\end{array}$$

(1)

where $I_x$ , $I_y$ , and $I_t$ are the derivatives of the image intensity values along the horizontal direction x, vertical direction y, and time t dimension, respectively, u and v are the horizontal and vertical optical flow, and α is a regularization constant.

Based on the optical flow, the histogram of the optical flow orientation (HOFO) [13] is computed to fuse the movement information as a high dimension vector. A $\mathrm{2}\times \mathrm{2}$ rectangular cell HOFO descriptor of the image is shown in Figure 3. The orientation is computed by horizontal and vertical optical flow and then it is voted into n bins. The number of bins n is set to 9 in this work. Each cell contains $h_c$ pixels in height dimension and $w_c$ in width dimension. Their values are set to a half of the image size. A block contains $h_b\times w_b$ cells, and it is fixed to be $\mathrm{2}\times \mathrm{2}$ in this work.

Figure 3

Histogram of the optical flow orientation (HOFO) feature descriptor of the image.

3. One-Class SVM with Multiple Kernels

In practice, most data belong to the normal frames of video sequences. It is reasonable to assume that these data lie in the high density zone of the underlying data distribution. Data of an abnormal frame will appear as an outlier of this data distribution. Thus, one-class classification methods, such as OCNM (one-class neighbor machine) and OCSVM (one-class support vector machine), are appropriate to deal with the abnormal event detection problem, by checking whether a data point lies in the characteristic region constructed with normal data. Specifically, in the literature [14–16], OCNM which asymptotically converges to the exact minimum volume set was proposed. Because the suboptimal performance of OCSVM may arise from the fact that its decision function is not based on the use of neighborhood measures, the OCNM performs consistently better than OCSVM. In this paper we focus on verifying the advantage of using the feature representation of an action from multiview cameras. Thus, the OCSVM is chosen as a baseline in this paper. OCNM, which potentially has more competent performance, will be studied as a future work.

The problem of nonlinear one-class SVM [17, 18] can be cast as a quadratic programming problem:

$$\begin{array}{cc}& \underset{\omega ,\xi ,\rho }{\min }\hspace{1em}\frac{\mathrm{1}}{\mathrm{2}}{‖\mathbf{w}‖}^{\mathrm{2}}+\frac{\mathrm{1}}{\nu n}{\displaystyle \sum }_{i=\mathrm{1}}^n{\xi }_i-\rho ,\\ \text{subject\hspace{0.17em}\hspace{0.17em}to}\hspace{1em}〈\mathbf{w},\Phi \left({\mathbf{x}}_i\right)〉\ge \rho -{\xi }_i,\hspace{1em}{\xi }_i\ge \mathrm{0},\end{array}$$

(2)

where ${\mathbf{x}}_i\in \mathcal{X}$ with $i=\mathrm{1},\dots ,n$ are the training samples in the original data space $\mathcal{X}$ and ${\xi }_i$ is the slack variable for penalizing the outliers. The hyperparameter $\nu \in (\mathrm{0,1}]$ is the weight for controlled slack variable and it tunes the number of acceptable outliers. The decision hyperplane is described by $〈\mathbf{w},\Phi ({\mathbf{x}}_i)〉-\rho =\mathrm{0}$ . Nonlinear function Φ maps the input data $\mathcal{X}$ into a high dimensional feature space $\mathcal{H}$ , where linear algorithms are supposed to be available for classification. The inner product in space $\mathcal{H}$ is defined by the kernel function $\kappa ({\mathbf{x}}_i,{\mathbf{x}}_j)=〈\Phi ({\mathbf{x}}_i),\Phi ({\mathbf{x}}_j)〉$ . Introducing Lagrangian multipliers ${\alpha }_i$ , the decision function in the data space $\mathcal{X}$ is

$$\begin{array}{cc}& f\left(\mathbf{x}\right)=\mathrm{sgn}\left({\displaystyle \sum }_{i=\mathrm{1}}^n{\alpha }_i\kappa \left({\mathbf{x}}_i,\mathbf{x}\right)-\rho \right).\end{array}$$

(3)

The Gaussian kernel is chosen to handle movement feature in this work. It is a semipositive definite kernel that satisfies the Mercer condition [19, 20] and is defined by

$$\begin{array}{cc}& \kappa \left({\mathbf{x}}_i,{\mathbf{x}}_j\right)=\exp \left(-\frac{{‖{\mathbf{x}}_i-{\mathbf{x}}_j‖}^2}{\mathrm{2}{\sigma }^{\mathrm{2}}}\right),\hspace{1em}\left({\mathbf{x}}_i,{\mathbf{x}}_j\right)\in \mathcal{X}\times \mathcal{X},\end{array}$$

(4)

where ${\mathbf{x}}_i$ , ${\mathbf{x}}_j$ are the data in the original data space $\mathcal{X}$ and the bandwidth σ indicates the scale at which the data should be clustered.

For constructing a more representative and discriminative feature descriptor for distributed camera networks, we take the scene captured by each view as a partial feature. The multikernel strategy considers a linear combination of candidates kernels:

$$\begin{array}{cc}& \kappa \left({\mathbf{x}}_i,{\mathbf{x}}_j\right)={\displaystyle \sum }_{s=\mathrm{1}}^m{\mu }_s{\kappa }_s\left({\mathbf{x}}_i,{\mathbf{x}}_j\right),\end{array}$$

(5)

where ${\kappa }_s$ with $s=\mathrm{1},\dots ,m$ are m candidate kernels that satisfy the Mercer condition and ${\mu }_s$ are nonnegative factors. Consequently, their combination κ is also a semipositive definite kernel. In this expression, the Gaussian kernel is adopted with

$$\begin{array}{cc}& {\kappa }_s\left({\mathbf{x}}_i,{\mathbf{x}}_j\right)=\exp \left(-\frac{{‖{\mathbf{x}}_{i_{\left[s\right]}}-{\mathbf{x}}_{j_{\left[s\right]}}‖}^2}{\mathrm{2}{\sigma }^{\mathrm{2}}}\right).\end{array}$$

(6)

Each sample vector ${\mathbf{x}}_i$ consists of m parts $[{\mathbf{x}}_{i_{\mathrm{1}}},{\mathbf{x}}_{i_{\mathrm{2}}},\dots ,{\mathbf{x}}_{i_m}]$ , where $i_{[s]}$ denotes the ith event which is monitored by the mth camera. For example, in a distributed camera network with 4 cameras, the complete feature descriptor ${\mathbf{x}}_{\mathrm{1}}$ is a set of 4 parts $[{\mathbf{x}}_{{\mathrm{1}}_{\mathrm{1}}},{\mathbf{x}}_{{\mathrm{1}}_{\mathrm{2}}},{\mathbf{x}}_{{\mathrm{1}}_{\mathrm{3}}},{\mathbf{x}}_{{\mathrm{1}}_{\mathrm{4}}}]$ .

For a given scene monitored by multiple cameras, suppose we can obtain a set of training frames. Based on the one-class SVM hypothesis, the abnormal behavior is the sample deviating from the training set. Take the plaza monitored by the three cameras shown in Figure 2; for example, if $s=\mathrm{1}$ , the frame captured by camera 1 is selected. We preset the weight ${\mu }_s$ according to the characteristics of the image to tune the importance of each view. By considering this combination, we will find that the resulting kernel outperforms each individual kernel ${\kappa }_s$ . Based on the histogram of the optical flow orientation feature descriptor and the nonlinear one-class SVM, the abnormal event detection method is summarized in Algorithm 1 and explained in the following.

Algorithm 1 (abnormal event detection algorithm).

 

Input. Image set captured by the cameras.

Algorithm. (1) Compute the optical flow of the training frame set $[I_{\mathrm{1}}^{v_i},\dots ,I_m^{v_i}]$ , where $v_i$ means the scene is monitored by camera i via the HS optical flow method:

$$\begin{array}{cc}& \left[I_{\mathrm{1}}^{v_i},\dots ,I_m^{v_i}\right]⟶\left[O_{\mathrm{1}}^{v_i},O_{\mathrm{2}}^{v_i},\dots ,O_m^{v_i}\right].\end{array}$$

(7)

(2) Compute the histogram of the optical flow orientation (HOFO) of the image in different views:

$$\begin{array}{cc}& \left[O_{\mathrm{1}}^{v_i},O_{\mathrm{2}}^{v_i},\dots ,O_m^{v_i}\right]⟶\left[H_{\mathrm{1}}^{v_i},\dots ,H_m^{v_i}\right].\end{array}$$

(8)

(3) Construct the feature samples of the image k under c for the distributed camera network:

$$\begin{array}{cc}& H_k=\left[H_k^{v_{\mathrm{1}}},H_k^{v_{\mathrm{2}}},\dots ,H_k^{v_c}\right].\end{array}$$

(9)

(4) Obtain the support vectors by training the feature samples with nonlinear one-class SVM method:

$$\begin{array}{cc}& \left[H_{\mathrm{1}},\dots ,H_m\right]⟶\text{support\hspace{0.17em}\hspace{0.17em}vector}\left[S_{\mathrm{1}}^{v_i},\dots ,S_{m_{\mathrm{1}}}^{v_i}\right].\end{array}$$

(10)

(5) Each incoming frame $H_{p,\dots ,q}$ is classified by the decision function of one-class SVM.

(6) The normal event or abnormal event is detected.

Step 1.

The optical flow feature is computed. The training frame set $[I_{\mathrm{1}}^{v_i},\dots ,I_m^{v_i}]$ monitored by the multicamera network describing the normal behavior is available. $I_j^{v_i}$ means that the jth image is captured by the ith camera. Horn-Schunck method is applied to acquire the optical flow feature. This step can be presented as

$$\begin{array}{cc}& \left[I_{\mathrm{1}}^{v_i},\dots ,I_m^{v_i}\right]⟶\left[O_{\mathrm{1}}^{v_i},O_{\mathrm{2}}^{v_i},\dots ,O_m^{v_i}\right],\end{array}$$

(11)

where $[I_{\mathrm{1}}^{v_i},\dots ,I_m^{v_i}]$ are m training frames of the ith view and $[O_{\mathrm{1}}^{v_i},O_{\mathrm{2}}^{v_i},\dots ,O_m^{v_i}]$ are the corresponding optical flows.

Step 2.

The second step consists of calculating the histogram of optical flow orientation (HOFO) of the training frames. It can be generalized as

$$\begin{array}{cc}& \left[O_{\mathrm{1}}^{v_i},O_{\mathrm{2}}^{v_i},\dots ,O_m^{v_i}\right]⟶\left[H_{\mathrm{1}}^{v_i},\dots ,H_m^{v_i}\right],\end{array}$$

(12)

where $[O_{\mathrm{1}}^{v_i},O_{\mathrm{2}}^{v_i},\dots ,O_m^{v_i}]$ are the optical flow of the training frames captured by camera i. $[H_{\mathrm{1}}^{v_i},\dots ,H_m^{v_i}]$ are the corresponding histogram of optical flow orientation (HOFO) feature.

Step 3.

The third step consists of fusing the $\mathrm{1}\mathrm{s}\mathrm{t}$ to cth HOFO feature into one high dimension feature vector. It is described in the following equation:

$$\begin{array}{cc}& H_k=\left[H_k^{v_{\mathrm{1}}},H_k^{v_{\mathrm{2}}},\dots ,H_k^{v_c}\right],\end{array}$$

(13)

where $H_k$ is the feature vector fusing the multiview movement information. $H_k^{v_{\mathrm{1}}}$ is the HOFO feature of the kth image in view i.

Step 4.

Nonlinear one-class SVM is applied to the training frame HOFO descriptor in multiview to obtain the support vectors. It is described as follows:

$$\begin{array}{cc}& \left[H_{\mathrm{1}},\dots ,H_m\right]⟶\text{support\hspace{0.17em}\hspace{0.17em}vector}\left[S_{\mathrm{1}},\dots ,S_{m_{\mathrm{1}}}\right],\end{array}$$

(14)

where $[H_{\mathrm{1}},\dots ,H_m]$ are the histogram of optical flow orientation descriptors of the training frames under multiview environment and $[S_{\mathrm{1}},\dots ,S_{m_{\mathrm{1}}}]$ are support vectors that are the minority of the training vectors contributing to the decision function.

Step 5.

In the online detection phase, based on the support vectors obtained in the training step, one-class SVM classifies each incoming frame feature $[H_p,\dots ,H_q]$ . The abnormal event will be detected in an online manner.

4. Abnormal Events Detection Results

We then conduct experiments to evaluate the performance of the one-class SVM classification method for abnormal frame event detection with the distributed camera network from the PETS dataset [1]. In the experiments of PETS [1] dataset, each event is represented by 3 separate scenes; thus, the event is described by a set of 3 separated HOFO features. We mark the use of these three features as “3 HOFO,” while we mark the use of a single feature as “1 HOFO.” If the multikernel strategy is used, we mark it as “3 kernels”; otherwise, we mark the single kernel strategy as “1 kernel.” The detection accuracy of the detection results are shown for the experiments.

The normal and abnormal events of sequence Time 14–17 in the PETS dataset are shown in Figure 4. The training samples and the normal samples for testing are the frames in 3 views chosen from the sequence (Time 14–55) where the people were walking in different directions. 400 training frames (Frame0000 to Frame0399) and 90 normal testing frames (Frame0400 to Frame0489) were selected from Time 14–55. The abnormal testing samples were selected from the sequence (Time 14–17) where the people were walking or running in the same direction. 89 abnormal testing frames (Frame0000 to Frame0089) were selected from Time 14–17.

Figure 4

Detection results of the normal and abnormal scenes of Time 14–17 captured by distributed camera network. (a, c, e) The detection results of one normal frame. (b, d, f) The detection results of one abnormal frame.

In the experiments, the parameters of the one-class SVM, ν and σ, were selected from a set of preassigned values to plot the ROC (receiver operating characteristic curve) and comparing the AUC (area under the curve) values from the curves. The AUC value of the abnormal detection results in different views and different multikernel strategies are shown in Table 1. “Single view” means “1 HOFO 1 kernel” strategy, and “multiview” means “3 HOFO 3 kernels.” “View 1” denotes the scene that was monitored by camera 1, and the abnormal detection results are shown in Figures 4(a) and 4(b). “View 2” denotes the scene that was monitored by camera 2, as shown in Figures 4(c) and 4(d). “View 3” denotes the scene that was monitored by camera 3, as shown in Figures 4(e) and 4(f). “Multiview (1)” denotes the multikernel strategy with the parameter setting ${\mu }_{\mathrm{1}}=\mathrm{0.5}$ , ${\mu }_{\mathrm{2}}=\mathrm{0.2}$ , ${\mu }_{\mathrm{3}}=\mathrm{0.3}$ . “Multiview (2)” denotes the multikernel strategy with ${\mu }_{\mathrm{1}}=\mathrm{1}/\mathrm{3}$ , ${\mu }_{\mathrm{2}}=\mathrm{1}/\mathrm{3}$ , ${\mu }_{\mathrm{3}}=\mathrm{1}/\mathrm{3}$ . The ROC of Time 14–17 results is shown in Figure 5. Clearly, selection of coefficients ${\mu }_s$ has an impact on the abnormal detection performance. In this work, these parameters were manually assigned according to the importance of the views and the camera locations.

Table 1

Comparison of the abnormal detection AUC values with the single view and with the distributed cameras using the multikernel strategy for sequence Time 14–17.

Strategy	Area under ROC
Single view	View 1	View 2	View 3
	0.9036	0.7807	0.9270
Multiview (1)	0.9441
Multiview (2)	0.9643

Figure 5

ROC of abnormal detection results of sequence Time 14–17 under different views and different kernel strategies.

View 2 has the lowest area under the ROC, for the reason that camera 2 faces the movement direction of the crowds, and the occlusion influences the computation of the optical flow. Thus, the HOFO feature based on the optical flow cannot represent the accurate movement information. The results show that the abnormal detection algorithm of HOFO feature can obtain satisfactory detection results. Moreover, the multikernel strategy can generally improve the performance.

The experiment detecting the running activity as the abnormal event is shown in Figures 6 and 7. The normal event corresponds to the frames where the people were walking. The training data are selected from the sequence (Time 14–17 and Time 14–31) in PETS dataset where the individuals were walking in one direction. In this experiment, 61 frames (Frame0000 to Frame0060) where people were walking from left to right in sequence Time 14–17 and 50 frames (Frame0000 to Frame0049) where the individuals were walking from right to left in sequence Time 14–31 are chosen as training samples. Correspondingly, 104 normal samples and 118 abnormal frames in sequence Time 14–16 are detected. The abnormal detection performance is improved by multikernel strategy also. The AUC values of the detection results are shown in Table 2.

Table 2

The abnormal detection results of sequence Time 14–16. The comparison of the abnormal frame event detection results in single view scene and in distributed camera scenes via multikernel strategies.

Strategy	Area under ROC
Single view	View 1	View 2	View 3
	0.8637	0.8071	0.9205
Multiview (1)	0.9312
Multiview (2)	0.9403

Figure 6

Detection results of the normal and abnormal scenes of Time 14–16 captured by the distributed camera network: the individuals are moving from right to left. (a, c, e) The detection result of one normal frame: the individuals are walking in one direction. (b, d, f) The detection result of one abnormal frame: the individuals are running in one direction.

Figure 7

Detection results of the normal and abnormal scenes of Time 14–16 captured by the distributed camera network: the individuals are moving from left to right. (a, c, e) The detection result of one normal frame: the individuals are walking in one direction. (b, d, f) The detection result of one abnormal frame: the individuals are running in one direction.

5. Conclusions

In this paper, we have proposed a method for abnormal frame event detection with distributed camera networks. The histogram of optical flow was computed as the descriptor to represent the movement of a frame. A multikernel strategy was presented to benefit from multiple views captured by the distributed camera network. The benchmark dataset PETS has been tested to demonstrate the effectiveness of the proposed algorithm.

In the future work, the optimal coefficients of the multikernel strategy should be obtained automatically based on the scene, while these parameters were preset according to the importance of each view in the current work. Additional application scenarios, such as single person action recognition or action tracking in multiview scenes, will be considered to show the advantages of distributed camera networks. Finally, advanced one-class classification techniques such as OCNM will also be investigated for the video anomaly detection.