Histogram of Maximal Optical Flow Projection for Abnormal Events Detection in Crowded Scenes

Abstract

Abnormal events detection plays an important role in the video surveillance, which is a challenging subject in the intelligent detection. In this paper, based on a novel motion feature descriptor, that is, the histogram of maximal optical flow projection (HMOFP), we propose an algorithm to detect abnormal events in crowded scenes. Following the extraction of the HMOFP of the training frames, the one-class support vector machine (SVM) classification method is utilized to detect the abnormality of the testing frames. Compared with other methods based on the optical flow, experiments on several benchmark datasets show that our algorithm is effective with satisfying results.

1. Introduction

Nowadays, more and more surveillance cameras have been used in public places. Behavior analysis in crowded scenes [1–5] becomes more and more popular and important for public safety. In order to eliminate the world representation layer which can be a significant source of errors for algorithm modeling, an approach based on modeling directly at the pixel level was described in [6]. In [7, 8], social force model was used in abnormal crowd behavior detection. In [9, 10], a model named social attribute-aware force model was proposed. In this model, in order to improve the algorithm performance for the interaction behavior of the crowd, social characteristics of crowd behavior were taken into account.

In [11], SIFT features were extracted for the Bag of Words (Bow) model with Spatial Pyramid Matching Kernel (SPM). Then a SVM classifier was used for cross-scene abnormal events detection. In [12], based on the fact that the occurrence of abnormal events is rare while the frequently occurring events are normal in general human perception, proximity clustering for abnormal events detection in video sequence was proposed. In [13], when labeled information about normal events was limited and information about abnormal events was not available, projection subspace associated with detectors was discovered by using both labeled and unlabeled segments. In wireless sensor networks, a fact has been observed that instead of being transient, most abnormal events persist over a considerable period time. Thus, a technique for handling data in a segment-based manner was introduced in [14]. Without using any tracking and motion features, a feature extraction and events detection method were presented in [15], where features were extracted from foreground blobs and then confined in SVM based models for real-time events detection.

Unlike most existing approaches used for abnormal events detection, sparse representation based approaches attracted many researchers in the recent years. In [16], a method to detect abnormal events by a sparse subspace clustering was proposed. In [17, 18], a model based on the optical flow was described, which utilized the sparse reconstruction cost (SRC) over the normal dictionary to measure the normalness of the tested samples. As we know, optical flow is the approximated motion vector at each pixel location, which can reflect the relative distances of moving objects. Therefore, it is important and useful in video surveillance and abnormal events detection. Other methods based on the histogram of optical flow were described in [19–21]. Also, it was improved and used in this paper.

Although the above approaches could successfully realize abnormal events detection, they were limited in some aspects. Some models were established complicatedly and others cost a long time in the detection process. Based on these, we propose a novel detection model in crowded scenes, which is relatively simple and time-saving in calculation. Similar to the approach introduced in [21], our algorithm is mainly based on a proper processing method in the optical flow field.

The rest of the paper is organized as follows. In Section 2, we present how to acquire the motion features. In Section 3, the theory of one-class SVM is reviewed. In Section 4, the algorithm of abnormal events detection is introduced in detail. Section 5 presents our experiment results. Finally, some conclusions are presented in Section 6.

2. Motion Feature Extraction

Optical flow field is the movement on the surface of grayscale images, which reflects the movement information of two consecutive frames. Optical flow provides the information of direction and amplitude of the moving object in a scene, which can describe the behavior of people very well. Optical flow is derived from the following basic equation:

$$\begin{array}{cc}& I_{x}u+I_{y}v+I_t=\mathrm{0},\end{array}$$

(1)

where $I_x$ , $I_y$ , and $I_t$ are the partial derivatives of the image grayscale value along the x, y, and t dimension, respectively; u and v are the horizontal (x dimension) and vertical (y dimension) components of the optical flow. Equation (1) is an ill-posed problem. In [22], Horn and Schunck proposed an algorithm. It is known as the HS algorithm to compute the optical flow by introducing a global constraint of smoothness, which is equal to the additional condition

$$\begin{array}{cc}& \min {\nabla }^{\mathrm{2}}u+{\nabla }^{2}v,\end{array}$$

(2)

where ${\nabla }^{\mathrm{2}}u$ and ${\nabla }^{\mathrm{2}}v$ are Laplace operators of u and v, respectively. The problem to get optical flow can be concluded as follows:

$$\begin{array}{cc}& \min {\displaystyle \iint }\left[{\left(I_{x}u+I_{y}v+I_t\right)}^{\mathrm{2}}+{\alpha }^{\mathrm{2}}{\left({\nabla }^{\mathrm{2}}u+{\nabla }^{\mathrm{2}}v\right)}^{\mathrm{2}}\right]dx\hspace{0.17em}dy,\end{array}$$

(3)

where α is the parameter that represents the weights of the regularization term. Then the Euler-Lagrange equations can be acquired, which are solved by utilizing the Gauss-Seidel method. It can get an iterative result to compute the optical flow:

$$\begin{array}{cc}& u^{n+\mathrm{1}}={\stackrel{-}{u}}^n-\frac{I_x\left(I_x{\stackrel{-}{u}}^n+I_y{\stackrel{-}{v}}^n+I_t\right)}{{\alpha }^{\mathrm{2}}+{I_x}^{\mathrm{2}}+{I_y}^{\mathrm{2}}},\\ v^{n+\mathrm{1}}={\stackrel{-}{v}}^n-\frac{I_y\left(I_x{\stackrel{-}{u}}^n+I_y{\stackrel{-}{v}}^n+I_t\right)}{{\alpha }^{\mathrm{2}}+{I_x}^{\mathrm{2}}+{I_y}^{\mathrm{2}}},\end{array}$$

(4)

where $\stackrel{-}{u}$ and $\stackrel{-}{v}$ are weighted average value of u and v, respectively, which are calculated in a neighborhood around the pixel location. n denotes the algorithm iteration number.

In this paper, we propose a novel motion feature descriptor, called histogram of maximal optical flow projection (HMOFP). Figure 1 briefly shows the process for computing the HMOFP.

Figure 1

The process for computing the HMOFP feature.

As shown in Figure 2, the optical flow field of frame s is divided into m image patches with overlap areas. Each block contains $B\times B$ pixels. Then we deal with the optical flow in each patch as follows: ${\mathrm{0}}^{°}\text{–}{\mathrm{360}}^{°}$ are segmented into p bins. For an image patch, the optical flow vector of each pixel must belong to a bin according to its direction. Thus, each bin may contain several optical flow vectors. We project all optical flow vectors in the same bin onto the angle bisector of this bin. Then the maximal projection vector is selected as the feature descriptor. For example, in Figure 3(a), there are two vectors $\overrightarrow{o{n}_1}$ and $\overrightarrow{o{n}_2}$ falling into the first bin. It is easy to know that the projection of $\overrightarrow{o{n}_2}$ is longer than the projection of $\overrightarrow{o{n}_1}$ . Thus, the length of the projection vector $\overrightarrow{o{n}_{2^{\prime }}}$ is selected as the feature descriptor of the first bin. After computing m patches, we obtain the feature descriptor vector of each image patch, denoted as $[h_1,h_2,\dots ,h_m{]}_{p\times m}$ , where $h_i=[h_i^1,\dots ,h_i^p{]}_{p\times 1}^{}$ . For the ith patch, $h_i^j$ , $1\le j\le p$ , $1\le i\le m$ denotes the maximal amplitude among all projection vectors in the jth bin. As shown in Figure 3(b), we take the concatenation of the m feature descriptor vectors, which is named $H_s$ , as the global HMOFP feature of the frame s.

Figure 2

Block-division of the optical flow field belonging to the frame s.

Figure 3

(a) The calculation of HMOFP in each bin. (b) Components of the global feature descriptor of the frame s.

In order to describe a crowd scene well, sufficient crowd movement information is required. On the other hand, for distinguishing two different scenes, detailed comparisons of them are needed and useless information in these two scenes should be eliminated. In the classification process, overlapping block-division can increase the number of significant motion features in two different frames such that these two frames can be more distinguishable. Thus, it is adopted in our algorithm since the optical information can be utilized sufficiently. Moreover, to describe the motion of a crowd, we need two factors: explicit directions and the moving distance along each direction. The operation of segmenting the $\mathrm{2}D$ space into p bins provides us ample information to describe the directions of moving people. To let the direction in each bin be unique, we select the p angle bisectors as the direction standard. Since there may be far more than one optical flow vector in each bin, in order to enhance the distinction between the normal scene and the abnormal scene, we select the maximal vector projection rather than the sum of all the vector projections on the bisector as the motion feature descriptor. If we ignore the background area, the amplitudes of motion vectors that belong to the normal area are very small in a normal frame and the motion vectors corresponding to the abnormal area are large in an abnormal frame. Usually, the number of normal motion vectors is much more than that of the abnormal area. If we use the sum of all projection vectors on the angel bisector as the feature descriptor of each bin, the accumulation of the massive small motion vectors in the normal frame may confuse the small number of large motion vectors in the abnormal frame; that is, the sum of all projection vectors on the angel bisector in each bin of the normal frame is likely to be close to that of the abnormal frame. Thus, in order to improve the distinguishability between the abnormal and normal frames, we select the maximal projection vector as the feature descriptor of each bin, as it was demonstrated in Figure 3.

3. One-Class SVM

SVM was initiated by Vapnik and Lerner [23]. Since the kernel methods were introduced, SVM has been applied extensively in nonliner classification problems [24–26]. In one-class classification problem, the substance is that the boundary, that is, an appropriate region, needs to be determined in the data space $\mathcal{X}$ , which contains most of the samples coming from an unknown probability distribution D. This goal can be realized by searching for an optimal decision hyperplane in the feature space, which is known as the Hilbert space $\mathcal{H}$ . This hyperplane can maximize the distance between itself and the original point, while only a small part of data falls between them [27]. The relationship between $\mathcal{X}$ and $\mathcal{H}$ is shown in Figure 4.

Figure 4

The correspondence between data space and feature space. (a) The boundary in the data space $\mathcal{X}$ . (b) The hyperplane in the feature space $\mathcal{H}$ .

One-class SVM problem can be presented as an optimization model:

$$\begin{array}{cc}& \underset{w,\xi ,\rho }{\min }\hspace{1em}\frac{\mathrm{1}}{\mathrm{2}}\left({\mathit{w}}^{\text{T}}\mathit{w}\right)-\rho +\frac{\mathrm{1}}{\nu l}\left({\displaystyle \sum }_{i=\mathrm{1}}^l{\xi }_i\right)\\ \text{s.t.}\hspace{1em}{\mathit{w}}^{\text{T}}\varphi \left({\mathit{x}}_i\right)\ge \rho -{\xi }_i,\hspace{1em}{\xi }_i\ge \mathrm{0},\end{array}$$

(5)

where ${\mathit{x}}_i\in \mathcal{X}$ , $i\in \left[1\cdots l\right]$ are training samples in the input data space $\mathcal{X}$ and $\varphi :\mathcal{X}\to \mathcal{H}$ can map a vector ${\mathit{x}}_i$ into the feature space $\mathcal{H}$ . ${\mathit{w}}^{\text{T}}\varphi \left({\mathit{x}}_i\right)-\rho =0$ is the decision hyperplane. ${\xi }_i$ is the slack variable for penalizing the outliers. $\nu \in \left(\mathrm{0,1}\right]$ is the hyperparameter, which is the weight for controlling slack variable and tunes the number of acceptable outliers. ϕ is a mapping function, which provides us a way to solve the nonlinear classification problem in the space $\mathcal{X}$ by a linear solution in the space $\mathcal{H}$ . By calculating dot product in $\mathcal{H}$ , the kernel function is defined as $k\left({\mathit{x}}_i,{\mathit{x}}_j\right)={\varphi }^{\text{T}}\left({\mathit{x}}_i\right)\varphi \left({\mathit{x}}_j\right)$ . The decision function in the space $\mathcal{X}$ with a Lagrangian multiplier ${\alpha }_i$ is defined as

$$\begin{array}{cc}& f\left(\mathit{x}\right)=\mathrm{sgn}\left({\displaystyle \sum }_{i=\mathrm{1}}^l{\alpha }_{i}k\left({\mathit{x}}_i,\mathit{x}\right)-\rho \right).\end{array}$$

(6)

In [28], it was introduced that if appropriate parameters were selected, polynomial and sigmoid kernels will result in similar results with Gaussian. We choose Gaussian kernel in our algorithm. This kernel is defined as

$$\begin{array}{cc}& k\left({\mathit{x}}_i,{\mathit{x}}_j\right)=\exp \left(-\frac{{‖{\mathit{x}}_i-{\mathit{x}}_j‖}^2}{2{\sigma }^2}\right),\end{array}$$

(7)

where ${\mathit{x}}_i$ and ${\mathit{x}}_j$ belong to the space $\mathcal{X}$ and σ is the scale factor at which the data should be clustered.

In our method, one-class SVM is utilized as follows. Firstly, the training set is used to establish a model. Then an appropriate boundary in the data space can be determined. The new incoming frames will be clustered by the following rule: if the HMOFP feature of the testing frame falls inside the boundary, it will be clustered as a normal frame. Otherwise, it is abnormal.

4. Abnormal Events Detection

In this section, an algorithm for abnormal events detection in surveillance video is described in detail. Suppose that for a given scene, there is a set of training frames $[f_1,\dots ,f_l]$ , which describe the normal behavior of crowded people. The general procedures for the abnormal events detection based on the histogram of maximal optical flow projection (HMOFP) are presented as follows.

Step 1.

Calculate the optical flow, that is, $[\mathrm{O}{\mathrm{P}}_{\mathrm{1}},\dots ,\mathrm{O}{\mathrm{P}}_{l-\mathrm{1}}]$ , by the HS method at each pixel of the first $l-\mathrm{1}$ frames:

$$\begin{array}{cc}& {\left[f_1,\dots ,f_l\right]}_{a\times b\times l}\stackrel{\text{HS}}{⟶}{\left[{\text{OP}}_1,\dots ,{\text{OP}}_{l-1}\right]}_{a\times b\times \left(l-1\right)},\end{array}$$

(8)

where $a\times b$ is the size of the frame image and l is the number of the frames in the training set. Our method to compute optical flow is based on the two consecutive frames, which is only effective to the first frame, so in the right side of (8), the maximal subscript is $l-\mathrm{1}$ .

Step 2.

Extract the motion features of the first $l-\mathrm{1}$ training frames. Then the HMOFP feature vectors of them can be obtained, which is denoted as the set ${\left[H_1,\dots ,H_{l-1}\right]}^{\text{T}}$ . Consider

$$\begin{array}{cc}& {\left[{\text{OP}}_1,\dots ,{\text{OP}}_{l-1}\right]}_{a\times b\times \left(l-1\right)}\stackrel{\text{HMOFP}}{\to }{\left[H_1,\dots ,H_{l-1}\right]}_{p\times m\times \left(l-1\right)}^{\text{T}}.\end{array}$$

(9)

Step 3.

Based on HMOFP, one-class SVM is utilized to calculate the optimal boundary of the set ${\left[H_1,\dots ,H_{l-1}\right]}^{\mathrm{T}}$ , which corresponds to the set of support vectors or the optimal hyperplane in the feature space.

Step 4.

Detect HMOFP of the testing frames based on the model trained by the motion feature of the first $l-\mathrm{1}$ training frames.

The whole procedure is illustrated in Figure 5.

Figure 5

The flowchart of the proposed abnormal events detection algorithm.

5. Experimental Results

In this section, based on the UMN dataset [29] and PETS2009 dataset [30], we evaluate our method for abnormal event detection. Image patch size is set as $\mathrm{64}\times \mathrm{64}$ and $\mathrm{128}\times \mathrm{128}$ , respectively, in the UMN dataset and PETS2009 dataset. 0°–360° are divided into 18 bins, that is, $p=\mathrm{18}$ . The overlapping proportion of two neighboring blocks is 50%. In the UMN dataset, the length of the HMOFP feature of each frame is 972 with a $\mathrm{320}\times \mathrm{240}$ resolution. In the PETS2009 dataset, the resolution of each frame is $\mathrm{768}\times \mathrm{576}$ , and the length of the HMOFP feature is 1584.

5.1. Experiments on the UMN Dataset

There are three different crowded scenes in the UMN dataset, which are named lawn, indoor, and plaza, respectively. In our experiments, we select a part of the normal frames of each scene as the training set and take the rest of the video sequence as the testing set.

5.1.1. Detection in the Lawn Scene

The video sequence of the lawn scene contains 1453 frames in total. The first 480 frames are taken as the training set. As shown in Figure 6, in the lawn scene, the normal event is that individuals walk in different directions. The abnormal event is that individuals suddenly run away. The detection results of the lawn scene are shown in Figure 7. The accuracy of the detection results is 95.5141%.

Figure 6

Two different scenes in the sequence of lawn.

Figure 7

Classification results of the lawn scene.

5.1.2. Detection in the Indoor Scene

The video sequence of the indoor scene contains 4144 frames in total. The first 319 frames are taken as the training set. As shown in Figure 8, in the indoor scene, the normal event is that some people are talking and standing in a relatively fixed location while some others are walking along the road in the hall. The abnormal event is that people run out of the doors suddenly. The detection results of the indoor scene are shown in Figure 9. The accuracy of the detection results is 91.2857%.

Figure 8

Two different scenes in the sequence of indoor.

Figure 9

Classification results of the indoor scene.

5.1.3. Detection in the Plaza Scene

The video sequence of the plaza scene contains 2412 frames in total. The first 550 frames are taken as the training set. As shown in Figure 10, in the plaza scene, the normal event is that people walk around the center of the square. The abnormal event is that people suddenly run away from the square. The detection results of the plaza scene are shown in Figure 11. The accuracy of the detection results is 94.3352%.

Figure 10

Two different scenes in the sequence of plaza.

Figure 11

Classification results of the plaza scene.

5.2. Experiments on the PETS2009 Dataset

In the following experiments, we can choose some specific scenes we are interested in as the targets in the detection progress. In the PETS2009 dataset, we firstly select the training set and the normal testing set, respectively, in the same scene. Then another video clip in a different scene is taken as the corresponding abnormal testing set. Our experiments and the detection results are shown as follows.

5.2.1. People Scatter Detection

In this part, the training set is the video sequence Time 14-16 (Frame 0 to Frame 222), where people are walking or running towards one direction. The normal testing set includes 41 frames (Frame 48 to Frame 88) of Time 14-17. 41 frames (Frame 337 to Frame 377) of Time 14-33 are labeled as abnormal for testing, in which people are scattered in all directions. The two different scenes are shown in Figure 12. The accuracy of the detection results is 97.5%, as shown in Figure 13.

Figure 12

Two different scenes in the same location.

Figure 13

The detection results of the sequence Time 14-33. “1” means normal and “−1” means abnormal.

5.2.2. Crowd Movement Direction Detection

In this part, the training set is the video sequence Time 14-55 (Frame 0 to Frame 399), where people are walking towards all directions. The normal testing set includes 89 frames (Frame 400 to Frame 488) of Time 14-55. 89 frames (Frame 0 to Frame 88) of the video sequence Time 14-17 are labeled as abnormal for testing, in which people are walking towards one direction. The two different scenes are shown in Figure 14. The accuracy of the detection results is 92.6136%, as shown in Figure 15.

Figure 14

Two different scenes in the same location.

Figure 15

The detection results of the sequence Time 14-17. “1” means normal and “−1” means abnormal.

5.2.3. People Running Detection

In this part, the training set contains 50 frames (Frame 0 to Frame 49) of the video sequence Time 14-31 and 61 frames (Frame 0 to Frame 60) of the video sequence Time 14-17, where people are walking from right to left and from left to right, respectively. The normal testing set includes 104 frames (Frame 0 to Frame 37 and Frame 108 to Frame 173) of Time 14-16. 119 frames (Frame 38 to Frame 107 and Frame 174 to Frame 222) of Time 14-16 are labeled as abnormal for testing, in which people are running towards one direction. The two different scenes are shown in Figure 16. The accuracy of the detection results is 93.6937%, as shown in Figure 17.

Figure 16

Two different scenes in the same location.

Figure 17

The detection results of the sequence Time 14-16. “1” means normal and “−1” means abnormal.

5.2.4. People Splitting Detection

In this part, the training set contains Frames 0 to 40 of the video sequence Time 14-16, where people are walking towards the same direction. The normal testing set includes 64 frames (Frame 0 to Frame 63) of Time 14-31. 66 frames (Frame 64 to Frame 129) of the video sequence Time 14-31 are labeled as abnormal for testing, in which the crowd is splitting. The normal scene and abnormal scene are shown in Figure 18. The accuracy of the detection results is 96.1538%, as shown in Figure 19.

Figure 18

Two different scenes in the same location.

Figure 19

The detection results of the sequence Time 14-31. “1” means normal and “−1” means abnormal.

5.2.5. Comparison

We compared our algorithm with the histogram of optical flow orientation (HOFO) method proposed in [21], as shown in Table 1. Most results of our algorithm are better than those of HOFO.

Table 1

The comparison of HMOFP with HOFO.

Accuracy	Sequence
	Time 14-33	Time 14-17	Time 14-16	Time 14-31
Method
HOFO	97.5%	90%	93.24%	94.6154%
HMOFP (ours)	97.5%	92.6136%	93.6937%	96.1538%

6. Conclusion

In this paper, we proposed an algorithm for abnormal events detection in crowded scenes with global-frame scale. Our method contains two main procedures: first is computing the histogram of maximal optical flow projection (HMOFP) descriptor of the input video sequence. Second, one-class SVM classifier is utilized for nonlinear classification of the testing sets. The proposed method has been tested on several surveillance video datasets with good detection accuracy.