GMM with parameters initialization based on SVD for network threat detection

Abstract

In the field of security, the data labels are unknown or the labels are too expensive to label, so that clustering methods are used to detect the threat behavior contained in the big data. The most widely used probabilistic clustering model is Gaussian Mixture Models(GMM), which is flexible and powerful to apply prior knowledge for modelling the uncertainty of the data. Therefore, in this paper, we use GMM to build the threat behavior detection model. Commonly, Expectation Maximization (EM) and Variational Inference (VI) are used to estimate the optimal parameters of GMM. However, both EM and VI are quite sensitive to the initial values of the parameters. Therefore, we propose to use Singular Value Decomposition (SVD) to initialize the parameters. Firstly, SVD is used to factorize the data set matrix to get the singular value matrix and singular matrices. Then we calculate the number of the components of GMM by the first two singular values in the singular value matrix and the dimension of the data. Next, other parameters of GMM, such as the mixing coefficients, the mean and the covariance, are calculated based on the number of the components. After that, the initialization values of the parameters are input into EM and VI to estimate the optimal parameters of GMM. The experiment results indicate that our proposed method performs well on the parameters initialization of GMM clustering using EM and VI for estimating parameters.

Keywords

Network threat detection gaussian mixture models expectation maximization variational inference singular value decomposition parameters initialization

Get full access to this article

View all access options for this article.

References

Fan

, Bouguila

and Sallay

, Anomaly intrusion detection using incremental learning of an infinite mixture model with feature selection, International Conference on Rough Sets and Knowledge Technology. Springer, Berlin, Heidelberg, 2013.

Tsai

C.F.

, Hsu

Y.F.

and Lin

C.Y.

, et al., Intrusion detection by machine learning: A review, Expert Systems with Applications 36(10) (2009), 11994–12000.

Mayhew

, Atighetchi

and Adler

, et al., Use of machine learning in big data analytics for insider threat detection, MILCOM 2015-2015 IEEE Military Communications Conference, IEEE, (2015), 915–922.

Agrawal

and Agrawal

, Survey on anomaly detection using data mining techniques, Procedia Computer Science 60 (2015), 708–713.

Nicholas

, Ooi

S.Y.

, Pang

Y.H.

, Hwang

S.O.

and Tan

S.Y.

, Study of long short-term memory in flow-based network intrusion detection system, Journal of Intelligent & Fuzzy Systems 35(6) (2018), 5947–5957.

O’Hagan

and White

, Improved model-based clustering performance using Bayesian initialization averaging, Computational Statistics 34(1) (2019), 201–231.

Chen

, An effective synchronization clustering algorithm, Applied Intelligence 46(1) (2017), 135–157.

Nguyen

H.L.

, Woon

Y.K.

and Ng

W.K.

, A survey on data stream clustering and classification, Knowledge and Information Systems 45(3) (2015), 535–569.

Chen

, Qian

and Wang

, et al., Large-scale fuzzy multiple-medoid clustering method, Journal of Intelligent and Fuzzy Systems 32(3) (2017), 1833–1845.

10.

Biernacki

and Celeux

, and G Govaert, Choosing starting values for the EM algorithm for getting the highest likelihood in multivariate Gaussian mixture models, Computational Statistics and Data Analysis 41(3-4) (2003), 561–575.

11.

Bagherinia

, Minaei-Bidgoli

and Hossinzadeh

, et al., Elite fuzzy clustering ensemble based on clustering diversity and quality measures, Applied Intelligence 49(5) (2019), 1724–1747.

12.

Bouveyron

, Girard

and Schmid

, High-dimensional data clustering, Computational Statistics and Data Analysis 52(1) (2007), 502–519.

13.

Melnykov

and Melnykov

, Initializing the EM algorithm in Gaussian mixture models with an unknown number of components, Computational Statistics and Data Analysis 56(6) (2012), 1381–1395.

14.

Melnykov

and Maitra

, Finite mixture models and model-based clustering, Statistics Surveys 4 (2010), 80–116.

15.

Zong

, Song

and Min

M.R.

, et al, Deep autoencoding gaussian mixture model for unsupervised anomaly detection, 2018.

16.

and Leijon

, Bayesian estimation of beta mixture models with variational inference, IEEE Transactions on Pattern Analysis and Machine Intelligence 33(11) (2011), 2160–2173.

17.

Nguyen

T.M.

, Wu

Q.M.J.

and Zhang

, Bounded generalized Gaussian mixture model, Pattern Recognition 47(9) (2014), 3132–3142.

18.

, Fan

and Du

J.X.

, et al., A novel statistical approach for clustering positive data based on finite inverted Beta-Liouville mixture models, Neurocomputing 333 (2019), 110–123.

19.

Yao

and Ge

, Scalable Semisupervised GMM for Big Data Quality Prediction in Multimode Processes, IEEE Transactions on Industrial Electronics 66(5) (2018), 3681–3692.

20.

, Lai

and Kleijn

B.W.

, et al., Variational Bayesian learning for Dirichlet process mixture of inverted Dirichlet distributions in non-Gaussian image feature modeling, IEEE Transactions on Neural Networks and Learning Systems 30(2) (2018), 449–463.

21.

Lai

, Ma

and Xu

, et al., Positive Data Modeling Using a Mixture of Mixtures of Inverted Beta Distributions, IEEE Access 7 (2019), 38146–38156.

22.

Fan

, Al-Osaimi

F.R.

and Bouguila

, et al., Proportional data modeling via entropy-based variational bayes learning of mixture models, Applied Intelligence 47(2) (2017), 473–487.

23.

Yang

M.S.

, Lai

C.Y.

and Lin

C.Y.

, A robust EM clustering algorithm for Gaussian mixture models, Pattern Recognition 45(11) (2012), 3950–3961.

24.

and Dy

J.G.

, In search of deterministic methods for initializing K-means and Gaussian mixture clustering, Intelligent Data Analysis 11(4) (2007), 319–338.

25.

Denisova

and Sergeyev

, Using hierarchical histogram representation for the EM clustering algorithm enhancement, Proceedings of the 10th International Symposium on Image and Signal Processing and Analysis. IEEE, (2017), 41–46.

26.

Nasios

and Bors

A.G.

, Variational learning for Gaussian mixture models, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 36(4) (2006), 849–862.

27.

Maitra

, Initializing partition-optimization algorithms, IEEE/ACM Transactions on Computational Biology and Bioinformatics 6(1) (2009), 144–157.

28.

Fraley

and Raftery

A.E.

, Model-based clustering, discriminant analysis, and density estimation, Journal of the American Statistical Association 97(458) (2002), 611–631.

29.

Wan

, Liu

and Wu

, et al., ICGT: A novel incremental clustering approach based on GMM tree, Data & Knowledge Engineering 117 (2018), 71–86.

30.

Maitra

and Melnykov

, Simulating data to study performance of finite mixture modeling and clustering algorithms, Journal of Computational and Graphical Statistics 19(2) (2010), 354–376.

31.

Karlis

and Xekalaki

, Choosing initial values for the EM algorithm for finite mixtures, Computational Statistics and Data Analysis 41(3-4) (2003), 577–590.

32.

Boutsidis

and Gallopoulos

, SVD based initialization: A head start for nonnegative matrix factorization, Pattern Recognition 41(4) (2008), 1350–1362.

33.

Becker

J.M.

and Menzel

, Rohlfing

, Complex SVD initialization for NMF source separation on audio spectrograms, DAGA, 2015.

34.

Zong

, Song

and Min

M.R.

, et al. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. 2018.

35.

Gaussian mixture models, https://scikit-learn.org/stable/modules/mixture.html#mixture.

36.

Bonilla

E.V.

, Chai

K.M.

and Williams

, Multi-task Gaussian process prediction, Advances in Neural Information Processing systems, (2008), 153–160.

37.

McLachlan

and Peel

, Finite mixture models, John Wiley and Sons, USA, 2004. https://www.tandfonline.com/doi/abs/10.1198/tech.2002.s651

38.

Bishop

C.M.

, Pattern recognition and machine learning, Springer, Berlin, 2006.

39.

, Taghia

and Guo

, On the Convergence of ExtendedVariational Inference for Non-Gaussian Statistical Models, arXiv preprint arXiv:1902,05068, 2019.

40.

Blei

D.M.

and Jordan

M.I.

, Variational inference for Dirichlet process mixtures, Bayesian Analysis 1(1) (2006), 121–143.

41.

Attias

, A variational baysian framework for graphical models, Advances in Neural Information Processing Systems, (2000), 209–215.

42.

Fan

, Sallay

and Bouguila

, et al., Variational learning of hierarchical infinite generalized Dirichlet mixture models and applications, Soft Computing 20(3) (2016), 979–990.

43.

Lim

Y.J.

and Teh

Y.W.

, Variational Bayesian approach to movie rating prediction, Proceedings of KDD cup and workshop 7 (2007), 15–21.

44.

Tzikas

D.G.

, Likas

A.C.

and Galatsanos

N.P.

, The variational approximation for Bayesian inference, IEEE Signal Processing Magazine 25(6) (2008), 131–146.

45.

Choudrey

R.A.

and Roberts

S.J.

, Variational mixture of Bayesian independent component analyzers, Neural Computation 15(1) (2003), 213–252.

46.

Peña

J.M.

and Sauer

, SVD update methods for large matrices and applications, Linear Algebra and its Applications 561 (2019), 41–62.

47.

Qiao

, New SVD based initialization strategy for non-negative matrix factorization, Pattern Recognition Letters 63 (2015), 71–77.

48.

Zhao

and Ye

, Singular value decomposition packet and its application to extraction of weak fault feature, Mechanical Systems and Signal Processing 70 (2016), 73–86.

49.

Kim

, Sparse inverse covariance learning of conditional Gaussian mixtures for multiple output regression, Applied Intelligence 44(1) (2016), 17–29.

50.

UCI Machine Learning Repository, http://archive.ics.uci.edu/ml/datasets.php.

51.

Raykov

Y.P.

, Boukouvalas

and Baig

, et al., What to do when k-means clustering fails: A simple yet principled alternative algorithm, PloS One 1(9) (2016), 1e0162259.

52.

Clustering performance evaluation, https://scikit-learn.org/stable/modules/clustering.html#clustering-evaluation.