Sage Journals: Discover world-class research

Abstract

This paper offers a recurrent neural network to support vector machine (SVM) learning in regression arising widespread applications in a variety of setting. The SVM learning problem in regression is first converted into an equivalent quadratic programming (QP) formulation. An artificial neural network for SVM learning is then proposed. The presented neural network framework guarantees to obtain the optimal solution of the support vector regression (SVR). The existence and convergence of the trajectories of the network are studied. The Lyapunov stability for the considered neural network is also shown. Two illustrative examples provide a further demonstration of the effectiveness of the method.

Keywords

Neural network support vector regression quadratic optimization convergent stability

Get full access to this article

View all access options for this article.

References

Ben-Yacoub

Abdeljaoued

and Mayoraz

, Fusion of face and speech data for person identity verification, IEEE Transactions on Neural Networks 10 (1999) 1065–1074.

Burges

, A tutorial on support vector machines for patter recognition, Data Mining and Knowledg Discovery 2 (1998), 1–47.

Cortes

and Vapnik

, Support-vector networks, Machine Learning 20 (1995), 273–297.

Chuang

C.C.

S.F.

Jeng

J.T.

and Hsiao

C.C.

, Robust support vector regression networks for function approximation with outliers, IEEE Transactions on Neural Networks 13 (2002), 1322–1330.

Fletcher

, Support Vector Machines Explained, UCL, 2009.

Gunn

S.R.

, Support Vector Machines for Classification and Regression, Univ. Southampton, Southhampton, U.K., Tech. Rep. May 1998. Image Speech and Intelligent Systems Research Group.

Kalouptisidis

, Signal Processing Systems, Theory and Design, New York: Wiley, 1997.

Keerthi

and DeCoste

, A modified finite Newton method for fast solution of large scale linear SVMs, Journal of Machine Learning Research 6(1) (2006), 341–361.

Lin

C.J.

, On the convergence of the decomposition methods for support vector machines, IEEE Transactions on Neural Networks 12 (2001), 1288–1298.

10.

Lin

C.J.

, A formal analysis of stopping criteria of decomposition methods for support vector machines, IEEE Transaction on Neural Networks 13 (2002), 1045–1052.

11.

Mangasarian

O.L.

and Musicant

D.R.

, Successive overrelaxation for support vector machines, IEEE Transactions on Neural Networks 10 (1999), 1032–1037.

12.

Mangasarian

, A finite Newton method for classification problems, Journal: Optimization Methods and Software 17 (2002), 913–929.

13.

Theiler

and Perkins

, Accurate on-line support vector regression, Neural Computation 15 (2003), 2683–2703.

14.

Martin

, On-line support vector machine regression, In Machine Learning: ECML 2002, Springer Berlin Heidelberg, 2002, pp. 282–294.

15.

Osuna

Freund

and Girosi

, Training support vector machines: An application to face detection, in Proc. Computer Vision Pattern Recognition, San Juan, Puerto Rico, June 1997.

16.

Platt

, Sequential minimal optimization: A fast algorithm for training support vector machines, in: Advances in Kernel Methods Support Vector Learning Scholkopf

Burges

C.J.C.

and Smola

A.J.

, eds, Cambridge, MA: MIT Press, 1999, pp. 185–208.

17.

Shevade

S.K.

Keerthi

S.S.

Bhattacharyya

and Murthy

K.R.K.

, Improvements to the SMO algorithm for SVM regression, IEEE Transactions on Neural Networks 11 (2000), 1188–1193.

18.

Suykens

J.A.K.

and Vandewalle

, Training multilayer perceptron classifiers based on a modified support vector method, IEEE Transactions on Neural Networks 10 (1999), 907–911.

19.

Scholkopf

Smola

Williamson

R.C.

and Bartlett

P.L.

, New support vector algorithms, Neural Computing 12 (2000), 1207–1245.

20.

Vapnik

, The Nature of Statistical Learning Theory, New York: Springer-Verlag, 1995.

21.

Vapnik

Golowich

and Smola

, Support vector method for function approximation, regression estimation and signal processing, in Advances in Neural Information Processing Systems. Cambridge, MA: MIT Press, 9, 1997.

22.

Harker

P.T.

and Pang

J.S.

, Finite-dimensional variational inequality and nonlinear complementarity problems: A survey of theory, algorithms, and applications, Mathematical Programming Series B 48 (1990), 161–220.

23.

B.S.

and Liao

L.-Z.

, Improvements of some projection methods for monotone nonlinear variational inequalities, Journal of Optimization Theory and Applications 112 (2002), 111–128.

24.

Solodov

M.V.

and Tseng

, Modified projection-type methods for monotone variational inequalities, SIAM Journal on Control and Optimization 2 (1996), 1814–1830.

25.

Konnov

I.V.

, A class of combined iterative methods for solving variational inequalities, Journal of Optimization Theory and Applications 94 (1997), 677–693.

26.

Cichocki

and Unbehauen

, Neural networks for optimization with bounded constraints, IEEE Transactions on Neural Networks 4 (1993), 293–304.

27.

Hopfield

J.J.

and Tank

D.W.

, Computing with neural circuits: a model, Science 233 (1986), 625–633.

28.

Tank

D.W.

and Hopfield

J.J.

, Simple neural optimization networks: An A/D converter, signal decision circuit, and a linear programming pircuit, IEEE Transactions on Circuits and Systems 33 (1986), 533–541.

29.

Anguita

and Boni

, Improved neural network for SVM learning, IEEE Transactions on Neural Networks 13 (2002), 1243–1244.

30.

Effati

and Nazemi

A.R.

, Neural network models and its application for solving linear and quadratic programming problems, Applied Mathematics and Computation 172 (2006), 305–331.

31.

Forti

Nistri

and Quincampoix

, Convergence of neural networks for programming problems via a nonsmooth Lojasiewicz inequality, IEEE Transactions on Neural Networks 17 (2006), 1471–1486.

32.

Friesz

T.L.

Bernstein

D.H.

Mehta

N.J.

Tobin

R.L.

and Ganjlizadeh

, Day-to-day dynamic network disequilibria and idealized traveler information systems, Operation Research 42 (1994), 1120–1136.

33.

Gao and

Liao

, A new projection-based neural network for constrained variational inequalities, IEEE Transactions on Neural Networks 20 (2009), 373–388.

34.

, Applications of the general projection neural network in solving extended linear-quadratic programming problems with linear constraints, Neurocomputing 72 (2009), 1131–1137.

35.

and Wang

, Design of general projection neural networks for solving monotone linear variational inequalities and linear and quadratic optimization problems, IEEE Transactions on Systems, Man, and Cybernetics, Part B 37 (2007), 1414–1421.

36.

Sun

and Zhang

, Design of recurrent neural networks for solving constrained least absolute deviation problems, IEEE Transactions on Neural Networks 21 (2010), 1073–1086.

37.

and Zhang

, An alternative recurrent neural network for solving variational inequalities and related optimization problems, IEEE Transactions on System Man Cybernetics, B 39 (2009), 1640–1645.

38.

Kennedy

M.P.

and Chua

L.O.

, Neural networks for nonlinear programming, IEEE Transactions on Circuits and Systems 35 (1988), 554–562.

39.

Lillo

W.E.

Loh

M.H.

Hui

and Zăk

S.H.

, On solving constrained optimization problems with neural networks: A penalty method approach, IEEE Transactions on Neural Networks 4 (1993), 931–939.

40.

Liu

Q.S.

and Wang

, A one-layer recurrent neural network with a discontinuous hard-limiting activation function for quadratic programming, IEEE Transactions on Neural Networks 19 (2008), 558–570.

41.

Liu

and Zhao

, A continuous-time recurrent neural network for real-time support vector regression, Computational Intelligence in Control and Automation (CICA), 2013 IEEE Symposium on, pp 189–193.

42.

Maa

C.Y.

and Shanblatt

M.A.

, Linear and quadratic programming neural network analysis, IEEE Transactions on Neural Networks 3 (1992), 580–594.

43.

Malek

Hosseinipour-Mahani

and Ezazipour

, Efficient recurrent neural network model for the solution of general nonlinear optimization problems, Optimization Methods and Software 25 (2010), 1–18.

44.

Nazemi

A.R.

, A dynamic system model for solving convex nonlinear optimization problems, Communications in Nonlinear Science and Numerical Simulation 17 (2012), 1696–1705.

45.

Nazemi

A.R.

, A neural network model for solving convex quadratic programming problems with some applications, Engineering Applications of Artificial Intelligence 32 (2014), 54–62.

46.

Nazemi

A.R.

and Nazemi

, A gradient-based neural network method for solving strictly convex quadratic programming problems, Cognitive Computation 6 (2014), 484–495.

47.

Nazemi

A.R.

, A dynamical model for solving degenerate quadratic minimax problems with constraints, Journal of Computational and Applied Mathematics 236 (2011), 1282–1295.

48.

Nazemi

A.R.

, Solving general convex nonlinear optimization problems by an efficient neurodynamic model, Engineering Applications of Artificial Intelligence 26 (2013), 685–696.

49.

Nazemi

A.R.

and Omidi

, A capable neural network model for solving the maximum flow problem, Journal of Computational and Applied Mathematics 236 (2012), 3498–3513.

50.

Nazemi

A.R.

and Omidi

, An efficient dynamic model for solving the shortest path problem, Transportation Research Part C 26 (2013), 1–19.

51.

Nazemi

A.R.

and Tahmasbi

, A computational intelligence method for solving a class of portfolio optimization problems, Soft Computing 18 (2014), 2101–2117.

52.

Nazemi

A.R.

Abbasi

and Omidi

, Solving portfolio selection models with uncertain returns using an artificial neural network scheme, Applied Intelligence (2014) (In Press).

53.

Perfetti

and Ricci

, An alog neural network for support vector machine learning, IEEE Transactions on Neural Networks 17 (2006), 1085–1091.

54.

Qin

Fan

and Liu

, A simplified recurrent neural network for pseudoconvex optimization subject to linear equality constraints, Communications in Nonlinear Science and Numerical Simulation 19 (2014), 789–798.

55.

Sun

Chen

J-S.

and Ko

C.-H.

, Neural networks for solving second-order cone constrained variational inequality problem, Computational Optimization and Applications 51 (2012), 623–648.

56.

Tan

, Xia

and Wang

, Neural network realization of support vector methods for pattern classification, in: Proceding IEEE International Joint Conference Neural Networks, 2000, pp. 411–416.

57.

Tao

Cao

and Sun

, A simple and high performance neural network for quadratic programming problems, Applied Mathematics and Computation 124 (2001) 251–260.

58.

Xia

, A new neural network for solving linear and quadratic programming problems, IEEE Transactions on Neural Networks 7 (1996), 1544–1547.

59.

Xia

and Wang

, A recurrent neural network for Solving nonlinear convex programs subject to linear constraints, IEEE Transactions on Neural Networks 16 (2005), 379–386.

60.

Xia

and Wang

, A one layer recurrent neural network for support vector machine learning, IEEE Transactions on Systems, Man Cybernetics – Part B 34 (2004) 1261–1269.

61.

Xia

Y.S.

and Wang

, A recurrent neural network for solving linear projection equations, Neural Networks 13 (2000), 337–350.

62.

Xia

Feng

and Wang

, A recurrent neural network with exponential convergence for solving convex quadratic program and related linear piecewise equation, Neural Networks 17 (2004), 1003–1015.

63.

Xue

and Bian

, A project neural network for solving degenerate convex quadratic program, Neurocomputing 70 (2007), 2449–2459.

64.

, Projection neural networks for solving constrained convex and degenerate quadratic problems (2010) Proceedings – 2010 IEEE International Conference on Intelligent Computing and Intelligent Systems, ICIS 2010, 3, art. no. 5658537, pp.91–96.

65.

Yang

and Hu

, A compact neural network for training support vector machines, Neurocomputing 86 (2012), 193–198.

66.

Yang

, Cao Xua

Hua

and Gao

, A new neural network for solving quadratic programming problemswith equality and inequality constraints, Mathematics and Computers in Simulation 101 (2014), 103–112.

67.

Zhao

and Liu

, Generalized recurrent neural network for ϵ-insensitive support vector regression, Mathematics and Computers in Simulation 86 (2012), 2–9.

68.

Ortega

T.M.

and Rheinboldt

W.C.

, Iterative solution of nonlinear equations in several variables, Academic Press, New York, 1970.

69.

Quarteroni

Sacco

and Saleri

, Numerical mathematics, Volume 37 of Texts in Applied Mathematics, Springer-Verlag, Berlin, second edition, 2007.

70.

Miller

R.K.

and Michel

A.N.

, Ordinary Differential Equations, Academic Press, NewYork, 1982.

71.

Hale

J.K.

, Ordinary Differential Equations, New York: Wiley-Interscience, 1969.

72.

Dierckx

, Curve and Surface Fitting With Splines. Oxford, U.K.: Clarendon, 1993.

73.

Bazaraa

M.S.

Sherali

H.D.

and Shetty

C.M.

, Nonlinear Programming-Theory and Algorithms, 2nd ed. New York: Wiley, 1993.

74.

Boyd

and Vandenberghe

, Convex Optimization, Cambridge University Press, Cambridge, 2004.

75.

Avriel

, Nonlinear Programming: Analysis and Methods, Englewood Cliffs, NJ: Prentice-Hall, 1976.

An application of a practical neural network model for solving support vector regression problems

Abstract

Keywords

Get full access to this article

References