Kernel based Support Vector Machines, SVM, one of the most popular machine learning models, usually achieve top performances in two-class classification and regression problems. However, their training cost is at least quadratic on sample size, making them thus unsuitable for large sample problems. However, Deep Neural Networks (DNNs), with a cost linear on sample size, are able to solve big data problems relatively easily. In this work we propose to combine the advanced representations that DNNs can achieve in their last hidden layers with the hinge and insensitive losses that are used in two-class SVM classification and regression. We can thus have much better scalability while achieving performances comparable to those of SVMs. Moreover, we will also show that the resulting Deep SVM models are competitive with standard DNNs in two-class classification problems but have an edge in regression ones.
SchölkopfBSmolaAJ. Learning with Kernels: support vector machines, regularization, optimization, and beyond. Adaptive computation and machine learning series. MIT Press; 2002.
2.
BordesAErtekinSWestonJBottouL. Fast kernel classifiers with online and active learning. Journal of Machine Learning Research. 2005; 6: 1579–1619.
3.
ClaesenMSmetFDSuykensJAKMoorBD. EnsembleSVM: a library for ensemble learning using support vector machines. Journal of Machine Learning Research. 2014; 15(1): 141–145.
WenZShiJLiQHeBChenJ. ThunderSVM: a fast SVM library on GPUs and CPUs. Journal of Machine Learning Research. 2018; 19(21): 1–5.
6.
YuHHsiehCChangKLinC. Large linear classification when data cannot fit in memory. TKDD. 2012; 5(4): 23:1–23:23.
7.
HsiehCChangKLinCKeerthiSSSundararajanS. A dual coordinate descent method for large-scale linear SVM. In: Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5–9, 2008; 2008. pp. 408–415.
8.
Shalev-ShwartzSSingerYSrebroNCotterA. Pegasos: primal estimated sub-gradient solver for SVM. Math Program. 2011; 127(1): 3–30.
9.
SteinwartIHushDRScovelC. Training SVMs without offset. J Mach Learn Res. 2011; 12: 141–202.
10.
TangY. Deep Learning using Support Vector Machines. CoRR. 2013; abs/1306.0239. Available from: http://arxiv.org/abs/1306.0239.
11.
ZhangSLiuCYaoKGongY. Deep neural support vector machines for speech recognition. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19–24, 2015; 2015. pp. 4275–4279.
12.
CholletF. Keras: Deep Learning library for Theano and TensorFlow; 2015. Available from: https://github.com/fchollet/keras.
13.
RudinCCarlsonD. The Secrets of Machine Learning: Ten Things You Wish You Had Known Earlier to be More Effective at Data Analysis. CoRR. 2019; abs/1906.01998. Available from: http://arxiv.org/abs/1906.01998.
Díaz-VicoDPradaJOmariADorronsoroJR. Deep Support Vector Classification and Regression. In: From Bioinspired Systems and Biomedical Applications to Machine Learning – 8th International Work-Conference on the Interplay Between Natural and Artificial Computation, IWINAC 2019, Almería, Spain, June 3–7, 2019, Proceedings, Part II; 2019. pp. 33–43.
16.
LeCunYBottouLBengioYHaffnerP. Gradient-based learning applied to document recognition. Proceedings of the IEEE. 1998; 86: 2278–2323.
17.
HeKZhangXRenSSunJ. Deep Residual Learning for Image Recognition. In: IEEE Conference on Computer Vision and Pattern Recognition; 2016. pp. 770–778.
18.
LázaroJLDorronsoroJR. Simple proof of convergence of the SMO algorithm for different SVM variants. IEEE Trans Neural Netw Learning Syst. 2012; 23(7): 1142–1147.
19.
MantonJHAmblardP. A primer on reproducing kernel hilbert spaces. Foundations and Trends in Signal Processing. 2015; 8(1–2): 1–126.
20.
KingmaDPBaJ. Adam: A Method for Stochastic Optimization. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings; 2015. Available from: http://arxiv.org/abs/1412.6980.
21.
ChangCCLinCJ. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology. 2011; 2(3): 27:1–27:27. Software available at http://www.csie.ntu.edu.tw/c̃jlin/libsvm.
22.
BenensonR. What is the Class of This Image? Available from: http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html.
23.
WanLZeilerMZhangSLeCunYFergusR. Regularization of Neural Networks Using Dropconnect. In: Proceedings of the 30th International Conference on International Conference on Machine Learning, Vol. 28; 2013. pp. III–1058–III–1066.
24.
LeCunY. The MNIST Database of Handwritten Digits. Available from: http://yann.lecun.com/exdb/mnist/.
25.
XiaoHRasulKVollgrafR. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. CoRR. 2017; abs/1708.07747.
26.
Fashion-MNISTZ. Available from: https://github.com/zalandoresearch/fashion-mnist.
27.
Keras Examples: CIFAR-10 ResNet. Available from: https://keras.io/examples/cifar10_resnet/.
28.
SunSChenWWangLLiuTY. Large Margin Deep Neural Networks: Theory and Algorithms. ArXiv. 2015; abs/1506. 05232.
29.
LiuWWenYYuZYangM. Large-Margin Softmax Loss for Convolutional Neural Networks. In: Proceedings of The 33rd International Conference on Machine Learning, Vol. 48 of Proceedings of Machine Learning Research; 2016. pp. 507–516.
30.
LiangXWangXLeiZLiaoSLiSZ. Soft-Margin Softmax for Deep Classification. In: LiuDXieSLiYZhaoDEl-AlfyESM, eds. ICONIP. Neural Information Processing. Springer International Publishing; 2017. pp. 413–421.
31.
ElsayedGKrishnanDMobahiHReganKBengioS. Large Margin Deep Networks for Classification. In: BengioSWallachHLarochelleHGraumanKCesa-BianchiNGarnettR, eds. Advances in Neural Information Processing Systems 31; 2018. pp. 842–852.
32.
JacotAGabrielFHonglerC. Neural Tangent Kernel: Convergence and Generalization in Neural Networks. In: BengioSWallachHLarochelleHGraumanKCesa-BianchiNGarnettR, eds. Advances in Neural Information Processing Systems 31; 2018. pp. 8571–8580.
33.
AroraSDuSSHuWLiZSalakhutdinovRWangR. On Exact Computation with an Infinitely Wide Neural Net. CoRR. 2019; abs/1904.11955.
34.
ZhangCBengioSHardtMRechtBVinyalsO. Understanding deep learning requires rethinking generalization. ArXiv. 2016; abs/1611.03530.
35.
OlsonMWynerABerkR. Modern Neural Networks Generalize on Small Data Sets. In: BengioSWallachHLarochelleHGraumanKCesa-BianchiNGarnettR, eds. Advances in Neural Information Processing Systems 31; 2018. pp. 3619–3628.
36.
AhmadlouMAdeliH. Enhanced probabilistic neural network with local decision circles: a robust classifier. Integrated Computer-Aided Engineering. 2010 8; 17: 197–210.
37.
RafieiMHAdeliH. A new neural dynamic classification algorithm. IEEE Trans Neural Networks Learn Syst. 2017; 28(12): 3074–3083.
AntoniadesASpyrouLMartín-LopezDValentínAAlarcónGSaneiS, et al. Deep neural architectures for mapping scalp to intracranial EEG. International Journal of Neural Systems. 2018; 28(8): 1850009:1–1850009:15.
40.
WuTBîlbîeFDPaunAPanLNeriF. Simplified and yet turing universal spiking neural p systems with communication on request. International Journal of Neural Systems. 2018; 28(8): 1850013:1–1850013:19.
41.
PupoOGRVenturaS. Performing multi-target regression via a parameter sharing-based deep network. International Journal of Neural Systems. 2019; 29(9): 1950014:1–1950014:22.
42.
NguyenaTKashaniaABordasbSNgoT. Deep neural network with higher order neuron for the prediction of foamed concrete strength. Computer-Aided Civil and Infrastructure Engineering. 2019; 34(5): 316–332.
43.
LiangX. Image-based post-disaster inspection of reinforced concrete bridge systems using deep learning with bayesian optimization. Computer-Aided Civil and Infrastructure Engineering. 2019; 34(5): 415–430.
44.
MaedaKOgawaTHaseyamaMTakahashiS. Convolutional sparse coding-based deep random vector functional link network for distress classification of road structures. Computer-Aided Civil and Infrastructure Engineering. 2019; 34(8): 654–676.
45.
GaoYKongBMosalamKM. Deep leaf-bootstrapping generative adversarial network for structural image data augmentation. Computer-Aided Civil and Infrastructure Engineering. 2019; 34(9): 755–773.
46.
WuRTAnkush SinglaAJahanshahiMRBertinoEKoBJVermaD. Pruning deep convolutional neural networks for efficient edge computing in condition assessment of civil infrastructures. Computer-Aided Civil and Infrastructure Engineering. 2019; 34(9): 774–789.
47.
ZhangYChengTRenY. A graph deep learning method for short-term traffic forecasting on large road networks. Computer-Aided Civil and Infrastructure Engineering. 2019; 34(10): 431–449.
