Sage Journals: Discover world-class research

Abstract

We propose a novel pruning method which uses the oscillations around 0, i.e. sign flips, that a weight has undergone during training in order to determine its saliency. Our method can perform pruning before the network has converged, requires little tuning effort due to having good default values for its hyperparameters, and can directly target the level of sparsity desired by the user. Our experiments, performed on a variety of object classification architectures, show that it is competitive with existing methods and achieves state-of-the-art performance for levels of sparsity of $99.6 %$ and above for 2 out of 3 of the architectures tested. Moreover, we demonstrate that our method is compatible with quantization, another model compression technique. For reproducibility, we release our code at https://github.com/AndreiXYZ/flipout.

Keywords

Deep learning network pruning quantization computer vision

Get full access to this article

View all access options for this article.

References

A.C.

Apostol,

M.C.

Stol and

Forré, FlipOut: Uncovering redundant weights via sign flipping, in: Artificial Intelligence and Machine Learning,

Baratchi,

Cao,

W.A.

Kosters,

Lijffijt,

J.N.

van Rijn and

F.W.

Takes, eds, Springer International Publishing, Cham, 2021, pp. 15–29. ISBN 978-3-030-76640-5. doi:10.1007/978-3-030-76640-5_2.

Bellec,

Kappel,

Maass and

Legenstein, Deep rewiring: Training very sparse deep networks, in: International Conference on Learning Representations, 2018.

Bottou, Online algorithms and stochastic approximations, in: Online Learning and Neural Networks, Cambridge University Press, Cambridge, UK, 1998, revised, Oct. 2012.

Brock,

Donahue and

Simonyan, Large scale GAN training for high fidelity natural image synthesis, in: International Conference on Learning Representations, 2019.

Ding,

Zhou,

Guo,

Han,

Liu et al., Global sparse momentum sgd for pruning very deep neural networks, in: Advances in Neural Information Processing Systems, 2019.

Frankle and

Carbin, The lottery ticket hypothesis: Finding sparse, trainable neural networks, in: International Conference on Learning Representations, 2019.

Goodfellow,

Pouget-Abadie,

Mirza,

Xu,

Warde-Farley,

Ozair,

Courville and

Bengio, Generative adversarial nets, in: Advances in Neural Information Processing Systems, 2014, pp. 2672–2680.

Han,

Pool,

Tran and

Dally, Learning both weights and connections for efficient neural network, in: Advances in Neural Information Processing Systems, 2015.

Hassibi and

D.G.

Stork, Second order derivatives for network pruning: Optimal brain surgeon, in: Advances in Neural Information Processing Systems, 1993.

10.

Hawks,

Duarte,

N.J.

Fraser,

Pappalardo,

Tran and

Umuroglu, Ps and Qs: Quantization-aware pruning for efficient low latency neural network inference, arXiv preprint (2021). arXiv:2102.11289.

11.

He,

Zhang,

Ren and

Sun, Deep residual learning for image recognition, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. doi:10.1109/CVPR.2016.90.

12.

He,

Zhang and

Sun, Channel pruning for accelerating very deep neural networks, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1389–1397.

13.

Hochreiter and

Schmidhuber, Long short-term memory, Neural Computation 9(8) (1997), 1735–1780. doi:10.1162/neco.1997.9.8.1735.

14.

Howard, Imagenette, 2019, Accessed April 6th, 2020 at https://github.com/fastai/imagenette.

15.

Huang,

Liu,

Van Der Maaten and

K.Q.

Weinberger, Densely connected convolutional networks, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. doi:10.1109/CVPR.2017.243.

16.

Ioffe and

Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: Proceedings of the 32nd International Conference on Machine Learning, Vol. 37, ICML’15, JMLR.org, 2015.

17.

Jacob,

Kligys,

Chen,

Zhu,

Tang,

Howard,

Adam and

Kalenichenko, Quantization and training of neural networks for efficient integer-arithmetic-only inference, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2704–2713.

18.

Kingma and

Ba, Adam: A method for stochastic optimization, in: International Conference on Learning Representations, 2015.

19.

D.P.

Kingma and

Welling, Auto-encoding variational Bayes, in: Proceedings of the 2nd International Conference on Learning Representations, 2014.

20.

Ko,

Oehmcke and

Gieseke, Magnitude and uncertainty pruning criterion for neural networks, in: 2019 IEEE International Conference on Big Data (Big Data), 2019. doi:10.1109/BigData47090.2019.9005692.

21.

Krishnamoorthi, Quantizing deep convolutional networks for efficient inference: A whitepaper, arXiv preprint (2018). arXiv:1806.08342.

22.

Krizhevsky, Learning multiple layers of features from tiny images, 2009.

23.

LeCun,

J.S.

Denker and

S.A.

Solla, Optimal brain damage, in: Advances in Neural Information Processing Systems, Vol. 2, 1990.

24.

Lee,

Ajanthan and

Torr, SNIP: Single-shot network pruning based on connection sensitivity, in: International Conference on Learning Representations, 2019.

25.

Li,

Kadav,

Durdanovic,

Samet and

H.P.

Graf, Pruning filters for efficient ConvNets, in: International Conference on Learning Representations, 2017.

26.

Liu,

Xu,

Shi,

R.C.C.

Cheung and

H.K.H.

So, Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers, in: International Conference on Learning Representations, 2020.

27.

Louizos,

Welling and

D.P.

Kingma, Learning sparse neural networks through

L_{0}

regularization, in: International Conference on Learning Representations, 2018.

28.

Martens, Deep learning via Hessian-free optimization, in: ICML, Vol. 27, 2010, pp. 735–742.

29.

Mishra,

J.A.

Latorre,

Pool,

Stosic,

Venkatesh,

Yu and

Micikevicius, Accelerating sparse deep neural networks, arXiv preprint (2021). arXiv:2104.08378.

30.

Molchanov,

Ashukha and

Vetrov, Variational dropout sparsifies deep neural networks, in: International Conference on Machine Learning, 2017.

31.

Nair and

G.E.

Hinton, Rectified linear units improve restricted Boltzmann machines, in: Proceedings of the 27th International Conference on Machine Learning (ICML-10), 2010.

32.

Neelakantan,

Vilnis,

Q.V.

Le,

Sutskever,

Kaiser,

Kurach and

Martens, Adding gradient noise improves learning for very deep networks, arXiv e-prints (2015).

33.

NVIDIA A100 tensor core GPU architecture. https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/nvidia-ampere-architecture-whitepaper.pdf.

34.

Paszke,

Gross,

Massa,

Lerer,

Bradbury,

Chanan,

Killeen,

Lin,

Gimelshein,

Antiga,

Desmaison,

Kopf,

Yang,

DeVito,

Raison,

Tejani,

Chilamkurthy,

Steiner,

Fang,

Bai and

Chintala, PyTorch: An imperative style, high-performance deep learning library, in: Advances in Neural Information Processing Systems, Vol. 32, Curran Associates, Inc., 2019. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf .

35.

Pytorch quantization official documentation. https://pytorch.org/docs/stable/torch.quantization.html.

36.

Saad, Iterative Methods for Sparse Linear Systems, SIAM, 2003.

37.

Simonyan and

Zisserman, Very deep convolutional networks for large-scale image recognition, in: International Conference on Learning Representations, 2015.

38.

Strubell,

Ganesh and

McCallum, Energy and policy considerations for deep learning in NLP, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. doi:10.18653/v1/P19-1355.

39.

N.C.

Thompson,

Greenewald,

Lee and

G.F.

Manso, The computational limits of deep learning, arXiv preprint (2020) arXiv:2007.05558.

40.

Tieleman and

Hinton, Lecture 6.5—rmsprop: Divide the gradient by a running average of its recent magnitude, 2012.

41.

Train CIFAR10 with PyTorch (GitHub Repository), Unknown Author, PyTorch CIFAR-10 GitHub Repository, 2017, Accessed April 6th, 2020 at https://github.com/kuangliu/pytorch-cifar/.

42.

Vaswani,

Shazeer,

Parmar,

Uszkoreit,

Jones,

A.N.

Gomez,

Ł.

Kaiser and

Polosukhin, Attention is all you need, in: Advances in Neural Information Processing Systems, 2017.

43.

Verdenius,

Stol and

Forré, Pruning via iterative ranking of sensitivity statistics, arXiv e-prints (2020). arXiv:2006.00896.

44.

Welling and

Y.W.

Teh, Bayesian learning via stochastic gradient Langevin dynamics, in: International Conference on Machine Learning, 2011.

45.

Yang,

Wen and

Li, DeepHoyer: Learning sparser neural network with differentiable scale-invariant sparsity measures, in: International Conference on Learning Representations, 2020.

46.

Zhou,

Lan,

Liu and

Yosinski, Deconstructing lottery tickets: Zeros, signs, and the supermask, in: Advances in Neural Information Processing Systems, 2019.

Pruning by leveraging training dynamics

Abstract

Keywords

Get full access to this article

References