Neural network pruning based on input importance

Abstract

In this paper we present a new algorithm called Neural Network Pruning Based on Input Importance (NNPII) that prunes the neural network based on the input importance. The algorithm depends on the frequency of using a certain value of an attribute in all the given instances in the dataset. Pruning will include only links between input layer and hidden layer. The algorithm has three phases, the first phase is the preprocessing phase, where the data inputs are replaced with their importance. The second phase is a forward pass, which is similar to forward pass in the backpropgation algorithm, but instead of using the real inputs as inputs, we use the input importance obtained in the preprocessing stage. The third pass is the backward phase which is again as backpropgation algorithm, but in this stage we use the input importance instead of real inputs, and β factor that measures the value changing for every input attribute, β will be incorporated in the formula in updating the weights between the input layer and the hidden layer. The elimination process is performed based on criterion that depends on Ω factor that represents a threshold value for a certain input attribute for all instances. It is worth mentioning that the pruning is performed within the usual training phases. The proposed algorithm has been tested through three types of experiments, a comparison between backpropgation and NNPII, Applying NNPII with various parameter values and finally comparing NNPII with other various pruning algorithms. Results show that NNPII performs well and compete with other pruning algorithms. NNPII outperforms all other algorithms when the classes are fairly distributed in the datasets.

Keywords

Neural networks pruning backpropgation

Get full access to this article

View all access options for this article.

References

Abbasi-Asl and

Yu , Structural Compression of Convolutional Neural Networks Based on Greedy Filter Pruning, Cornell University arXiv:1705.07356v3 [cs.CV], 21 Jul 2017.

Abbasi-Asl and

Yu , Interpreting Convolutional Neural Networks Through Compression, Cornell University, arXiv:1711.02329v1 [stat.ML], presented at NIPS 2017 Symposium on Interpretable Machine Learning, 7 Nov 2017.

Augasta and

Kathirvalavakumar , A novel pruning algorithm for optimizing feedforward neural network of classification problems, Neural Processing Letters 34(3) (2011), 241–258.

Augasta and

Kathirvalavakumar , Pruning algorithms of neural networks - a comparative study, Central European Journal of Computer Science 3(3) (2013), 105–115.

Babaeizadeh ,

Smaragdis and

Campbell , NoiseOut: A SimpleWay to Prune Neural Networks, in the proceedings of the 29th Conference on Neural Information Processing Systems (NIPS), Barcelona, Spain, 2016.

Bondarenko ,

Borisov and

Aleksejeva , Neurons vs Weights Pruning in Artificial Neural Networks, Environment Technology, Resources III (2015), 22–28.

Castellano ,

Fanelli and

Pelillo , An iterative pruning algorithm for feedforwrd neural networks, IEEE Transactions on Neural Networks 8 (1997), 519–531.

Engelbrecht , A new pruning heuristic based on variance analysis of sensitivity information, IEEE Transactions on Neural Networks 12(6) (2001), 1386–1399.

Hagiwara , A simple and effective method for removal of hidden units and weights, Neurocomputing 6 (1994), 207–218.

10.

Han ,

Pool ,

Tran and

Dally , Learning both Weights and Connections for Efficient Neural Networks, Proceedings of Neural Information Processing System Conference, NIPS 2015, poster, 2015.

11.

Hassibi ,

D.G.

Stork and

G.J.

Wolf , Optimal brain surgeon and general network pruning, Proceedings of IEEE ICNN’93,1, WDS’08 Proceedings of Contributed Papers, Part I, (2008), 293–299.

12.

Lazarevic and

Obradovic , Effective Pruning of Neural Network Classifier Ensembles, Proceedings of International Joint Conference on Neural Networks, Washington, DC, USA 15–19 July 2001.

13.

Le Cun ,

J.S.

Denker and

S.A.

Solla , In.

D. S.

Touretzky (Ed.), Optimal brain damage, Advances in neural information processing systems (Morgan Kaufmann, San Mateo, 1990) 2 (1990), 598–605.

14.

Molchanov ,

Tyree ,

Karras ,

Aila and

Kautz , Pruning convolutional neural networks for resource efficient inference, 5th International Conference on Learning Representations (ICLR 2017), Toulon, France, April 24 - 26, 2017.

15.

Pan ,

Dong and

Guo , DropNeuron: Simplifying the Structure of Deep Neural Networks, Cornell University, arXiv, :1606.07326v3 [cs.CV] 3 Jul 2016.

16.

Ponnapallii ,

Ho and

Thomson , A formal selection and pruning algorithm for feedforward artificial neural network optimization, IEEE Transactions on Neural Networks 10(4) (1999), 964–968.

17.

Sabo and

Yu , A new pruning algorithm for neural network dimension analysis, International Joint Conference on Neural Networks (2008), 3313–3318.

18.

Schiavo , Two methods for pruning neural networks: A performance evaluation, Statistica Applicata 13(1) (2001), 75–88.

19.

Stepniewski and

Keane , Pruning backpropagation neural networks using modern stochastic optimization techniques, Neural Computing and Applications 5(2) (1997), 76–98.

20.

Suzuki ,

Horiba and

Sugie , A Simple Neural Network Pruning Algorithm with Application to Filter Synthesis, Neural Processing Letters 13 (2001), 43–53.

21.

UCI Machine Learning repository, https://archive.ics.uci.edu/ml/datasets.html

22.

H.-J.

Xing and

B.-G.

Hu , Two phase construction of multilayer perceptrons using Information Theory, IEEE Transactions on Neural Network 20(4) (2009), 715–721.

23.

Yilmaz , Artificial neural networks pruning approach for geodetic velocity field determination, BCG - Boletim de Ciências Geodésicas 19(4) (2013), 558–573.