Pruning of neural networks is undoubtedly a popular approach to cope with the current compression of large-scale, high-cost network models. However, most of the existing methods require a high level of human-regulated pruning criteria, which requires a lot of human effort to figure out a reasonable pruning strength. One of the main reasons is that there are different levels of sensitivity distribution in the network. Our main goal is to discover compression methods that adapt to this distribution to avoid deep architectural damage to the network due to unnecessary pruning. In this paper, we propose a filter texture distribution that affects the training of the network. We also analyze the sensitivity of each of the diverse states of this distribution. To do so, we first use a multidimensional penalty method that can analyze the potential sensitivity based on this texture distribution to obtain a pruning-friendly sparse environment. Then, we set up a lightweight dynamic threshold container in order to prune the sparse network. By providing each filter with a suitable threshold for that filter at a low cost, a massive reduction in the number of parameters is achieved without affecting the contribution of certain pruning-sensitive layers to the network as a whole. In the final experiments, our two methods adapted to texture distribution were applied to ResNet Deep Neural Network (DNN) and VGG-16, which were deployed on the classical CIFAR-10/100 and ImageNet datasets with excellent results in order to facilitate comparison with good cutting-edge pruning methods. Code is available at https://github.com/wangyuzhe27/CDP-and-DTC.
E.L.Denton, W.Zaremba, J.Bruna, Y.LeCun and R.Fergus, Exploiting linear structure within convolutional networks for efficient evaluation, Advances in neural information processing systems27 (2014).
4.
X.Ding, Y.Guo, G.Ding and J.Han, ACNet: Strengthening the kernel skeletons for powerful CNN via asymmetric convolution blocks, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 1911–1920.
5.
A.Graves, A.-r.Mohamed and G.Hinton, Speech recognition with deep recurrent neural networks, in: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, 2013, pp. 6645–6649. doi:10.1109/ICASSP.2013.6638947.
6.
S.Han, X.Liu, H.Mao, J.Pu, A.Pedram, M.A.Horowitz and W.J.Dally, EIE: Efficient inference engine on compressed deep neural network, ACM SIGARCH Computer Architecture News44(3) (2016), 243–254. doi:10.1145/3007787.3001163.
7.
S.Han, J.Pool, J.Tran and W.Dally, Learning both weights and connections for efficient neural network, Advances in neural information processing systems28 (2015).
8.
B.Hassibi and D.Stork, Second order derivatives for network pruning: Optimal brain surgeon, Advances in neural information processing systems5 (1992).
9.
Y.He, Y.Ding, P.Liu, L.Zhu, H.Zhang and Y.Yang, Learning filter pruning criteria for deep convolutional neural networks acceleration, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 2009–2018.
10.
Y.He, G.Kang, X.Dong, Y.Fu and Y.Yang, Soft filter pruning for accelerating deep convolutional neural networks, 2018, arXiv preprint arXiv:1808.06866.
11.
Y.He, J.Lin, Z.Liu, H.Wang, L.-J.Li and S.Han, Amc: Automl for model compression and acceleration on mobile devices, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 784–800.
12.
Y.He, P.Liu, Z.Wang, Z.Hu and Y.Yang, Filter pruning via geometric median for deep convolutional neural networks acceleration, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4340–4349.
13.
H.Hu, R.Peng, Y.-W.Tai and C.-K.Tang, Network trimming: A data-driven neuron pruning approach towards efficient deep architectures, 2016, arXiv preprint arXiv:1607.03250.
14.
Z.Huang and N.Wang, Data-driven sparse structure selection for deep neural networks, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 304–320.
15.
A.Krizhevsky, I.Sutskever and G.E.Hinton, ImageNet classification with deep convolutional neural networks, Communications of the ACM60(6) (2017), 84–90. doi:10.1145/3065386.
16.
A.Kusupati, V.Ramanujan, R.Somani, M.Wortsman, P.Jain, S.Kakade and A.Farhadi, Soft threshold weight reparameterization for learnable sparsity, in: International Conference on Machine Learning, PMLR, 2020, pp. 5544–5555.
17.
V.Lebedev and V.Lempitsky, Fast ConvNets using group-wise brain damage, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2554–2564.
18.
Y.LeCun, B.Boser, J.Denker, D.Henderson, R.Howard, W.Hubbard and L.Jackel, Handwritten digit recognition with a back-propagation network, Advances in neural information processing systems2 (1989).
19.
N.Lee, T.Ajanthan and P.H.Torr, Snip: Single-shot network pruning based on connection sensitivity, 2018, arXiv preprint arXiv:1810.02340.
20.
H.Li, A.Kadav, I.Durdanovic, H.Samet and H.P.Graf, Pruning filters for efficient convnets, 2016, arXiv preprint arXiv:1608.08710.
21.
M.Lin, R.Ji, Y.Wang, Y.Zhang, B.Zhang, Y.Tian and L.Shao, Hrank: Filter pruning using high-rank feature map, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 1529–1538.
22.
S.Lin, R.Ji, C.Yan, B.Zhang, L.Cao, Q.Ye, F.Huang and D.Doermann, Towards optimal structured cnn pruning via generative adversarial learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2790–2799.
23.
N.Liu, X.Ma, Z.Xu, Y.Wang, J.Tang and J.Ye, Autocompress: An automatic DNN structured pruning framework for ultra-high compression rates, in: Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, 2020, pp. 4876–4883.
24.
Z.Liu, J.Li, Z.Shen, G.Huang, S.Yan and C.Zhang, Learning efficient convolutional networks through network slimming, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2736–2744.
25.
Z.Liu, M.Sun, T.Zhou, G.Huang and T.Darrell, Rethinking the value of network pruning, 2018, arXiv preprint arXiv:1810.05270.
26.
Z.Liu, J.Xu, X.Peng and R.Xiong, Frequency-domain dynamic pruning for convolutional neural networks, Advances in neural information processing systems31 (2018).
27.
J.-H.Luo, J.Wu and W.Lin, ThiNet: A filter level pruning method for deep neural network compression, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 5058–5066.
28.
H.Mao, S.Han, J.Pool, W.Li, X.Liu, Y.Wang and W.J.Dally, Exploring the granularity of sparsity in convolutional neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp. 13–20.
29.
F.Meng, H.Cheng, K.Li, H.Luo, X.Guo, G.Lu and X.Sun, Pruning filter in filter, Advances in Neural Information Processing Systems33 (2020), 17629–17640.
30.
P.Molchanov, S.Tyree, T.Karras, T.Aila and J.Kautz, Pruning convolutional neural networks for resource efficient inference, 2016, arXiv preprint arXiv:1611.06440.
31.
C.Szegedy, V.Vanhoucke, S.Ioffe, J.Shlens and Z.Wojna, Rethinking the inception architecture for computer vision, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2818–2826.
32.
M.Tukan, L.Mualem and A.Maalouf, Pruning neural networks via coresets and convex geometry: Towards no assumptions, 2022, arXiv preprint arXiv:2209.08554.
33.
C.Wang, G.Zhang and R.Grosse, Picking winning tickets before training by preserving gradient flow, 2020, arXiv preprint arXiv:2002.07376.
34.
H.Wang, C.Qin, Y.Zhang and Y.Fu, Neural pruning via growing regularization, in: International Conference on Learning Representations, 2021.
35.
H.Wang, Q.Zhang, Y.Wang, L.Yu and H.Hu, Structured pruning for efficient convnets via incremental regularization, in: 2019 International Joint Conference on Neural Networks (IJCNN), IEEE, 2019, pp. 1–8.
36.
W.Wen, C.Wu, Y.Wang, Y.Chen and H.Li, Learning structured sparsity in deep neural networks, Advances in neural information processing systems29 (2016).
37.
J.Yao, P.Li, X.Kang and Y.Wang, A pruning method based on the dissimilarity of angle among channels and filters, 2022, arXiv preprint arXiv:2210.16504.
38.
Z.You, K.Yan, J.Ye, M.Ma and P.Wang, Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks, Advances in neural information processing systems32 (2019).
39.
R.Yu, A.Li, C.-F.Chen, J.-H.Lai, V.I.Morariu, X.Han, M.Gao, C.-Y.Lin and L.S.Davis, NISP: Pruning networks using neuron importance score propagation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 9194–9203.
40.
M.Yuan and Y.Lin, Model selection and estimation in regression with grouped variables, Journal of the Royal Statistical Society: Series B (Statistical Methodology)68(1) (2006), 49–67. doi:10.1111/j.1467-9868.2005.00532.x.
