Deep Neural Network Channel Pruning Compression Method for Filter Elasticity

doi:10.3778/j.issn.1002-8331.2210-0420

Abstract

Abstract: Deep neural network (DNN) has achieved great success in various fields. Due to its high computing and storage costs, it is difficult to directly deploy them to resource constrained mobile devices. To solve this problem, the importance evaluation of the global filter in the network is studied, and a channel pruning compression method with filter elasticity is proposed to reduce the size of the neural network. Firstly, the method sets the local dynamic threshold between layers to improve the shortage of over pruning in L1 regularization (L1 lasso) sparse training. Then, its output is multiplied by the channel scaling factor to replace the ordinary convolution layer module. The importance of the global filter is defined by the elastic size of the filter. Its values are estimated and ranked by Taylor formula. At the same time, a new iterative pruning framework of the filter is designed to balance the contradiction between the pruning performance and the pruning speed. Finally, the improved L1 regularization training and the importance of the global filter are used to prune the composite channels. VGG-16 is tested on CIFAR-10 using the proposed method, which reduces 80.2% of floating-point operations (FLOPs) and 97.0% of parameter quantities, without significant loss of accuracy, indicating the effectiveness of the method, which can compress neural networks in a large scale, and can be deployed to resource constrained terminal devices.

Key words: model compression, filter importance, channel pruning, scaling factor, elastic

摘要： 深度神经网络（deep neural network，DNN）在各个领域获得了巨大成功，由于其需要高额的计算和储存成本，难以直接将它们部署到资源受限的移动设备端。针对这个问题，对网络中的全局滤波器重要性评估进行了研究，提出滤波器弹性的通道剪枝压缩方法以轻量化神经网络的规模。该方法先设置层间局部动态阈值改进L1正则化（L1 lasso）稀疏训练中剪枝过度的不足；然后将其输出乘以通道缩放因子替换普通的卷积层模块，利用滤波器的弹性大小定义全局滤波器的重要性，其数值由泰勒公式估计得出并排序，同时设计新的滤波器迭代剪枝框架，以平衡剪枝性能和剪枝速度的矛盾；最后利用改进的L1正则化训练和全局滤波器重要性程度进行复合通道剪枝。在CIFAR-10上使用所提方法对VGG-16进行实验，减少了80.2%的浮点运算次数（FLOPs）和97.0%的参数量，而没有明显的准确性损失，表明了方法的有效性，能大规模地压缩神经网络，可部署于资源受限的终端设备。

关键词: 模型压缩, 滤波器重要性, 通道剪枝, 缩放因子, 弹性

LI Ruiquan, ZHU Lu, LIU Yuanyuan. Deep Neural Network Channel Pruning Compression Method for Filter Elasticity[J]. Computer Engineering and Applications, 2024, 60(6): 163-171.

李瑞权, 朱路, 刘媛媛. 滤波器弹性的深度神经网络通道剪枝压缩方法[J]. 计算机工程与应用, 2024, 60(6): 163-171.

References

[1] SINGH B, NAJIBI M, DAVIS L S. Sniper: efficient multi-scale training[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018.
[2] HE K M, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
[3] WANG J, XU C, YANG X, et al. A novel pruning algorithm for smoothing feedforward neural networks based on group lasso method[J]. IEEE Transactions on Neural Networks and Learning Systems, 2017, 29(5): 2012-2024.
[4] FRANKLE J, CARBIN M. The lottery ticket hypothesis: finding sparse, trainable neural networks[EB/OL].(2018-03-09). https://doi.org/10.48550/arXiv.1803.03635.
[5] KRISHNAMOORTHI R. Quantizing deep convolutional networks for efficient inference: a whitepaper[EB/OL].(2018-06-21).https://doi.org/10.48550/arXiv.1806.08342.
[6] KIM H, KARIM M U, KYUNG C M. Efficient neural network compression[EB/OL].(2019-04-12).https://doi.org/10.
48550/arXiv.1811.12781.
[7] CROWLEY E J, GRAG G, STORKEY A J. Moonshine: distilling with cheap convolutions[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018: 2893-2903.
[8] XU L, ZHU X, GONG S. Knowledge distillation by on-the-fly native ensemble[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018: 7517-7527.
[9] HOWARD A G, ZHU M, CHEN B, et al. MobileNets: efficient convolutional neural networks for mobile vision applications[EB/OL].(2017?04?17).https://doi.org/10.48550/arXiv.1704.
04861.
[10] MURAVEV A, RAITOHARJU J, GABBOUJ M. Neural architecture search by estimation of network structure distributions[J]. IEEE Access, 2021, 9: 15304-15319.
[11] HE Y M, ZHANG X, SUN J. Channel pruning for accelerating very deep neural networks[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 1389-1397.
[12] SRINIVAS S, SUBRAMANYA A. Training sparse neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017: 138-145.
[13] LIU Z, LI J, SHEN Z, et al. Learning efficient convolutional networks through network slimming[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 2736-2744.
[14] HAN S, MAO H, DALLY W J. Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding[EB/OL]. (2015-10-01).https://doi.org/10.48550/arXiv.1510.00149.
[15] LI H, KADAV A, DURDANOVIC I, et al. Pruning filters for efficient convnets[EB/OL]. (2016-08-31). https://doi.org/10.48550/arXiv.1608.08710.
[16] LIU Z, SUN M, ZHOU T, et al. Rethinking the value of network pruning[EB/OL]. (2018-10-11).https://doi.org/10.48550/
arXiv.1810.05270.
[17] HAN B, ZHANG Z, XU C, et al. Deep face model compression using entropy-based filter selection[C]//International Conference on Image Analysis and Processing. Cham:Springer, 2017: 127-136.
[18] HE Y M, ZHANG X, SUN J. Channel pruning for accelerating very deep neural networks[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 1389-1397.
[19] LUO J H, WU J, LIN W. Thinet: a filter level pruning method for deep neural network compression[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 5058-5066.
[20] HUANG J, SUN W, HUANG L. Deep neural networks compression learning based on multi-objective evolutionary algorithms[J]. Neurocomputing, 2020, 378:260-269.
[21] LI T, WU B, YANG Y, et al. Compressing convolutional neural networks via factorized convolutional filters[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 3977-3986.
[22] ZHUANG Z W, TAN M K, ZHUANG B. Discrimination-aware channel pruning for deep neural networks[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurIPS), 2018: 883-894.
[23] DONG X, HUANG J, YANG Y, et al. More is less: a more complicated network with less inference complexity[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 5840-5848.
[24] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. Learning multiple layers of features from tiny images[J]. Communications of the ACM, 2017, 60(6): 84-90.
[25] LIN S, JI R, YAN C, et al. Towards optimal structured DNN pruning via generative adversarial learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 2790-2799.
[26] HE Y, LIU P, WANG Z, et al. Filter pruning via geometric median for deep convolutional neural networks acceleration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 4340-4349.
[27] LIN M, JI R, WANG Y, et al. Hrank: filter pruning using high-rank feature map[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 1529-1538.
[28] LIN M, JI R, LI S, et al. Network pruning using adaptive exemplar filters[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(12): 7357-7366.
[29] LIN M, CAO L, ZHANG Y, et al. Pruning networks with cross-layer ranking & k-reciprocal nearest filters[J].IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(11): 9139-9148.