CNN Pruning Method Based on Information Fusion Strategy

doi:10.3778/j.issn.1002-8331.2203-0289

Abstract

Abstract: Current structured pruning methods rely too much on the pre-training models and directly discard unimportant convolution kernel information, resulting in performance degradation that cannot be ignored. Aiming at the problems, a convolution neural network pruning method（APBM） is proposed to reduce model complexity and achieve model acceleration. The APBM method introduces the concept of information entropy to represent the similarity distribution of the convolution kernels, then uses the relative entropy between the distributions to dynamically measure the importance of the convolution kernels. At the same time, in order to reduce the information loss in the pruning process and improve the fault tolerance of the method, the information fusion strategy, fusing the unimportant convolution kernels information and the important convolution kernels, is adopted in the forward propagation of model training. Experiments are carried out on the CIFAR-10 and CIFAR-100 datasets. The experimental results show that the APBM method achieves less training time, higher model compression rate, and best accuracy compared to the network pruning algorithms such as HRank, Polarization, and SWP.On CIFAR-10, the method effectively reduces the parameters of VGG16 and ResNet56 by 92.74% and 48.84% respectively without significant decrease in accuracy. On CIFAR-100, this method reduces the parameters of VGG16 and ResNet56 by 72.91% and 44.18% respectively with a little accuracy loss.

Key words: structured pruning, information entropy, model complexity, model acceleration, information fusion

摘要： 针对现有结构化剪枝方法过度依赖预训练模型和直接丢弃不重要卷积核的信息造成了明显的性能下降的问题，提出一种基于信息融合策略的卷积神经网络剪枝方法（APBM），以较小精度损失降低模型复杂度、实现模型加速。首先APBM方法引入信息熵概念以表示卷积核的相似度分布，并使用分布之间的相对熵动态衡量卷积核的重要程度；同时在训练的前向传播中采用信息融合策略：融合非重要卷积核信息与重要卷积核信息，以减少剪枝过程中的信息损失和提高剪枝的容错性。在CIFAR10和CIFAR100数据集上进行验证和对比实验。实验结果表明：相对于HRank、Polarization、SWP等剪枝算法，APBM方法训练时间更少、模型压缩率更高，精度保持最佳。在基于CIFAR10的剪枝任务中，对VGG16和ResNet56分别剪掉92.74%和48.84%的参数量；在基于CIFAR100的剪枝任务中，对VGG16和ResNet56分别剪掉72.91%和44.18%的参数量。

关键词: 结构化剪枝, 信息熵, 模型复杂度, 模型加速, 信息融合

QU Haicheng, ZHANG Xuecong, WANG Yuping. CNN Pruning Method Based on Information Fusion Strategy[J]. Computer Engineering and Applications, 2022, 58(24): 125-133.

曲海成, 张雪聪, 王宇萍. 基于信息融合策略的卷积神经网络剪枝方法[J]. 计算机工程与应用, 2022, 58(24): 125-133.

References

[1] GAO S H，CHENG M M，ZHAO K，et al.Res2net：a new multi-scale backbone architecture[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2019，43（2）：652-662.
[2] REDMON J，FARHADI A.YOLOv3：an incremental improvement[C]//IEEE Conference on Computer Vision and Pattern Recognition（CVPR），June 18-22，2018，Salt Lake City，UT，USA.New York：IEEE，2018：1-6.
[3] 谭显东，彭辉.改进YOLOv5的SAR图像舰船目标检测[J].计算机工程与应用，2022，58（4）：247-254.
TAN X D，PENG H.Improved YOLOv5 ship target detection in SAR image[J].Computer Engineering and Applications，2022，58（4）：247-254.
[4] KANG S，IWANA B K，UCHIDA S.Complex image processing with less data—document image binarization by integrating multiple pre-trained u-net modules[J].Pattern Recognition，2021，109：107577.
[5] LIU Z，LI J G，SHEN Z Q，et al.Learning efficient convolutional networks through network slimming[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision.Washington：IEEE Computer Society，2017：2755-2763.
[6] LIN M B，JI R R，WANG Y，et al.Hrank：filter pruning using high-rank feature map[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Los Angeles：IEEE Computer Society，2020：1529-1538.
[7] LIN M B，JI R R，ZHANG Y X，et al.Channel pruning via automatic structure search[C]//Proceedings of the International Joint Conference on Artificial Intelligence，2020：673-679.
[8] ZHUANG T，ZHANG Z X，HUANG Y H，et al.Neuron-level structured pruning using polarization regularizer[J].Advances in Neural Information Processing Systems，2020，33：9865-9877.
[9] MENG F X，CHENG H，LI K，et al.Pruning filter in filter[J].Advances in Neural Information Processing Systems，2020，33：17629-17640.
[10] 权宇，李志欣，张灿龙，等.融合深度扩张网络和轻量化网络的目标检测模型[J].电子学报，2020，48（2）：390-397.
QUAN Y，LI Z　X，ZHANG C L，et al.Fusing deep dilated convolutions network and light-weight network for object detection[J].Acta Electronica Sinica，2020，48（2）：390-397.
[11] 郎磊，夏应清.紧凑的神经网络模型设计研究综述[J].计算机科学与探索，2020，14（9）：1456-1470.
LANG L，XIA Y Q.Survey on compact neural network model design[J].Journal of Frontiers of Computer Science and Technology，2020，14（9）：1456-1470.
[12] HU Q H，WANG P S，CHENG J.From hashing to CNNs：training binary weight networks via hashing[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence.Palo Alto：AAAI Press，2018：3247-3254.
[13] KIM S，GHOLAMI A，YAO Z，et al.I-bert：integer-only bert quantization[C]//International Conference on Machine Learning，2021：5506-5518.
[14] SHEN Z Q，HE Z K，XUE X Y.Meal：multi-model ensemble via adversarial learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2019：4886-4893.
[15] TURC I，CHANG M W，LEE K，et al.Well-read students learn better：on the importance of pre-training compact models[J].arXiv：1908.08962，2019.
[16] LEBEDEV V，GANIN Y，RAKHUBA M，et al.Speeding-up convolutional neural networks using fine-tuned CP-decomposition[C]//3rd International Conference on Learning Representations，2015.
[17] IDELBAYEV Y，CARREIRA-PERPINáN M A.Low-rank compression of neural nets：learning the rank of each layer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：8049-8059.
[18] AGHASI A，ABDI A，NGUYEN N，et al.Net-trim：convex pruning of deep neural networks with performance guarantee[C]//31st Conference on Neural Information Processing Systems，2017.
[19] LIU Z H，XU J Z，PENG X L，et al.Frequency-domain dynamic pruning for convolutional neural networks[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems，2018：1051-1061.
[20] ZHANG T Y，YE S K，ZHANG K Q，et al.A systematic DNN weight pruning framework using alternating direction method of multipliers[C]//Proceedings of the European Conference on Computer Vision，2018：184-199.
[21] ZHANG Y X，LIN M B，CHAO F，et al.Efficient weight pruning using pre-trained lottery jackpots[J].arXiv：2104.
08700，2021.
[22] LIN M B，CAO L J，LI S J，et al.Filter sketch for network pruning[J].IEEE Transactions on Neural Networks and Learning Systems，2021：1-10.
[23] 陆林，季繁繁，袁晓彤.随机初始化神经网络剪枝的稀疏二值规划方法[J/OL].计算机工程与应用：1-12[2022-03-13].http：//kns.cnki.net/kcms/detail/11.2127.TP.20220302.1228.002.html.
LU L，JI F F，YUAN X T.Sparse binary programming method for pruning of randomly initialized neural networks[J].Computer Engineering and Applications：1-12[2022-03-13].http：//kns.cnki.net/kcms/detail/11.2127.TP.20220302.
1228.002.html.
[24] HINTON G，VINYALS O，DEAN J.Distilling the knowledge in a neural network[J].Computer Science，2015，14（7）：38-39.
[25] LEE W，XIANG D.Information-theoretic measures for anomaly detection[C]//Proceedings 2001 IEEE Symposium on Security and Privacy，2000：130-143.
[26] HE Y，KANG G L，DONG X Y，et al.Soft filter pruning for accelerating deep convolutional neural networks[J].arXiv：1808.06866，2018.
[27] LI H，KADAV A，DURDANOVIC I，et al.Pruning filters for efficient convnets[J].arXiv：1608.08710，2016.
[28] MENG F X，CHENG H，LI K，et al.Filter grafting for deep neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：6599-6607.
[29] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[C]//Proceedings of the 3rd International Conference on Learning Representations，San Diego，CA，USA，May 7-9，2015：1-12.
[30] HE K M，ZHANG X Y，REN S Q，et al.Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition，Las Vegas，Jun 27-30，2016.Washington：IEEE Computer Society，2016：770-778.
[31] WANG H，QIN C，ZHANG Y L，et al.Neural pruning via growing regularization[C]//International Conference on Learning Representations，2020.
[32] TIWARI R，BAMBA U，CHAVAN A，et al.Chipnet：budget-aware pruning with heaviside continuous approximations[J].arXiv：2102.07156，2021.
[33] MA N N，ZHANG X Y，ZHENG H T，et al.Shufflenet v2：practical guidelines for efficient cnn architecture design[C]//Proceedings of the European Conference on Computer Vision，2018：116-131.