Object Detection Optimization Research for Mobile Terminals

doi:10.3778/j.issn.1002-8331.2206-0139

Abstract

Abstract: In order to meet the demand for real-time object detection in the era of mobile intelligence, numerous solutions have been proposed for object detection optimization for mobile terminals. Optimization ideas can be classified into two categories：one is based on manual or AutoML methods, using a lightweight convolution design to build a lightweight network at the beginning of network design；the other is to use tensor decomposition, model pruning, parameter quantization or other compression methods to adjust the existing object detection model to optimize the detection performance. Then, considering that the development laws of optimization methods are different and related, different analysis angles and comparison dimensions are adopted respectively. Finally, the current situation of the mobile-oriented object detection industrial in China is analyzed from the perspective of market, and the potential problems and development directions of optimization research are discussed.

Key words: object detection, lightweight network, tensor decomposition, model pruning, parameter quantization

摘要： 为适应移动智能时代对实时目标检测的需求，人们针对面向移动端的目标检测优化问题提出了众多解决思路。其优化思路可归纳为轻量化网络设计和模型压缩两类：一类是基于手工设计或自动化机器学习（AutoML）手段，在网络设计之初就采用轻量化卷积设计构建轻量化网络；另一类是借助张量分解、模型剪枝、参数量化等压缩手段，调整现有的目标检测模型来优化检测性能。考虑到优化方法的发展规律不尽相同且彼此之间有所关联，分别采取了不同的分析角度和对比维度。从市场角度剖析了国内面向移动端的目标检测产业化现状，并对其优化研究的潜在问题和发展方向进行了总结与展望。

关键词: 目标检测, 轻量化网络, 张量分解, 模型剪枝, 参数量化

HAN Jingjing, LIU Jiangyue, GONG Weijun, WEI Hongyang, QIAN Yurong. Object Detection Optimization Research for Mobile Terminals[J]. Computer Engineering and Applications, 2022, 58(24): 12-28.

韩晶晶, 刘江越, 公维军, 魏宏杨, 钱育蓉. 面向移动端的目标检测优化研究[J]. 计算机工程与应用, 2022, 58(24): 12-28.

References

[1] 丁培，阿里甫·库尔班，耿丽婷，等.自然环境下实时人脸口罩检测与规范佩戴识别[J].计算机工程与应用，2021，57（24）：268-275.
DING P，ARIF K，GENG L T，et al.Real-time face mask detection and standard wearing recognition method in natural environment[J].Computer Engineering and Applications，2021，57（24）：268-275.
[2] 闵巍庆，刘林虎，刘宇昕，等.食品图像识别方法综述[J].计算机学报，2022，45（3）：542-566.
MIN W Q，LIU L H，LIU Y X，et al.A survey on food image recognition[J].Chinese Journal of Computers，2022，45（3）：542-566.
[3] LECUN Y，BENGIO Y，HINTON G.Deep learning[J].Nature，2015，521（7553）：436-444.
[4] 张政馗，庞为光，谢文静，等.面向实时应用的深度学习研究综述[J].软件学报，2020，31（9）：2654-2677.
ZHANG Z K，PANG W G，XIE W J，et al.Deep learning for real-time applications：a suvery[J].Journal of Software，2020，31（9）：2654-2677.
[5] 张展，张宪琦，左德承，等.面向边缘计算的目标追踪应用部署策略研究[J].软件学报，2020，31（9）：2691-2708.
ZHANG Z，ZHANG X Q，ZUO D C，et al.Research on target tracking application deployment strategy for edge computing[J].Journal of Software，2020，31（9）：2691-2708.
[6] HINTON G E，SALAKHUTDINOV R R.Reducing the dimensionality of data with neural networks[J].Science，2006，313（5786）：504-507.
[7] PAPAGEORGIOU C P，OREN M，POGGIO T.A general framework for object detection[C]//International Conference on Computer Vision，1998：555-562.
[8] LOWE D G.Object recognition from local scale-invariant features[C]//International Conference on Computer Vision，1998：1150-1157.
[9] OJALA T，PIETIKAINEN M，MAENPAA T.Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[J].Transactions on Pattern Analysis and Machine Intelligence，2002，24（7）：971-987.
[10] KE Y，SUKTHANKAR R.PCA-SIFT：a more distinctive representation for local image descriptors[C]//Computer Vision and Pattern Recognition，2004：506-513.
[11] BAY H，TUYTELAARS T，GOOL L.SURF：speeded up robust features[C]//European Conference on Computer Vision，2006：404-417.
[12] VIOLA P，JONES M J.Robust real-time face detection[J].International Journal of Computer Vision，2004，57（2）：137-154.
[13] FREUND Y.Experiments with a new boosting algorithm[C]//International Conference on Machine Learning，1996：148-156.
[14] DALAL N，TRIGGS B.Histograms of oriented gradients for human detection[C]//Computer Vision and Pattern Recognition，2005：886-893.
[15] HEARST M A，DUMAIS S T，OSMAN E，et al.Support vector machines[J].Intelligent Systems and Their Applications，1998，13（4）：18-28.
[16] FELZENSZWALB P F，GIRSHICK R B.Object detection with discriminatively trained part-based models[J].Transactions on Pattern Analysis and Machine Intelligence，2010，32（9）：1627-1645.
[17] FELZENSZWALB P F，MCALLESTER D A，RAMANAN D.A discriminatively trained，multiscale，deformable part model[C]//Computer Vision and Pattern Recognition，2008：1-8.
[18] GRAEFE G.The?cascades?framework?for?query?optimization[J].Data Engineering Bulletin，1995，18（3）：19-29.
[19] FELZENSZWALB P F，GIRSHICK R B.Cascade object detection with deformable part models[C]//Computer Vision and Pattern Recognition，2010：2241-2248.
[20] LECUN Y，BOTTOU L，BENGIO Y，et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE，1998，86（11）：2278-2324.
[21] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.Imagenet classification with deep convolutional neural networks[J].Advances in Neural Information Processing Systems，2012，25（2）：1097-1105.
[22] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Computer Vision and Pattern Recognition，2016：770-778.
[23] GIRSHICK R，DONAHUE J.Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Computer Vision and Pattern Recognition，2014：580-587.
[24] HE K M，ZHANG X Y.Spatial pyramid pooling in deep convolutional networks for visual recognition[J].Transactions on Pattern Analysis and Machine Intelligence，2015，37（9）：1904-1916.
[25] GIRSHICK R.Fast R-CNN[C]//International Conference on Computer Vision，2015：1440-1448.
[26] REN S，HE K，GIRSHICK R，et al.Faster R-CNN：towards real-time object detection with region proposal networks[J].Transactions on Pattern Analysis and Machine Intelligence，2017，39（6）：1137-1149.
[27] LIN T Y，DOLLáR P.Feature pyramid networks for object detection[C]//Computer Vision and Pattern Recognition，2017：2117-2125.
[28] HE K M，GKIOXARI G，DOLLáR P，et al.Mask R-CNN[C]//International Conference on Computer Vision，2017：2961-2969.
[29] CAI Z，VASCONCELOS N.Cascade R-CNN：delving into high quality object detection[C]//Computer Vision and Pattern Recognition，2018：6154-6162.
[30] LIU S，QI L，QIN H，et al.Path aggregation network for instance segmentation[C]//Computer Vision and Pattern Recognition，2018：8759-8768.
[31] ZHOU X，WANG D，KR?HENBüHL P.Objects as points[J].arXiv：1904.07850，2019.
[32] ZHANG H，CHANG H，MA B，et al.Dynamic R-CNN：towards high quality object detection via dynamic training[C]//European Conference on Computer Vision，2020：260-275.
[33] LI Z，WANG F，WANG N.LiDAR R-CNN：an efficient and universal 3D object detector[C]//Computer Vision and Pattern Recognition，2021.
[34] SUN P，ZHANG R，JIANG Y，et al.Sparse R-CNN：end-to-end object detection with learnable proposals[C]//Computer Vision and Pattern Recognition，2021：14454-14463.
[35] YANG C，HUANG Z，WANG N.QueryDet：cascaded sparse query for accelerating high-resolution small object detection[C]//Computer Vision and Pattern Recognition，2022：13668-13677.
[36] 王兵，乐红霞，李文璟，等.改进YOLO轻量化网络的口罩检测算法[J].计算机工程与应用，2021，57（8）：62-69.
WANG B，LE H X，LI W J，et al.Mask detection algorithm based on improved YOLO lightweight network[J].Computer Engineering and Applications，2021，57（8）：62-69.
[37] REDMON J，DIVV S，GIRSHICK R，et al.You only look once：unified，real-time object detection[C]//Computer Vision and Pattern Recognition，2016：779-788.
[38] REDMON J，FARHADI A.YOLO9000：better，faster，stronger[C]//Computer Vision and Pattern Recognition，2017：7263-7271.
[39] REDMON J，FARHADI A.Yolov3：an incremental improvement[J].arXiv：1804.02767，2018.
[40] BOCHKOVSKIY A，WANG C Y，LIAO H.Yolov4：optimal speed and accuracy of object detection[J].arXiv：2004.
10934，2020.
[41] LIU W，ANGUELOV D，ERHAN D，et al.SSD：single shot multibox detector[C]//International Conference on Computer Vision，2016：21-37.
[42] FU C Y，LIU W，RANGA A，et al.DSSD：deconvolutional single shot detector[J].arXiv：1701.06659，2017.
[43] LI Z X，ZHOU F Q.FSSD：feature fusion single shot multibox detector[J].arXiv：1712.00960，2017.
[44] GE Z，LIU S，WANG F，et al.Yolox：exceeding yolo series in 2021[J].arXiv：2107.08430，2021.
[45] CHEN Q，WANG Y，YANG T，et al.You only look one-level feature[C]//Computer Vision and Pattern Recognition，2021：13039-13048.
[46] XU S，WANG X，LV W，et al.PP-YOLOE：an evolved version of YOLO[J].arXiv：2203.16250，2022.
[47] 赵鹏飞，谢林柏，彭力.融合注意力机制的深层次小目标检测算法[J].计算机科学与探索，2022，16（4）：927-937.
ZHAO P F，XIE L B，PENG L，et al.Deep small object detection algorithm integrating attention mechanism[J].Journal of Frontiers of Computer Science and Technology，2022，16（4）：927-937.
[48] 陈科圻，朱志亮，邓小明，等.多尺度目标检测的深度学习研究综述[J].软件学报，2021，32（4）：1201-1227.
CHEN K Q，ZHU Z L，DENG X M，et al.Deep learning for multi-scale object detection：a survey[J].Journal of Software，2021，32（4）：1201-1227.
[49] SIFRE L，MALLAT S.Rigid-motion scattering for texture classification[J].Computer Science，2014，3559（2014）：501-515.
[50] ZOPH B，LE P V.Neural architecture search with reinforcement learning[J].arXiv：1611.01578，2016.
[51] IANDOLA F N，HAN S，MOSKEWICZ M W，et al.SqueezeNet：AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size[J].arXiv：1602.07360，2016.
[52] HOWARD A G，ZHU M，CHEN B，et al.MobileNets：efficient convolutional neural networks for mobile vision applications[J].arXiv：1704.04861，2017.
[53] SANDLER M，HOWARD A，ZHU M，et al.MobileNet V2：inverted residuals and linear bottlenecks[C]//Computer Vision and Pattern Recognition，2018：4510-4520.
[54] ZHANG X，ZHOU X，LIN M，et al.ShuffleNet：an extremely efficient convolutional neural network for mobile devices[C]//Computer Vision and Pattern Recognition，2018：6848-6856.
[55] MA N，ZHANG X，ZHENG H T，et al.ShuffleNet V2：practical guidelines for efficient CNN architecture design[C]//European Conference on Computer Vision，2018：116-131.
[56] HE Y，LIN J，LIU Z，et al.AMC：AutoML for model compression and acceleration on mobile devices[C]//European Conference on Computer Vision，2018：784-800.
[57] HOWARD A，SANDLER M，CHEN B，et al.Searching for MobileNetV3[C]//International Conference on Computer Vision，2019：1314-1324.
[58] HU J，SHEN L，SUN G.Squeeze-and-excitation networks[C]//Computer Vision and Pattern Recognition，2018：7132-7141.
[59] TAN M，CHEN B，PANG R，et al.MnasNet：platform-aware neural architecture search for mobile[C]//Computer Vision and Pattern Recognition，2019：2820-2828.
[60] HAN K，WANG Y，TIAN Q，et al.GhostNet：more features from cheap operations[C]//Computer Vision and Pattern Recognition，2020：1580-1589.
[61] YU C，XIAO B，GAO C，et al.Lite-hrnet：a lightweight high-resolution network[C]//Computer Vision and Pattern Recognition，2021：10440-10450.
[62] MA H，XIA X，WANG X，et al.MoCoViT：mobile convolutional vision transformer[J].arXiv：2205.12635，2022.
[63] TUCKER L R.Some mathematical notes on three-mode factor analysis[J].Psychometrika，1966，31（3）：279-311.
[64] KLEMA V，LAUB A.The singular value decomposition：its computation and some applications[J].Transactions on Automatic Control，1980，25（2）：164-176.
[65] NOVIKOV A，PODOPRIKHIN D，OSOKIN A，et al.Tensorizing neural networks[C]//Natural Information Processing Canada，2015：442-450.
[66] WANG W，AGGARWAL V，AERON S.Efficient low rank tensor ring completion[C]//International Conference on Computer Vision，2017：5697-5705.
[67] DENTON E L，ZAREMBA W，BRUNA J.Exploiting linear structure within convolutional networks for efficient evaluation[C]//Neural Information Processing Systems，2014：1269-1277.
[68] GARIPOV T，PODOPRIKHIN D，NOVIKOV A，et al.Ultimate tensorization：compressing convolutional and FC layers alike[J].arXiv：1611.03214，2016.
[69] ZHAO Q，SUGIYAMA M，YUAN L，et al.Learning efficient tensor representations with ring-structured networks[C]//International Conference on Acoustics，Speech and Signal Processing，2019.
[70] CHIEN J T，BAO Y T.Tensor-factorized neural networks[J].Transactions on Neural Networks and Learning Systems，2017，29（5）：1998-2011.
[71] HUANG H，NI L，WANG K，et al.A highly parallel and energy efficient three-dimensional multilayer CMOS-RRAM accelerator for tensorized neural network[J].IEEE Transactions on Nanotechnology，2017，17（4）：645-656.
[72] CHEN Y，JIN X，KANG B，et al.Sharing residual units through collective tensor factorization to improve deep neural networks[C]//International Joint Conference on Artificial Intelligence，2018：635-641.
[73] LATHAUWER L D.Decompositions of a higher-order tensor in block terms-Part II：definitions and uniqueness[J].Siam Journal on Matrix Analysis and Applications，2008，30（3）：1022-1032.
[74] CAI H Q，CHAO Z，HUANG L，et al.Fast robust tensor principal component analysis via fiber cur decomposition[C]//International Conference on Computer Vision，2021：189-197.
[75] YIN M，SUI Y，LIAO S，et al.Towards efficient tensor decomposition-based DNN model compression with optimization framework[C]//Computer Vision and Pattern Recognition，2021：10674-10683.
[76] BOYD S，PARIKH N，CHU E，et al.Distributed optimization and statistical learning via the alternating direction method of multipliers[J].Foundations and Trends in Machine Learning，2010，3（1）：1-122.
[77] KHOUJA R，MATTEI P A，MOURRAIN B.Tensor decomposition for learning Gaussian mixtures from moments[J].arXiv：2106.00555，2021.
[78] HUANG H，NI L，WANG K，et al.A highly parallel and energy efficient three-dimensional multilayer CMOS-RRAM accelerator for tensorized neural network[J].Transactions on Nanotechnology，2017，17（4）：645-656.
[79] ZHONG Z，WEI F，LIN Z，et al.ADA-tucker：compressing deep neural networks via adaptive dimension adjustment tucker decomposition[J].Neural Networks，2019，110：104-115.
[80] 刘海洋，王志海，张志东.基于ReliefF剪枝的多标记分类算法[J].计算机学报，2019，42（3）：483-496.
LIU H Y，WANG Z H，ZHANG Z D.ReliefF based pruning model for multi-label classification[J].Chinese Journal of Computers，2019，42（3）：483-496.
[81] HANSON S J，PRATT L Y.Comparing biases for minimal network construction with back-propagation[C]//Neural Information Processing Systems，USA，1988.
[82] LECUN Y.Optimal brain damage[J].Neural Information Proceeding Systems，1990，279（2）：598-605.
[83] HASSIBI B，STORK D G.Second order derivatives for network pruning：optimal brain surgeon[C]//Neural Information Processing Systems.Morgan Kaufmann：ACM，1992：164-171.
[84] HAN S，POOL J，TRAN J，et al.Learning both weights and connections for efficient neural networks[J].Neural Information Processing Systems，2015，28：1135-1143.
[85] HAN S，MAO H.Deep compression：compressing deep neural networks with pruning，trained quantization and huffman coding[C]//Conference on Learning Representations，2016.
[86] LEBEDEV V，LEMPITSKY V.Fast ConvNets using group-wise brain damage[C]//Computer Vision and Pattern Recognition，2015：2554-2564.
[87] ZHOU H，ALVAREZ J M，PORIKLI F.Less is more：towards compact CNNs[C]//European Conference on Computer Vision，2016：662-677.
[88] HE Y，LIN J，LIU Z，et al.AMC：AutoML for model compression and acceleration on mobile devices[C]//European Conference on Computer Vision，2018：794-800.
[89] YU J，HUANG T.AutoSlim：towards one-shot architecture search for channel numbers[C]//Computer Vision and Pattern Recognition，2019.
[90] LIU N，MA X，XU Z，et al.AutoCompress：an automatic DNN structured pruning framework for ultra-high compression rates[C]//Conference on Artificial Intelligence，2020：4876-4883.
[91] MENG F，CHENG H.Pruning filter in filter[J].Advances in Neural Information Processing Systems，2020，33：17629-17640.
[92] WANG W，CHEN M，ZHAO S，et al.Accelerate CNNs from three dimensions：a comprehensive pruning framework[C]//International Conference on Machine Learning，2021：10717-10726.
[93] GAO S，HUANG F，CAI W，et al.Network pruning via performance maximization[C]//Computer Vision and Pattern Recognition，2021：9270-9280.
[94] MYUNG S，HUH I，JANG W，et al.PAC-Net：a model pruning approach to inductive transfer learning[C]//International Conference on Machine Learning，2022.
[95] LIU S，CHEN T，CHEN X，et al.The unreasonable effectiveness of random pruning：return of the most Naive baseline for sparse training[J].arXiv：2202.02643，2022.
[96] DENG L，LI G，HAN S，et al.Model compression and hardware acceleration for neural networks：a comprehensive survey[J].Proceedings of the IEEE，2020，108（4）：485-532.
[97] ZHU C，HAN S，MAO H，et al.Trained ternary quantization[J].arXiv：1612.01064，2016.
[98] WEN W，XU C，YAN F，et al.Terngrad：ternary gradients to reduce communication in distributed deep learning[C]//Neural Information Processing Systems，2017：1509-1519.
[99] LENG C，DOU Z，LI H，et al.Extremely low bit neural network：squeeze the last bit out with admm[C]//AAAI Conference on Artificial Intelligence，2018.
[100] MISHRA A，NURVITADHI E，COOK J J，et al.WRPN：wide reduced-precision networks[J].arXiv：1709.01134，2017.
[101] CAI Z，HE X，JIAN S，et al.Deep learning with low precision by half-wave Gaussian quantization[C]//Computer Vision and Pattern Recognition，2017：5918-5926.
[102] ZHOU S，WU Y，NI Z，et al.Dorefa-net：training low bitwidth convolutional neural networks with low bitwidth gradients[J].arXiv：1606.06160，2016.
[103] WU S，LI G，CHEN F，et al.Training and inference with integers in deep neural networks[J].arXiv：1802.04680，2018.
[104] KAPUR S，MISHRA A.Low precision RNNs：quantizing RNNS without losing accuracy[J].arXiv：1710.07706，2017.
[105] FIESLER E，CHOUDRY A，CAULFIELD H J.Weight discretization paradigm for optical neural networks[C]//International Society for Optics and Photonics，1990：164-173.
[106] GUPTA S，AGRAWAL A，GOPALAKRISHNAN K，et al.Deep learning with limited numerical precision[C]//International Conference on Machine Learning，2015：1737-1746.
[107] NAGEL M，AMJAD R A，VAN BAALEN M，et al.Up or down? adaptive rounding for post-training quantization[C]//International Conference on Machine Learning，2020：7197-7206.
[108] MIYASHITA D，LEE E H，MURMANN B.Convolutional neural networks using logarithmic data representation[J].arXiv：1603.01025，2016.
[109] GONG J，SHEN H，ZHANG G，et al.Highly efficient 8-bit low precision inference of convolutional neural networks with intelcaffe[C]//Proceedings of the 1st Reproducible Quality-Efficient Systems Tournament on Co-designing Pareto-efficient Deep Learning，2018.
[110] LIU X，YE M，ZHOU D，et al.Post-training quantization with multiple points：mixed precision without mixed precision[J].arXiv：2002.09049，2020.
[111] LIN X，ZHAO C，PAN W.Towards accurate binary convolutional neural network[C]//Neural Information Processing Systems，2017：30-38.
[112] CHOUKROUN Y，KRAVCHIK E.Low-bit quantization of neural networks for efficient inference[C]//International Conference on Computer Vision Workshop，2019：3009-3018.
[113] BANNER R，NAHSHAN Y，HOFFER E，et al.Post-training 4-bit quantization of convolution networks for rapid-deployment[J].arXiv：1810.05723，2018.
[114] FANG J，SHAFIEE A，AZIZ H，et al.Post-training piecewise linear quantization for deep neural networks[C]//European Conference on Computer Vision，2020：69-86.
[115] LEE J，KIM D，HAM B.Network quantization with element-wise gradient scaling[C]//Computer Vision and Pattern Recognition，2021：6448-6457.
[116] 张帆，黄赟，方子茁，等.卷积神经网络的损失最小训练后参数量化方法[J].通信学报，2022，43（4）：114-122.
ZHANG F，HUANG Y，FANG Z Z，et al.Lost-minimum post-training parameter quantization method for convolutional neural network[J].Journal on Communications，2022，43（4）：114-122.
[117] NAHSHAN Y，CHMIEL B，BASKIN C，et al.Loss aware post-training quantization[J].Machine Learning，2021，110（11）：3245-3262.
[118] CHOI J，WANG Z，VENKATARAMANI S，et al.Pact：parameterized clipping activation for quantized neural networks[J].arXiv：1805.06085，2018.
[119] GONG R，LIU X，JIANG S，et al.Differentiable soft quantization：bridging full-precision and low-bit neural networks[C]//International Conference on Computer Vision，2019：4852-4861.
[120] GUO C，QIU Y，LENG J，et al.SQuant：on-the-fly data-free quantization via diagonal hessian approximation[J].arXiv：2202.07471，2022.
[121] YANG J，SHEN X，XING J，et al.Quantization networks[C]//Computer Vision and Pattern Recognition，2019：7308-7316.
[122] ZHUO L，ZHANG B，YANG L，et al.Cogradient descent for bilinear optimization[C]//Computer Vision and Pattern Recognition，2020：7959-7967.
[123] ORNHAG M V，OLSSON C.A unified optimization framework for low-rank inducing penalties[C]//Computer Vision and Pattern Recognition，2020：8474-8483.
[124] 董文轩，梁宏涛，刘国柱，等.深度卷积应用于目标检测算法综述[J].计算机科学与探索，2022，16（5）：1025-1042.
DONG W X，LIANG H T，LIU G D，et al.Review of deep convolution applied to target detection algorithms[J].Journal of Frontiers of Computer Science and Technology，2022，16（5）：1025-1042.