Convolutional Neural Network Optimization Method Based on Momentum Fractional Order Gradient Descent Algorithm

doi:10.3778/j.issn.1002-8331.2103-0118

Abstract

Abstract: Aiming at the slow convergence speed of convolutional neural network trained by traditional gradient descent algorithm, a momentum fractional order gradient descent algorithm is proposed. The definition of fractional order calculus is introduced, and according to the problem description, the momentum thought of integer order gradient descent algorithm is applied to the fractional order gradient descent algorithm through algorithm derivation, then the momentum fractional order gradient descent algorithm is designed. The convergence of the algorithm is verified by using the test function, and the influence of different fractional orders and momentum coefficients on the algorithm are analyzed. The momentum fractional order gradient descent algorithm is used to compare with the traditional gradient descent algorithm and momentum gradient descent algorithm on three datasets. The experimental data shows that the momentum fractional order gradient descent algorithm can greatly improve the convergence speed of convolutional neural network on datasets with different complexity levels, while ensuring high classification accuracy, and saving a lot of training time.卷积神经网络；分数阶；梯度下降；动量

Key words: convolutional neural network, fractional order, gradient descent, momentum

摘要： 针对采用传统梯度下降算法训练卷积神经网络收敛速度慢的问题，提出了动量分数阶梯度下降算法。介绍了分数阶微积分的定义，并依据问题描述，通过算法推导，将整数阶梯度下降算法中的动量思想应用到分数阶梯度下降算法中，设计出动量分数阶梯度下降算法。使用测试函数验证算法的收敛性，并分析不同分数阶阶次和动量项系数对算法收敛性的影响。在三个数据集上使用动量分数阶梯度下降算法与传统梯度下降算法、动量梯度下降算法作对比实验，实验数据表明，动量分数阶梯度下降算法可以在不同复杂程度的数据集上，在保证较高分类准确率的前提下，极大提高卷积神经网络的收敛速度，为训练卷积神经网络节省大量时间成本。

关键词: 卷积神经网络, 分数阶, 梯度下降, 动量

GUO Mingxiao, WANG Hongwei, WANG Jia, LI Haozhe, YANG Shiqi. Convolutional Neural Network Optimization Method Based on Momentum Fractional Order Gradient Descent Algorithm[J]. Computer Engineering and Applications, 2022, 58(6): 80-87.

郭明霄, 王宏伟, 王佳, 李昊哲, 杨仕旗. 基于动量分数阶梯度的卷积神经网络优化方法[J]. 计算机工程与应用, 2022, 58(6): 80-87.

References

[1] LECUN Y，BOTTOU L，BENGIO Y，et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE，1998，86（11）：2278-2324.
[2] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems.NewYork：Curran Associates Inc，2012：1097-1105.
[3] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[C]//International Conference on Learning Representations（ICLR），San Diego，USA，2015.
[4] HE K M，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Las Vegas：IEEE，2016：770-778.
[5] SZEGEDY C，LIU W，JIA Y Q，et al.Going deeper with convolutions[C]//IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Boston：IEEE，2015：1-9.
[6] LONG J，SHELHAMER E.DARRELL T.Fully convolutional networks for semantic segmentation[C]//IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Boston：IEEE，2015：3431-3440.
[7] HUANG G，LIU Z，MAATEN L V D，et al.Densely connected convolutional networks[C]//IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Honolulu：IEEE，2017：2261-2269.
[8] 赫孝良，葛照强.最优化与最优控制[M].西安：西安交通大学出版社，2015：31-32.
HE X L，GE Z Q.Optimization and optimal control[M].2nd ed.Xi’an：Xi’an Jiaotong University Press，2015：31-32.
[9] QIAN N.On the momentum term in gradient descent learning algorithms[J].Neural Networks，1999，12（1）：145-151.
[10] DUCHI J，HAZAN E，SINGER Y.Adaptive subgradient methods for online learning and stochastic optimization[J].Journal of Machine Learning Research，2011，12（61）：2121-2159.
[11] ZEILER M D.ADADELTA：an adaptive learning rate method[J].arXiv：1212.5701，2012.
[12] KINGMA D P，LEI BA J.Adam：a method for stochastic optimization[J].arXiv：1412.6980，2014.
[13] ZU?IGA AGUILAR C J，GóMEZ-AGUILAR J F，ALVARADO-MARTíNEZ V M，et al.Fractional order neural networks for system identification[J].Chaos，Solitons & Fractals，2020，130.
[14] YANG Q，ZHANG Y Z，ZHAO T B，et al.Single image super-resolution using self-optimizing mask via fractional-order gradient interpolation and reconstruction[J].ISA Transactions，2018，82：163-171.
[15] SAADIA A，RASHDI A.Incorporating fractional calculus in echo-cardiographic image denoising[J].Computers and Electrical Engineering，2018，67：134-144.
[16] XU C，CHENG S J，WANG K L，et al.A fractional-order model for liquid metal batteries[J].Energy Procedia，2019，158：4690-4695.
[17] REN H P，WANG X，FAN J T，et al.Fractional order sliding mode control of a pneumatic position servo system[J].Journal of the Franklin Institute，2019，356（12）：6160-6174.
[18] AMéLIE C，CLOVIS F，COPOT C，et al.Fractional-order PID design：towards transition from state-of-art to state-of-use[J].ISA Transactions，2019，84：178-186.
[19] WEI Y H，KANG Y，YIN W D，et al.Generalization of the gradient method with fractional order gradient direction[J].Journal of the Franklin Institute，2020，357（4）：2514-2532.
[20] CHEN Y Q，GAO Q，WEI Y H，et al.Study on fractional order gradient methods[J].Applied Mathematics and Computation，2017，314：310-321.
[21] CHENG S S，WEI Y H，CHEN Y Q，et al.A novel fractional order normalized LMS algorithm with direction optimization[J].IFAC-PapersOnLine，2016，49（9）：180-185.
[22] WANG J，WEN Y Q，GOU Y D，et al.Fractional-order gradient descent learning of BP neural networks with Caputo derivative[J].Neural Networks，2017，89：19-30.
[23] SHENG D，WEI Y H，CHEN Y Q，et al.Convolutional neural networks with fractional order gradient method[J].Neurocomputing，2020，408：42-50.
[24] 薛定宇.分数阶微积分学与分数阶控制[M].北京：科学出版社，2018：31.
XUE D Y.Fractional calculus and fractional control[M].Beijing：Science Press，2018：31.