AdaSVRG: Accelerating SVRG by Adaptive Learning Rate

doi:10.3778/j.issn.1002-8331.2104-0264

Abstract

Abstract: In deep learning tasks, stochastic variance-reduced gradient methods have better stability and higher computational efficiency by reducing variance of the stochastic gradients. However, a constant learning rate is employed in the learning process, which may reduce their computational efficiency of these variance-reduced methods. Based on the stochastic variance-reduced method, a new variance-reduced method is developed by utilizing the idea of momentum acceleration and the strategy of weighted average, referred to as adaptive stochastic variance-reduced gradient method (AdaSVRG). The effectiveness of the proposed AdaSVRG is verified based on MNIST and CIFAR-10 data sets. Experimental results show that AdaSVRG outperforms the stochastic variance-reduced gradient method and the stochastic gradient descent method in terms of convergence speed and stability.

Key words: deep learning, stochastic variance-reduced gradient method, adaptive learning rate, momentum

摘要： 在深度学习任务中，随机方差衰减梯度法通过降低随机梯度方差，因此，其具有较好的稳定性和较高的计算效率。然而，这类方法在学习过程中均使用恒定的学习率，降低了随机方差衰减梯度法的计算效率。基于随机方差衰减梯度法，借鉴动量加速思想并对梯度估计采取加权平均策略，对学习率利用历史梯度信息进行自动调整，提出了自适应随机方差衰减梯度法。基于MNIST和CIFAR-10数据集，验证提出的自适应随机方差衰减梯度法的有效性。实验结果表明，自适应随机方差衰减梯度法在收敛速度和稳定性方面优于随机方差衰减梯度法和随机梯度下降法。

关键词: 深度学习, 随机方差衰减梯度法, 自适应学习率, 动量法

JI Meng, HE Qinglong. AdaSVRG: Accelerating SVRG by Adaptive Learning Rate[J]. Computer Engineering and Applications, 2022, 58(9): 83-90.

吉梦, 何清龙. AdaSVRG：自适应学习率加速SVRG[J]. 计算机工程与应用, 2022, 58(9): 83-90.

References

[1] DENG J，DONG W，SOCHER R，et al.Imagenet：a large-scale hierarchical image database[C]//Proceedings IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2009：1-8.
[2] KRIZHEVSKY A，SUTSKEVER I，GEOFFREY E H.Imagenet classification with deep convolutional neural networks[J].Communications of the ACM，2012，60（6）：1097-1105.
[3] HUANG G B，MARWAN M，TAMARA B，et al.Labeled faces in the wild：a database for studying face recognition in unconstrained environments[D].University of Massachusetts，Amherst，2007：1-13.
[4] NEERAJ K，ALEXANDER C B，PETER N B，et al.Attribute and simile classifiers for face verification[C]//IEEE International Conference on Computer Vision，2009：1-8.
[5] DONG C，XU D C，FANG W，et al.Blessing of dimensionality：high dimensional feature and its efficient compression for face verification[C]//IEEE International Conference on Computer Vision and Pattern Recognition，2013：3025-3031.
[6] SUN Y，WANG X G，TANG X O.Deeply learned face representations are sparse，selective，and robust[C]//IEEE International Conference on Computer Vision and Pattern Recognition，2015：2892-2899.
[7] YANN L C，LéON B，BENGIO Y，et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE，1998，86（11）：2278-2324.
[8] SEBASTIAN R.An overview of gradient descent optimization algorithms[J].arXiv：1609.04747，2016.
[9] CAUCHY A L.Methode generale pour la resolution des systemes d’equations simultanees[J].Comptes rendus des seances de l’Academie des sciences de Paris，2000，1847（25）：536-538.
[10] NESTEROV Y.Introductory lectures on convex optimization：a basic course[M].[S.l.]：Kluwer Academic Publishers，2004.
[11] ROUX N L，SCHMIDT M，BACH F.A stochastic gradient method with an exponential convergence rate for finite training sets[C]//26th Annual Conference on Neural Information Processing Systems，2012.
[12] LéON B，YANN L C.Large scale online learning[J].Advances in Neural Information Processing Systems，2003：217-224.
[13] LéON B，FRANK E C，JORGE N.Optimization methods for large-scale machine learning[J].Siam Review，2018，60（2）：223-311.
[14] ROBBINS H，MONRO S.A stochastic approximation method[J].The Annals of Mathematical Statistics，1951，22（3）：400-407.
[15] ZHANG T.Solving large scale linear prediction problems using stochastic gradient descent algorithms[C]//Proceedings of the Twenty-First International Conference on Machine Learning，2004.
[16] MENG X R，BRADLEY J K，YAVUZ B.MLlib：machine learning in Apache Spark[J].Computer Science，2016，17（1）：1235-1241.
[17] SCHMIDT M，ROUX N L，BACH F.Minimizing finite sums with the stochastic average gradient[J].arXiv：1309. 2388，2013.
[18] SCHWARTZ S S，ZHANG T.Stochastic dual coordi-nate ascent methods for regularized loss minimization[J].Journal of Machine Learning Research，2013，14：567-599.
[19] MAIRAL J.Optimization with first-order surrogate functions[C]//International Conference on Machine Learning，2013.
[20] DEFAZIO A，BACH F，JULIEN S L.Saga：a fast incremental gradient method with support for non-strongly convex composite objectives[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems，2014.
[21] 王建飞，亢良伊，刘杰，等.分布式随机方差消减梯度下降算法topkSVRG*[J].计算机科学与探索，2018，12（7）：1047-1054.
WANG J F，KANG L Y，LIU J.Distributed stochastic variance reduction gradient descent algorithm topk-SVRG*[J].Journal of Frontiers of Computer Science and Technology，2018，12（7）：1047-1054.
[22] 宋杰，朱勇，许冰.等.批量减数更新方差缩减梯度下降算法BSUG[J].计算机工程与应用，2020，56（22）：117-123.
SONG J，ZHU Y，XU B.Batch subtraction update variance reduction gradient descent algorithm BSUG[J].Computer Engineering and Applications，2020，56（22）：117-123.
[23] JOHNSON R，ZHANG T.Accelerating stochastic gradient descent using predictive variance reduction[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems，2013.
[24] HARIKANDEH R B，AHMED M O，VIRANI A，et al.Stop wasting my gradients：practical SVRG[J].arXiv：1511.01942，2015.
[25] 邱锡鹏.神经网络与深度学习[M].北京：机械工业出版社，2020：160-169.
QIU X P.Neural networks and deep learning[M].Beijing：China Machine Press，2020：160-169.
[26] KONECNY J，RICHTSRIK P.Semi-stochastic gradient descent methods[J].arXiv：1312.1666，2013.
[27] TIELEMAN T，HINTON G.Lecture 6.5-RMSProp：divide the gradient by a running average of its recent magnitude[J].COURSERA：Neural Networks for Machine Learning，2012，4：26-30.
[28] MATTHEW D Z.ADADELTA：an adaptive learning rate method[J].arXiv：1212.5701，2012.
[29] DIEDERIK P K，JIMMY L B.ADAM：a method for stochastic optimization[C]//International Conference on Learning Representations，2015：1-13.