关键词: 机器学习, 优化, 小批量, 减数更新, 随机方差缩减梯度法（SVRG）

Abstract:

Machine learning problems are often converted to solving an object function problem. After SGD（Stochastic Gradient Descent）, SVRG（Stochastic Variance Reduction Gradient） has become the mainstream algorithm to optimize the parameters of the objective function. Because it can achieve linear convergence without the influence of variance, it has been widely studied. It leads to the emergence of new variance reduction algorithms such as SAGA（Stochastic Average Gradient Average） and SCSG（Stochastically Controlled Stochastic Gradient）. They have problems with excessive memory consumption and slow iteration. In order to realize the purpose of low cost storage and fast iteration, a novel variance reduction algorithm BSUG（Batch Subtraction Update Gradient） based on SVRG is designed. The improvement is that the average gradient is calculated by using small batch samples instead of all samples, and the average gradient is updated by subtraction. In each iteration, a batch of small samples are randomly selected to calculate the average gradient, and the average gradient is updated by discarding the previous model gradient during the internal iteration. Memory storage and iteration times can be reduced by appropriately reducing batch size [B]. The convergence of the algorithm is analyzed theoretically and implemented based on Python. The effectiveness of the BSUG algorithm is proved by making comparisons with the algorithms of Mini-Batch SGD, AdaGrad, RMSProp, SVRG and SCSG, and the stability of the algorithm is proved by exploring the superparameters.

Key words: machine learning, optimization, mini batch, reduction update, Stochastic Variance Reduction Gradient（SVRG）