Abstract:

In this paper, the AlexNet convolutional neural network is improved and optimized by three aspects：Normali-
zation, optimizer and activation function. Firstly, there is no learnable parameter for LRN（Local Response Normalization）. It is proposed to use WN（Weight Normalization） instead of LRN, and put WN after all the pooling layer. The accuracy of the training of AlexNet model is improved. Secondly, the effects of Adam, RMSProp and Momentum on the training of AlexNet model under different learning rates are compared and analyzed, and the corresponding optimization interval of learning rate is obtained. It improves the accuracy of AlexNet’s learning rate interval selection in Optimizer. Finally, for the partial weight of the ReLU activation function in AlexNet can not be updated and the gradient explosion problem, it proposes a fusion segmentation function algorithm of ReLU6 and Swish, which improves the AlexNet model training convergence speed and accuracy while also alleviating the occurrence of overfitting.

Key words: AlexNet, Convolutional Neural Network（CNN）, Normalization, optimizer, activation function

摘要：

通过对Normalization、优化器、激活函数三方面对AlexNet卷积神经网络进行了改进及优化。针对LRN（Local Response Normalization）不存在可学习参数，提出了用WN（Weight Normalization）来代替LRN，同时将WN置于所有池化层（Pooling layer）之后，提高了AlexNet模型训练的准确率；通过对比分析Adam、RMSProp、Momentum三种优化器在不同学习率（Learning rate）下对AlexNet模型训练的影响，并得出了相应的学习率的优化区间，提高了AlexNet在Optimizer的学习率区间选择上的准确性；针对AlexNet中ReLU激活函数存在的部分权重无法更新以及梯度爆炸问题，提出了ReLU6与Swish的融合分段函数算法，提升了AlexNet模型训练收敛速度以及准确率的同时也缓解了过拟合现象的发生。

关键词: AlexNet, 卷积神经网络（CNN）, Normalization, 优化器, 激活函数