面向预测的长短时神经网络记忆增强机制

doi:10.3778/j.issn.1002-8331.2101-0461

摘要/Abstract

摘要：

基于时序数据建模的长短时神经网络（LSTM）可用于预测类问题。现实场景中，LSTM预测精度往往与输入序列长度相关，有效的历史信息会被新输入的数据淹没。针对此问题，提出在LSTM节点中构建强化门实现对遗忘信息的提取，并与记忆信息按比例选取、融合、输入记忆单元，增加学习过程中的梯度传导能力，使网络对相对较远的信息保持敏感以提升记忆能力。实验采用工业故障数据，当序列长度超过100时，具有强化门机制的改进模型预测误差低于其他LSTM模型。预测精度的差距随序列增加而增大，当序列长度增至200时，改进模型的预测误差（RMSE/MAE）较原模型分别降低了26.98%与35.85%。

关键词: 长短时神经网络, 时间序列预测模型, 记忆增强机制, 深度学习

Abstract:

Based on time series data modeling, Long Short-Term Memory neural network（LSTM） can be used to solve prediction-oriented problems. In real scenarios, the prediction accuracy of LSTM is often related to the length of the input sequence, and valid historical information will be overwhelmed by the newly input data. To solve this problem, it is proposed to construct a reinforcement gate in the LSTM node to extract the forgotten information, and to select, merge, and enter the memory unit in proportion to the memory information to increase the gradient conduction ability in the learning process, so that the network can deal with relatively distant information, stay sensitive to improve memory ability. The experiment uses industrial fault data. When the sequence length exceeds 100, the prediction error of the improved model with enhanced gate mechanism is lower than other LSTM models. The gap in prediction accuracy increases with the increase of the sequence. When the sequence length increases to 200, the prediction error（RMSE/MAE） of the improved model is reduced by 26.98% and 35.85% respectively compared with the original model.

Key words: long short-term memory neural network, time series prediction model, memory enhaunce mechanism, deep learning

吴明慧，侯凌燕，王超. 面向预测的长短时神经网络记忆增强机制[J]. 计算机工程与应用, 2021, 57(21): 109-115.

WU Minghui, HOU Lingyan, WANG Chao. Improved Mechanism of Prediction-Oriented Long Short-Term Memory Neural Network[J]. Computer Engineering and Applications, 2021, 57(21): 109-115.

参考文献

[1] CAI X，ZHANG N，VENAYAGAMOORTHY G K，et al.Time series prediction with recurrent neural networks trained by a hybrid pso-ea algorithm[J].Neurocomputing，2007，70（13/15）：2342-2353.
[2] HOCHREITER S，SCHMIDHUBER J.Long short-term memory[J].Neural Computation，1997，9（8）：1735-1780.
[3] GERS F A，SCHMIDHUBER J，CUMMINS　F.Learning to forget：continual prediction with LSTM[C]//9th International Conference on Artificial Neural Networks，1999.
[4] MARTINETZ T M，BERKOVICH S G，SCHULTEN K J，et al.Neural-gas’network for vector quantization and its application to time-series prediction[J].IEEE Transactions on Neural Networks，1993，4（4）：558-569.
[5] BROCKWELL P J，DAVIS R A，CALDER M V.Introduction to time series and forecasting[M].Berlin：Springer，2002.
[6] TYRALIS H，PAPACHARALAMPOUS G，LANGOUSIS A.Super learning for daily streamflow forecasting：large-scale demonstration and comparison with multiple machine learning algorithms[J].arXiv：1909.04131，2019.
[7] 李洁，林永峰.基于多时间尺度RNN的时序数据预测[J].计算机应用与软件，2018，35（7）：33-37.
LI J，LIN Y F.Prediction of time series data based on multi-time scale RNN[J].Computer Applications and Software，2018，35（7）：33-37.
[8] 张洋，姬波，卢红星，等.基于LSTM神经网络的短期高压负荷电流预测方法[J].计算机科学，2019，46（6）：49-54.
ZHANG Y，JI B，LU H X，et al.Short-term high-voltage load current prediction method based on LSTM neural network[J].Computer Science，2019，46（6）：49-54.
[9] MOORE D H.Classification and regression trees，by Leo Breiman，Jerome H.Friedman，Richard A.Olshen，and Charles J.Stone.Brooks/Cole Publishing，Monterey，1984，358 pages[J].Cytometry Part A，1987，8（5）：534-535.
[10] ZHANG S，WANG Y，LIU M，et al.Data-based line trip fault prediction in power systems using LSTM networks and SVM[J].IEEE Access，2017，6：7675-7686.
[11] JIANG C，JIN P，KANG Y，et al.Fault diagnosis and prediction method of SPC for engine block based on LSTM neural network[C]//International Conference on Intelligent Robotics and Applications.Cham：Springer，2019：483-492.
[12] ZHENG H，DAI Y，ZHOU Y.Fault prediction of fan gearbox based on K-means clustering and LSTM[C]//IOP Conference Series：Materials Science and Engineering，2019.
[13] HUANG Z，XU W，YU K.Bidirectional LSTM-CRF models for sequence tagging[J].arXiv：1508.01991，2015.
[14] WIGINGTON C，STEWART S，DAVIS B，et al.Data augmentation for recognition of handwritten words and lines using a CNN-LSTM network[C]//2017 14th IAPR International Conference on Document Analysis and Recognition（ICDAR），2017：639-645.
[15] YUAN M，WU Y，LIN L.Fault diagnosis and remaining useful life estimation of aero engine using LSTM neural network[C]//2016 IEEE International Conference on Aircraft Utility Systems（AUS），2016：135-140.
[16] CHO K，COURVILLE A，BENGIO Y.Describing multimedia content using attention-based encoder-decoder networks[J].IEEE Transactions on Multimedia，2015，17（11）：1875-1886.
[17] CHOROWSKI J，BAHDANAU D，SERDYUK D，et al.Attention-based models for speech recognition[J].arXiv：1506.07503，2015.
[18] WANG Q，BU S，HE Z，et al.Toward the prediction level of situation awareness for electric power systems using CNN-LSTM Network[J].IEEE Transactions on Industrial Informatics，2021，17（10）：6951-6961.
[19] LI S，XIE G，REN J，et al.Urban PM2.5 concentration prediction via attention-based CNN-LSTM[J].Applied Sciences，2020，10（6）：1953.
[20] SHI X，CHEN Z，WANG H，et al.Convolutional LSTM network：a machine learning approach for precipitation nowcasting[M].[S.l.]：MIT Press，2015.
[21] SIAMI-NAMINI S，TAVAKOLI N，NAMIN A S.The performance of LSTM and BiLSTM in forecasting time series[C]//2019 IEEE International Conference on Big Data，2019：3285-3292.
[22] DING X，FENG L，ZOU Y，et al.Deep learning aided spectrum prediction for satellite communication systems[J].IEEE Transactions on Vehicular Technology，2020，69（12）：16314-16319.
[23] WANG M，CHENG J，ZHAI H.Life prediction for machinery components based on CNN-BiLSTM network and attention model[C]//2020 IEEE 5th Information Technology and Mechatronics Engineering Conference（ITOEC），Chongqing，China，2020：851-855.
[24] JIANG Y，LYU Y，WANG Y，et al.Fusion network combined with bidirectional LSTM network and multiscale CNN for remaining useful life estimation[C]//2020 12th International Conference on Advanced Computational Intelligence（ICACI），Dali，China，2020：620-627.
[25] MA F L，CHITTA R，ZHOU J，et al.Dipole：diagnosis prediction in healthcare via attention-based bidirectional recurrent neural networks[J].arXiv：1706.05764，2017.
[26] FAN G，DIAO X，YU H，et al.Software defect prediction via attention-based recurrent neural network[J].Scientific Programming，2019.
[27] ZHENG L，CHEN J，CHEN F，et al.Rotating machinery fault prediction method based on Bi-LSTM and attention mechanism[C]//2019 IEEE International Conference on Energy Internet（ICEI），2019：53-58.
[28] 熊良鹏，刘晓丽.从属解析函数族的有限阶哈达玛乘积[J].数学物理学报，2014，34（1）：152-158.
XIONG L P，LIU X L.Finite-order hadamard product of a family of associated analytic functions[J].Acta Mathematica Sinica，2014，34（1）：152-158.