Early Event Detection Based on Multivariate Spatial-Temporal Fusion

doi:10.3778/j.issn.1002-8331.2206-0429

Abstract

Abstract: Aiming at the problem that existing early event detection algorithms do not fully exploit the spatial-temporal relationships of multivariate time series, an early event detection method based on spatial-temporal fusion is proposed. Time convolutional network（TCN） is adopted for sequence prediction by learning the temporal patterns from the input temporal events described by multivariate time series. Both the original and predicted time series are concatenated along the temporal dimension and fed into a novel multivariate spatial-temporal neural network（MSTNN） for classification. Experimental results for recognizing partial events in multiple scenarios, including action classification, handwritten character recognition, and spoken digit recognition, demonstrate the benefits of the proposed method compared to existing approaches, which achieves an average accuracy of 93.2% when 2/3 of the temporal events have been observed. The performance on the handwritten character recognition dataset is comparable with the case when the complete event is observed.

Key words: multivariate time series, early event detection, spatial-temporal fusion, sequence prediction, deep neural network

摘要： 针对事件早期识别算法对多维变量序列时空依赖关系挖掘不足的问题，提出了一种时空融合的事件早期识别算法。将多变量时间序列规整为矩阵化数据输入时间卷积网络，通过学习时间序列的时域特征，对时间序列进行预测。将包含已知值和预测值在内的多变量时间序列送入融合时空信息的深度神经网络进行识别。在动作识别、手写字符识别、口语数字识别等多个任务上的实验结果表明，所提算法的性能优于主流的算法。当事件仅发生2/3时，能以平均93.2%的准确率识别多类别事件，在手写字符识别数据集上的早期识别准确率甚至与完整事件被观测到时的识别准确率相当。

关键词: 多变量时间序列, 事件早期识别, 时空信息融合, 序列预测, 深度神经网络

LIU Xuying, LU Wenda, WANG Jian, WANG Xue, WANG Qing. Early Event Detection Based on Multivariate Spatial-Temporal Fusion[J]. Computer Engineering and Applications, 2023, 59(17): 116-122.

刘绪颖, 卢文达, 王剑, 王雪, 王庆. 融合多变量序列时空信息的事件早期识别方法[J]. 计算机工程与应用, 2023, 59(17): 116-122.

References

[1] 耿小航，彭冬亮，张震，等.基于图卷积网络的重大事件趋势预测[J].计算机工程与设计，2021，42（11）：3196-3203.
GENG X H，PENG D L，ZHANG Z，et al.Major event trend prediction based on graph convolution network[J].Computer Engineering and Design，2021，42（11）：3196-3203.
[2] 刘伟，张晓蕾，孙士保，等.基于随机失活的循环神经网络交通事件预测[J].计算机仿真，2021，38（6）：78-82.
LIU W，ZHANG X L，SUN S B，et al.The forecast of the traffic event of the recurrent neural network based on the random deactivation[J].Computer Simulation，2021，38（6）：78-82.
[3] HOAI M，DE LA TORRE F.Max-margin early event detectors[C]//Proceedings of CVPR，2012：2863-2870.
[4] 李蓉蓉，戴永.基于LSTM和时间序列分析法的短期风速预测[J].计算机仿真，2020，37（3）：393-398.
LI R R，DAI Y.Short-term wind speed prediction based on short and long time memory network and time series analysis method[J].Computer Simulation，2020，37（3）：393-398.
[5] 李霞.基于连续密度隐马尔可夫的时间序列分类算法[J].计算机仿真，2021，38（1）：291-294.
LI X.Time series classification algorithm based on continuous density hidden markov[J].Computer Simulation，2021，38（1）：291-294.
[6] MOHAMED M A，SHOUKRY A，ISMAIL M A.EKNN：ensemble classifier incorporating connectivity and density into kNN with application to cancer diagnosis[J].Artificial Intelligence in Medicine，2021，111：101985.
[7] 杨艳超，任秀丽.基于时间序列相似度的无线传感网故障检测算法[J].计算机应用研究，2021，38（8）：2401-2406.
YANG Y C，REN X L.Fault detection algorithm based on time series similarity in WSN[J].Application Research of Computers，2021，38（8）：2401-2406.
[8] 张雅雯，王志海，刘海洋，等.基于多尺度残差FCN的时间序列分类算法[J].软件学报，2022，33（2）：555-570.
ZHANG Y W，WANG Z H，LIU H Y，et al.Time series classification algorithm based on multiscale residual full convolutional neural network[J].Journal of Software，2022，33（2）：555-570.
[9] 滕伟，黄乙珂，吴仕明，等.基于XGBoost与LSTM的风力发电机绕组温度预测[J].中国电力，2021，54（6）：95-103.
TENG W，HUANG Y K，WU S M，et al.Wind-driven generator winding temperature prediction based on XGBoost and LSTM[J].Electric Power，2021，54（6）：95-103.
[10] KARIM F，MAJUMDAR S，DARABI H，et al.LSTM fully convolutional networks for time series classification[J].IEEE Access，2018，6：1662-1669.
[11] 赵彦明.基于时空相关性的LSTM算法及PM2.5浓度预测应用[J].计算机应用与软件，2021，38（6）：249-255.
ZHAO Y M.LSTM algorithm based on spatio-temporal correlation and its application of PM2.5 concentration prediction[J].Computer Applications and Software，2021，38（6）：249-255.
[12] SHOREWALA V.Early detection of coronary heart disease using ensemble techniques[J].Informatics in Medicine Unlocked，2021，26：100655.
[13] FAGERSTROM J，BANG M，WIHELMS D，et al.LiSep LSTM：a machine learning algorithm for early detection of septic shock[J].Scientific Reports，2019，9：15132.
[14] BAI L，YAO L，LI C，et al.Adaptive graph convolutional recurrent network for traffic forecasting[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems（NeurIPS），2020：17804-17815.
[15] WU Z，PAN S，LONG G，et al.Connecting the dots multivariate time series forecasting with graph neural networks[C]//Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining，2022：753-763.
[16] LEA C，FLYNN M D，VIDAL R，et al.Temporal convolutional networks for action segmentation and detection[C]//Proceedings of CVPR，2017：156-165.
[17] SZEGEDY C，IOFFE S，VANHOUCKE V，et al.Inception-v4，inception-ResNet and the impact of residual connections on learning[C]//Thirty-First AAAI Conference on Artificial Intelligence，2017：4278-4284.
[18] KARIM F，MAJUMDAR S，DARABI H，et al.Multivariate LSTM-FCNs for time series classification[J].Neural Networks，2019，116：237-245.
[19] VELICKOVIC P，CUCURULL G，CASANOVA A，et al.Graph attention networks[C]//Proceedings of ICLR，2018.
[20] BAHDANAU D，CHO K，BENGIO Y.Neural machine translation by jointly learning to align and translate[C]//Proceedings of ICLR，2015.
[21] BIANCHI F M，SCARDAPANE S， LOKSE S，et al.Reservoir computing approaches for representation and classification of multivariate time series[J].IEEE Transactions on Neural Networks and Learning Systems，2021，32（5）：2169-2179.
[22] ZHAO B，LU H，CHEN S，et al.Convolutional neural networks for time series classification[J].Journal of Systems Engineering and Electronics，2017，28（1）：162-169.
[23] WANG Z，YAN W，OATES T.Time series classification from scratch with deep neural networks：a strong baseline[C]//International Joint Conference on Neural Networks，2017：1578-1585.
[24] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of CVPR，2016：770-778.