T-Transformer Model for Predicting Tensor Time Series

doi:10.3778/j.issn.1002-8331.2211-0168

Abstract

Abstract: Tensor time series shows the co-evolution characteristics of multiple time series with time changes, and tensor time series prediction has become an important problem. Aiming at the high-dimensional characteristics of tensor time series, T-Transformer, a tensor time series prediction method based on Transformer, is proposed, which integrates tensor operation and Transformer into a unified framework. The method first uses tensors to represent higher-order time series, uses tensor slices and vectorization to convert tensor time series into vectors, and then inputs the vectors into Transformer model after coding, and finally obtains the predicted value of tensor time series. Experiments on three open data sets show that the proposed method achieves better prediction results than the benchmark method.

Key words: tensor time series, tensor vectorization, attention, Transformer model, time series prediction

摘要： 张量时间序列呈现了多个时间序列随着时间变化协同演化的特征，张量时间序列预测已成为一个重要的问题。针对张量时间序列的高维特征，提出了基于Transformer的张量时间序列预测方法T-Transformer，将张量运算和Transformer共同集成到一个统一的框架中。该方法用张量表示的高阶时间序列，运用张量的切片和向量化，将张量时间序列转换成向量，将该向量经过编码后输入到Transformer模型，得到张量时间序列预测值。在三个公开数据集上进行的实验表明，与基准方法相比较，所提出的方法取得了更好的预测效果。

关键词: 张量时间序列, 张量向量化, 注意力机制, Transformer模型, 时间序列预测

LI Wen, CHEN Jiawei, LIU Ruixue, HOU Yuguo, DU Shouguo. T-Transformer Model for Predicting Tensor Time Series[J]. Computer Engineering and Applications, 2023, 59(11): 57-62.

李文, 陈佳伟, 刘瑞雪, 侯玉国, 杜守国. 张量时间序列预测T-Transformer模型[J]. 计算机工程与应用, 2023, 59(11): 57-62.

References

[1] JING P，SU Y，JIN X，et al.High-order temporal correlation model learning for time-series prediction[J].IEEE Transactions on Cybernetics，2018，49（6）：2385-2397.
[2] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems，2017.
[3] WEN Q，ZHOU T，ZHANG C，et al.Transformers in time series：a survey[J].arXiv：2202.07125，2022.
[4] 李文，邓升，段妍，等.时间序列预测与深度学习：文献综述与应用实例[J].计算机应用与软件，2020（10）：64-70.
LI Wen，DENG Sheng，DUAN Yan，et al.Time series forecasting and deep learning：literature review and empirical example[J].Computer Application and Software，2020（10）：64-70.
[5] 朱刚，李文，杜守国，等.基于深度学习模型DeepAR的时间序列预测及应用实例[J].电子商务，2020（7）：83-86.
ZHU Gang，LI Wen，DU Shouguo，et al.Time series forecasting and empirical example based on deep learning model DeepAR[J].E-Business，2020（7）：83-86.
[6] ROGERS M，LI L，RUSSELL S J.Multilinear dynamical systems for tensor time series[C]//Advances in Neural Information Processing Systems，2013.
[7] SHI Q，YIN J，CAI J，et al.Block Hankel tensor ARIMA for multiple short time series forecasting[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2020：5758-5766.
[8] LIU J，MUSIALSKI P，WONKA P，et al.Tensor completion for estimating missing values in visual data[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2012，35（1）：208-220.
[9] CHEN X，SUN L.Low-rank autoregressive tensor completion for multivariate time series forecasting[J].arXiv：2006.10436，2020.
[10] JING B，TONG H，ZHU Y.Network of tensor time series[C]//Proceedings of the Web Conference 2021，2021：2425-2437.
[11] DU H，DU S，LI W.Probabilistic time series forecasting with deep non‐linear state space models[J].CAAI Transactions on Intelligence Technology，2023，8（1）：3-13.
[12] YU H F，RAO N，DHILLON I S.Temporal regularized matrix factorization for high-dimensional time series prediction[C]//Advances in Neural Information Processing Systems，2016.
[13] LI S，JIN X，XUAN Y，et al.Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting[C]//Advances in Neural Information Processing Systems，2019.
[14] ZHOU H，ZHANG S，PENG J，et al.Informer：beyond efficient transformer for long sequence time-series forecasting[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2021：11106-11115.
[15] WU H，XU J，WANG J，et al.AutoFormer：decomposition transformers with auto-correlation for long-term series forecasting[C]//Advances in Neural Information Processing Systems，2021：22419-22430.
[16] DOSOVITSKIY A，BEYER L，KOLESNIKOV A，et al.An image is worth 16x16 words：transformers for image recognition at scale[J].arXiv：2010.11929，2020.
[17] 张贤达.矩阵分析与应用[M].2版.北京：清华大学出版社，2013.
ZHANG Xianda.Matrix analysis and applications[M].
2nd ed.Beijing：Tsinghua University Press，2013.
[18] ZERVEAS G，JAYARAMAN S，PATEL D，et al.A transformer-based framework for multivariate time series representation learning[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining，2021：2114-2124.
[19] KATHAROPOULOS A，VYAS A，PAPPAS N，et al.Transformers are RNNs：fast autoregressive transformers with linear attention[C]//International Conference on Machine Learning，2020：5156-5165.
[20] CHOROMANSKI K，LIKHOSHERSTOV V，DOHAN D，et al.Rethinking attention with performers[J].arXiv：2009. 14794，2020.
[21] CHEN X，SUN L.Bayesian temporal factorization for multidimensional time series prediction[J].arXiv：1910. 06366，2019.