Traffic Flow Forecasting Model for Improved Spatio-Temporal Transformer

doi:10.3778/j.issn.1002-8331.2203-0290

Abstract

Abstract: To address the low performance problem of traffic flow prediction model based on spatio-temporal Transformer model, an improved spatio-temporal Transformer model（ISTTM） based on encoder-decoder is proposed. The encoder encodes the historical traffic features and the decoder predicts the future sequences. Firstly, the encoder combines spatial sparse self-attentiveness and temporal hierarchical diffusion convolution to capture the dynamic spatial correlation and local spatial features of traffic flows, and then uses temporal self-attentiveness to model the nonlinear temporal correlation. Then, the decoder mines the spatio-temporal features of the input sequences similarly to the encoder. Finally, based on the spatio-temporal features extracted by the encoder-decoder, the impact of historical traffic observations on future forecasts is simulated using double cross-attention, modeling the direct relationship between each historical time step and each future time step and the impact on the whole future time period, and the final representation of the future traffic flow is output. To confirm the effectiveness of ISTTM, experiments are executed on two real-world large-scale datasets, METR-LA and NE-BJ, and the ISTTM results outperform the six state-of-the-art baselines.

Key words: traffic forecasting, spatio-temporal feature, sparse self-attention, diffusion convolution

摘要： 针对基于时空Transformer模型的交通流量预测模型性能不高的问题，提出了一种基于编解码器的改进的时空Transformer模型（improved spatio-temporal Transformer model，ISTTM）。编码器对历史流量特征进行编码，解码器预测未来序列。编码器将空间稀疏自注意力和时间层次扩散卷积相结合，捕捉交通流量的动态空间相关性和局部空间特征，再利用时间自注意力建模非线性时间相关性；解码器与编码器类似地挖掘出输入序列的时空特征。基于编解码器提取的时空特征，采用双重交叉注意力模拟历史交通观测对未来预测的影响，建模每个历史时间步和每个未来时间步的直接关系以及对整个未来时间段的影响，并输出未来交通流量的最终表示。为了证实ISTTM的有效性，在METR-LA和NE-BJ两个真实世界的大规模数据集上进行实验，ISTTM结果优于6个先进的基线。

关键词: 交通流量预测, 时空特征, 稀疏自注意力, 扩散卷积

GAO Rong, WAN Yiliang, SHAO Xiongkai, Wu Xinyun. Traffic Flow Forecasting Model for Improved Spatio-Temporal Transformer[J]. Computer Engineering and Applications, 2023, 59(7): 250-260.

高榕, 万以亮, 邵雄凯, 吴歆韵. 面向改进的时空Transformer的交通流量预测模型[J]. 计算机工程与应用, 2023, 59(7): 250-260.

References

[1] LI Y，YU R，SHAHABI C，et al.Diffusion convolutional recurrent neural network：data-driven traffic forecasting[C]//Proceedings of 6th International Conference on Learning Representations，Vancouver，BC，Canada，2018.
[2] WU Z H，PAN S R，LONG G D，et al.Graph WaveNet for deep spatial-temporal graph modeling[C]//Proceedings of the 28th International Joint Conference on Artificial Intelligence，2019：1907-1913.
[3] PARK C，LEE C，BAHNG H，et al.ST-GRAT：a novel spatio-temporal graph attention networks for accurately forecasting dynamically changing road speed[C]//Proceedings of the 29th ACM International Conference on Information and Knowledge Management，2020.
[4] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Proceedings of the 31st Advances in Neural Information Processing Systems，2017：5998-6008.
[5] ZHENG C P，FAN X L，WANG C，et al.GMAN：a graph multi-attention network for traffic prediction[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence，2020：1234-1241.
[6] ZENG Z，ZHAO W，QIAN P S，et al.Robust traffic prediction from spatial-temporal data based on conditional distribution learning[J].IEEE Transactions on Cybernetics，2021，52（12）：13458-13471.
[7] YING C X，CAI T L，LUO S J，et al.Do transformers really perform badly for graph representation?[C]//Advances in Neural Information Processing Systems，2021.
[8] ZHOU H Y，ZHANG S H，PENG J Q，et al.Informer：beyond efficient transformer for long sequence time-series forecasting[C]//Proceedings of the 35th AAAI Conference on Artificial Intelligence，Palo Alto，California USA，2021.
[9] 刘世泽，秦艳君，王晨星，等.基于深度残差长短记忆网络交通流量预测算法[J].计算机应用，2021，41（6）：1566-1572.
LIU S Z，QIN Y J，WANG C X，et al.Traffic flow prediction algorithm based on deep residual long short-term memory network[J].Journal of Computer Applications，2021，41（6）：1566-1572.
[10] WANG J，DENG W，GUO Y T.New bayesian combination method for short-term traffic flow forecasting[J].Transportation Research Part C Emerging Technologies，2014，43：79-94.
[11] WANG X Y，MA Y，WANG Y Q，et al.Traffic flow prediction via spatial temporal graph neural network[C]//Proceedings of 29th The Web Conference 2020：International World Wide Web Conference，2020：1082-1092.
[12] HOCHREITER S，SCHMIDHUBER J.Long short-term memory[J].Neural Computation，1997，9（8）：1935-1980.
[13] CHO K，VAN MERRIENBOER B，GULCEHRE C，et al.Learning phrase representations using RNN encoder-decoder for statistical machine translation[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing（EMNLP），2014.
[14] 陈丹蕾，陈红，任安虎.考虑时空影响下的图卷积网络短时交通流预测[J].计算机工程与应用，2021，57（13）：269-275.
CHEN D L，CHEN H，REN A H.Short-time traffic flow prediction of graph convolutional network considering influence of space and time[J].Computer Engineering and Applications，2021，57（13）：269-275.
[15] YU B，YIN H T，ZHU Z X.Spatio-temporal graph convolutional networks：a deep learning framework for traffic forecasting[C]//Proceedings of the 27th International Joint Conference on Artificial Intelligence，Stockholm，Sweden，2018：3634-3640.
[16] ZHANG Q，CHANG J L，MENG G F，et al.Spatio-Temporal graph structure learning for traffic forecasting[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence，2020：1177-1185.
[17] ZHAO L，SONG Y J，ZHANG C，et al.T-GCN：a temporal graph convolutional network for traffic prediction[J].IEEE Transactions on Intelligent Transportation Systems，2019，21（9）：3848-3858.
[18] WU Z H，PAN S R，LONG G D，et al.Connecting the dots：multivariate time series forecasting with graph neural networks[C]//Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining，2020.
[19] 张晓旭，马志强，刘志强，等.Transformer在语音识别任务中的研究现状与展望[J].计算机科学与探索，2021，15（9）：1578-1594.
ZHANG X X，MA Z Q，LIU Z Q，et al.Research status and prospect of Transformer in speech recognition[J].Journal of Frontiers of Computer Science and Technology，2021，15（9）：1578-1594.
[20] 高金金，李潞洋.一种改进的点云Transformer深度学习模型[J].中北大学学报：（自然科学版），2021，42（6）：9-15.
GAO J J，LI L Y.Deep learning model based on improved point cloud Transformer[J].Journal of North University of China（Social Science Edition），2021，42（6）：9-15.
[21] Veli?kovi? P，CUCURULL G，CASANOVA A，et al.Graph attention networks[C]//Proceedings of the International Conference on Learning Representations，Vancouver Convention Center，Vancouver，BC，Canada，2018.
[22] ZHANG J N，SHI X J，XIE J Y，et al.GaAN：gated attention networks for learning on large and spatiotemporal graphs[C]//Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence，2018.
[23] GUO S N，LIN Y F，FENG N.et al.Attention based spatial-temporal graph convolutional networks for traffic flow forecasting[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence，2019：922-929.
[24] SONG C，LIN Y F，GUO S N，et al.Spatial-temporal synchronous graph convolutional networks：a new framework for spatial-temporal network data forecasting[C]// Proceedings of the 34th AAAI Conference on Artificial Intelligence，2020：914-921.
[25] YE J X，ZHAO J J，YE K J，et al.How to build a graph-based deep learning architecture in traffic domain：a survey[J].arXiv：2005.11692，2020.