混合特征平衡图注意力网络日志异常检测模型

doi:10.3778/j.issn.1002-8331.2310-0210

摘要/Abstract

摘要： 针对现有方法忽略了日志异常数据不平衡和日志特征间的关联性，导致异常检测准确率低的问题。提出一种基于混合特征平衡图注意力网络的日志异常检测模型（HBGATLog）。构建混合日志图构建模块，通过混合特征提取模块提取日志数据的语义信息、日志序列和时间结构，增强日志特征间的关联性，并采用日志图构建模块构建日志图，有效保留空间结构特征。设计平衡日志图生成模块，解决不平衡的日志数据导致检测结果偏向多数类问题。采用图日志异常检测模块进行异常检测。使用BGL、Thunderbird和HDFS三个公共数据集对HBGATLog进行验证，实验结果表明，F1 score分别达到了99.0%、98.7%和98.1%。证明HBGATLog不但能够解决日志数据不平衡问题，充分考虑日志数据特征的关联性，而且有效降低了漏检率。

关键词: 日志异常检测, 日志分析, 图神经网络, 混合特征提取, 数据不平衡

Abstract: The existing methods neglect the imbalance in log abnormal data and the correlation between log features, leading to the problem of low accuracy in anomaly detection. This paper proposes a log anomaly detection model based on a hybrid graph attention network with balanced features (HBGATLog). Firstly, the hybrid log graph construction module is established, it extracts the semantic information, log sequence and time structure of log data through a hybrid feature extraction module, which enhances the correlation between log features. In addition, a log graph construction module is employed to build a log graph, which can effectively preserve spatial structural features. Secondly, a balanced log graph generation module is designed to solve the problem that unbalanced log data leads to detection results biased towards the majority classes. Thirdly, the graph log anomaly detection module is used for anomaly detection. Finally, three public datasets, BGL, Thunderbird and HDFS, are used to validate HBGATLog, and the experimental results show that the F1 score reaches 99.0%, 98.7% and 98.1%, respectively. It is proved that HBGATLog can not only solve the problem of log data imbalance, fully consider the correlation of log data features, but also effectively reduce the missing rate.

Key words: log anomaly detection, log analysis, graph neural networks, hybrid feature extraction, data imbalance

陈旭, 张硕, 景永俊, 王叔洋. 混合特征平衡图注意力网络日志异常检测模型[J]. 计算机工程与应用, 2025, 61(1): 308-320.

CHEN Xu, ZHANG Shuo, JING Yongjun, WANG Shuyang. Log Anomaly Detection Model Based on Hybrid Feature Balanced Graph Attention Network[J]. Computer Engineering and Applications, 2025, 61(1): 308-320.

参考文献

[1] MENG W, LIU Y, HUANG Y, et al. A semantic-aware representation framework for online log analysis[C]//Proceedings of the 2020 29th International Conference on Computer Communications and Networks (ICCCN), Honolulu, HI, USA, 2020: 1-7.
[2] LIANG Y, ZHANG Y, XIONG H, et al. Failure prediction in IBM BlueGene/L event logs[C]//Proceedings of the Seventh IEEE International Conference on Data Mining (ICDM 2007), Omaha, NE, USA, 2007: 583-588.
[3] XU W, HUANG L, FOX A, et al. Detecting large-scale system problems by mining console logs[C]//Proceedings of the ACM SIGOPS 22nd Symposium on Operating Systems Principles, 2009: 117-132.
[4] ZHOU T, ZHANG X X, LU H, et al. GMRE-iUnet: isomorphic Unet fusion model for PET and CT lung tumor images[J]. Computers in Biology and Medicine, 2023: 107514.
[5] MENG W, LIU Y, ZHU Y, et al. LogAnomaly: unsupervised detection of sequential and quantitative anomalies in unstructured logs[C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, Macao, China, 2019: 4739-4745.
[6] ZHANG X, XU Y, LIN Q, et al. Robust log-based anomaly detection on unstable log data[C]//Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Tallinn, Estonia, 2019: 807-817.
[7] ZHOU P, SHI W, TIAN J, et al. Attention-based bidirectional long short-term memory networks for relation classification[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Berlin, Germany, 2016: 207-212.
[8] LE V H, ZHANG H. Log-based anomaly detection without log parsing[C]//Proceedings of the 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), Melbourne, Australia, 2021: 492-504.
[9] XIE Y, ZHANG H, BABAR M A. LogGD: detecting anomalies from system logs with graph neural networks[C]//Proceedings of the 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS), Guangzhou, China, 2022: 299-310.
[10] ZHANG C, PENG X, SHA C, et al. DeepTraLog: trace-log combined microservice anomaly detection through graph-based deep learning[C]//Proceedings of the 44th International Conference on Software Engineering, Pittsburgh, Pennsylvania, 2022: 623-634.
[11] SUN P, YUE P E, LI T, et al. Context-aware learning for anomaly detection with imbalanced log data[C]//Proceedings of the 2020 IEEE 22nd International Conference on High Performance Computing and Communications; IEEE 18th International Conference on Smart City; IEEE 6th International Conference on Data Science and Systems (HPCC/SmartCity/DSS), Yanuca Island, Cuvu, Fiji, 2020: 449-456.
[12] 姚杰, 程春玲, 韩静, 等. 云工作流中基于多任务时序卷积网络的异常检测方法[J], 计算机应用, 2021, 41(6): 1701-1708.
YAO J, CHENG C L, HAN J, et al. Anomaly detection method based on multi-task temporal convolutional network in cloud workflow[J]. Journal of Computer Applications, 2021, 41(6): 1701-1708.
[13] LIN Q, ZHANG H, LOU J G, et al. Log clustering based problem identification for online service systems[C]//Proceedings of the 38th International Conference on Software Engineering Companion, Austin, Texas, 2016: 102-111.
[14] DU M, LI F, ZHENG G, et al. DeepLog: anomaly detection and diagnosis from system logs through deep learning[C]//Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, Dallas, Texas, USA, 2017: 1285-1298.
[15] GUO H, YUAN S, WU X. LogBERT: log anomaly detection via BERT[C]//Proceedings of the 2021 International Joint Conference on Neural Networks (IJCNN), Shenzhen, China, 2021: 1-8.
[16] 何书前, 孙学朝, 蒋文娟, 等. 一种改进的模糊聚类日志异常检测方法[J]. 现代电子技术, 2022, 45(16): 30-34.
HE S Q, SUN X C, JIANG W J, et al. Method of improved fuzzy clustering log anomaly detection[J]. Modern Electronics Technique, 2022, 45(16): 30-34.
[17] YANG L, CHEN J, WANG Z, et al. Semi-supervised log-based anomaly detection via probabilistic label estimation[C]//Proceedings of the 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), Madrid, ES, 2021: 1448-1460.
[18] ZHANG M, CHEN J, LIU J, et al. LogST: log semi-supervised anomaly detection based on sentence-BERT[C]//Proceedings of the 2022 7th International Conference on Signal and Image Processing (ICSIP), 2022: 356-361.
[19] VELICKOVIC P, CUCURULL G, CASANOVA A, et al. Graph attention networks[J]. arXiv:1710.10903, 2017.
[20] 束文豪, 奚雪峰, 崔志明, 等. 图神经网络在命名实体识别中的应用研究[J]. 计算机工程与应用, 2023, 59(19): 52-65.
SHU W H, XI X F, CUI Z M, et al. Study of named entity recognition based on graph neural network[J]. Computer Engineering and Applications, 2023, 59(19): 52-65.
[21] LIU M, JIN S, JIN L, et al. Imbalanced nodes classification for graph neural networks based on valuable sample mining[C]//Proceedings of the 2022 6th International Conference on Electronic Information Technology and Computer Engineering, Xiamen, China, 2022: 1957-1962.
[22] SHI M, TANG Y, ZHU X, et al. Multi-class imbalanced graph convolutional network learning[C]//Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, Yokohama, Japan, 2020: 2879-2885.
[23] SHI S, QIAO K, YANG J, et al. Over-sampling strategy in feature space for graphs based class-imbalanced bot detection[J]. arXiv:2302.06900, 2023.
[24] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[J]. arXiv:1810.04805, 2018.
[25] HE P, ZHU J, ZHENG Z, et al. Drain: an online log parsing approach with fixed depth tree[C]//Proceedings of the 2017 IEEE International Conference on Web Services (ICWS), Honolulu, HI, USA, 2017: 33-40.
[26] HE S, ZHU J, HE P, et al. Loghub: a large collection of system log datasets towards automated log analytics[J]. arXiv:2008.06448, 2020.