Abstract: As an effective defense method of network security, intrusion detection technology is an essential part of network security system. With the drastic development of the Internet, the amount of network data increases rapidly, and network attacks tend to be more complex and diversified, consequently, current intrusion detection technologies cannot identify all kinds of attacks effectively. Owing to the unbalanced problem between normal traffic and attack traffic in the real network environment and the low detection rate of attack traffic, this paper proposes a CBL_DDQN detection model based on improved double deep Q-network which is based on deep reinforcement learning. A hybrid model consisting of one-dimensional convolutional neural network and bi-directional long short-term memory network is utilized in the DDQN framework of deep reinforcement learning, then the feedback learning and strategy-generating mechanism of deep reinforcement learning is used for training the agent to classify different types of attack samples, which can greatly weaken the dependence on data labels in the process of training model. In the meantime, the Borderline-SMOTE algorithm is used to reduce data imbalance so as to improve the detection rate of rare attack traffic. The performance of the model evaluated by NSL_KDD and UNSW_NB15 datasets shows that the model performs well in accuracy, precision and recall. The detection result of the model is far better than that of Adam-BNDNN, KNN, SVM and other detection methods, which implies the intrusion detection model proposed in this paper is efficient.

Key words: intrusion detection, deep reinforcement learning, double deep Q-network, convolutional neural network（CNN）, long short-term memory network（LSTM）

摘要： 入侵检测技术作为网络安全有效的防御手段，是网络安全体系中的重要组成部分。随着互联网的快速发展，网络数据量快速增加，网络攻击更加趋于复杂化和多元化，目前主流的入侵检测技术无法有效识别各种攻击。针对实际网络环境中正常流量和攻击流量数据不平衡，且对攻击类流量检测率低的问题，基于深度强化学习提出一种基于改进双重深度Q网络的CBL_DDQN网络入侵检测模型。该模型将一维卷积神经网络和双向长短期记忆网络的混合网络模型引入深度强化学习的DDQN框架，并使用深度强化学习中的反馈学习和策略生成机制训练智能体来对不同类别的攻击样本进行分类，在一定程度上减弱了训练模型过程中对数据标签的依赖性。采用Borderline-SMOTE算法降低数据的不平衡度，从而提高稀有攻击的检测率。通过NSL_KDD和UNSW_NB15数据集对模型的性能进行评估，结果表明：该模型在准确率、精确率、召回率这三项指标上均取得了良好的结果，检测效果远优于Adam-BNDNN、KNN、SVM等检测方法，是一种高效的网络入侵检测模型。

关键词: 入侵检测, 深度强化学习, 双重深度Q网络, 卷积神经网络（CNN）, 长短期记忆网络（LSTM）