关键词: TLS流量识别, 恶意加密流量, 传统机器学习, 深度神经网络, 特征自动挖掘

Abstract:

To address the problem that using traditional machine learning methods to identify malicious TLS traffic is greatly affected by expert experience, and the identification and classification results are not satisfactory, a Hybrid Neural Network Identification Model（HNNIM） for identification and classification is proposed. The model consists of two layers, the first layer is used to extract features and the second layer is used for identification and classification. In the first layer, the final extracted features are composed of two parts：one part is automatically mined by deep neural network; the other part is selected according to expert experience and further screened by the deep neural network. The second layer aggregates the features screened from the first layer, using a fully connected deep neural network for further learning and fitting. By analyzing a large number of TLS traffic samples, the ClientHello and ServerHello message and TCP protocol interactions information in TLS traffic are selected as the feature space. The experimental results show that the F1 value of HNNIM regarding malicious samples on the malicious TLS traffic identification task is 0.989, which is 0.016, 0.016, 0.019, 0.043 higher than the random forest, SVM, XGBoost, Convolutional Neural Network models, respectively; the average accuracy on the multi-classification task is 89.28%, which is 9.92%, 9.09%, 11.31%, 7.03% higher than the random forest, SVM, XGBoost, Convolutional Neural Network models.

Key words: TLS traffic identification, malicious encryption traffic, traditional machine learning, deep neural network, automatic feature mining