TLS Malicious Encrypted Traffic Identification Research

doi:10.3778/j.issn.1002-8331.2110-0029

Abstract

Abstract: With the advent of the 5G era and the increasing public awareness of the Internet, the public has paid more and more attention to the protection of personal privacy. Due to malicious communication in the process of data encryption, to ensure data security and safeguard social and national interests, the research work on encrypted traffic identification is particularly important. Therefore, this paper describes the TLS traffic in detail and analyzes the improved technology of early identification method, including common traffic detection technology, DPI detection technology, proxy technology, and certificate detection technology. It also introduces machine learning models for selecting different TLS encrypted traffic characteristics, as well as many recent research results of deep learning models without feature selection. The deficiencies of the related research work are summarized, and the future research work and development trend of the technology have been prospected.

Key words: 5G era, personal privacy, malicious traffic, data security, TLS encrypted traffic identification

摘要： 随着5G时代的来临，以及公众对互联网的认识日益加深，公众对个人隐私的保护也越来越重视。由于数据加密过程中存在着恶意通信，为确保数据安全，维护社会国家利益，加密流量识别的研究工作尤为重要。针对TLS流量详细的阐述，分析了早期识别方法的改进技术，包括常见的流量检测技术、DPI检测技术、代理技术以及证书检测技术。介绍了选取不同TLS加密流量特征的机器学习模型，以及无需特征选择的深度学习模型等诸多最新研究成果。对相关研究工作的不足进行总结，并对未来技术的研究工作和发展趋势进行了展望。

关键词: 5G时代, 个人隐私, 恶意流量, 数据安全, TLS加密流量识别

KANG Peng, YANG Wenzhong, MA Hongqiao. TLS Malicious Encrypted Traffic Identification Research[J]. Computer Engineering and Applications, 2022, 58(12): 1-11.

康鹏, 杨文忠, 马红桥. TLS协议恶意加密流量识别研究综述[J]. 计算机工程与应用, 2022, 58(12): 1-11.

References

[1] MEEKER M.Internet trends online[EB/OL].（2019-06-11）.https：//www.bondcap.com/report/itr19/.
[2] ECKERSLEY P.How unique is your web browser?[C]//10th International Conference on Privacy Enhancing Technologies.Berlin，German：Springer，2010：1-18.
[3] The TLS protocol version 1.0：RFC 2246[S/OL].（1999）.https：//tools.ietf.org/html/rfc2246.
[4] The Transport Layer Security（TLS） protocol version 1.3：RFC 8446[S/OL].[2018].https：//tools.ietf.org/html/rfc8446.
[5] 沈若愚，卢盛祺，赵运磊.TLS1.3协议更新发展及其攻击与防御研究[J].计算机应用与软件，2017，34（11）：264-269.
SHEN R Y，LU S Q，ZHAO Y L.The developments of TLS1.3 and its attack and defense[J].Computer Applications and Software，2017，34（11）：264-269.
[6] AVIRAM N，GELLERT K，JAGER T.Session resumption protocols and efficient forward security for TLS 1.3 0-RTT[C]//Annual International Conference on the Theory and Applications of Cryptographic Techniques.Berlin，German：Springer，2019：117-150.
[7] CREMERS C，HORVAT M，HOYLAND S，et al.Comprehensive symbolic analysis of TLS 1.3[C]//Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security.New York，NY：ACM，2017：1773-1788.
[8] BHARGAVAN K，LEURENT G.Transcript collision attacks：breaking authentication in TLS，IKE and SSH[C]//Network and Distributed System Security Symposium.San Diego，CA，USA：NDSS，2016：21-24.
[9] SHERRY J，LAN C，POPA P A，et al.BlindBox：deep packet inspection over encrypted traffic[C]//Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication，2015：213-226.
[10] LEVILLAIN O，GOURDIN B，DEBAR H.TLS record protocol：security analysis and defense-in-depth countermeasures for HTTPS[C]//Proceedings of the 10th ACM Symposium on Information Computer and Communications Security.New York，NY：ACM，2015：225-236.
[11] FISCHLIN M，GNTHER F.Replay attacks on zero round-trip time：the case of the TLS1.3 handshake candidates[C]//IEEE European Symposium on Security and Privacy.New York，NY：IEEE Communications Societ，2017：82-113.
[12] The many flaws of Dual_EC_DRBG[EB/OL].[2013-09-18].https：//blog.cryptographyengineering.com/2013/09/18/the-many-flaws-of-dualecdrbg/2013.
[13] The vulnerability of SSL to chosen plaintext attack[EB/OL].[2004].http：//eprint.iacr.org/2004/1112004.
[14] BARD G.A challenging but feasible blockwise-adaptive chosen-plaintext attack on SSL[C]//Proceedings of the International Conference on Security and Cryptography，2006：99-109.
[15] BLEICHENBACHER D.Chosen ciphertext attacks against protocols based on the RSA encryption standard PKCS #1[C]//18th Annual International Cryptology Conference.Santa Barbara，California，USA：CRYPTO，1998：1-12.
[16] CANVEL B，HILTGEN A，VAUDENAY S，et al.Password interception in a SSL/TLS channel[C]//Advances in Cryptology-CRYPTO 2003，23rd Annual International Cryptology Conference.Santa Barbara，California，USA：CRYPTO，2003：583-599.
[17] Security of CBC Ciphersuites in SSL/TLS：problems and counter measures[EB/OL].[2004].http：//www.openssl.org/~
bodo/tls-cbc.txt2004.
[18] VAUDENAY S.Security flaws induced by CBC padding-applications to SSL，IPSEC，WTLS[C]//International Conference on the Theory and Applications of Cryptographic Techniques.Amsterdam，The Netherlands：EUROCRYPT，2002：534-546.
[19] NADHEM J，ALFARDAN N T，PATERSON K G.Lucky thirteen：breaking the TLS and DTLS record protocols[C]//2013 IEEE Symposium on Security and Privacy.Berkeley，CA，USA：IEEE，2013：526-540.
[20] AVIRAM N，SCHINZEL S，SOMOROVSKY J，et al.DROWN：breaking TLS with SSLv2[C]//5th USENIX Security Symposium.Austin，TX：USENIX，2016：689-706.
[21] BHARGAVAN K，LEURENT G.On the practical （in-）security of 64-bit block ciphers：collision attacks on HTTP over TLS and Open VPN[C]//Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security.New York，NY，USA：ACM，2016：456-467.
[22] BHARGAVAN K，LEURENT G.Transcript collision attacks：breaking authentication in TLS，IKE，and SSH[C]//Network and Distributed System Security Symposium（NDSS 2016）.San Diego，CA，USA：NDSS，2016：1-17.
[23] GARMAN C，PATERSON K，MERWE T V D.Attacks only get better：password recovery attacks against RC4 in TLS[C]//24th USENIX Security Symposium.Washington，D C：USENIX，2015：113-128.
[24] JAGER T，SCHWENK J，SOMOROVSKY J.On the security of TLS 1.3 and QUIC against weaknesses in PKCS#1 v1.5 encryption[C]//Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security.CO，USA：ACM，2015：1185-1196.
[25] Attacking SSL when using RC4[EB/OL].[2015].https：//www.imperva.com/docs/HII_Attacking_SSL_when_using_RC4.pdf2015.
[26] MAVROGIANNOPOULOS N，VERCAUTEREN F，VELICHKOV V，et al.A cross-protocol attack on the TLS protocol[C]//the ACM Conference on Computer and Communications Security.Raleigh，NC，USA：ACM，2012：62-72.
[27] This POODLE bites：exploiting the SSL 3.0 fallback[EB/OL].[2014].https：//www.openssl.org/~bodo/ssl-poodle.
pdf2014.
[28] 张兴隆，程庆丰，马建峰.TLS 1.3协议研究进展[J].武汉大学学报（理学版），2018，64（6）：471-484.
ZHANG X L，CHENG Q F，MA J F.Advance in TLS 1.3 protocol studies[J].Journal of Wuhan University（Science Edition），2018，64（6）：471-484.
[29] LEVILLAIN O.Implementation flaws in TLS stacks：lessons learned and study of TLS 1.3 benefits[M]//Risks and security of internet and systems.Berlin，German：Springer，2020：87-104.
[30] AKHMETZYANOVA L，ALEKSEEV E，SMYSHLYAEVA E，et al.On post-handshake authentication and external PSKs in TLS 1.3[J].Journal of Computer Virology and Hacking Techniques，2020，16：269-274.
[31] LIU R，YU X Z.A survey on encrypted traffic identification[C]//Proceedings of the 2020 International Conference on Cyberspace Innovation of Advanced Technologies.New York，NY：ACM，2020：159-163.
[32] ANDERSON B，MCGREW D.Identifying encrypted malware traffic with contextual flow data[C]//Proceedings of the 2016 ACM Workshop on Artificial Intelligence and Security.New York，NY：ACM，2016：36-46.
[33] HURLEY J，PALACIOS E G，SEZER S.Host-based P2P flow identification and use in real-time[J].ACM Transactions on the Web，2011，5（2）：1-7.
[34] QIN T，WANG L，LIU Z，et al.Robust application identification methods for P2P and VoIP traffic classification in backbone networks[J].Knowledge Based Systems，2015，82（7）：152-162.
[35] 饶瑾.深度包检测（DPI）技术浅谈及应用[J].信息通信，2014（11）：245-246.
RAO J.Discussion and application of deep packet inspection（DPI） technology[J].Information and Communication，2014（11）：245-246.
[36] CANARD S，DIOP A，KHEIR N，et al.Blindids：market-compliant and privacy-friendly intrusion detection system over encrypted traffic[C]//ACM Symposium on Information，Computer and Communications Security.New York，NY，USA：ACM，2017：561-574.
[37] SHERRY J，LAN C，POPA P A，et al.BlindBox：deep packet inspection over encrypted traffic[C]//Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication，2015：213-226.
[38] 孙中军，翟江涛，戴跃伟.一种基于DPI和负载随机性的加密流量识别方法[J].应用科学学报，2019，37（5）：711-720.
SUN Z J，ZHAI J T，DAI Y W.An encrypted traffic identification method based on DPI and load randomness[J].Journal of Applied Sciences，2019，37（5）：711-720.
[39] BUTLER J M，WELLS D.Finding hidden threats by decrypting SSL/TLS[EB/OL].[2013-11-08].https：//www.sans.org/webcasts/finding-hidden-threats-decrypting-ssl-tls-97315.
[40] Snort and SSL/TLS inspection[EB/OL].[2017].https：//www.sans.org/reading-room/whitepapers/detection/paper/37735/.
[41] RADIVILOVA T，KIRICHENKO L，AGEYEV D，et al.Decrypting SSL/TLS traffic for hidden threats detection[C]//2018 IEEE 9th International Conference on Dependable Systems.New York，NY，USA：ACM，2018：143-146.
[42] BAEK J，KIM J，SUSILO W.Inspecting TLS anytime anywhere：a new approach to TLS interception[C]//Proceedings of the 15th ACM Asia Conference on Computer and Communications Security.New York，NY，USA：ACM，2020：116-126.
[43] RUOTI S，NEILL M O，ZAPPALA D，et al.User attitudes toward the inspection of encrypted traffic[J].arXiv：1510.
04921.2015.
[44] HUANG L S，RICE A，ELLINGSEN E，et al.Analyzing forged SSL certificates in the wild[C]//2014 IEEE Symposium on Security and Privacy，2014：83-97.
[45] HUNT T，ZHU Z，XU Y，et al.Ryoan：a distributed sandbox for untrusted computation on secret data[J].Association for Computing Machinery，2018，35（4）：1-34.
[46] JAMSHED M A，MOON Y，KIM D，et al.MOS：a reusable networking stack for flow monitoring middleboxes[C]//14th USENIX Symposium on Networked Systems Design and Implementation.Boston，MA：USENIX，2017：113-129.
[47] SSL/TLS interception proxies and transitive trust[EB/OL].[2012].https：//media.blackhat.com/bh-eu-12/Jarmoc/bh-eu-12-Jarmoc-SSL_TLS_Interception-WP.pdf.
[48] NAYLOR D，SCHOMP K，VARVELLO M，et al.Multi-context TLS（mcTLS）：enabling secure in-network functionality in TLS[C]//Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication，2015：199-212.
[49] SHERRY J，LAN C，POPA R A.BlindBox：deep packet inspection over encrypted traffic[C]//Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication，2015：213-226.
[50] WEAVER N，KREIBICH C，DAM M，et al.Here be web proxies[M]//Passive and active measurement.Berlin，German：Springer，2014：183-192.
[51] DURUMERIC Z，MA Z，SPRINGALL D，et al.The security impact of HTTPS interception[C]//Network and Distributed Systems Symposium.San Diego，CA，USA：NDSS，2017.
[52] MANI A，VAIDYA T，DWORKEN D，et al.An extensive evaluation of the Internet’s open proxies[C]//Proceedings of the 34th Annual Computer Security Applications Conference.New York，NY，USA：ACM，2018：252-265.
[53] HAN J，KIM S，HA J，et al.SGX-box：enabling visibility on encrypted traffic using a secure middlebox module[C]//Proceedings of the First Asia-Pacific Workshop on Networking.New York，NY，USA：ACM，2017：99-105.
[54] HUSáK M，CERMáK M，JIRSíK T，et al.HTTPS traffic analysis and client identification using passive SSL/TLS fingerprinting[J].EURASIP Journal on Information Security，2016：1-14.
[55] ERQUIAGA M J，GARCíA S，GARINO C.Observer effect：how intercepting HTTPS traffic forces malware to change their behavior[C]//Communications in Computer and Information Science.Berlin，German：Springer，2017：272-281.
[56] TSIRANTONAKIS G，ILIA P，IOANNIDIS S，et al.A large-scale analysis of content modifcation by open http proxies[C]//Network and Distributed Systems Security.San Diego，CA，USA：NDSS，2018：1-15.
[57] ANDERSON B，PAUL S，MCGREW D.Deciphering malware’s use of TLS（without decryption）[J].Journal of Computer Virology and Hacking Techniques，2018，14：195-211.
[58] KOTZIAS P，RAZAGHPANAH A，AMANN J，et al.Coming of age：a longitudinal study of TLS deployment[C]//Proceedings of the Internet Measurement Conference.New York，NY，USA：ACM，2018：415-428.
[59] MATOUSEK P，BURGETOVA I，RYSAVY O，et al.On reliability of JA3 hashes for fingerprinting mobile applications[C]//Digital Forensics and Cyber Crime.Berlin，German：Springer，2021：1-22.
[60] 翟明芳，张兴明，赵博.基于深度学习的加密恶意流量检测研究[J].网络与信息安全学报，2020，6（3）：66-77.
ZHAI M F，ZHANG X M，ZHAO B.Survey of encrypted malicious traffic detection based on deep learning[J].Chinese Journal of Network and Information Security，2020，6（3）：66-77.
[61] 曾勇，吴正远，董丽华，等.加密流量中的恶意流量识别技术[J].西安电子科技大学报（自然科学版），2021，48（3）：170-187.
ZENG Y，WU Z Y，DONG L H，et al.Research on malicious traffic identification technology in encrypted traffic[J].Journal of Xidian University（Natural Science），2021，48（3）：170-187.
[62] DAI R，GAO C，LANG B.SSL malicious traffic detection based on multi-view features[C]//Proceedings of the 2019 the 9th International Conference on Communication and Network Security.New York，NY，USA：ACM，2019：40-46.
[63] CTU malware capture facility project[EB/OL].[2019].https：//mcfp.weebly.com/the-ctu-13-dataset-a-labeled-dataset-with-botnet-normal-and-background-traffic.html.
[64] ANDERSON B，MCGREW D.Identifying encrypted malware traffic with contextual flow data[C]//Proceedings of the 2016 ACM Workshop on Artificial Intelligence and Security.New York，NY，USA：ACM，2016：35-46.
[65] RAZA S M，CABALLERO J.Malware traffic classification：evaluation of algorithms and an automated ground-truth generation pipeline[J].arXiv：2010.11627，2020.
[66] ZHENG R，LIU J，LI K，et al.Detecting malicious TLS network traffic based on communication channel features[C]//2020 IEEE 8th International Conference on Information Communication and Networks.New York，NY：IEEE Communications Society，2020：14-19.
[67] A source for pcap files and malware samples[EB/OL].[2020].https：//www.malware-traffic-analysis.net/.
[68] Malware capture facility project[EB/OL].[2020].https：//www.stratosphereips.org/datasets-malware.
[69] Canadian institute for cybersecurity[EB/OL].[2020].https：//www.unb.ca/cic/datasets/index.html，2020.
[70] 李慧慧，张士庚，宋虹，等.结合多特征识别的恶意加密流量检测方法[J].信息安全学报，2021，6（2）：129-142.
LI H H，ZHANG S G，SONG H，et al.Robust malicious encrypted traffic detection based with multiple features[J].Journal of Cyber Security，2021，6（2）：129-142.
[71] 胡斌，周志洪，姚立红，等.结合报文负载与流指纹特征的恶意流量检测[J].计算机工程，2020，46（11）：157-163.
HU B，ZHOU Z H，YAO L H，et al.Malicious traffic detection combining features of packet payload and stream fingerprint[J].Computer Engineering，2020，46（11）：157-163.
[72] 骆子铭，许书彬，刘晓东.基于机器学习的TLS恶意加密流量检测方案[J].网络与信息安全学报，2020，6（1）：77-83.
LUO Z M，XU S B，LIU X D.Scheme for identifying malware traffic with TLS data based on machine learning[J].Chinese Journal of Network and Information Security，2020，6（1）：77-83.
[73] 蒋彤彤，尹魏昕，蔡冰，等.基于层次时空特征与多头注意力的恶意加密流量识别[J].计算机工程，2021，47（7）：101-108.
JIANG T T，YIN W X，CAI B，et al.Encrypted maliclous traffic identification based on hierarchical spatiotemporal feature and multi-head attention[J].Computer Engineering，2021，47（7）：101-108.
[74] LASHKARI A H，KADIR A F A，TAHERI L，et al.Toward developing a systematic approach to generate benchmark android malware datasets and classification[C]//International Carnahan Conference on Security Technology.New York，NY：IEEE Communications Society，2018：1-7.
[75] 韦佶宏，郑荣锋，刘嘉勇.基于混合神经网络的恶意TLS流量识别研究[J].计算机工程与应用，2021，57（7）：107-114.
WEI J H，ZHENG R F，LIU J Y.Research on malicious TLS traffic identification based on hybrid neural network[J].Computer Engineering and Applications，2021，57（7）：107-114.
[76] Malware-traffic-analysis[EB/OL].[2019].https：//www.malware-traffic-analysis.net.
[77] YANG Y，KANG C，GOU G，et al.TLS/SSL encrypted traffic classification with autoencoder and convolutional neural network[C]//2018 IEEE 20th International Conference on High Performance Computing and Communications；IEEE 16th International Conference on Smart City；IEEE 4th International Conference on Data Science and Systems.New York，NY：IEEE Communications Society，2018：362-369.
[78] ZENG Y，GU H，WEI W，et al.Deep-full-range：a deep learning based network encrypted traffic classification and intrusion detection framework[J].IEEE Access，2019，7：45182-45190.
[79] DRAPER-GIL G，LASHKARI A H，MAMUN M S I，et al.Characterization of encrypted and VPN traffic using time-related[C]//The International Conference on Information Systems Security and Privacy，2016：407-414.
[80] SHIRAVI A，SHIRAVI H，TAVALLAEE M，et al.Toward developing a systematic approach to generate benchmark datasets forintrusion detection[J].Computers Security，2012，31（3）：357-374.
[81] TORROLEDO I，CAMACHO L D，CORREABAHNSEN A.Hunting malicious TLS certificates with deep neural networks[C]//Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security.New York，NY，USA：ACM，2018：64-73.
[82] LOTFOLLAHI M，SIAVOSHANI M J，SABERIAN M.Deep packet：a novel approach for encrypted traffic classification using deep learning[J].Soft Computing，2020，24：1999-2012.
[83] VELAN P，CELEDA P，DRASAR M，et al.A survey of methods for encrypted traffic classification and analysis[J].International Journal of Network Management，2015，25（5）：355-374.
[84] POH G，DIVAKARAN D，LIM H，et al.A survey of privacy-preserving techniques for encrypted traffic inspection over network middleboxes[J].arXiv：2101.04338.2021.
[85] GHARIB A，SHARAFALDIN I，HABIBI L A，et al.An evaluation framework for intrusion detection dataset[C]//2016 International Conference on Information Science and Security（ICISS）.New York，NY：IEEE Communications Society，2016：1-6.
[86] PENDLEBURY F，PIERAZZI F，JORDANEY R.TESSERACT：eliminating experimental bias in malware classification across space and time[C]//Usenix Security Symposium，2019.
[87] SINHA G，KANAGARATHINAM M R，JAYASEELAN S R，et al.CQUIC：cross-layer QUIC for next generation mobile networks[C]//2020 IEEE Wireless Communications and Networking Conference （WCNC）.Seoul，Korea（South）：IEEE，2020：1-8.
[88] Draft-ietf-quic-http-34，hypertext transfer protocol version 3（HTTP/3）[EB/OL].[2021].https：//datatracker.ietf.org/doc/draft-ietf-quic-http.