Survey on Automated Recognition and Extraction of TTPs

doi:10.3778/j.issn.1002-8331.2309-0489

Abstract

Abstract: In the ever-evolving landscape of cyber threats, tactics, techniques and procedures (TTPs) play a crucial role in understanding malicious activities, providing a fine-grained perspective on the status of cybersecurity, and comprehensively illustrating cyber attack behaviors. Despite significant research efforts in the field of automated identification and extraction of TTPs, a comprehensive systematic review is currently lacking. This paper presents an in-depth analysis of the progress in this area by employing three principal approaches：traditional natural language processing, machine learning, and large language models. The study categorizes the tasks into information extraction, text classification, and text generation, and presents a summary of the general framework for identification and extraction processes. It offers a clear scope of unstructured text and TTPs, while refining the processing and analysis procedures, as well as innovative directions for each approaches. Moreover, building upon existing research, the paper identifies current challenges and proposes future research directions and development opportunities. This comprehensive survey serves as a valuable literature review to support readers in applying advanced technologies and methods for advancing research in this field.

Key words: cyber threat intelligence (CTI), tactics, techniques and procedures (TTPs), deep learning, large language models (LLMs), natural language processing (NLP)

摘要： 当今网络威胁不断涌现，网络威胁技战术情报能够多维度挖掘网络恶意活动，细粒度展示网络安全态势，全方位刻画网络攻击行为。目前对于网络威胁技战术情报自动化识别提取任务的研究成果较多，但缺乏系统化梳理。基于传统自然语言处理、机器学习和大语言模型三种研究思路，深入分析了相关研究进展，对应信息抽取、文本分类、文本生成三类任务，概括了一般识别提取流程框架，明确了非结构化文本、网络威胁技战术情报范围，细化了每种技术方法的处理分析实践流程及创新方向，并基于现有研究工作，提出了当前研究存在的问题及未来的研究和发展方向，为读者运用新技术新方法促进领域研究水平提升提供了文献综述支持。

关键词: 网络威胁情报, 网络威胁技战术情报（TTPs）, 深度学习, 大语言模型, 自然语言处理

YU Fengrui. Survey on Automated Recognition and Extraction of TTPs[J]. Computer Engineering and Applications, 2024, 60(13): 1-22.

于丰瑞. 网络威胁技战术情报自动化识别提取研究综述[J]. 计算机工程与应用, 2024, 60(13): 1-22.

References

[1] 黄克振, 连一峰, 冯登国, 等. 基于区块链的网络安全威胁情报共享模型[J]. 计算机研究与发展, 2020, 57(4): 836-846.
HUANG K Z, LIAN Y F, FENG D G, et al. Cyber security threat intelligence sharing model based on blockchain[J]. Journal of Computer Research and Development, 2020, 57(4): 836-846.
[2] SCHLETTE D, B?HM F, CASELLI M, et al. Measuring and visualizing cyber threat intelligence quality[J]. International Journal of Information Security, 2021, 20(1): 21-38.
[3] SAHROM ABU M, RAHAYU SELAMAT S, ARIFFIN A, et al. Cyber threat intelligence-issue and challenges[J]. Indonesian Journal of Electrical Engineering and Computer Science, 2018, 10(1): 371.
[4] BIANCO D J. The pyramid of pain[EB/OL]. (2014-01-17)[2023-09-21]. https://detect-respond.blogspot.com/2013/03/the-pyramid-of-pain. html.
[5] OOSTHOEK K, DOERR C. Cyber threat intelligence: a product without a process?[J]. International Journal of Intelligence and CounterIntelligence, 2021, 34(2): 300-315.
[6] CONTI M, DARGAHI T, DEHGHANTANHA A. Cyber threat intelligence: challenges and opportunities[M]//Advances in information security. Cham: Springer, 2018.
[7] BROWN R, STIRPARO P. SANS 2022 Cyber threat intelligence survey[EB/OL]. (2023-02-23)[2023-09-21]. https://www.sans.org/white-papers/sans-2022-cyber-threat-intelligence-survey/.
[8] CISA. Defending against malicious cyber activity originating from Tor[EB/OL]. (2021-08-02)[2023-09-21]. https://www.cisa.gov/news-events/cybersecurity-advisories/aa20-183a.
[9] MITRE. MITRE ATT&CK[DB/OL]. [2023-09-21]. https://attack.mitre.org/.
[10] CISA. CISA analysis: fiscal year 2022 risk and vulnerability assessments[EB/OL]. (2023-06)[2023-09-21]. https://www.cisa.gov/sites/default/files/2023-07/FY22-RVA-Analysis-Final_508c.pdf.
[11] RAMSDALE A, SHIAELES S, KOLOKOTRONIS N. A comparative analysis of cyber-threat intelligence sources, formats and languages[J]. Electronics, 2020, 9(5): 824.
[12] ALAM M T, BHUSAL D, PARK Y, et al. Looking beyond IoCs: automatically extracting attack patterns from external CTI[J]. arXiv:2211.01753, 2022.
[13] GRO? S. Research directions in cyber threat intelligence[J]. arXiv:2001.06616, 2020.
[14] 董聪, 姜波, 卢志刚, 等. 面向网络空间安全情报的知识图谱综述[J]. 信息安全学报, 2020, 5(5): 56-76.
DONG C, JIANG B, LU Z G, et al. Knowledge graph for cyberspace security intelligence: a survey [J]. Journal of Cyber Security, 2020, 5(5): 56-76.
[15] DOUGLAS M R. Large language models[J]. arXiv:2307.
05782, 2023.
[16] LOU R, ZHANG K, YIN W. Is prompt all you need? no. a comprehensive and broader view of instruction learning[J]. arXiv:2303.10475, 2023.
[17] NAVEED H, KHAN A U, QIU S, et al. A comprehensive overview of large language models[J]. arXiv:2307.06435, 2023.
[18] ROY S, PANAOUSIS E, NOAKES C, et al. SoK: the MITRE ATT&CK framework in research and practice[J]. arXiv:2304.07411, 2023.
[19] HUSARI G, AL-SHAER E, AHMED M, et al. TTPDrill: automatic and accurate extraction of threat actions from unstructured text of CTI sources[C]//Proceedings of the 33rd Annual Computer Security Applications Conference, Orlando, FL, USA, 2017: 103-115.
[20] ROBERTSON S E, WALKER S. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval[C]//Proceedings of SIGIR’94. London: Springer London, 1994: 232-241.
[21] MACDONALD M, FRANK R, MEI J, et al. Identifying digital threats in a hacker web forum[C]//Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, Paris, France, 2015: 926-933.
[22] HUGHES J, AYCOCK S, CAINES A, et al. Detecting trending terms in cybersecurity forum discussions[C]//Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020), 2020: 107-115.
[23] SUFI F. Novel application of open-source cyber intelligence[J]. Electronics, 2023, 12(17): 3610.
[24] MITRE. Comparing STIX 1. X/CybOX 2. X with STIX 2[EB/OL]. (2023-07-25)[2023-08-16]. https://oasis-open.github.io/cti-documentation/stix/compare.
[25] 杨沛安, 刘宝旭, 杜翔宇. 面向攻击识别的威胁情报画像分析[J]. 计算机工程, 2020, 46(1): 136-143.
YANG P A, LIU B X, DU X Y. Portrait analysis of threat intelligence for attack recognition[J]. Computer Engineering, 2020, 46(1) : 136-143.
[26] 高见, 王安. 基于本体的网络威胁情报分析技术研究[J]. 计算机工程与应用, 2020, 56(11): 112-117.
GAO J, WANG A. Research on ontology-based network threat intelligence analysis technology[J]. Computer Engineering and Applications, 2020, 56(11): 112-117.
[27] TOUNSI W, RAIS H. A survey on technical threat intelligence in the age of sophisticated cyber attacks[J]. Computers & Security, 2018, 72: 212-233.
[28] XU Z, WU Z, LI Z, et al. High fidelity data reduction for big data security dependency analyses[C]//Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, 2016: 504-516.
[29] AL-MOSLMI T, OCA?A M G, OPDAHL A L, et al. Named entity extraction for knowledge graphs: a literature overview[J]. IEEE Access, 2020, 8: 32862-32881.
[30] ZHOU M, DUAN N, LIU S, et al. Progress in neural NLP: modeling, learning, and reasoning[J]. Engineering, 2020, 6(3): 275-290.
[31] GAO P, SHAO F, LIU X, et al. A system for efficiently hunting for cyber threats in computer systems using threat intelligence[C]//Proceedings of the 2021 IEEE 37th International Conference on Data Engineering (ICDE), 2021: 2705-2708.
[32] NIAKANLAHIJI A, WEI J, CHU B T. A natural language processing based trend analysis of advanced persistent threat techniques[C]//Proceedings of the 2018 IEEE International Conference on Big Data (Big Data), Seattle, WA, USA, 2018: 2995-3000.
[33] HUSARI G, AL-SHAER E, CHU B, et al. Learning APT chains from cyber threat intelligence[C]//Proceedings of the 6th Annual Symposium on Hot Topics in the Science of Security , Nashville, Tennessee, 2019.
[34] GAO P, LIU X, CHOI E, et al. ThreatKG: a threat knowledge graph for automated open-source cyber threat intelligence gathering and management[J]. arXiv:2212.10388, 2022.
[35] Snorkel-Team. snorkel[CP/OL]. (2023-02-18)[2023-09-15]. https://github.com/snorkel-team/snorkel.
[36] GHAZI Y, ANWAR Z, MUMTAZ R, et al. A supervised machine learning based approach for automatically extracting high-level threat intelligence from unstructured sources[C]//Proceedings of the 2018 International Conference on Frontiers of Information Technology (FIT), Islamabad, Pakistan, 2018: 129-134.
[37] Kaspersky. Kaspersky threat intelligence[EB/OL]. (2023-08-16)[2023-08-20]. https://usa.kaspersky.com/enterprise-security/apt-intelligence-reporting.
[38] BANDLA K. APTnotes[DB/OL]. (2019-09-18)[2023-08-25]. https://github.com/kbandla/aptnotes/.
[39] SUTTON C. An introduction to conditional random fields[J]. Foundations and Trends in Machine Learning, 2012, 4(4): 267-373.
[40] PIPLAI A, MITTAL S, JOSHI A, et al. Creating cybersecurity knowledge graphs from malware after action reports[J]. IEEE Access, 2020, 8: 211691-211703.
[41] MICROSOFT. security-bulletin[EB/OL]. (2023-08-25)[2023-08-26]. https://msrc.microsoft.com/blog/tags/security-bulletin/.
[42] Adobe Product Security Incident Response Team. Security updates[EB/OL]. (2023-09-13)[2023-09-25]. https://helpx.adobe.com/security.html.
[43] Cveproject. cvelist[EB/OL]. (2023-09-25)[2023-09-25]. https://github.com/CVEProject/cvelist.
[44] MIKOLOV T, SUTSKEVER I, CHEN K, et al. Distributed representations of words and phrases and their compositionality[C]//Advances in Neural Information Processing Systems, 2013, 26.
[45] OASIS-Open. STIX2.1 examples[EB/OL]. (2023-07-25)[2023-09-25]. https://oasis-open.github.io/cti-documentation/stix/examples.html.
[46] SYED Z, PADIA A, FININ T, et al. UCO: a unified cybersecurity ontology[C]//Proceedings of the AAAI Workshop on Artificial Intelligence for Cyber Security, 2016.
[47] LIU Y, SHI R, CHEN Y, et al. APTTOOLNER: a Chinese dataset of cyber security tool for NER task[C]//Proceedings of the 2023 3rd Asia-Pacific Conference on Communications Technology and Computer Science (ACCTCS), 2023: 368-373.
[48] HUANG Y, SU M, XU Y, et al. NER in cyber threat intelligence domain using transformer with TSGL[J]. Journal of Circuits, Systems and Computers, 2023: 2350201.
[49] WANG X, LIU J. A novel feature integration and entity boundary detection for named entity recognition in cybersecurity[J]. Knowledge-Based Systems, 2023, 260: 110114.
[50] ZHOU Y, TANG Y, YI M, et al. CTI View: APT threat intelligence analysis system[J]. Security and Communication Networks, 2022, 2022: 1-15.
[51] LI Y, GUO Y, FANG C, et al. Feature-enhanced document-level relation extraction in threat intelligence with knowledge distillation[J]. Electronics, 2022, 11(22): 3715.
[52] LI Y, GUO Y, FANG C, et al. A novel threat intelligence information extraction system combining multiple models[J]. Security and Communication Networks, 2022.
[53] SATVAT K, GJOMEMO R, VENKATAKRISHNAN V N. Extractor: extracting attack behavior from threat reports[C]//Proceedings of the 2021 IEEE European Symposium on Security and Privacy (EuroS&P), 2021: 598-615.
[54] DARPA. Transparent computing[EB/OL]. [2023-08-16]. https://www.darpa.mil/program/transparentcomputing.
[55] MILAJERDI S M, ESHETE B, GJOMEMO R, et al. POIROT: aligning attack behavior with kernel audit records for cyber threat hunting[C]//Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. London United Kingdom: ACM, 2019: 1795-1812.
[56] RAYMOND J W, WILLETT P. Maximum common subgraph isomorphism algorithms for the matching of chemical structures[J]. Journal of Computer-Aided Molecular Design, 2002, 16(7): 521-533.
[57] XIANG G, SHI C, ZHANG Y. An APT event extraction method based on BERT-BiGRU-CRF for APT attack detection[J]. Electronics, 2023, 12(15): 3349.
[58] DU Y, HUANG C, LIANG G, et al. ExpSeeker: extract public exploit code information from social media[J]. Applied Intelligence, 2023, 53(12): 15772-15786.
[59] JO H, LEE Y, SHIN S. Vulcan: automatic extraction and analysis of cyber threat intelligence from unstructured text[J]. Computers & Security, 2022, 120: 102763.
[60] LI Z, ZENG J, CHEN Y, et al. AttacKG: constructing technique knowledge graph from cyber threat intelligence reports[C]//Proceedings of the 27th European Symposium on Research in Computer Security. Cham: Springer, 2022: 589-609.
[61] 刘强, 祝鹏程. 基于联合学习的端到端威胁情报知识图谱构建方法[J]. 现代计算机, 2021(16): 16-21.
LIU Q, ZHU P C. End to end threat intelligence knowledge graph construction method based on joint learning[J]. Modern Computer, 2021(16): 16-21.
[62] GUO Y, LIU Z, HUANG C, et al. A framework for threat intelligence extraction and fusion[J]. Computers & Security, 2023, 132: 103371.
[63] WANG X, XIONG M, LUO Y, et al. Joint learning for document-level threat intelligence relation extraction and coreference resolution based on GCN[C]//Proceedings of the 2020 IEEE 19th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), Guangzhou, China, 2020: 584-591.
[64] MARCHIORI F, CONTI M, VERDE N V. STIXnet: a novel and modular solution for extracting All STIX objects in CTI reports[J]. arXiv:2303.09999, 2023.
[65] WEERAWARDHANA S, MUKHERJEE S, RAY I, et al. Automated extraction of vulnerability information for home computer security[M]//CUPPENS F, GARCIA-ALFARO J, ZINCIR HEYWOOD N, et al. Foundations and practice of security. Cham: Springer, 2015: 356-366.
[66] LI T, GUO Y, JU A. A self-attention-based approach for named entity recognition in cybersecurity[C]//Proceedings of the 2019 15th International Conference on Computational Intelligence and Security (CIS), 2019.
[67] WANG X, LIU R, YANG J, et al. Cyber threat intelligence entity extraction based on deep learning and field knowledge engineering[C]//Proceedings of the 2022 IEEE 25th International Conference on Computer Supported Cooperative Work in Design (CSCWD), 2022: 406-413.
[68] ZHOU Y, REN Y, YI M, et al. CDTier: a Chinese dataset of threat intelligence entity relationships[J]. IEEE Transactions on Sustainable Computing, 2023: 1-13.
[69] RANADE P, PIPLAI A, JOSHI A, et al. CyBERT: contextualized embeddings for the cybersecurity domain[C]//Proceedings of the 2021 IEEE International Conference on Big Data (Big Data), Orlando, FL, USA, 2021: 3334-3342.
[70] LI K, WEN H, LI H, et al. Security OSIF: toward automatic discovery and analysis of event based cyber threat intelligence[C]//Proceedings of the 2018 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), Guangzhou, China, 2018: 741-747.
[71] SARHAN I, SPRUIT M. Open-CyKG: an open cyber threat intelligence knowledge graph[J]. Knowledge-Based Systems, 2021, 233: 107524.
[72] HANKS C, MAIDEN M, RANADE P, et al. Recognizing and extracting cybersecurtity-relevant entities from text[J]. arXiv:2208.01693, 2022.
[73] SUN T, YANG P, LI M, et al. An automatic generation approach of the cyber threat intelligence records based on multi-source information fusion[J]. Future Internet, 2021, 13(2): 40.
[74] KOLOVEAS P, CHANTZIOS T, ALEVIZOPOULOU S, et al. inTIME: a machine learning-based framework for gathering and leveraging web data to cyber-threat intelligence[J]. Electronics, 2021, 10(7): 818.
[75] 孙天放. 基于深度学习的威胁情报信息抽取研究[J]. 现代计算机, 2021(16): 59-64.
SUN T F. Threat intelligence information extraction based on deep learning[J]. Modern Computer, 2021(16): 59-64.
[76] 张红斌, 尹彦, 赵冬梅, 等. 基于威胁情报的网络安全态势感知模型[J]. 通信学报, 2021, 42(6): 182-194.
ZHANG H B, YIN Y, ZHAO D M, et al. Network security situational awareness model based on threat intelligence[J]. Journal on Communications, 2021, 42(6): 182-194.
[77] WAGNER T D, MAHBUB K, PALOMAR E, et al. Cyber threat intelligence sharing: survey and research directions[J]. Computers & Security, 2019, 87: 101589.
[78] YOO S, LEE T. A study of the ordinal scale classification algorithm for cyber threat intelligence based on deception technology[J]. Electronics, 2023, 12(11): 2474.
[79] AMPEL B, SAMTANI S, ZHU H, et al. Labeling hacker exploits for proactive cyber threat intelligence: a deep transfer learning approach[C]//Proceedings of the 2020 IEEE International Conference on Intelligence and Security Informatics (ISI), Arlington, VA, USA, 2020: 1-6.
[80] RAPTIS G E, KATSINI C, ALEXAKOS C, et al. CAVeCTIR: matching cyber threat intelligence reports on connected and autonomous vehicles using machine learning[J]. Applied Sciences, 2022, 12(22): 11631.
[81] ZHAO J, YAN Q, LI J, et al. TIMiner: automatically extracting and analyzing categorized cyber threat intelligence from social data[J]. Computers & Security, 2020, 95: 101867.
[82] YAN J, DU Z, LI J, et al. A threat intelligence analysis method based on feature weighting and BERT-BiGRU for industrial Internet of things[J]. Security and Communication Networks, 2022.
[83] BARNUM S. Common attack pattern enumeration and classification (CAPEC) schema[Z]. Department of Homeland Security, 2008.
[84] RAHMAN M R, WILLIAMS L. From threat reports to continuous threat intelligence: a comparison of attack technique extraction methods from textual artifacts[J]. arXiv:2210.02601, 2022.
[85] AYOADE G, CHANDRA S, KHAN L, et al. Automated threat report classification over multi-source data[C]//Proceedings of the 2018 IEEE 4th International Conference on Collaboration and Internet Computing (CIC), Philadelphia, PA, 2018: 236-245.
[86] LIU J, YAN J, JIANG J, et al. TriCTI: an actionable cyber threat intelligence discovery system via trigger-enhanced neural network[J]. Cybersecurity, 2022, 5(1): 8.
[87] KIM H, KIM H. Comparative experiment on TTP classification with class imbalance using oversampling from CTI dataset[J]. Security and Communication Networks, 2022.
[88] CHAWLA N V, BOWYER K W, HALL L O, et al. SMOTE: synthetic minority over-sampling technique[J]. Journal of Artificial Intelligence Research, 2002, 16: 321-357.
[89] WEI J, ZOU K. Eda: easy data augmentation techniques for boosting performance on text classification tasks[J]. arXiv:1901.11196, 2019.
[90] IRSHAD E, SIDDIQUI A B. Cyber threat attribution using unstructured reports in cyber threat intelligence[J]. Egyptian Informatics Journal, 2023, 24(1): 43-59.
[91] SHIN C, LEE I, CHOI C. Exploiting TTP co-occurrence via GloVe-based embedding with MITRE ATT&CK framework[J]. IEEE Access, 2023, 11: 100823-100831.
[92] SHARMA Y, GIUNCHIGLIA E, BIRNBACH S, et al. To TTP or not to TTP? Exploiting TTPs to improve ML-based malware detection[C]//Proceedings of the 2023 IEEE International Conference on Cyber Security and Resilience (CSR), Venice , 2023: 8-15.
[93] BAGUI S, MINK D, BAGUI S, et al. Detecting reconnaissance and discovery tactics from the MITRE ATT&CK framework in Zeek Conn Logs using Spark’s machine learning in the big data framework[J]. Sensors, 2022, 22(20): 7999.
[94] NOOR U, ANWAR Z, AMJAD T, et al. A machine learning-based FinTech cyber threat attribution framework using high-level indicators of compromise[J]. Future Generation Computer Systems, 2019, 96: 227-242.
[95] YOU Y, JIANG J, JIANG Z, et al. TIM: threat context-enhanced TTP intelligence mining on unstructured threat data[J]. Cybersecurity, 2022, 5(1): 3.
[96] ZHANG H, SHEN G, GUO C, et al. EX-action: automatically extracting threat actions from cyber threat intelligence report based on multimodal learning[J]. Security and Communication Networks, 2021, 2021: 1-12.
[97] ZHU Z, DUMITRA? T. FeatureSmith: automatically engineering features for malware detection by mining the security literature[C]//Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, 2016: 767-778.
[98] ALSAEDI M, GHALEB F A, SAEED F, et al. Cyber threat intelligence-based malicious url detection model using ensemble learning[J]. Sensors, 2022, 22(9): 3373.
[99] IMRAN M, SIDDIQUI H U R, RAZA A, et al. A performance overview of machine learning-based defense strategies for advanced persistent threats in industrial control systems[J]. Computers & Security, 2023, 134: 103445.
[100] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems, 2014, 27.
[101] TANG B, WANG J, QIU H, et al. Attack behavior extraction based on heterogeneous cyberthreat intelligence and graph convolutional networks[J]. Computers, Materials & Continua, 2023, 74(1): 235-252.
[102] LIU C, WANG J, CHEN X. Threat intelligence ATT&CK extraction based on the attention transformer hierarchical recurrent neural network[J]. Applied Soft Computing, 2022, 122: 108826.
[103] YU Z, WANG J F, TANG B H, et al. Tactics and techniques classification in cyber threat intelligence[J]. The Computer Journal, 2023, 66(8): 1870-1881.
[104] ORBINATO V, BARBARACI M, NATELLA R, et al. Automatic mapping of unstructured cyber threat intelligence: an experimental study: (practical experience report)[C]//Proceedings of the 2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE), 2022: 181-192.
[105] 于忠坤, 王俊峰, 唐宾徽, 等. 基于注意力机制和特征融合的网络威胁情报技战术分类研究[J]. 四川大学学报 (自然科学版), 2022, 59(5): 96-103.
YU Z K, WANG J F, TANG B H, et al. Research on the classification of cyber threat intelligence techniques and tactics based on attention mechanism and feature fusion[J]. Journal of Sichuan University (Natural Science Edition), 2022, 59(5): 96-103.
[106] 葛文翰, 王俊峰, 唐宾徽, 等. 基于关联增强的网络威胁情报技战术分类[J]. 四川大学学报 (自然科学版), 2022, 59(2): 100-108.
GE W H, WANG J F, TANG B H, et al. RENet: tactics and techniques classifications for cyber threat intelligence with relevance enhancement[J]. Journal of Sichuan University (Natural Science Edition), 2022, 59(2): 100-108.
[107] LEGOY V, CASELLI M, SEIFERT C, et al. Automated retrieval of att&ck tactics and techniques for cyber threat reports[J]. arXiv:2004.14322, 2020.
[108] CHERQI O, MOUKAFIH Y, GHOGHO M, et al. Enhancing cyber threat identification in open-source intelligence feeds through an improved semi-supervised generative adversarial learning approach with contrastive learning[J]. IEEE Access, 2023, 11: 84440-84452.
[109] GE W, WANG J, LIN T, et al. Explainable cyber threat behavior identification based on self-adversarial topic generation[J]. Computers & Security, 2023, 132: 103369.
[110] PAN J. What in-context learning “learns” in-context: disentangling task recognition and task learning[D]. Princeton: Princeton University, 2023.
[111] CHEN L, CHEN J, GOLDSTEIN T, et al. InstructZero: efficient instruction optimization for black-box large language models[J]. arXiv:2306.03082, 2023.
[112] WHITE J, FU Q, HAYS S, et al. A prompt pattern catalog to enhance prompt engineering with ChatGPT[J]. arXiv:2302.11382, 2023.
[113] ZHOU Y, MURESANU A I, HAN Z, et al. Large language models are human-level prompt engineers[J]. arXiv:2211.
01910, 2022.
[114] HU Z, LAN Y, WANG L, et al. LLM-adapters: an adapter family for parameter-efficient fine-tuning of large language models[J]. arXiv:2304.01933, 2023.
[115] VOS D, D?HMEN T, SCHELTER S. Towards parameter-efficient automation of data wrangling tasks with prefix-tuning[C]//Proceedings of the NeurIPS 2022 First Table Representation Workshop, 2022.
[116] TOPSAKAL O, AKINCI T C. Creating large language model applications utilizing LangChain: a primer on developing LLM apps fast[C]//Proceedings of the International Conference on Applied Engineering and Natural Sciences, 2023: 1050-1056.
[117] DONG Q, LI L, DAI D, et al. A survey for in-context learning[J]. arXiv:2301.00234, 2022.
[118] KRISHNA C S. Prompt generate train (PGT): a framework for few-shot domain adaptation, alignment, and uncertainty calibration of a retriever augmented generation (RAG) model for domain specific open book question-answering[J]. arXiv:2307.05915, 2023.
[119] ?TEFáNIK M, KADL?íK M. Concept-aware training improves in-context learning ability of language models[J]. arXiv:2305.13775, 2023.
[120] ZENG F, GAO W. Prompt to be consistent is better than self-consistent? few-shot and zero-shot fact verification with pre-trained language models[J]. arXiv:2306.02569, 2023.
[121] FERRAG M A, NDHLOVU M, TIHANYI N, et al. Revolutionizing cyber threat detection with large language models[J]. arXiv:2306.14263, 2023.
[122] KOIDE T, FUKUSHI N, NAKANO H, et al. Detecting phishing sites using ChatGPT[J]. arXiv:2306.05816, 2023.
[123] ZHANG Z, ZHANG A, LI M, et al. Automatic chain of thought prompting in large language models[J]. arXiv:2210.03493, 2022.
[124] ZHANG Z, ZHANG A, LI M, et al. Multimodal chain-of-thought reasoning in language models[J]. arXiv:2302.00923, 2023.
[125] CHARAN P V, CHUNDURI H, ANAND P M, et al. From text to MITRE techniques: exploring the malicious use of large language models for generating cyber attack payloads[J]. arXiv:2305.15336, 2023.
[126] LIU Y, DENG G, XU Z, et al. Jailbreaking ChatGPT via prompt engineering: an empirical study[J]. arXiv:2305.13860, 2023.
[127] SHEN X, CHEN Z, BACKES M, et al. “Do anything now”: characterizing and evaluating in-the-wild jailbreak prompts on large language models[J]. arXiv:2308.03825, 2023.
[128] BHARDWAZ S, KUMAR J. An extensive comparative analysis of Chatbot technologies-ChatGPT, Google BARD and Microsoft Bing[C]//Proceedings of the 2023 2nd International Conference on Applied Artificial Intelligence and Computing (ICAAIC), 2023: 673-679.
[129] SIRACUSANO G, SANVITO D, GONZALEZ R, et al. Time for aCTIon: automated analysis of cyber threat intelligence in the wild[J]. arXiv:2307.10214, 2023.
[130] FAYYAZI R, YANG S J. On the uses of large language models to interpret ambiguous cyberattack descriptions[J]. arXiv:2306.14062, 2023.
[131] LIU Y, OTT M, GOYAL N, et al. RoBERTa: a robustly optimized BERT pretraining approach[J]. arXiv:1907.11692, 2019.
[132] BAYER M, FREY T, REUTER C. Multi-level fine-tuning, data augmentation, and few-shot learning for specialized cyber threat intelligence[J]. Computers & Security, 2023, 134: 103430.
[133] DAI H, LIU Z, LIAO W, et al. AugGPT: leveraging ChatGPT for text data augmentation[J]. arXiv:2302.13007, 2023.
[134] BROWN T, MANN B, RYDER N, et al. Language models are few-shot learners[C]//Advances in Neural Information Processing Systems, 2020, 33: 1877-1901.
[135] PICUS Security. The red report 2023[EB/OL]. [2023-08-18]. https://www.picussecurity.com/resource/report/the-red-report-2023.
[136] AGHAEI E, NIU X, SHADID W, et al. SecureBERT: a domain-specific language model for cybersecurity[C]//Proceedings of the International Conference on Security and Privacy in Communication Systems. Cham: Springer, 2022: 39-56.
[137] ZXHANG Y X, HAXO Y M, MAT Y X. Falcon LLM: a new frontier in natural language processing[J]. AC Investment Research Journal, 2023, 220(44).
[138] FERRAG M A, BATTAH A, TIHANYI N, et al. SecureFalcon: the next cyber reasoning system for cyber security[J]. arXiv:2307.06616, 2023.
[139] FERRAG M A, FRIHA O, HAMOUDA D, et al. Edge-IIoTset: a new comprehensive realistic cyber security dataset of IoT and IIoT applications for centralized and federated learning[J]. IEEE Access, 2022, 10: 40281-40306.
[140] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems, 2017, 30.
[141] YE J, CHEN X, XU N, et al. A comprehensive capability analysis of GPT-3 and GPT-3. 5 series models[J]. arXiv:2303.10420, 2023.
[142] TONY C, MUTAS M, FERREYRA N E D, et al. LLMSec-
Eval: a dataset of natural language prompts for security evaluations[J]. arXiv:2303.09384, 2023.
[143] YANG R, SONG L, LI Y, et al. Gpt4tools: teaching large language model to use tools via self-instruction[J]. arXiv:2305.18752, 2023.
[144] ZHANG R, HAN J, ZHOU A, et al. LLaMA-adapter: efficient fine-tuning of language models with zero-init attention[J]. arXiv:2303.16199, 2023.
[145] ZENG A, LIU X, DU Z, et al. Glm-130b: an open bilingual pre-trained model[J]. arXiv:2210.02414, 2022.
[146] WEN J Z H, DENG L T, XIN M F, et al. HackMentor: fine-tuning large language models for cybersecurity[C]//Proceedings of the 2023 IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), 2023.
[147] CHAI Y, ZHOU Y, LI W, et al. An explainable multi-modal hierarchical attention model for developing phishing threat intelligence[J]. IEEE Transactions on Dependable and Secure Computing, 2021, 19(2): 790-803.
[148] REN Y, XIAO Y, ZHOU Y, et al. CSKG4APT: a cybersecurity knowledge graph for advanced persistent threat organization attribution[J]. IEEE Transactions on Knowledge and Data Engineering, 2023, 35(6): 5695-5709.
[149] CHEN Y, GUO S, LIU K, et al. Large language models and knowledge graphs[C]//Proceedings of the 22nd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum), 2023: 67-76.
[150] LI Z X, LI Y J, LIU Y W, et al. K-CTIAA: automatic analysis of cyber threat intelligence based on a knowledge graph[J]. Symmetry, 2023, 15(2): 337.
[151] WANG Y, PAN Y, YAN M, et al. A survey on ChatGPT: AI-generated contents, challenges, and solutions[J]. arXiv:2305.18339, 2023.
[152] WU J, GAN W, CHEN Z, et al. AI-generated content (AIGC): a survey[J]. arXiv:2304.06632, 2023.
[153] 崔琳, 杨黎斌, 何清林, 等. 基于开源信息平台的威胁情报挖掘综述[J]. 信息安全学报, 2022, 7(1): 1-26.
CUI L, YANG L B, HE Q L, et al. Survey of cyber threat intelligence mining based on open source information platform[J]. Journal of Cyber Security, 2022, 7(1): 1-26.