Survey of Deep Learning Applied in Code Representation

doi:10.3778/j.issn.1002-8331.2106-0368

Abstract

Abstract:

Source code representation is an important technology of code numerization, which is the foundation of code cloning detection, code recommendation, code plagiarism and other applications in software engineering domain. It helps programmers to generate or analyze code. It has become a core technology and a hot topic in the field of software engineering. Researchers have conducted a series of researches on code representation. The methods can be divided into text-based representation, syntactic based representation, semantic based representation and function based representation according to different ways of using code information, can be divided into words based representation, statement based representation and function based representation; according to representation granularity, and can be divided into statistical based model, natural language based model and deep learning based representation according to representation methods. In this paper, it first investigates the recent research work of deep learning based code representation which maps source code into a set of continuous space vectors to extract the underlying intrinsic properties. Then it discusses the granularity of representation, abstract level, representation model and application. Finally, this paper summarizes the future development trend of deep learning based code representation.

Key words: deep learning, code representation, representation model, representation granularity

摘要：

代码表征是对代码数值化的一种技术，把代码映射为一组连续的实值向量，提取隐藏在代码内部的属性，辅助程序员生成或分析代码，是代码克隆、代码推荐、代码剽窃等软件工程任务的核心技术和研究热点。研究者们对代码表征方面进行了一系列研究，根据源代码抽取信息的方式，分为基于文本的表征、基于语法的表征、基于语义的表征和基于功能的表征；根据表征粒度的大小，分为基于词汇的表征、基于语句的表征、基于函数的表征等不同等级；根据表征方法的不同，分为基于统计的模型、基于自然语言的模型和基于深度学习的模型。对近几年基于深度学习的代码表征研究进展进行了综述，并从表征粒度、表征层次、表征模型、应用场景等方面对现有工作进行了概括、比较和分析。对基于深度学习的代码表征的未来发展趋势进行分析和展望。

关键词: 深度学习, 代码表征, 表征模型, 表征粒度

XIE Chunli, LIANG Yao, WANG Xia. Survey of Deep Learning Applied in Code Representation[J]. Computer Engineering and Applications, 2021, 57(20): 53-63.

谢春丽，梁瑶，王霞. 深度学习在代码表征中的应用综述[J]. 计算机工程与应用, 2021, 57(20): 53-63.

References

[1] JIANG L，MISHERGHI G，SU Z，et al.Deckard：scalable and accurate tree-based detection of code clones[C]//Proc of the 29th International Conference on Software Engineering.Piscataway，NJ：IEEE，2007：20-26.
[2] HINDLE A，BARR E T，SU Z，et al.On the naturalness of software[C]//Proc of the 29th International Conference on Software Engineering.Piscataway，NJ：IEEE，2007：837-847.
[3] WHITE M，TUFANO M，VENDOME C，et al.Deep learning code fragments for code clone detection[C]//Proc of the 31st IEEE/ACM International Conference on Automated Software Engineering.Piscataway，NJ：IEEE，2016：87-98.
[4] HARER J，KIM L，RUSSELL R L，et al.Automated software vulnerability detection with machine learning[J].arXiv：1803.04497，2018.
[5] KAMIYA T，KUSUMOTO S，INOUE K，et al.CCFinder：a multilinguistic token-based code clone detection system for large scale source code[J].IEEE Transactions on Software Engineering，2002，28（7）：654-670.
[6] LI Z，LU S，MYAGMAR S，et al.CP-Miner：finding copy-paste and related bugs in large-scale software code[J].IEEE Transactions on Software Engineering，2006，32（3）：176-192.
[7] NGUYEN S，NGUYEN T N，LI Y，et al.Combining program analysis and statistical language model for code statement completion[C]//Proc of the 34th IEEE/ACM International Conference on Automated Software Engineering.Piscataway，NJ：IEEE，2019：710-721.
[8] DAM H K，TRAN T，GRUNDY J，et al.DeepSoft：a vision for a deep model of software[C]//Proc of ACM SIGSOFT International Symposium on Foundations of Software Engineering.New York，NY：ACM，2016：944-947.
[9] BEN-NUN T，JAKOBOVITS A S，HOEFLER T，et al.Neural code comprehension：a learnable representation of code semantics[C]//Proc of the 32nd Conference on Neural Information Processing Systems.Cambridge，MA：MIT Press，2018：3585-3597.
[10] ZHANG J，WANG X，ZHANG H，et al.A novel neural source code representation based on abstract syntax tree[C]//Proc of the IEEE/ACM 41st International Conference on Software Engineering.Piscataway，NJ：IEEE，2019.
[11] PENG H，MOU L，LI G，et al.Building program vector representations for deep learning[C]//International Conference on Knowledge Science，Engineering and Management.Berlin：Springer，2016：547-553.
[12] DEFREEZ D，THAKUR A，RUBIOGONZALEZ C，et al.Path-based function embedding and its application to specification mining[C]//Proc of ACM SIGSOFT International Symposium on Foundations of Software Engineering.NewYork：ACM，2018：423-433.
[13] ARROYOFERNANDEZ I，MENDEZCRUZ C，SIERRA G，et al.Unsupervised sentence representations as word information series：revisiting TF-IDF[J].Computer Speech & Language，2019，56：107-129.
[14] SAJNANI H，SAINI V，SVAJLENKO J，et al.SourcererCC：scaling code clone detection to big-code[C]//Proc of the 38th International Conference on Software Engineering.Piscataway，NJ：IEEE，2016：1157-1168.
[15] 刘芳，李戈，胡星，等.基于深度学习的程序理解研究进展[J].计算机研究与发展，2019，56（8）：1605-1620.
LIU F，LI G，HU X，et al.Program comprehension based on deep learning[J].Journal of Computer Research and Develpoment，2019，56（8）：1605-1620.
[16] BAXTER I D，YAHIN A，DE M L，et al.Clone detection using abstract syntax trees[C]//Proc of International Conference on Software Maintenance.Piscataway，NJ：IEEE，1998：368-377.
[17] KOMONDOOR R，HORWITZ S.Using slicing to identify duplication in source code[C]//Proc of International Symposium on Static Analysis.Berlin：Springer，2001：40-56.
[18] KRINKE J.Identifying similar code with program dependence graphs[C]//Proc of the 8th Working Conf on Reverse Engineering.Piscataway，NJ：IEEE，2001：301-309.
[19] ALON U，ZILBERSTEIN M，LEVY O，et al.Code2vec：learning distributed representations of code[C]//Proc of ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages.New York：ACM，2019.
[20] ALON U，ZILBERSTEIN M，LEVY O，et al.A general path-based representation for predicting program properties[C]//Proc of Programming Language Design and Implementation.New York，NY：ACM，2018：404-419.
[21] HINDLE A，BARR E T，SU Z，et al.On the naturalness of software[J].Communications of the ACM，2016，59（5）：122-131.
[22] VINCENT H，PREMKUMAR D.Are deep neural networks the best choice for modeling source code?[C]//Proc of the 11th Joint Meeting on Foundations of Software Engineering，Paderborn，Germany.New York，ACM，2017：763-773.
[23] ALLAMANIS M，BARR E T，DEVANBU P，et al.A survey of machine learning for big code and naturalness[J].ACM Computing Surveys，2018，51（4）：1-37.
[24] ALLAMANIS M，BROCKSCHMIDT M，KHADEMI M，et al.Learning to represent programs with graphs[C]//Proc of the 6th International Conference on Learning Representations，2018.
[25] KARAMPATSIS R M，SUTTON C.Maybe deep neural networks are the best choice for modeling source code[J].arXiv：1903.05734，2019.
[26] BABII H，JANES A，ROBBES R.Modeling vocabulary for big code machine learning[J].arXiv：1904.01873v1，2019.
[27] WAINAKH Y，RAUF M，PRADEL M.Evaluating semantic representations of source code[J].arXiv：1910.05177，2019.
[28] WANG K，CHRISTODORESCU M.Coset：a benchmark for evaluating neural program embeddings[J].arXiv：1905.
11445，2019.
[29] EFSTATHIOU V，SPINELLIS D.Semantic source code models using identifier embeddings[C]//Proc of 16th International Conference on Mining Software Repositories.Piscataway，NJ：IEEE，2019：29-33.
[30] NGUYEN T T，NGUYEN A T，NGUYEN H A，et al.A statistical semantic language model for source code[C]//Proc of ACM SIGSOFT Symposium on the Foundation of Software Engineering.New York，NY，ACM，2013：532-542.
[31] NGUYEN A T，NGUYEN T T，NGUYEN T N，et al.Lexical statistical machine translation for language migration[C]//Proc of ACM SIGSOFT Symposium on the Foundation of Software Engineering.New York，NY：ACM，2013：651-654.
[32] NGUYEN A T，NGUYEN T T，NGUYEN T N，et al.Migrating code with statistical machine translation[C]//Proc of 36th International Conference on Software Engineering.Piscataway，NJ：IEEE，2014：544-547.
[33] NGUYEN A T，NGUYEN H A，NGUYEN T T，et al.Statistical learning approach for mining API usage mappings for code migration[C]//Proc of 29th IEEE/ACM International Conference on Automated Software Engineering.Piscataway，NJ：IEEE，2014：457-468.
[34] TU Z，SU Z，DEVANBU P，et al.On the localness of software[C]//Proc of the ACM SIGSOFT Symposium on the Foundation of Software Engineering.New York，NY：ACM，2014：269-280.
[35] FRANKS C，TU Z，DEVANBU P，et al.CACHECA：a cache language model based code suggestion tool[C]//Poc of the 37th International Conference on Software Engineering.Piscataway，NJ：IEEE，2015：705-708.
[36] ALLAMANIS M，SUTTON C.Mining source code repositories at massive scale using language modeling[C]//Proc of the 10th Working Conference on Mining Software Repositories.Piscataway，NJ：IEEE，2013：207-216.
[37] ALLAMANIS M，BARR E T，BIRD C，et al.Suggesting accurate method and class names[C]//Proc of the ACM SIGSOFT Symposium on the Foundations of Software Engineering.New York，NY：ACM，2015：38-49.
[38] RAYCHEV V，VECHEV M，YAHAV E，et al.Code completion with statistical language models[C]//Proc of ACM SIGPLAN Symposium on Programming Language Design & Implementation.New York，NY：ACM，2014：419-428.
[39] YANG Y，JIANG Y，GU M，et al.A language model for statements of software code[C]//Proc of the 32nd IEEE/ACM International Conference on Automated Software Engineering.New York，NY：ACM，2017：682-687.
[40] KARNALIM O.Syntax trees and information retrieval to improve code similarity detection[C]//Proc of 22nd Australasian Computing Education Conference.New York，NY：ACM，2020：48-55.
[41] ALON U，BRODY S，LEVY O，et al.code2seq：generating sequences from structured representations of code[C]//Proc of the 7th International Conference on Learning Representations，2019.
[42] WHITE M，VENDOME C，LINARESVASQUEZ M，et al.Toward deep learning software repositories[C]//Proc of the 12th IEEE Working Conference on Mining Software Repositories.Piscataway，NJ：IEEE，2015：334-345.
[43] MOU L，LI G，ZHANG L，et al.Convolutional neural networks over tree structures for programming language processing[C]//Proc of the 30th AAAI Conference on Artificial Intelligence.Palo Alto，CA：AAAI Press，2014：1287-1293.
[44] ALLAMANIS M，PENG H，SUTTON C，et al.A convolutional attention network for extreme summarization of source code[C]//Proc of the 33rd International Conference on Machine Learning.New York，NY：ACM，2016：2091-2100.
[45] NGUYEN A T，NGUYEN T D，PHAN H D，et al.A deep neural network language model with contexts for source code[C]//Proc of IEEE International Conference on Software Analysis Evolution and Reengineering.Piscataway，NJ：IEEE，2018：323-334.
[46] ZHAO G，HUANG J.DeepSim：deep learning code functional similarity[C]//Proc of the 26th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering.New York，NY：ACM，2018：141-151.
[47] WAN Y，SHU J，SUI Y，et al.Multi-modal attention network learning for semantic source code retrieval[C]//Proc of the 34th IEEE/ACM International Conference on Automated Software Engineering.Piscataway，NJ：IEEE，2019：13-25.
[48] TUFANO M，WATSON C，BAVOTA G，et al.Deep learning similarities from different representations of source code[C]//Proc of the 15th International Conference on Mining Software Repositories.Piscataway，NJ：IEEE，2018：542-553.
[49] WHITE M，TUFANO M，MARTINEZ M，et al.Sorting and transforming program repair ingredients via deep learning code similarities[C]//Proc of the 26th International Conference on Software Analysis，Evolution and Reengineering.Piscataway，NJ：IEEE，2019：479-490.
[50] WANG S，LIU T，TAN L，et al.Automatically learning semantic features for defect prediction[C]//Proc of the 38th IEEE International Conference on Software Engineering.Piscataway，NJ：IEEE，2016：297-308.
[51] HELLENDOORN V J，SUTTON C，SINGH R，et al.Global relational models of source code[C]//Proc of the 8th International Conference on Learning Representations，2020.
[52] SHI Z，SWERSKY K，TARLOW D，et al.Learning execution through neural code fusion[C]//Proc of the 8th International Conference on Learning Representations，2020.
[53] 杨博，张能，李善平，等.智能代码补全研究综述[J].软件学报，2020，31（5）：1435-1453.
YANG B，ZHANG N，LI S P，et al.Survey of intelligent code completion[J].Journal of Software，2020，31（5）：1435-1453.
[54] KANG H J，BISSYANDE T F，Lo D，et al.Assessing the generalizability of code2vec token embeddings[C]//Proc of the 34th ACM/IEEE International Conference on Automated Software Engineering.Piscataway，NJ：IEEE，2019：1-12.
[55] FENG Z，GUO D，TANG D，et al.CodeBERT：a pre-trained model for programming and natural languages[C]//Findings of the 2020 Conference on Empirical Methods in Natural Language Processing，2020.