Survey of Research on Neural Network Verification and Testing Technology

doi:10.3778/j.issn.1002-8331.2106-0342

Abstract

Abstract:

Neural network technology has made remarkable achievements in the fields of image processing, text analysis and speech recognition. With the application of neural network technology to some security related fields, how to ensure the quality of these software applications is particularly important. Software based on neural network technology is essentially different from traditional software in development and programming. Traditional testing technology is difficult to be directly applied to this kind of software. It is necessary to study the verification and testing evaluation technology for neural network. To effectively evaluate and test neural networks, this paper reviews the research status of neural network verification and testing technology, summarizes and classifies the verification technology, testing technology based on coverage, testing technology based on adversarial sample, and fusing traditional testing technology. The basic idea and implementation of some key technologies are briefly introduced, and some testing frameworks and tools are listed. The challenges of neural network verification and testing are summarized, which can provide reference for researchers in this field.

Key words: neural networks, test evaluation, verification technology, testing technology

摘要：

神经网络技术在图像处理、文本分析和语音识别等领域取得了令人瞩目的成就，随着神经网络技术应用到一些安全攸关的领域，如何保证这些软件应用的质量就显得尤为重要。基于神经网络技术的软件在开发和编程上和传统软件有着本质的区别，传统测试技术很难直接应用到此类软件中，研究针对神经网络的验证和测试评估技术十分必要。从有效评估和测试神经网络出发，对神经网络验证和测试技术的研究现状进行梳理，分别从验证技术、基于覆盖的测试技术、基于对抗样本的测试技术、融合传统测试技术等方面进行了归纳和分类。对其中一些关键技术的基本思想和实现做了简明扼要的介绍，并列举了一些测试框架和工具，总结了神经网络验证和测试工作面临的挑战，为该领域的研究人员提供参考。

关键词: 神经网络, 测试评估, 验证技术, 测试技术

LI Duo, DONG Chaoqun, SI Pinchao, HE Man, LIU Qianchao. Survey of Research on Neural Network Verification and Testing Technology[J]. Computer Engineering and Applications, 2021, 57(22): 53-67.

李舵，董超群，司品超，何曼，刘钱超. 神经网络验证和测试技术研究综述[J]. 计算机工程与应用, 2021, 57(22): 53-67.

References

[1] MASUDA S，ONO K，YASUE T，et al.A survey of software quality for machine learning applications[C]//2018 IEEE International Conference on Software Testing，Verification and Validation Workshops，2018：279-284.
[2] ZHANG J M，HARMAN M，MA L，et al.Machine learning testing：survey，landscapes and horizons[J].arXiv：1906.10742，2019.
[3] MARIJAN D，GOTLIEB A.Software testing for machine learning[C]//34th AAAI Conference on Artificial Intelligence，2020：13576-13582.
[4] HUANG X W，KROENING D，RUAN W J，et al.A survey of safety and trustworthiness of deep neural networks：verification，testing，adversarial attack and defence，and interpretability[J].Computer Science Review，2020，37：100270.
[5] 王赞，闫明，刘爽，等.深度神经网络测试研究综述[J].软件学报，2020，31（5）：1255-1275.
WANG Z，YAN M，LIU S，et al.Survey on testing of deep neural networks[J].Journal of Software，2020，31（5）：1255-1275.
[6] 曾春艳，严康，王志锋，等.深度学习模型可解释性研究综述[J].计算机工程与应用，2021，57（8）：1-9.
ZENG C Y，YAN K，WANG Z F，et al.Survey of interpretability research on deep learning models[J].Computer Engineering and Applications，2021，57（8）：1-9.
[7] RUAN W，HUANG X，KWIATKOWSKA M，et al.Reachability analysis of deep neural networks with provable guarantees[C]//27th International Joint Conference on Artificial Intelligence，2018：2651-2659.
[8] HUANG X，KWIATKOWSKA M，WANG S，et al.Safety verification of deep neural networks[C]//International Conference on Computer Aided Verification，2018：3-29.
[9] KATZ G，BARRETT C，DILL D，et al.Reluplex：an efficient SMT solver for verifying deep neural networks[C]//International Conference on Computer Aided Verification，2017：97-117.
[10] EHLERS R.Formal verification of piece-wise linear feed-forward neural networks[C]//5th International Symposium on Automated Technology for Verification and Analysis，2017：269-286.
[11] CHENG C H，NüHRENBERG G，HUANG C H，et al.Verification of binarized neural networks via inter-neuron factoring[C]//10th International Conference on Verified Software：Theories，Tools，and Experiments，2018：279-290.
[12] NARODYTSKA N，KASIVISWANATHAN S，RYZHYK L，et al.Verifying properties of binarized deep neural networks[C]//32nd AAAI Conference on Artificial Intelligence，2018：6615-6624.
[13] NARODYTSKA N.Formal analysis of deep binarized neural networks[C]//27th International Joint Conference on Artificial Intelligence，2018：5692-5696.
[14] LOMUSCIO A，MAGANTI L.An approach to reachability analysis for feed-forward ReLU neural networks[J].arXiv：1706.07351，2017.
[15] BUNEL R，TURKASLAN I，PHILIP H，et al.A unified view of piecewise linear neural network verification[C]//32nd Conference on Neural Information Processing Systems，2018：4795-4804.
[16] BUNEL R，LU J M，TURKASLAN I，et al.Branch and bound for piecewise linear neural network verification[J].Journal of Machine Learning Research，2020，21：42.
[17] CHENG C H，NüHRENBERG G，RUESS H.Maximum resilience of artificial neural networks[C]//15th International Symposium on Automated Technology for Verification and Analysis，2017：251-268.
[18] DUTTA S，JHA S，SANAKARANARAYANAN S，et al.Output range analysis for deep neural networks[J].arXiv：1709.09130，2018.
[19] TJENG V，KAI Y，TEDRAKE R.Evaluating robustness of neural networks with mixed integer programming[C]//7th International Conference on Learning Represrntations，2019.
[20] COUSOT P，COUSOT R.Abstract interpretation：a unified lattice model for static analysis of programs by constructionor approximation of fixpoints[C]//4th ACM Symposium on Principles of Programming Languages，1977：238-252.
[21] GEHR T，MIRMAN M，DRACHSLER D，et al.AI2：safety and robustness certification of neural networks with abstract interpretation[C]//2018 IEEE Symposium on Security and Privacy，2018：948-963.
[22] SINGH G，GEHR T，MIRMAN M，et al.Fast and effective robustness certification[C]//32nd Conference on Neural Information Processing Systems，2018.
[23] SINGH G，GEHR T，PüSCHEL M，et al.An abstract domain for certifying neural networks[C]//ACM Symposium on Principles of Programming Languages，2019.
[24] MIRMAN M，GEHR T，VECHEV M.Differentiable abstract interpretation for provably robust neural networks[C]//35th International Conference on Machine Learning，2018：3578-3586.
[25] LI J，LIU J，YANG P，et al.Analyzing deep neural networks with symbolic propagation：towards higher precision and faster verification[C]//International Static Analysis Symposium，2019：296-319.
[26] WENG T W，ZHANG H，CHEN H.Towards fast computation of certified robustness for Relu networks[C]//35th International Conference on Machine Learning，2018：5273-5282.
[27] ZHANG H，WENG T W，CHEN P Y.Efficient neural network robustness certi?cation with general activation functions[C]//32nd Conference on Neural Information Processing Systems，2018.
[28] BOOPATHY A，WENG T W，CHEN P Y.CNN-Cert：an efficient framework for certifying robustness of convolutional neural networks[C]//33rd AAAI Conference on Arti?cial Intelligence，2019：3240-3247.
[29] DVIJOTHAM K，STANFORTH R，GOWAL S，et al.A dual approach to scalable verification of deep networks[C]//34th Conference on Uncertainty in Artificial Intelligence，2018：550-559.
[30] WONG E，KOLTER Z.Provable defenses against adversarial examples via the convex outer adversarial polytope[C]//35th International Conference on Machine Learning，2018：5283-5292.
[31] RAGHUNATHAN A，STEINHARDT J，LIANG P.Semidefinite relaxations for certifying robustness to adversarial examples[C]//32nd Conference on Neural Information Processing Systems，2018.
[32] RICHARD Y，ZHANG Y.On the tightness of semidefinite relaxations for certifying robustness to adversarial examples[C]//34th Conference on Neural Information Processing Systems，2020.
[33] HEIN M，ANDRIUSHCHENKO M.Formal guarantees on the robustness of a classifier against adversarial manipulation[C]//31st International Conference on Neural Information Processing Systems，2017：2263-2273.
[34] WENG T W，ZHANG H，CHEN P Y，et al.Evaluating the robustness of neural networks：an extreme value theory approach[C]//6th International Conference on Learning Representations，2018.
[35] WU M，WICKER M，RUAN W，et al.A game-based approximate verification of deep neural networks with provable guarantees[J].Theoretical Computer Science，2020，807：298-329.
[36] RUAN W，HUANG X，KWIATKOWSKA M.Reachability analysis of deep neural networks with provable guarantees[C]//28th International Joint Conference on Artificial Intelligence for Improving Human Well-being，2018：5944-5952.
[37] WANG S Q，PEI K X，WHITEHOUSE J，et al.Efficient formal safety analysis of neural networks[C]//32nd Conference on Neural Information Processing Systems，2018.
[38] SINGH G，GEHR T，PUSCHEL M.Boosting robustness certification of neural networks[C]//2019 International Conference on Learning Representations，2019.
[39] PEI K X，CAO Y Z，YANG J F，et al.DeepXplore：automated whitebox testing of deep learning systems[C]//26th Symposium on Operating Systems Principles，2017：1-18.
[40] TIAN Y C，PEI K X，JANA S，et al.DeepTest：automated testing of deep-neural-network-driven autonomous cars[C]//40th International Conference on Software Engineering，2018：303-314.
[41] SUN Y，HUANG X，KROENING D.Testing deep neural networks[J].arXiv：1803.04792，2018.
[42] MA L，XU J F，ZHANG F Y，et al.DeepGauge：multi-granularity testing criteria for deep learning systems[C]//33rd ACM/IEEE International Conference on Automated Software Engineering，2018：120-131.
[43] MA L，XU J F，XUE M，et al.DeepCT：tomographic combinatorial testing for deep learning systems[C]//2019 26th International Conference on Software Analysis，Evalution and Reengineering，2019：614-618.
[44] LEE S，CHA S，LEE D，et al.Effective white-box testing of deep neural networks with adaptive neuron-selection strategy[C]//29th ACM SIGSOFT International Symposium on Software Testing and Analysis，2020：165-176.
[45] XIE X F，MA L，XU J F，et al.DeepHunter：a coverage-guided fuzz testing framework for deep neural networks[C]//28th ACM SIGSOFT International Symposium on Software Testing and Analysis，2019：146-157.
[46] KIM J，FELDT R，YOO S.Guiding deep learning system testing using surprise adequacy[C]//41st International Conference on Software Engineering，2019：1039-1049.
[47] WANG D，WANG Z，FANG C，et al.DeepPath：path-driven testing criteria for deep neural networks[C]//2019 IEEE International Conference on Artificial Intelligence Testing，2019：119-120.
[48] TIAN Y，ZHONG Z，ORDONEZ V，et al.Testing deep neural network based image classifiers[J].arXiv：1905.07831，
2019.
[49] DU X，XIE X，LI Y，et al.DeepCruiser：automated guided testing for stateful deep learning systems[J].arXiv：1812.
05339，2018.
[50] CHENG C H，HUANG C H，YASUOKA H.Quantitative projection coverage for testing ML-enabled autonomous systems[C]//International Symposium on Automated Technology for Verification and Analysis，2018.
[51] DONG Y Z，ZHANG P Y，WANG J Y，et al.There is limited correlation between coverage and robustness for deep neural networks[J].arXiv：1911.05904，2019.
[52] LI Z N，MA X X，XU C，et al.Structural coverage criteria for neural networks could be misleading[C]//41st International Conference on Software Engineering，2019：89-92.
[53] HAREL F，WANG L X，GULZAR M，et al.Is neuron coverage a meaningful measure for testing deep neural networks?[C]//28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering，2020：851-862.
[54] YAN S，TAO G，LIU X，et al.Correlations between deep neural network model coverage criteria and model quality[C]//28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering，2020：775-787.
[55] SZEGEDY C，ZAREMBA W，SUTSKEVER I，et al.Intriguing properties of neural networks[C]//2014 International Conference on Learning Representations，2014.
[56] GOODFELLOW I，SHLENS J，SZEGEDY C.Explaining and harnessing adversarial examples[C]//2015 International Conference on Learning Representations，2015.
[57] KURAKIN A，GOODFELLOW I，BENGIO S.Adversarial examples in the physical world[C]//2017 International Conference on Learning Representations，2017.
[58] DONG Y，LIAO F，PANG T，et al.Boosting adversarial attacks with momentum[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition，2018：9185-9193.
[59] PAPERNOT N，MCDANIEL P，JHA S，et al.The limitations of deep learning in adversarial settings[C]//2016 IEEE European Symposium on Security and Privacy，2016：372-387.
[60] MOOSAVIDEZFOOLI S M，FAWZI A，FROSSARD P.DeepFool：a simple and accurate method to fool deep neural networks[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition，2016：2574-2582.
[61] CARLINI N，WAGNER D.Towards evaluating the robustness of neural networks[C]//2017 IEEE Symposium on Security and Privacy，2017.
[62] PAPERNOT N，MCDANIEL P，GOODFELLOW I，et al.Practical black-box attacks against machine learning[C]//2017 ACM Asia Conference on Computer and Communications Security，2017：506-519.
[63] NARODYTSKA N，KASIVISWANATHAN S.Simple black-box adversarial attacks on deep neural networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops，2017：1310-1318.
[64] SU J，VARGAS D V，SAKURAI K.One pixel attack for fooling deep neural networks[J].IEEE Transactions on Evolutionary Computation，2019，23（5）：828-841.
[65] ZHANG F Y，CHOWDHURY S，CHRISTAKIS M.DeepSearch：a simple and effective blackbox attack for deep neural networks[C]//28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering，2020：800-812.
[66] ENGSTROM L，TSIPRAS D，SCHMIDT L，et al.A rotation and a translation suffice：fooling CNNs with simple transformations[J].arXiv：1712.02779，2017.
[67] XIAO C，ZHU J Y，LI B，et al.Spatially transformed adversarial examples[C]//2018 International Conference on Learning Representations，2018.
[68] WANG Z，ZHENG M，SONG M，et al.advPattern：physical world attacks on deep person re-identification via adversarially transformable patterns[C]//2019 International Conference on Computer Vision，2019：8340-8349.
[69] PAPERNOT N，FAGHRI F，CARLINI N，et al.Technical report on the CleverHans v2.1.0 adversarial examples library[J].arXiv：1610.00768，2018.
[70] GOODMAN D，XIN H，YANG W，et al.Advbox：a toolbox to generate adversarial examples that fool neural networks[J].arXiv：2001.05574，2020.
[71] NICOLAE M，SINN M，TRAN M，et al.Adversarial robustness toolbox v1.0.0[J].arXiv：1807.01069，2018.
[72] LING X，JI S L，ZOU J X，et al.DEEPSEC：a uniform platform for security analysis of deep learning models[C]//2019 IEEE Symposium on Security and Privacy，2019：673-690.
[73] ZENG G Y，QI F C，ZHOU Q R，et al.OpenAttack：an open-source textual adversarial attack toolkit[J].arXiv：2009.09191，2020.
[74] GUO J，JIANG Y，ZHAO Y，et al.DLFuzz：differential fuzzing testing of deep learning systems[C]//2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering，2018：739-743.
[75] ODENA A，OLSSON C，ANDERSEN D，et al.TensorFuzz：debugging neural networks with coverage-guided fuzzing[C]//2019 International Conference on Machine Learning，2019：4901-4911.
[76] ZHANG P Y，DAI Q Y，PELLICCIONE P.CAGFuzz：coverage-guided adversarial generative fuzzing testing of deep learning systems[J].arXiv：1911.07931，2019.
[77] DING J，KANG X，HU X H.Validating a deep learning framework by metamorphic testing[C]//IEEE/ACM 2nd International Workshop on Metamorphic Testing，2017：28-34.
[78] DWARAKANATH A，AHUJA M，SIKAND S，et al.Identifying implementation bugs in machine learning based image classifiers using metamorphic testing[C]//27th ACM SIGSOFT International Symposium on Software Testing and Analysis，2018：118-128.
[79] ZHU H，LIU D，BAYLEY I，et al.Datamorphic testing：a method for testing intelligent applications[C]//2019 International Conference on Artificial Intelligence Testing，2019：149-156.
[80] SHARMA A，WEHRHEIM H.Testing machine learning algorithms for balanced data usage[C]//12th IEEE Conference on Software Testing，Validation and Verification，2019：125-135.
[81] 刘佳洛，姚奕，黄松，等.机器学习图像分类程序的蜕变测试框架[J].计算机工程与应用，2020，56（17）：69-77.
LIU J L，YAO Y，HUANG S，et al.Metamorphic testing framework for machine learning image classification program[J].Computer Engineering and Applications，2020，56（17）：69-77.
[82] MA L，ZHANG F，SUN J，et al.DeepMutation：mutation testing of deep learning systems[C]//IEEE 29th International Symposium on Software Reliability Engineering，2018：100-111.
[83] SHEN W，WAN J，CHEN Z.MuNN：mutation analysis of neural networks[C]//2018 IEEE International Conference on Software Quality，Reliability and Security Companion，2018：108-115.
[84] JAHANGIROVA G，TONELLA P.An empirical evaluation of mutation operators for deep learning systems[C]//13th International Conference on Software Testing，Validation and Verification，2020：74-84.
[85] SUN Y，WU M，RUAN W，et al.Concolic testing for deep neural networks[C]//33rd ACM/IEEE International Conference on Automated Software Engineering，2018：109-119.
[86] GOPINATH D，WANG K，ZHANG M，et al.Symbolic execution for deep neural networks[J].arXiv：1807.10439，2018.
[87] AGARWAL A，LOHIA P，NAGAR S，et al.Automated test generation to detect individual discrimination in AI models[J].arXiv：1809.03260，2018.