Review of Research on Generative Adversarial Networks and Its Application

doi:10.3778/j.issn.1002-8331.2104-0248

Abstract

Abstract:

The theoretical research and applications of generative adversarial networks have been continuously successful and have become one of the current hot spots of research in the field of deep learning. This paper provides a systematic review of the theory of generative adversarial networks and their applications in terms of types of models, evaluation criteria and theoretical research progress; analyzing the strengths and weaknesses of generative models with explicit and implicit density-based, respectively; summarizing the evaluation criteria of generative adversarial networks, interpreting the relationship between the criteria, and introduces the research progress of the generative adversarial network in image generation from the application level, that is, through the image conversion, image generation, image restoration, video generation, text generation and image super-resolution applications; analyzing the theoretical research progress of generative adversarial networks from the perspectives of interpretability, controllability, stability and model evaluation methods. Finally, the paper discusses the challenges of studying generative adversarial networks and looks forward to the possible future directions of development.

Key words: generate model, Generative Adversarial Network（GAN）, model evaluation, image generation applications

摘要：

生成对抗网络的理论研究与应用不断获得成功，已经成为当前深度学习领域研究的热点之一。对生成对抗网络理论及其应用从模型的类型、评价标准和理论研究进展等方面进行系统的综述：分别分析基于显式密度和基于隐式密度的生成模型的优缺点；总结生成对抗网络的评价标准，解读各标准之间的关系，并从应用层面介绍生成对抗网络在图像及其他领域中的研究进展，即通过图像转换、图像生成、图像修复、视频生成、文本生成及图像超分辨率等的应用；从模型的结构表示、训练控制、性能稳定以及评价标准等角度分析生成对抗网络的理论研究进展。研究讨论生成对抗网络的挑战，展望未来可能存在的发展方向。

关键词: 生成模型, 生成对抗网络, 模型评价, 图像生成应用

WEI Fuqiang, Gulanbaier Tuerhong, Mairidan Wushouer. Review of Research on Generative Adversarial Networks and Its Application[J]. Computer Engineering and Applications, 2021, 57(19): 18-31.

魏富强，古兰拜尔·吐尔洪，买日旦·吾守尔. 生成对抗网络及其应用研究综述[J]. 计算机工程与应用, 2021, 57(19): 18-31.

References

[1] DOERSCH C.Tutorial on variational autoencoders[J].arXiv：1606.05908，2016.
[2] ZHU J Y，PARK T，ISOLA P，et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2223-2232.
[3] ZHANG Z，XIE Y，YANG L.Photographic text-to-image synthesis with a hierarchically-nested adversarial network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：6199-6208.
[4] WELANDER P，KARLSSON S，EKLUND A.Generative adversarial networks for image-to-image translation on multi-contrast MR images—A comparison of CycleGAN and UNIT[J].arXiv：1806.07777，2018.
[5] TULYAKOV S，LIU M Y，YANG X，et al.Mocogan：Decomposing motion and content for video generation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：1526-1535.
[6] VONDRICK C，PIRSIAVASH H，TORRALBA A.Generating videos with scene dynamics[J].arXiv：1609.02612，2016.
[7] WALKER J，MARINO K，GUPTA A，et al.The pose knows：Video forecasting by generating pose futures[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：3332-3341.
[8] VOLKHONSKIY D，NAZAROV I，BURNAEV E.Steganographic generative adversarial networks[C]//Proceedings of the Twelfth International Conference on Machine Vision（ICMV 2019），2019.
[9] SHI H，DONG J，WANG W，et al.SSGAN：Secure steganography based on generative adversarial networks[C]//Proceedings of the Pacific Rim Conference on Multimedia，2017：534-544.
[10] DAI B，FIDLER S，URTASUN R，et al.Towards diverse and natural image descriptions via a conditional gan[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2970-2979.
[11] OORD A，DIELEMAN S，ZEN H，et al.Wavenet：A generative model for raw audio[J].arXiv：1609.03499，2016.
[12] GOODFELLOW I J，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial networks[J].arXiv：1406.2661，2014.
[13] RADFORD A，METZ L，CHINTALA S.Unsupervised representation learning with deep convolutional generative adversarial networks[J].arXiv：1511.06434，2015.
[14] SALIMANS T，GOODFELLOW I，ZAREMBA W，et al.Improved techniques for training gans[J].arXiv：1606.03498，2016.
[15] QUAN H，TU D N，LE T，et al.MGAN：Training generative adversarial nets with multiple generators[C]//Proceedings of International Conference on Learning Representations，2018.
[16] ENGELHARDT S，SIMONE R D，FULL P M，et al.Improving surgical training phantoms by hyperrealism：Deep unpaired image-to-image translation from real surgeries[C]//Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention，2018：747-755.
[17] METZ L，POOLE B，PFAU D，et al.Unrolled generative adversarial networks[J].arXiv：1611.02163，2016.
[18] WANG S，ZHANG L，FU J R.CatGAN：Coupled adversarial transfer for domain generation[J].arXiv：1711.08904，2017.
[19] OLMOS R，TABIK S，LAMAS A，et al.A binocular image fusion approach for minimizing false positives in handgun detection with deep learning[J].Information Fusion，2019，49：271-280.
[20] ZHAO J，MATHIEU M，LECUN Y.Energy-based generative adversarial network[J].arXiv：1609.03126，2016.
[21] BALAJI Y，HASSANI H，CHELLAPPA R，et al.Entropic GANs meet VAEs：A statistical approach to compute sample likelihoods in GANs[C]//Proceedings of International Conference on Machine Learning，2019：414-423.
[22] SCH?FER F，ZHENG H，ANANDKUMAR A.Implicit competitive regularization in GANs[J].arXiv：1910.05852，2019.
[23] DUMOULIN V，VISIN F.A guide to convolution arithmetic for deep learning[J].arXiv：1603.07285，2016.
[24] KARRAS T，LAINE S，AILA T.A style-based generator architecture for generative adversarial networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：4401-4410.
[25] LEDIG C，THEIS L，HUSZáR F，et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：4681-4690.
[26] ZHAO W，XU W，YANG M，et al.Dual learning for cross-domain image captioning[C]//Proceedings of the 2017 ACM on Conference on Information and Knowledge Management，2017：29-38.
[27] WANG W，HUANG Q，YOU S，et al.Shape inpainting using 3d generative adversarial network and recurrent convolutional networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2298-2306.
[28] ZHANG H，XU T，LI H，et al.Stackgan：Text to photo-realistic image synthesis with stacked generative adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：5907-5915.
[29] LUC P，COUPRIE C，CHINTALA S，et al.Semantic segmentation using adversarial networks[J].arXiv：1611.08408，2016.
[30] MOKHAYERI F，GRANGER E.A paired sparse representation model for robust face recognition from a single sample[J].Pattern Recognition，2020，100：107129.
[31] 李健伟，曲长文，彭书娟，等.基于生成对抗网络和线上难例挖掘的SAR图像舰船目标检测[J].电子与信息学报，2019，41（1）：143-149.
LI J W，QU C W，PENG S J，et al.Ship detection in SAR images based on generative adversarial network and online hard examples mining[J].Journal of Electronics and Information Technology，2019，41（1）：143-149.
[32] CHAKRABORTY A，ALAM M，DEY V，et al.Adversarial attacks and defences：A survey[J].arXiv：1810.00069，2018.
[33] YANG Z，CHEN W，WANG F，et al.Improving neural machine translation with conditional sequence generative adversarial nets[J].arXiv：1703.04887，2017.
[34] MA J，YU W，CHEN C，et al.Pan-GAN：An unsupervised pan-sharpening method for remote sensing image fusion[J].Information Fusion，2020，62：110-120.
[35] ZHAN F，ZHU H，LU S.Spatial fusion gan for image synthesis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：3653-3662.
[36] JOO D，KIM D，KIM J.Generating a fusion image：One’s identity and another’s shape[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：1635-1643.
[37] REY-AREA M，GUIRADO E，TABIK S，et al.FuCiTNet：Improving the generalization of deep learning networks by the fusion of learned class-inherent transformations[J].Information Fusion，2020，63：188-195.
[38] PARK H S，BAEK J，YOU S K，et al.Unpaired image denoising using a generative adversarial network in X-ray CT[J].IEEE Access，2019，7：110414-110425.
[39] 王曙燕，金航，孙家泽.GAN图像对抗样本生成方法[J].计算机科学与探索，2021，15（4）：702-711.
WANG S Y，JIN H，SUN J Z.Method for image adversarial samples generating based on GAN[J].Journal of Frontiers of Computer Science and Technology，2021，15（4）：702-711.
[40] 彭晏飞，高艺，杜婷婷，等.生成对抗网络的单图像超分辨率重建方法[J].计算机科学与探索，2020，14（9）：1612-1620.
PENG Y F，GAO Y，DU T T，et al.Single image super-resolution reconstruction method for generative adversarial network[J].Journal of Frontiers of Computer Science and Technology，2020，14（9）：1612-1620.
[41] ROYER A，BOUSMALIS K，GOUWS S，et al.XGAN：Unsupervised image-to-image translation for many-to-many mappings[M]//Domain Adaptation for Visual Understanding.Cham：Springer：2020：33-49.
[42] HUANG H，YU P S，WANG C.An introduction to image synthesis with generative adversarial nets[J].arXiv：1803. 04469，2018.
[43] GOODFELLOW I.Nips 2016 tutorial：Generative adversarial networks[J].arXiv：1701.00160，2017.
[44] CRESWELL A，WHITE T，DUMOULIN V，et al.Generative adversarial networks：An overview[J].IEEE Signal Processing Magazine，2018，35（1）：53-65.
[45] ALQAHTANI H，KAVAKLI-THORNE M，KUMAR G.Applications of generative adversarial networks（GANS）：An updated review[J].Archives of Computational Methods in Engineering，2021，28：525-552.
[46] HUGHES R T，ZHU L，BEDNARZ T.Generative adversarial networks enabled human artificial intelligence collaborative applications for creative and design industries：A systematic review of current approaches and trends[J].Frontiers in Artificial Intelligence，2021，4.
[47] GUI J，SUN Z，WEN Y，et al.A review on generative adversarial networks：Algorithms，theory，and applications[J].arXiv：2001.06937，2020.
[48] GAO Y，GONG M，XIE Y，et al.An attention-based unsupervised adversarial model for movie review spam detection[J].IEEE Transactions on Multimedia，2020，23：784-796.
[49] YANG W，TAN R T，WANG S，et al.Single image deraining：From model-based to data-driven and beyond[J].arXiv：1912.07150，2019.
[50] PICCIALLI F，DI SOMMA V，GIAMPAOLO F，et al.A survey on deep learning in medicine：Why，how and when?[J].Information Fusion，2021，66：111-137.
[51] DI MATTIA F，GALEONE P，DE SIMONI M，et al.A survey on GANS for anomaly detection[J].arXiv：1906.11632，2019.
[52] HONG Y，HWANG U，YOO J，et al.How generative adversarial networks and their variants work：An overview[J].ACM Computing Surveys，2019，52（1）：1-43.
[53] WANG Z，SHE Q，WARD T E.Generative adversarial networks in computer vision：A survey and taxonomy[J].arXiv：1906.01529，2019.
[54] WIATRAK M，ALBRECHT S V.Stabilizing generative adversarial network training：A survey[J].arXiv：1910. 00927，2019.
[55] HAN J，ZHANG Z，CUMMINS N，et al.Adversarial training in affective computing and sentiment analysis：Recent advances and perspectives[J].IEEE Computational Intelligence Magazine，2019，14（2）：68-81.
[56] YI X，WALIA E，BABYN P.Generative adversarial network in medical imaging：A review[J].Medical Image Analysis，2019，58：101552.
[57] ZENATI H，FOO C S，LECOUAT B，et al.Efficient gan-based anomaly detection[J].arXiv：1802.06222，2018.
[58] KURACH K，LU?I? M，ZHAI X，et al.A large-scale study on regularization and normalization in GANs[C]//Proceedings of International Conference on Machine Learning，2019：3581-3590.
[59] WU X，XU K，HALL P.A survey of image synthesis and editing with generative adversarial networks[J].Tsinghua Science and Technology，2017，22（6）：660-674.
[60] LECUN Y，BOTTOU L，BENGIO Y，et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE，1998，86（11）：2278-2324.
[61] KRIZHEVSKY A，HINTON G.Learning multiple layers of features from tiny images[J].Handbook of Systemic Autoimmune Diseases，2009，1（4）.
[62] CHEN Z，JIANG C.Building occupancy modeling using generative adversarial network[J].Energy and Buildings，2018，174：372-379.
[63] DENTON E，CHINTALA S，SZLAM A，et al.Deep generative image models using a laplacian pyramid of adversarial networks[J].arXiv：1506.05751，2015.
[64] WU J，ZHANG C，XUE T，et al.Learning a probabilistic latent space of object shapes via 3D generative-adversarial modeling[J].arXiv：1610.07584，2016.
[65] ISOLA P，ZHU J Y，ZHOU T，et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：1125-1134.
[66] ZHU J Y，PARK T，ISOLA P，et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2223-2232.
[67] ZHU J Y，ZHANG R，PATHAK D，et al.Toward multimodal image-to-image translation[J].arXiv：1711.11586，2017.
[68] ODENA A，OLAH C，SHLENS J.Conditional image synthesis with auxiliary classifier gans[C]//Proceedings of International Conference on Machine Learning，2017：2642-2651.
[69] MARIANI G，SCHEIDEGGER F，ISTRATE R，et al.Bagan：Data augmentation with balancing gan[J].arXiv：1803.09655，2018.
[70] KARACAN L，AKATA Z，ERDEM A，et al.Learning to generate images of outdoor scenes from attributes and semantic layouts[J].arXiv：1612.00215，2016.
[71] 吴春梅，胡军浩，尹江华.利用改进生成对抗网络进行人体姿态识别[J].计算机工程与应用，2020，56（8）：96-103.
WU C M，HU J H，YIN J H.Using improved generative adversarial network for human pose estimation[J].Computer Engineering and Applications，2020，56（8）：96-103.
[72] LARSEN A B L，S?NDERBY S K，LAROCHELLE H，et al.Autoencoding beyond pixels using a learned similarity metric[C]//Proceedings of International Conference on Machine Learning，2016：1558-1566.
[73] RUSU A A，RABINOWITZ N C，DESJARDINS G，et al.Progressive neural networks[J].arXiv：1606.04671，2016.
[74] SHAMSOLMOALI P，ZAREAPOOR M，WANG R，et al.G-GANISR：Gradual generative adversarial network for image super resolution[J].Neurocomputing，2019，366：140-153.
[75] ZAREAPOOR M，ZHOU H，YANG J.Perceptual image quality using dual generative adversarial network[J].Neural Computing and Applications，2020，32（18）：14521-14531.
[76] DASH A，GAMBOA J，AHMED S，et al.TAC-GAN-text conditioned auxiliary classifier generative adversarial network[J].arXiv：1703.06412，2017.
[77] JING Y，YANG Y，FENG Z，et al.Neural style transfer：A review[J].IEEE Transactions on Visualization and Computer Graphics，2019，26（11）：3365-3385.
[78] ESTEBAN C，HYLAND S L，R?TSCH G.Real-valued（medical） time series generation with recurrent conditional gans[J].arXiv：1706.02633，2017.
[79] BISSOTO A，PEREZ F，VALLE E，et al.Skin lesion synthesis with generative adversarial networks[M]//OR 2.0 context-aware operating theaters，computer assisted robotic endoscopy，clinical image-based procedures，and skin image analysis.Cham：Springer，2018：294-302.
[80] YANG Z，WEN J，DAVATZIKOS C.Smile-GANs：Semi-supervised clustering via GANs for dissecting brain disease heterogeneity from medical images[J].arXiv：2006. 15255，2020.
[81] HAN C，KITAMURA Y，KUDO A，et al.Synthesizing diverse lung nodules wherever massively：3D multi-conditional GAN-based CT image augmentation for object detection[C]//2019 International Conference on 3D Vision（3DV），2019：729-737.
[82] LIU Z，BICER T，KETTIMUTHU R，et al.TomoGAN：Low-dose synchrotron x-ray tomography with generative adversarial networks：discussion[J].Journal of the Optical Society of America A，2020，37（3）：422-434.
[83] SANDFORT V，YAN K，PICKHARDT P J，et al.Data augmentation using generative adversarial networks（CycleGAN） to improve generalizability in CT segmentation tasks[J].Scientific Reports，2019，9（1）：1-9.
[84] HAN C，RUNDO L，MURAO K，et al.MADGAN：Unsupervised medical anomaly detection GAN using multiple adjacent brain MRI slice reconstruction[J].arXiv：2007. 13559，2020.
[85] WELANDER P，KARLSSON S，EKLUND A.Generative adversarial networks for image-to-image translation on multi-contrast MR images—A comparison of CycleGAN and UNIT[J].arXiv：1806.07777，2018.
[86] TANG Y，CAI J，LU L，et al.CT image enhancement using stacked generative adversarial networks and transfer learning for lesion segmentation improvement[C]//International Workshop on Machine Learning in Medical Imaging，2018：46-54.
[87] SCHLEGL T，SEEB?CK P，WALDSTEIN S M，et al.f-AnoGAN：Fast unsupervised anomaly detection with generative adversarial networks[J].Medical Image Analysis，2019，54：30-44.
[88] DOU Q，OUYANG C，CHEN C，et al.Pnp-adanet：Plug-and-play adversarial domain adaptation network with a benchmark at cross-modality cardiac segmentation[J].arXiv：1812.07907，2018.
[89] SMITH E J，MEGER D.Improved adversarial systems for 3D object generation and reconstruction[C]//Proceedings of Conference on Robot Learning，2017：87-96.
[90] YANG B，WEN H，WANG S，et al.3d object reconstruction from a single depth view with adversarial learning[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops，2017：679-688.
[91] GALAMA Y，MENSINK T.IterGANs：Iterative GANs to learn and control 3D object transformation[J].Computer Vision and Image Understanding，2019，189：102803.
[92] HERMOZA R，SIPIRAN I.3D reconstruction of incomplete archaeological objects using a generative adversarial network[J].arXiv：1711.06363，2017.
[93] MA J，YU W，LIANG P，et al.FusionGAN：A generative adversarial network for infrared and visible image fusion[J].Information Fusion，2019，48：11-26.
[94] ZHANG H，LE Z，SHAO Z，et al.MFF-GAN：An unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion[J].Information Fusion，2021，66：40-53.
[95] CLARK A，DONAHUE J，SIMONYAN K.Efficient video generation on complex datasets[J].arXiv：1907.06571，2019.
[96] CHANDAK V，SAXENA P，PATTANAIK M，et al.Semantic image completion and enhancement using deep learning[C]//2019 10th International Conference on Computing，Communication and Networking Technologies，2019：1-6.
[97] 王海涌，梁红珠.基于改进的GAN的局部遮挡人脸表情识别[J].计算机工程与应用，2020，56（5）：141-146.
WANG H Y，LIANG H Z.Recognition of local occluded facial expressions based on improved generative adversarial network[J].Computer Engineering and Applications，2020，56（5）：141-146.
[98] LI Y，LIU S，YANG J，et al.Generative face completion[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：3911-3919.
[99] HONG X，XIONG P，JI R，et al.Deep fusion network for image completion[C]//Proceedings of the 27th ACM International Conference on Multimedia，2019：2033-2042.
[100] LEE C H，LIU Z，WU L，et al.Maskgan：Towards diverse and interactive facial image manipulation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：5549-5558.
[101] YANG W，ZHOU F，ZHU R，et al.Deep learning for image super-resolution[J].Neurocomputing，2020，398：291-292.
[102] SHIN H，LEE J K，KIM J，et al.Continual learning with deep generative replay[J].arXiv：1705.08690，2017.
[103] GOMEZ A N，HUANG S，ZHANG I，et al.Unsupervised cipher cracking using discrete gans[J].arXiv：1801.04883，2018.