Survey of Generative Adversarial Networks

doi:10.3778/j.issn.1002-8331.2205-0097

Abstract

Abstract: With its strong adversary learning ability, generative adversarial networks（GAN） is favored by more and more researchers in many fields. This paper expounds the development background, framework and objective function of GAN, analyzes the causes of pattern collapse and gradient disappearance in the training process, and introduces in detail the GAN derived model proposed through the change of architecture and the modification of objective function. Then, it summarizes some standards used to evaluate the quality and diversity of generated images, and summarizes the wide application of GAN in different fields, Finally, this paper summarizes and puts forward some prospects for the future research direction in this field.

Key words: machine learning, generative adversarial networks, image generation, unsupervised learning

摘要： 生成式对抗网络（GAN）凭借其强大的对抗学习能力受到越来越多研究者的青睐，并在诸多领域内展现出巨大的潜力。阐述了GAN的发展背景、架构、目标函数，分析了训练过程中出现模式崩溃和梯度消失的原因，并详细介绍了通过架构变化和目标函数修改而提出GAN衍生模型，对一些用来评估生成图像质量和多样性的标准进行了小结，总结了GAN在不同领域的广泛应用，总结全文并对该领域未来的研究方向提出一些展望。

关键词: 机器学习, 生成式对抗网络, 图像生成, 无监督学习

SUN Shukui, FAN Jing, QU Jinshuai, LU Peidong. Survey of Generative Adversarial Networks[J]. Computer Engineering and Applications, 2022, 58(18): 90-103.

孙书魁, 范菁, 曲金帅, 路佩东. 生成式对抗网络研究综述[J]. 计算机工程与应用, 2022, 58(18): 90-103.

References

[1] ALPAYDIN E.Machine learning[M].[S.l.]：MIT Press，2021.
[2] SMOLENSKY P.Information processing in dynamical systems：foundations of harmony theory[R].Colorado University.Boulder Department of Computer Science，1986.
[3] HINTON G E，OSINDERO S，TEH Y W.A fast learning algorithm for deep belief nets[J].Neural Computation，2006，18（7）：1527-1554.
[4] HINTON G E，SALAKHUTDINOV R.A better way to pretrain deep boltzmann machines[J].Advances in Neural Information Processing Systems，2012，25.
[5] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial nets[J].Advances in Neural Information Processing Systems，2014，27.
[6] 梁俊杰，韦舰晶，蒋正锋.生成对抗网络GAN综述[J].计算机科学与探索，2020，14（1）：1-17.
LIANG J J，WEI J J，JIANG Z F.Generative adversarial networks GAN overview[J].Journal of Frontiers of Computer Science and Technology，2020，14（1）：1-17.
[7] 夏萌霏，叶子鹏，赵旺，等.几何视角下深度神经网络的对抗攻击与可解释性研究进展[J].中国科学（信息科学），2021，51（9）：1411-1437.
XIA M F，YE Z P，ZHAO W.Adversarial attack and interpretability of the deep neural network from the geometric perspective[J].Science in China（Information Sciences），2021，51（9）：1411-1437.
[8] BROWNLEE J.Generative adversarial networks with python：deep learning generative models for image synthesis and image translation[M]//Machine learning mastery，2019.
[9] 陈佛计，朱枫，吴清潇，等.生成对抗网络及其在图像生成中的应用研究综述[J].计算机学报，2021，44（2）：347-369.
CHEN F J，ZHU F，WU Q X，et al.A survey about image generation with generative adversarial nets[J].Chinese Journal of Computers，2021，44（2）：347-369.
[10] CHEN X，DUAN Y，HOUTHOOFT R，et al.Infogan：interpretable representation learning by information maximizing generative adversarial nets[J].arXiv：1606.03657，2016.
[11] ARJOVSKY M，CHINTALA S，BOTTOU L.Wasserstein generative adversarial networks[C]//Proceedings of the 34th International Conference on Machine Learning，2017：214-223.
[12] LARSEN A B L，S?NDERBY S K，LARO-CHELLE H，et al.Autoencoding beyond pixels using a learned similarity metric[C]//International Conference on Machine Learning，2016：1558-1566.
[13] TRAN N T，BUI T A，CHEUNG N M.Dist-GAN：an improved gan using distance constraints[C]//Proceedings of the European Conference on Computer Vision，2018：370-385.
[14] GUO X，HONG J，LIN T，et al.Relaxed Wasserstein with applications to GANs[C]//2021 IEEE International Conference on Acoustics，Speech and Signal Processing，2021：3325-3329.
[15] NOWOZIN S，CSEKE B，TOMIOKA R.f-GAN：training generative neural samplers usingvariational divergence minimization[J].Advances in Neural Information Processing Systems，2016，29.
[16] MESCHEDER L，NOWOZIN S，GEIGER A.The numerics of GANs[J].Advances in Neural Information Processing Systems，2017，30.
[17] MESCHEDER L，GEIGER A，NOWOZIN S.Which training methods for GANs do actually converge?[C]//International Conference on Machine Learning，2018：3481-3490.
[18] UEHARA M，SATO I，SUZUKI M，et al.Generative adversarial nets from a density ratio estimation perspective[J].arXiv：1610.02920，2016.
[19] LIU M Y，TUZEL O.Coupled generative adversarial networks[J].Advances in Neural Information Processing Systems，2016，29.
[20] MAO X，LI Q，XIE H，et al.Least squares generative adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2794-2802.
[21] MISHRA D，PRATHOSH A P，ARAVIND J，et al.Unsupervised conditional generation using noise engineered mode matching GAN[C]//Proceedings of International Conference on Learning Representations，2018.
[22] LIN M.Softmax GAN[J].arXiv：1704.06191，2017.
[23] MIRZA M，OSINDERO S.Conditional generative adversarial nets[J].arXiv：1411.1784，2014.
[24] 马永杰，徐小冬，张茹，等.生成式对抗网络及其在图像生成中的研究进展[J].计算机科学与探索，2021，15（10）：1795-1811.
MA Y J，XU X D，ZHANG R，et al.Generative adversarial network and its research progress in image generation[J].Journal of Frontiers of Computer Science and Technology，2021，15（10）：1795-1811.
[25] GHOJOGH B，GHODSI A，KARRAY F，et al.Generative adversarial networks and adversarial autoencoders：tutorial and survey[J].arXiv：2111.13282，2021.
[26] ADLER J，LUNZ S.Banach Wasserstein GAN[J].Advances in Neural Information Processing Systems，2018，31.
[27] LECUN Y，BOTTOU L，BENGIO Y，et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE，1998，86（11）：2278-2324.
[28] RADFORD A，METZ L，CHINTALA S.Unsupervised representation learning with deep convolutional generative adversarial networks[J].arXiv：1511.06434，2015.
[29] MAKHZANI A，SHLENS J，JAITLY N，et al.Adversarial autoencoders[J].arXiv：1511.05644，2015.
[30] 叶晨，关玮.生成式对抗网络的应用综述[J].同济大学学报（自然科学版），2020，48（4）：591-601.
YE C，GUAN W.A review of application of generative adversarial networks[J].Journal of Tongji University（Natural Science），2020，48（4）：591-601.
[31] 魏丙财，张立晔，孟晓亮，等.基于深度残差生成对抗网络的运动图像去模糊[J].液晶与显示，2021，36（12）：1693-1701.
WEI B C，ZHANG L Y，MENG X L，et al.Motion image deblurring based on depth residual generative adversarial network[J].Chinese Journal of Liquid Crystals and Displays，2021，36（12）：1693-1701.
[32] GULRAJANI I，AHMED F，ARJOVSKY M，et al.Improved training of wasserstein gans[J].Advances in Neural Information Processing Systems，2017，30.
[33] QI G J.Loss-sensitive generative adversarial networks on lipschitz densities[J].International Journal of Computer Vision，2020，128（5）：1118-1140.
[34] MIYATO T，KATAOKA T，KOYAMA M，et al.Spectral normalization for generative adversarial networks[J].arXiv：1802.05957，2018.
[35] SALIMANS T，GOODFELLOW I，ZAREMBA W，et al.Improved techniques for training GANs[J].Advances in Neural Information Processing Systems，2016，29.
[36] SOLOVEITCHIK M，DISKIN T，MORIN E，et al.Conditional frechet inception distance[J].arXiv：2103.11521，2021.
[37] DZIUGAITE G K，ROY D M，GHAHRAMANI Z.Training generative neural networks via maximum mean discrepancy optimization[J].arXiv：1505.03906，2015.
[38] XU Q，HUANG G，YUAN Y，et al.An empirical study on evaluation metrics of generative adversarial networks[J].arXiv：1806.07755，2018.
[39] DOSSELMANN R，YANG X D.A comprehensive assessment of the structural similarity index[J].Signal，Image and Video Processing，2011，5（1）：81-91.
[40] WANG Z，SIMONCELLI E P，BOVIK A C.Multiscale structural similarity for image quality assessment[C]//The Thrity-Seventh Asilomar Conference on Signals，Systems & Computers，2003：1398-1402.
[41] KURACH K，LU?I? M，ZHAI X，et al.A large-scale study on regularization and normalization in GANs[C]//International Conference on Machine Learning，2019：3581-3590.
[42] SATHE S，AGGARWAL C C.Nearest neighbor classifiers versus random forests and support vector machines[C]//2019 IEEE International Conference on Data Mining，2019：1300-1305.
[43] ISOLA P，ZHU J Y，ZHOU T，et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：1125-1134.
[44] SIDDIQUE N，PAHEDING S，ELKIN C P，et al.U-net and its variants for medical image segmentation：a review of theory and applications[J].IEEE Access，2021.
[45] CHANDALIYA P K，NAIN N.Child face age progression and regression using self-attention multi-scale patch GAN[C]//International Joint Conference on Biometrics（IJCB 2021），2021：1-8.
[46] YU W，CHEN F，CHOI J.Multi-pose face recognition based on TP-GAN[C]//International Conference on Intelligent and Fuzzy Systems，2021：725-732.
[47] ZENG Y，FU J，CHAO H，et al.Aggregated contextual transformations for high-resolution image inpainting[J].IEEE Transactions on Visualization and Computer Graphics，2022.
[48] YANG C，LU X，LIN Z，et al.High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：6721-6729.
[49] IIZUKA S，SIMO-SERRA E，ISHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Graphics（ToG），2017，36（4）：1-14.
[50] YU F，KOLTUN V.Multi-scale context aggregation by dilated convolutions[J].arXiv：1511.07122，2015.
[51] YAN Y，SHEN G，ZHANG S，et al.Sequence generative adversarial nets with a conditional discriminator[J].Neurocomputing，2021，429：69-76.
[52] MOGREN O.C-RNN-GAN：continuous recurrent neural networks with adversarial training[J].arXiv：1611.09904，2016.
[53] HOCHREITER S，SCHMIDHUBER J.Long short-term memory[J].Neural Computation，1997，9（8）：1735-1780.
[54] LEE S，HWANG U，MIN S，et al.A seqgan for polyphonic music generation[J].arXiv：1710.11418，2017.
[55] LIN K，LI D，HE X，et al.Adversarial ranking for language generation[J].Advances in Neural Information Processing Systems，2017，30.
[56] GUIMARAES G L，SANCHEZ-LENGELING B，OUTEIRAL C，et al.Objective-reinforced generative adversarial networks（ORGAN） for sequence generation models[J].arXiv：1705.10843，2017.
[57] HSU C，HWANG H T，WU Y C，et al.Voice conversion from unaligned corpora using variational autoencoding wasserstein generative adversarial networks[J].arXiv：1704.
00849，2017.
[58] 庄跃生，林珊玲，林志贤，等.生成对抗网络在数据异常检测中的研究[J].计算机工程与应用，2022，58（4）：143-149.
ZHUANG Y S，LIN S L，LIN Z X，et al.Study on generative adversarial network for data anomaly detection[J].Computer Engineering and Applications，2022，58（4）：143-149.
[59] MOTI Z，HASHEMI S，KARIMIPOUR H，et al.Generative adversarial network to detect unseen Internet of things malware[J].Ad Hoc Networks，2021，122：102591.
[60] 万梦翔，姚寒冰.面向恶意网页训练数据生成的GAN模型[J].计算机工程与应用，2021，57（6）：124-130.
WAN M X，YAO H B.GAN model for malicious Web training data generative[J].Computer Engineering and Applications，2021，57（6）：124-130.
[61] SHIN H，LEE J K，KIM J，et al.Continual learning with deep generative replay[J].Advances in Neural Information Processing Systems，2017，30.
[62] 吴辰文，梁雨欣，田鸿雁.改进卷积神经网络的COVID-19 CT影像分类方法研究[J].计算机工程与应用，2022，58（2）：225-234.
WU C W，LIANG Y X，TIAN H Y.Research on COVID-19 CT image classification method based on improved convolutional neural network[J].Computer Engineering and Applications，2022，58（2）：225-234.
[63] MENON S，MANGALAGIRI J，GALITA J，et al.CCS-GAN：COVID-19 CT-scan classification with very few positive training images[J].arXiv：2110.01605，2021.
[64] SILVA V L S，HEANEY C E，LI Y，et al.Data assimilation predictive GAN（DA-PredGAN）：applied to determine the spread of COVID-19[J].arXiv：2105.07729，2021.
[65] WAHEED A，GOYAL M，GUPTA D，et al.CovidGAN：data augmentation using auxiliary classifier GAN for improved Covid-19 detection[J].IEEE Access，2020，8：91916-91923.
[66] LEE K，CHANG H，JIANG L，et al.Vitgan：training GANs with vision transformers[J].arXiv：2107.04589，2021.
[67] GONG X，CHANG S，JIANG Y，et al.Autogan：Neural architecture search for generative adversarial networks[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：3224-3234.
[68] LI C，XU T，ZHU J，et al.Triple generative adversarial nets[J].Advances in Neural Information Processing Systems，2017，30.