New GAN-Based Partial Realistic Anime Image Style Transfer

doi:10.3778/j.issn.1002-8331.2104-0083

Abstract

Abstract: Generative adversarial network（GAN） is one of the research hotspots in computer vision to transfer the style of images from real world to high quality animation style. The popular AnimeGAN and CartoonGAN anime generating networks have suffered from serious detail loss and color distortion during image migration. In this paper, a new ExpressionGAN is proposed to solve the problem of serious detail loss in AnimeGAN transfer images by introducing the SE-Residual Block, the face detection mechanism and optimizing the loss function. By adding DSCONV（distributed shift convolution）, it is proposed that SceneryGAN speeds up the training and eliminates the ambiguous pixel blocks in the CartoonGAN migration images. The image fusion boundary is optimized by convolution. At the same time, a new partial realistic cartoon model is proposed, which deals with and integrates the original image characters and environment respectively. The experimental results show that, compared with AnimeGAN and CartoonGAN, the proposed method has significantly improved the training speed, anime image generation quality and local realism of anime images.

Key words: image style transfer, generative adversarial network, anime style, partial sense of reality, AnimeGAN, CartoonGAN

摘要： 利用生成对抗网络对图像进行风格迁移，将真实世界的图像直接转换为高品质动漫风格，是当今计算机视觉的研究热点之一。针对目前流行的AnimeGAN和CartoonGAN漫画生成对抗网络在图像迁移中存在细节丢失严重、色彩失真等问题。通过引入SE-Residual Block（挤压激励残差块）、漫画脸部检测机制并优化损失函数提出全新的ExpressionGAN解决了AnimeGAN迁移图像细节丢失严重的问题。通过加入DSConv（分布偏移卷积）提出SceneryGAN 加快了训练速度并消除了CartoonGAN迁移图像中的歧义像素块。通过卷积优化了图像的融合边界。同时，提出了一种新的对原始图像人物和环境分别处理并融合的局部写实主义漫画模型。实验结果表明，与AnimeGAN和CartoonGAN相比，该方法在训练速度、漫画图像生成质量和图像局部写实感方面都有了明显的提升。

关键词: 图像风格迁移, 生成对抗网络, 动漫风格, 局部真实感, AnimeGAN, CartoonGAN

SUN Tianpeng, ZHOU Ningning, HUANG Guofang. New GAN-Based Partial Realistic Anime Image Style Transfer[J]. Computer Engineering and Applications, 2022, 58(14): 167-176.

孙天鹏, 周宁宁, 黄国方. 新的基于GAN的局部写实感漫画图像风格迁移[J]. 计算机工程与应用, 2022, 58(14): 167-176.

References

[1] 陈淑環，韦玉科，徐乐，等.基于深度学习的图像风格迁移研究综述[J].计算机应用研究，2019，36（8）：2250-2255.
CHEN S H，WEI Y K，XU L，et al.A review of image style transfer based on deep learning[J].Application Research of Computers，2019，36（8）：2250-2255.
[2] STROTHOTTE T，SCHLECHTWEG S.Non-photorealistic computer graphics：modeling，rendering，and animation[M].[S.l.]：Morgan Kaufmann，2002.
[3] 陈淮源，张广驰，陈高，等.基于深度学习的图像风格迁移进展[J].计算机工程与应用，2021，57（11）：37-45.
CHEN H Y，ZHANG G C，CHEN G，et al.Research progress of image style transfer based on deep learning[J].Computer Engineering and Applications，2021，57（11）：37-45.
[4] GATYS L A，ECKER A S，BETHGE M.Image style transfer using convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：2414-2423.
[5] JOHNSON J，ALAHI A，FEI-FEI L.Perceptual losses for real-time style transfer and super-resolution[C]//European Conference on Computer Vision.Cham：Springer，2016：694-711.
[6] LUAN F，PARIS S，SHECHTMAN E，et al.Deep photo style transfer[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：4990-4998.
[7] LI C，WAND M.Combining Markov random fields and convolutional neural networks for image synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：2479-2486.
[8] HUANG X，BELONGIE S.Arbitrary style transfer in real-time with adaptive instance normalization[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：1501-1510.
[9] LI Y，FANG C，YANG J，et al.Universal style transfer via feature transforms[C]//Advances in Neural Information Processing Systems，2017：386-396.
[10] RADFORD A，METZ L，CHINTALA S.Unsupervised representation learning with deep convolutional generative adversarial networks[J].arXiv：1511.06434，2015.
[11] ZHU J Y，PARK T，ISOLA P，et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2223-2232.
[12] CHEN Y，LAI Y K，LIU Y J.CartoonGAN：generative adversarial networks for photo cartoonization[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：9465-9474.
[13] CHEN J，LIU G，CHEN X.AnimeGAN：a novel lightweight GAN for photo animation[C]//International Symposium on Intelligence Computation and Applications.Singapore：Springer，2019：242-256.
[14] YOSINSKI J，CLUNE J，BENGIO Y，et al.How transferable are features in deep neural networks?[J].arXiv：1411.1792，2014.
[15] CRESWELL A，WHITE T，DUMOULIN V，et al.Generative adversarial networks：an overview[J].IEEE Signal Processing Magazine，2018，35（1）：53-65.
[16] HU J，SHEN L，SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：7132-7141.
[17] WOO S，PARK J，LEE J Y，et al.CBAM：convolutional block attention module[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：3-19.
[18] NASCIMENTO M G，FAWCETT R，PRISACARIU V A.DSConv：efficient convolution operator[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：5148-5157.