Review of Neural Style Transfer Models

doi:10.3778/j.issn.1002-8331.2105-0296

Abstract

Abstract:

Neural Style Transfer（NST） technique is used to simulate different art styles of images and videos, which is a popular topic in computer vision. This paper aims to provide a comprehensive overview of the current progress towards NST. Firstly, the paper reviews the Non-Photorealistic Rendering（NPR） technique and traditional texture transfer. Then, the paper categorizes current major NST methods and gives a detailed description of these methods along with their subsequent improvements. After that, it discusses various applications of NST and presents several evaluation methods which compares different style transfer models both qualitatively and quantitatively. In the end, it summarizes the existing problems and provides some future research directions for NST.

Key words: Neural Style Transfer（NST）, deep learning, convolution neural network, generative model, generative adversarial network

摘要：

神经风格迁移技术主要用于对图像、视频等进行风格化，使其具有艺术美感，该领域极具应用价值，是人工智能的热门研究领域之一。为推动神经风格迁移领域的研究发展，对神经风格迁移技术进行了全面概述。简述了非真实感渲染技术和传统的纹理迁移技术。对现有神经风格迁移模型进行了分类整理，并详细探讨了各类代表性模型的算法原理及后续改进，分析了神经风格迁移技术的应用市场。提出对风格迁移模型质量的评判应该从定性评估和定量评估两个方面来考虑，并从各个角度讨论了现阶段风格迁移技术存在的问题以及未来研究方向。最后强调应提高模型的综合能力，在保证生成质量的情况下提升生成速度以及泛化能力。

关键词: 神经风格迁移, 深度学习, 卷积神经网络, 生成模型, 生成对抗网络

TANG Renwei, LIU Qihe, TAN Hao. Review of Neural Style Transfer Models[J]. Computer Engineering and Applications, 2021, 57(19): 32-43.

唐稔为，刘启和，谭浩. 神经风格迁移模型综述[J]. 计算机工程与应用, 2021, 57(19): 32-43.

References

[1] GOOCH B，GOOCH A.Non-photorealistic rendering[M].[S.l.]：CRC Press，2001.
[2] STROTHOTTE T，SCHLECHTWEG S.Non-photorealistic computer graphics：modeling，rendering，and animation[M].San Francisco：Morgan Kaufmann，2002.
[3] ROSIN P L，COLLOMOSSE J.Image and video-based artistic stylisation[M].[S.l.]：Springer Science & Business Media，2012..
[4] EFROS A A，FREEMAN W T.Image quilting for texture synthesis and transfer[C]//Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques，2001：341-346.
[5] DRORI I，COHEN-OR D，YESHURUN H.Example-based style synthesis[C]//2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2003.
[6] FRIGO O，SABATER N，DELON J，et al.Split and match：Example-based adaptive patch sampling for unsupervised style transfer[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：553-561.
[7] MEIER B J.Painterly rendering for animation[C]//Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques，1996：477-484.
[8] HERTZMANN A，JACOBS C E，OLIVER N，et al.Image analogies[C]//Proceedings of the 28th Annual Conference on Computer Graphics Andinteractive Techniques，2001：327-340.
[9] WINNEM?LLER H，OLSEN S C，GOOCH B.Real-time video abstraction[J].ACM Transactions on Graphics（TOG），2006，25（3）：1221-1226.
[10] EFROS A A，LEUNG T K.Texture synthesis by non-parametric sampling[C]//Proceedings of the 7th IEEE International Conference on Computer Vision，1999：1033-1038.
[11] WEI L Y，LEVOY M.Fast texture synthesis using tree-structured vector quantization[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques，2000：479-488.
[12] ASHIKHMIN M.Synthesizing natural textures[C]//Proceedings of the 2001 Symposium on Interactive 3D Graphics，2001：217-226.
[13] 徐晓刚，鲍虎军，马利庄.基于相关性原理的多样图纹理合成方法[J].自然科学进展，2002，12（6）：665-668.
XU X G，BAO H J，MA L Z.Texture synthesis method of diverse graphs based on correlation principle[J].Progress in Natural Science，2002，12（6）：665-668.
[14] HAN C，RISSER E，RAMAMOORTHI R，et al.Multiscale texture synthesis[C]//Proceedings of ACM SIGGRAPH 2008，2008：1-8.
[15] GATYS L A，ECKER A S，BETHGE M.A neural algorithm of artistic style[J].arXiv：1508.06576，2015.
[16] GATYS L A，ECKER A S，BETHGE M.Texture synthesis using convolutional neural networks[J].arXiv：1505.07376，2015.
[17] GATYS L A，ECKER A S，BETHGE M.Image style transfer using convolutional neural networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition，2016：2414-2423．
[18] GATYS L A，ECKER A S，BETHGE M，et al.Controlling perceptual factors in neural style transfer[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：3985-3993.
[19] LI C，WAND M.Combining markov random fields and convolutional neural networks for image synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：2479-2486.
[20] JOHNSON J，ALAHI A，FEI-FEI L.Perceptual losses for real-time style transfer and super-resolution[C]//European Conference on Computer Vision，2016：694-711.
[21] ULYANOV D，LEBEDEV V，VEDALDI A，et al.Texture networks：Feed-forward synthesis of textures and stylized images[J].arXiv：1603.03417，2016.
[22] MIRZA M，OSINDERO S.Conditional generative adversarial nets[J].arXiv：1411.1784，2014.
[23] CHOI Y，CHOI M，KIM M，et al.Stargan：Unified generative adversarial networks for multi-domain image-to-image translation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：8789-8797.
[24] REINHARD E，ADHIKHMIN M，GOOCH B，et al.Color transfer between images[J].IEEE Computer Graphics and Applications，2001，21（5）：34-41.
[25] MORDVINTSEV A，OLAH C，TYKA M.Inceptionism：Going deeper into neural networks[EB/OL].[2021-04-01].http：//ai.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html.
[26] BERGER G，MEMISEVIC R.Incorporating long-range consistency in cnn-based texture generation[J].arXiv：1606.01286，2016.
[27] RISSER E，WILMOT P，BARNES C.Stable and controllable neural texturesynthesis and style transfer using histogram losses[EB/OL].（2017-02-03）[2021-04-01].https：//arxiv.org/pdf/1701.08893.pdf.
[28] LI S，XU X，NIE L，et al.Laplacian-steered neural style transfer[C]//Proceedings of the 33rd International Conference on Machine Learning，2017.
[29] CASTILLO C，DE S，HAN X，et al.Son of zorn’s lemma：Targeted style transfer using instance-awaresemantic segmentation[C]//IEEE International Conference on Acoustics，Speech and Signal Processing（ICASSP），2017：1348-1352.
[30] LUAN F，PARIS S，SHECHTMAN E，et al.Deep photo style transfer[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：4990-4998.
[31] PENHOU?T S，SANZENBACHER P.Automated deep photo style transfer[J].arXiv：1901.03915，2019.
[32] LI Y，WANG N，LIU J，et al.Demystifying neural style transfer[J].arXiv：1701.01036，2017.
[33] CHAMPANDARD A J.Semantic style transfer and turning two-bit doodles into fine artworks[J].arXiv：1603.01768，2016.
[34] CHEN Y L，HSU C T.Towards deep style transfer：A content-aware perspective[C]//Proceedings of the British Machine Vision Conference，2016.
[35] MECHREZ R，TALMI I，ZELNIK-MANOR L.The contextual loss for image transformation with non-aligned data[C]//Proceedings of the European Conference on Computer Vision，2018：768-783.
[36] LIAO J，YAO Y，YUAN L，et al.Visual attribute transfer through deep image analogy[J].arXiv：1705.01088，2017.
[37] GU S，CHEN C，LIAO J，et al.Arbitrary style transfer with deep feature reshuffle[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：8222-8231.
[38] RADFORD A，METZ L，CHINTALA S.Unsupervised representation learning with deep convolutional generative adversarial networks[J].arXiv：1511.06434，2015.
[39] ULYANOV D，VEDALDI A，LEMPITSKY V.Improved texture networks：Maximizing quality and diversity in feed-forward stylization and texture synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：6924-6932.
[40] IOFFE S，SZEGEDY C.Batch normalization：Accelerating deep network training by reducing internal covariate shift[C]//Proceedings of the International Conference on Machine Learning，2015：448-456.
[41] DUMOULIN V，SHLENS J，KUDLUR M.A learned representation for artistic style[J].arXiv：1610.07629，2016.
[42] CHEN D，YUAN L，LIAO J，et al.Stylebank：An explicit representation for neural image style transfer[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：1897-1906.
[43] LI Y，FANG C，YANG J，et al.Diversified texture synthesis with feed-forward networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：3920-3928.
[44] ZHANG H，DANA K.Multi-style generative network for real-time transfer[C]//Proceedings of the European Conference on Computer Vision，2018.
[45] CHEN T Q，SCHMIDT M.Fast patch-based style transfer of arbitrary style[J].arXiv：1612.04337，2016.
[46] HUANG X，BELONGIE S.Arbitrary style transfer in real-time with adaptive instance normalization[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：1501-1510.
[47] LU M，ZHAO H，YAO A B，et al.Decoder network over lightweight reconstructed feature for fast semantic style transfer[C]//Proceedings of the IEEE International Conference on Computer Vision（ICCV），2017.
[48] LI Y，FANG C，YANG J，et al.Universal style transfer via feature transforms[J].arXiv：1705.08086，2017.
[49] LI X，LIU S，KAUTZ J，et al.Learning linear transformations for fast arbitrary style transfer[J].arXiv：1808.04537，2018.
[50] SHEN F，YAN S，ZENG G.Neural style transfer via meta networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：8061-8069.
[51] PARK D Y，LEE K H.Arbitrary style transfer with style-attentional networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：5880-5888.
[52] GOODFELLOW I J，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial nets[EB/OL]（2014-06-11）[2021-04-01].https：//arxiv.org/pdf/1406.2661v1.pdf.
[53] LI C，WAND M.Precomputed real-time texture synthesis with markovian generative adversarial networks[C]//Proceedings of the European Conference on Computer Vision，2016
[54] ZHU J Y，PARK T，ISOLA P，et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of IEEE International Conference on Computer Vision，2017：2242-2251．
[55] KIM T，CHA M，KIM H，et al.Learning to discover cross-domain relations with generative adversarial networks[EB/OL].（2017-05-15）[2021-04-01].https：//arxiv.org/pdf/1703.05192.pdf．
[56] YI Z，ZHANG H，TAN P，et al.DualGAN：Unsupervised dual learning for image-to-image translation[J].arXiv：1704.02510，2017.
[57] LIU M Y，BREUEL T，KAUTZ J.Unsupervised image-to-image translation networks[J].arXiv：1703.00848，2017.
[58] CHEN Y，LAI Y K，LIU Y J.Cartoongan：Generative adversarial networks for photo cartoonization[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：9465-9474.
[59] P??KO M，SVYSTUN A，ANDRUSZKIEWICZ P，et al.Comixify：Transform video into comics[J].Fundamenta Informaticae，2019，168：311-333.
[60] LI X Y，ZHANG W，SHEN T，et al.Everyone is a cartoonist：Selfie cartoonization with attentive adversarial networks[C]//Proceedings of the IEEE International Conference on Multimedia，2019：652-657.
[61] WANG X，YU J.Learning to cartoonize using white-box cartoon representations[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020.
[62] ZHAO Y，WU R，DONG H.Unpaired image-to-image translation using adversarial consistency loss[C]//Proceedings of European Conference on Computer Vision，2020：800-815.
[63] CAO K，JING L，LU Y.CariGANs：Unpaired photo-to-caricature translation[J].ACM Transactions on Graphics，2018，37（6）：1-14.
[64] SHI Y，DEB D，JAIN A K.WarpGAN：Automatic caricature generation[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019.
[65] 曹毅.基于图像的中国水墨画绘制方法的研究[D].长春：吉林大学，2012.
CAO Y.Research on image based rendering of Chinese ink painting[D].Changchun：Jilin University，2012.
[66] 王涛，邓丽君.一种实现流体风格（涡旋状）梵高油画特效的方法[J].信息通信，2014（7）：38-40.
WANG T，DENG L J.A method to realize the special effect of fluid style（vortex） Van Gogh oil painting[J].Information and Communication，2014（7）：38-40.
[67] CASELLES V，MOREL J M.Introduction to the special issue on partial diffusion in image processing and analysis[J].IEEE Transactions on Image Processing，1998，7（3）：269-273.
[68] WEICKERT J.Anisotropic diffusion in image processing[J].FCMI Series，1998，16（1）：272.
[69] LIN D，WANG Y，XU G，et al.Transform a simple sketch to a Chinese painting by a multiscale deep neural network[J].Algorithms，2018，11（1）：4.
[70] LI B，XIONG C M，WU T F，et al.Neural abstract style transfer for Chinese traditional painting[C]//Proceedings of the 14th Asian Conference on Computer Vision，2018：212-227.
[71] TANG Y，ZHANG Y，SHI X Y，et al.2015.Multi-style video stylization based on texture advection[J].Science China（Information Sciences），2015，11：90-102.
[72] RUDER M，DOSOVITSKIY A，BROX T.Artistic style transfer for videos[C]//Proceedings of German Conference on Pattern Recognition，2016.
[73] HUANG H，WANG H，LUO W，et al.Real-time neural style transfer for videos[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：783-791.
[74] GUPTA A，JOHNSON J，ALAHI A，et al.Characterizing and improving stability in neural style transfer[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：4067-4076.
[75] XU K，WEN L，LI G，et al.Learning self-supervised space-time CNN for fast video style transfer[J].IEEE Transactions on Image Processing，2021，30：2501-2512.
[76] Prisma：Turn memories into art using artificial intelligence[CP/OL].（2016）[2021-04-01].http：//prisma-ai.com.
[77] DeepArt[CP/OL].（2016）[2021-04-01].https：//deepart.io/.
[78] JING Y，YANG Y，FENG Z，et al.Neural style transfer：A review[J].IEEE Transactions on Visualization and Computer Graphics，2019，26（11）：3365-3385.
[79] LIN T Y，MAIRE M，BELONGIE S，et al.Microsoft coco：Common objects in context[C]//Proceedings of European Conference on Computer Vision，2014：740-755.
[80] DENG J，DONG W，SOCHER R，et al.Imagenet：A large-scale hierarchical image database[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition，2009：248-255.
[81] CIMPOI M，MAJI S，KOKKINOS I，et al.Describing textures in the wild[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2014：3606-3613.
[82] DUCK S Y.Painter by numbers[DB/OL].（2016-06）[2021-04-01].https：//www.kaggle.com/c/painter-by-numbers.
[83] CORDTS M，OMRAN M，RAMOS S，et al.The cityscapes dataset for semantic urban scene understanding[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：3213-3223.
[84] TYLECEK R，SARA R.Spatial pattern templates for recognition of objects with regular structure[C]//Proceedings of the German Conference on Pattern Recognition，2013：364-374.
[85] YU A，GRAUMAN K.Fine-grained visual comparisons with local learning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2014：192-199.
[86] ZHANG R，ISOLA P，EFROS A A.Colorful image colorization[C]//Proceedings of the European Conference on Computer Vision，2016：649-666.
[87] LUAN F，PARIS S，SHECHTMAN E，et al.Deep painterly harmonization[J].Computer Graphics Forum，2018，37（4）：95-106.
[88] ZHANG Y，FANG C，WANG Y，et al.Multimodal style transfer via graph cuts[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：5943-5951.