Image Arbitrary Style Transfer via Super-Resolution Reconstruction

doi:10.3778/j.issn.1002-8331.2305-0042

Abstract

Abstract: Image style transfer refers to the transformation of an ordinary photograph into an image with other artistic style effects. To address the problems of low definition and lack of texture details in the generated image due to the inability to reconstruct the resolution of the generated image in style transfer algorithms, an image arbitrary style transfer via super-resolution reconstruction is proposed. The multi-channel feature processing module added in the model enhances the feature expression by calculating the feature self-similarity, and a feature fusion module is proposed to improve the feature fusion effect. A feature decoder module is proposed to realize the image super-resolution reconstruction, in which features are fused several times to improve the stylized image quality. In the loss function, generative adversarial loss and whitening process are added to further improve the stylization effect. The experiment shows the model has great arbitrary style transfer effect, and the stylized image during resolution reconstruction has rich details and clear texture.

Key words: arbitrary style transfer, super-resolution reconstruction, generative adversarial network, self-attention mechanism

摘要： 图像风格迁移是指将一张普通照片转化为具有其他艺术风格效果的图像。针对风格迁移算法中无法重构生成图像的分辨率而造成生成图像清晰度低、纹理细节表现不丰富的问题，提出一种融合超分辨率重构的图像任意风格迁移模型。模型中加入的多支路特征处理模块通过计算特征的自相似性以增强特征的表达，提出新的特征融合模块以提升特征融合效果，提出特征解码模块来实现图像的超分辨率重构，并在其中多次进行特征融合以提升风格化图像的质量；在损失函数中加入生成对抗损失和白化处理来进一步提升风格化效果。实验表明，模型具有较好的任意风格迁移效果，分辨率重构后的风格化图像的细节丰富、纹理清晰。

关键词: 任意风格迁移, 超分辨率重构, 生成对抗网络, 自注意力机制

TAN Run, TIAN Qichuan, LIAN Lu, ZHANG Xiaohang. Image Arbitrary Style Transfer via Super-Resolution Reconstruction[J]. Computer Engineering and Applications, 2024, 60(15): 170-179.

谭润, 田启川, 廉露, 张晓行. 融合超分辨率重构的图像任意风格迁移[J]. 计算机工程与应用, 2024, 60(15): 170-179.

References

[1] 陈淮源, 张广驰, 陈高, 等. 基于深度学习的图像风格迁移研究进展[J]. 计算机工程与应用, 2021, 57(11): 37-45.
CHEN H Y, ZHANG G C, CHEN G, et al. Research progress of image style transfer based on deep learning[J]. Computer Engineering and Applications, 2021, 57(11): 37-45.
[2] HERTZMANN A, JACOBS C E, OLIVER N, et al. Image analogies[C]//The 28th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 2001: 327-340.
[3] GATYS L A, ECKER A S, BETHGE M. Image style transfer using convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE Press, 2016: 2414-2423.
[4] JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution[C]//European Conference on Computer Vision, 2016: 694-711.
[5] LIU X C, CHENG M M, LAI Y K, et al. Depth-aware neural style transfer[C]//Proceedings of the Symposium on Non-Photorealistic Animation and Rendering. Los Angeles: IEEE Press, 2017: 1-10.
[6] JING Y, LIU Y, YANG Y, et al. Stroke controllable fast style transfer with adaptive receptive fields[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 238-254.
[7] KOTOVENKO D, SANAKOYEU A, LANG S, et al. Content and style disentanglement for artistic style transfer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul: IEEE Press, 2019: 4422-4431.
[8] CHEN D, YUAN L, LIAO J, et al. Stylebank: an explicit representation for neural image style transfer[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 1897-1906.
[9] LI Y, FANG C, YANG J, et al. Diversified texture synthesis with feed-forward networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 3920-3928.
[10] LI Y, FANG C, YANG J, et al. Universal style transfer via feature transforms[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 385-395.
[11] HUANG X, BEONGIE S. Arbitrary style transfer in real-time with adaptive instance normalization[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 1501-1510.
[12] JING Y, LIU X, DING Y, et al. Dynamic instance normalization for arbitrary style transfer[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New York: AAAI, 2020: 4369-4376.
[13] CHEN T Q, SCHMIDT M. Fast patch-based style transfer of arbitrary style[C]//30th Conference on Neural Information Processing Systems, 2016.
[14] PARK D Y, LEE K H. Arbitrary style transfer with style-attentional networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 5880-5888.
[15] LIU S, LIN T, HE D, et al. Adaattn: revisit attenti-on mechanism in arbitrary neural style transfer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 6649-6658.
[16] LUO X, HAN Z, YANG L, et al. Consistent style transfer[EB/OL]. (2022-01-06)[2023-06-12]. https://arxiv.org/abs/2201.02233.
[17] 李鑫, 普园媛, 赵征鹏, 等. 内容语义和风格特征匹配一致的艺术风格迁移[J]. 图学学报, 2023, 44(4): 699-709.
LI X, PU Y Y, ZHAO Z P, et al. Content semantics and style features match consistent artistic style transfer[J]. Journal of Graphics , 2023, 44(4): 699-709.
[18] LI Y, CHEN X, ZHU Z, et al. Attention-guided unified network for panoptic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 7026-7035.
[19] CHIU T Y. Understanding generalized whitening and coloring transform for universal style transfer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 4452-4460.
[20] KARRAS T, LAINE S, AITTALA M, et al. Analyzing and improving the image quality of stylegan[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 8110-8119.
[21] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[EB/OL]. (2014-06-10)[2023-06-12]. https://arxiv.org/abs/1406.2661.
[22] HEUSEL M, RAMSAUER H, UNTERTHINER T, et al. GANs trained by a two time-scale update rule converge to a local nash equilibrium[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 6629-6640.
[23] WANG Z, ZHAO L, CHEN H, et al. Evaluate and improve the quality of neural style transfer[J]. Computer Vision and Image Understanding, 2021, 207: 103203.