融合多注意力机制的语义调整风格迁移网络

doi:10.3778/j.issn.1002-8331.2311-0047

摘要/Abstract

摘要： 风格迁移是一种计算机视觉技术，旨在将一幅图像的风格迁移到另一幅图像上，从而创造出拥有新风格的图像。但当前任意风格迁移网络中还存在一些问题，如融合后的风格化图像语义不清晰以及整体风格不一致等。为了解决这些问题，提出了一种新的多注意力风格迁移网络MatST。该网络结合了语义调整的方法，通过引入一系列注意力机制来改进风格迁移的效果。提出了RCCAB模块，通过结合交叉卷积和通道注意力机制，解决图像定位和细节表示的问题。结合窗口自注意力、重叠交叉窗口注意力OCAB和多头注意力块MHAB，设计了多注意力模块MAB作为Transformer编码器的子层。MAB模块从多个维度提取图像特征，解决图像网格化和风格化不细致的问题。设计了风格化图像语义调整器，通过反馈传播的方式来调整风格化图像的语义信息，生成语义清晰且更符合人眼感知的风格化图像。实验结果表明，相对于StyTr2网络，MatST网络在COCO数据集上内容损失降低0.172?5，同时风格损失减少0.075?7。经实验验证，该网络在获得较好风格化图像的同时，能够保留清晰的内容语义，具有良好的任意风格迁移效果。

关键词: 风格迁移, 语义调整, 注意力机制, 交叉卷积

Abstract: Style transfer is a computer vision technique aimed at transferring the style of one image to another, thereby creating an image with a new style. However, current arbitrary style transfer networks still suffer from some issues, such as unclear semantic fusion in stylized images and overall inconsistency in style. To address these problems, this paper proposes a new multi-attention style transfer network, MatST. This network integrates semantic adjustment methods and improves style transfer by introducing a series of attention mechanisms. Firstly, the RCCAB module is proposed to address the issues of image localization and detail representation by combining cross convolution and channel attention mechanisms. Secondly, by combining window self-attention, overlapping cross-window attention (OCAB), and multi-head attention block (MHAB), a multi-attention module (MAB) is designed as a sub-layer of the Transformer encoder. The MAB module extracts image features from multiple dimensions, addressing the problem of image gridization and coarse stylization. Finally, a stylized image semantic adjuster is designed to adjust the semantic information of stylized images through feedback propagation, generating stylized images with clear semantics and closer alignment with human perception. Experimental results show that compared to the StyTr2 network, the MatST network reduces content loss by 0.172?5 and style loss by 0.075?7 on the COCO dataset. Through experimental validation, the network in this paper retains clear semantic content while achieving good arbitrary style transfer effects, resulting in high-quality stylized images.

Key words: style transfer, semantic adjustment, attention mechanism, cross-convolution

张彩灯, 徐杨, 莫寒, 冯明文. 融合多注意力机制的语义调整风格迁移网络[J]. 计算机工程与应用, 2025, 61(8): 204-214.

ZHANG Caideng, XU Yang, MO Han, FENG Mingwen. Semantic Adjusted Style Transfer Network with Multi-Attention Mechanisms[J]. Computer Engineering and Applications, 2025, 61(8): 204-214.

参考文献

[1] 王小明, 毛语实, 徐斌, 等. 内容结构保持的图像风格迁移方法[J]. 计算机工程与应用, 2023, 59(6): 146-154.
WANG X M, MAO Y S, XU B, et al. Content structure preserved image style transfer method[J]. Computer Engineering and Applications, 2023, 59(6): 146-154.
[2] 谭润, 田启川, 廉露, 等. 融合超分辨率重构的图像任意风格迁移[J]. 计算机工程与应用, 2024, 60(15): 170-179.
TAN R, TIAN Q C, LIAN L, et al. Image arbitrary style transfer via super-resolution reconstruction[J]. Computer Engineering and Applications, 2024, 60(15): 170-179.
[3] ZHU J Y, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 2223-2232.
[4] HUANG X, LIU M Y, BELONGIE S, et al. Multimodal unsupervised image-to-image translation[C]//Proceedings of the European Conference on Computer Vision, 2018: 172-189.
[5] 孙泽, 张俊. 艺术图像生成的DeepDream技术研究与实现[J]. 电子技术与软件工程, 2021(18): 148-151.
SUN Z, ZHANG J. Research and implementation of DeepDream technology for artistic image generation[J]. Electronic Technology & Software Engineering, 2021(18): 148-151.
[6] KARRAS T, AITTALA M, LAINE S, et al. Alias-free generative adversarial networks[C]//Proceedings of the 35th Conference on Neural Information Processing Systems, 2021: 852-863.
[7] CHEN Y, LAI Y K, LIU Y J, et al. CartoonGAN: generative adversarial networks for photo cartoonization[C]//Proceedings of the 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 9465-9474.
[8] 朱渊略. 基于开关式生成对抗网络的人脸编辑[D]. 深圳: 深圳大学, 2021.
ZHU Y L. Face editing based on switchable generative adversarial networks[D]. Shenzhen: Shenzhen University, 2021.
[9] RUDER M, DOSOVITSKIY A, BROX T. Artistic style transfer for videos[C]//Proceedings of the 38th German Conference on Pattern Recognition, 2016: 26-36.
[10] GATYS L A, ECKER A S, BETHGE M. A neural algorithm of artistic style[J]. arXiv:1508.06576, 2015.
[11] EFROS A A, FREEMAN W T. Image quilting for texture synthesis and transfer[C]//Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, 2023: 571-576.
[12] GATYS L A, ECKER A S, BETHGE M. Image style transfer using convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 2414-2423.
[13] JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of the European Conference on Computer Vision, 2016: 694-711.
[14] YAO Y, REN J, XIE X, et al. Attention-aware multi-stroke style transfer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 1467-1475.
[15] PARK D Y, LEE K H. Arbitrary style transfer with style-attentional networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 5880-5888.
[16] DENG Y, TANG F, DONG W, et al. Stytr2: image style transfer with transformers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 11326-11336.
[17] ULYANOV D, VEDALDI A, LEMPITSKY V. Improved texture networks: maximizing quality and diversity in feed-forward stylization and texture synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6924-6932.
[18] LI Y, FANG C, YANG J, et al. Universal style transfer via feature transforms[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 385-395.
[19] LIU S, LIN T, HE D, et al. AdaAttN: revisit attention mechanism in arbitrary neural style transfer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 6649-6658.
[20] HUANG X, BELONGIE S. Arbitrary style transfer in real-time with adaptive instance normalization[C]//Proceedings of the 16th IEEE International Conference on Computer Vision, 2017: 1510-1519.
[21] DENG Y, TANG F, DONG W, et al. Arbitrary video style transfer via multi-channel correlation[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2021: 1210-1217.
[22] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 6000-6010.
[23] 谢刚, 王荃毅, 谢新林, 等. 融合多尺度深度卷积的轻量级Transformer交通场景语义分割算法[J]. 通信学报, 2023, 44(10): 1-13.
XIE G, WANG Q Y, XIE X L, et al. Lightweight Transformer traffic scene semantic segmentation algorithm integrating multi-scale depth convolution[J]. Journal on Communications, 2023, 44(10): 1-13.
[24] LIU Z, LIN Y, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 10012-10022.
[25] DENG Y, TANG F, DONG W, et al. Arbitrary style transfer via multi-adaptation network[C]//Proceedings of the 28th ACM International Conference on Multimedia, 2020: 2719-2727.
[26] CHEN H, WANG Z, ZHANG H, et al. Artistic style transfer with internal-external learning and contrastive learning[C]// Advances in Neural Information Processing Systems, 2021: 26561-26573.
[27] WU X, HU Z, SHENG L, et al. StyleFormer: real-time arbitrary style transfer via parametric style composition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 14618-14627.
[28] ZHANG Y, LI K, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the European Conference on Computer Vision, 2018: 286-301.
[29] CHEN X, WANG X, ZHOU J, et al. Activating more pixels in image super-resolution transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 22367-22377.
[30] LIU Y, JIA Q, FAN X, et al. Cross-SRN: structure-preserving super-resolution network with cross convolution[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 32(8): 4927-4939.
[31] ZHANG Y, TANG F, DONG W, et al. Domain enhanced arbitrary image style transfer via contrastive learning[C]//Proceedings of the Special Interest Group for Computer Graphics Conference, 2022: 1-8.
[32] WEN L, GAO C, ZOU C. CAP-VSTNet: content affinity preserved versatile style transfer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 18300-18309.
[33] ZHANG Q, YANG Y B. Rest: an efficient transformer for visual recognition[C]//Advances in Neural Information Processing Systems, 2021: 15475-15485.