Content Structure Preserved Image Style Transfer Method

doi:10.3778/j.issn.1002-8331.2110-0107

Abstract

Abstract: Most existing style transfer algorithms realize the conversion task from art works to real scenes, such as photos. Under this experiment setting, the content structure of the image may change to a certain extent. However, this change is not conducive to the completion of downstream tasks, so it is not available for most industrial scenes. Therefore, a new structure preserving style transfer method is proposed, which contains a frequency domain constrained image generation module and a semantic matching module based on memory bank. The former is used to ensure the consistency of the overall semantic structure of the image before and after transfer, while the latter ensures the semantic and style matching, so that the internal structure of the object is consistent. By comparing the frequency domain similarity between the original image and the migrated image and the similarity between the generated image and the corresponding category in memory bank, the purpose of keeping the image structure unchanged is achieved. In order to verify the effect this method produced in the industrial scene, the photovoltaic panel dataset is used. The proposed style transfer algorithm can not only complete the style conversion, but also better resolve the problem of image structure deformation, so as to meet the needs of subsequent tasks.

Key words: image style transfer, generative adversarial network, computer vision

摘要： 现有的风格迁移算法大多是实现艺术作品到真实场景，如照片等的转换任务。在这种任务设定下，图像内容的边界等结构信息可能会出现一定程度的变化，然而这种变化不利于下游任务的完成，因而不适用于大部分工业场景。为此提出一种新的结构保持的风格迁移方法，该方法分为频域约束的图像生成模块和基于memory bank机制的语义匹配模块。前者用以保证转换前后图像的整体语义结构一致性，而后者保证了图像的语义与风格匹配，从而物体内部结构一致。为了验证该方法在工业场景中的迁移效果，采用光伏板数据集，在虚拟到真实的迁移方向上，所提出的风格迁移算法能在完成风格转换的同时较好地解决图像结构形变问题，进而满足后续任务的需求。

关键词: 图像风格迁移, 生成对抗网络, 计算机视觉

WANG Xiaoming, MAO Yushi, XU Bin, WANG Zilei. Content Structure Preserved Image Style Transfer Method[J]. Computer Engineering and Applications, 2023, 59(6): 146-154.

王小明, 毛语实, 徐斌, 王子磊. 内容结构保持的图像风格迁移方法[J]. 计算机工程与应用, 2023, 59(6): 146-154.

References

[1] PANG Y，LIN J，QIN T，et al.Image-to-image translation：methods and applications[J].arXiv：2101.08629，2021.
[2] PARK T，LIU M Y，WANG T C，et al.Semantic image synthesis with spatially-adaptive normalization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，Long Beach，Jun 15-21，2019：2337-2346.
[3] 田敏，刘名果，陈立家，等.面向样本扩充的新型风格迁移网络研究[J].计算机工程与应用，2023，59（1）：228-235.
TIAN M，LIU M G，CHEN L J，et al.Research on new style transfer network for sample expansion[J].Computer Engineering and Applications，2023，59（1）：228-235.
[4] LI S，GUNEL S，OSTREK M，et al.Deformation-aware unpaired image translation for pose estimation on laboratory animals[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，Seattle，Jun 16-19，2020：13158-13168.
[5] MUREZ Z，KOLOURI S，KRIEGMAN D，et al.Image to image translation for domain adaptation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Salt Lake City，June 18-23，2018：4500-4509.
[6] DENTON E，BIRODKAR V.Unsupervised learning of disentangled representations from video[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems，Long Beach，Dec 4-9，2017：4417-4426.
[7] CHEN X，DUAN Y，HOUTHOOFT R，et al.Infogan：interpretable representation learning by information maximizing generative adversarial nets[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems，Barcelona，Dec 5-10，2016：2180-2188.
[8] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial nets[C]//Advances in Neural Information Processing Systems，Montréal，Dec 8-13，2014：2672-2680.
[9] CHANG H Y，WANG Z，CHUANG Y Y.Domain-specific mappings for generative adversarial style transfer[C]//European Conference on Computer Vision，Glasgow，Aug 13-18，2020：573-589.
[10] 李美丽，杨传颖，石宝.基于语义分割的图像风格迁移技术研究[J].计算机工程与应用，2020，56（24）：207-213.
LI M L，YANG C Y，SHI B.Research on image style transfer technology based on semantic segmentation[J].Computer Engineering and Applications，2020，56（24）：207-213.
[11] 林晓，屈时操，黄伟，等.显著区域保留的图像风格迁移算法[J].图学学报，2021，42（2）：190-197.
LIN X，QU S C，HUANG W，et al.Style transfer algorithm for salient region preservation[J].Journal of Graphics，2021，42（2）：190-197.
[12] MIRZA M，OSINDERO S.Conditional generative adversarial nets[J].arXiv：1411.1784，2014.
[13] ARJOVSKY M，CHINTALA S，BOTTOU L.Wasserstein generative adversarial networks[C]//International Conference on Machine Learning，Sydney，Aug 7-9，2017：214-223.
[14] MIYATO T，KATAOKA T，KOYAMA M，et al.Spectral normalization for generative adversarial networks[C]//International Conference on Learning Representations，Vancouver，Apr 30-May 3，2018.
[15] ZHU J Y，PARK T，ISOLA P，et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision，Venice，October 22-29，2017：2223-2232.
[16] HUANG X，LIU M Y，BELONGIE S，et al.Multimodal unsupervised image-to-image translation[C]//Proceedings of the European Conference on Computer Vision，Munich，Sep 8-14，2018：172-189.
[17] 兰天，辛月兰，殷小芳，等.基于生成对抗网络的无监督图像风格迁移[J].计算机工程与科学，2021，43（10）：1789-1795.
LAN T，XIN Y L，YIN X F，et al.Unsupervised image style transfer based on generating adversarial network[J].Computer Engineering & Science，2021，43（10）：1789-1795.
[18] LI M，HUANG H，MA L，et al.Unsupervised image-to-image translation with stacked cycle-consistent adversarial networks[C]//Proceedings of the European Conference on Computer Vision，Munich，Sep 8-14，2018：184-199.
[19] LEE H Y，TSENG H Y，MAO Q，et al.Drit++：diverse image-to-image translation via disentangled representations[J].International Journal of Computer Vision，2020，128（10）：2402-2417.
[20] TOMEI M，CORNIA M，BARALDI L，et al.Art2real：unfolding the reality of artworks via semantically-aware image-to-image translation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，Long Beach，Jun 15-21，2019：5849-5859.
[21] JOHNSON J，DOUZE M，JéGOU H.Billion-scale similarity search with Gpus[J].arXiv：1702.08734v1，2017.
[22] YE K，YE Y，YANG M，et al.Independent encoder for deep hierarchical unsupervised image-to-image translation[J].arXiv：2107.02494，2021.
[23] HAN J，SHOEIBY M，PETERSSON L，et al.Dual contrastive learning for unsupervised image-to-image translation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，June19-25，2021：746-755.
[24] HEUSEL M，RAMSAUER H，UNTERTHINER T，et al.Gans trained by a two time-scale update rule converge to a local nash equilibrium[C]//Advances in Neural Information Processing Systems，2017.
[25] SZEGEDY C，VANHOUCKE V，IOFFE S，et al.Rethinking the inception architecture for computer vision[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition，2016：2818-2826.