Attribute Editable Person Image Synthesis Based on Spatial Transformation

doi:10.3778/j.issn.1002-8331.2009-0394

Abstract

Abstract: Aiming at the problem that the existing pose-guided human body image synthesis methods cannot flexibly edit the appearance attributes of the human body, the appearance attributes of different parts can be obtained by semantic segmentation of the input human image, and the purpose of simultaneously editing the posture and appearance attributes of human body can be achieved by replacing these attributes. In order to solve the problem that the pattern on the clothes can not be generated well by the purely convolutional structure of the generative adversarial networks, a new spatial transformation algorithm is proposed to transform the multi-source feature map, so that the pattern on the clothes can be preserved. Experimental results show that the proposed method achieves the purpose of controlling human body attributes and has better visual effects than the previous methods.

Key words: generative adversarial networks, spatial transformation, person image synthesis

摘要： 针对现有姿态引导的人体图像合成方法无法灵活地编辑人体外观属性的问题，通过对输入的人体图像进行语义分割得到不同部位的外观属性，替换这些属性即可达到同时编辑人体姿态与外观属性的目的。针对纯卷积结构的生成对抗网络无法很好地生成衣服上的图案的问题，提出了新的空间变换算法对多个源属性的特征图进行空间变换，使衣服上的图案得到了保留。实验结果表明，所提出的方法实现了控制人体属性的目的，并且相比于之前的方法具有更好的视觉效果。

关键词: 生成对抗网络, 空间变换, 人体图像合成

WEI Chengfeng, DONG Hongwei, XU Xiaochun. Attribute Editable Person Image Synthesis Based on Spatial Transformation[J]. Computer Engineering and Applications, 2022, 58(6): 219-226.

魏程峰, 董洪伟, 徐小春. 基于空间变换的属性可编辑的人体图像合成[J]. 计算机工程与应用, 2022, 58(6): 219-226.

References

[1] JADERBERG M，SIMONYAN K，ZISSERMAN A.Spatial transformer networks[C]//Advances in Neural Information Processing Systems，2015：2017-2025.
[2] LIN C H，LUCEY S.Inverse compositional spatial transformer networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：2568-2576.
[3] JIANG W，SUN W，TAGLIASACCHI A，et al.Linearized multi-sampling for differentiable image transformation[C]//Proceedings of the IEEE International Conference on Computer Vision，2019：2988-2997.
[4] ZHOU T，TULSIANI S，SUN W，et al.View synthesis by appearance flow[C]//European Conference on Computer Vision.Cham：Springer，2016：286-301.
[5] WANG T C，LIU M Y，ZHU J Y，et al.Video-to-video synthesis[J].arXiv：1808.06601，2018.
[6] SIAROHIN A，LATHUILIèRE S，TULYAKOV S，et al.Animating arbitrary objects via deep motion transfer[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2019：2377-2386.
[7] MA L，JIA X，SUN Q，et al.Pose guided person image generation[C]//Advances in Neural Information Processing Systems，2017：406-416.
[8] ESSER P，SUTTER E，OMMER B.A variational u-net for conditional appearance and shape generation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：8857-8866.
[9] SIAROHIN A，SANGINETO E，LATHUILIERE S，et al.Deformable gans for pose-based human image generation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：3408-3416.
[10] ZHU Z，HUANG T，SHI B，et al.Progressive pose attention transfer for person image generation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2019：2347-2356.
[11] HAN X，HU X，HUANG W，et al.Clothflow：a flow-based model for clothed person generation[C]//Proceedings of the IEEE International Conference on Computer Vision，2019：10471-10480.
[12] REN Y，YU X，ZHANG R，et al.Structureflow：image inpainting via structure-aware appearance flow[C]//Proceedings of the IEEE International Conference on Computer Vision，2019：181-190.
[13] REN Y，YU X，CHEN J，et al.Deep image spatial transformation for person image generation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：7690-7699.
[14] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial nets[C]//Advances in Neural Information Processing Systems，2014：2672-2680.
[15] JOHNSON J，ALAHI A，LI F F.Perceptual losses for real-time style transfer and super-resolution[C]//European Conference on Computer Vision.Cham：Springer，2016：694-711.
[16] LIU Z，LUO P，QIU S，et al.Deepfashion：powering robust clothes recognition and retrieval with rich annotations[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：1096-1104.
[17] GONG K，LIANG X，LI Y，et al.Instance-level human parsing via part grouping network[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：770-785.
[18] LIU W，PIAO Z，MIN J，et al.Liquid warping GAN：a unified framework for human motion imitation，appearance transfer and novel view synthesis[C]//Proceedings of the IEEE International Conference on Computer Vision，2019：5904-5913.
[19] MEN Y，MAO Y，JIANG Y，et al.Controllable person image synthesis with attribute-decomposed gan[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：5084-5093.