Photorealistic Human Body Animation Synthesis Based on Single Image

doi:10.3778/j.issn.1002-8331.2104-0305

Abstract

Abstract: Because a single image lacks three-dimensional information and complete texture information, the synthesis of realistic three-dimensional human body animation based on a single image is extremely challenging. Aiming at the problem of the lack of 3D information in a single image, a 3D human body geometric reconstruction method based on the SMPL parameter model is proposed. This method takes a single image as input, first generates a three-dimensional geometric model of the front and back surfaces that are consistent with the target contour according to the SMPL parameter model of the human body contour information deformation standard of the input image, then uses the mesh splice fusion algorithm based on B-spline interpolation to splice the front and back 3D geometry, and finally in order to restore the correct hand geometry, uses the mesh stitching fusion algorithm based on B-spline interpolation to correct the wrong hand geometry on the reconstructed model with the correct hand geometry on the standard SMPL parameter model. At the same time, in response to the problem of texture loss in a single image, a confrontation generation network called FBN（front to back network） is proposed to restore the occluded back texture of the human body. Experimental results show that the human body geometry with complete texture generated by this method can be driven by 3D motion data to generate a highly realistic three-dimensional human body animation.

Key words: human body animation generation, 3D human body reconstruction, whole body texture generation, mesh fusion, deep learning

摘要： 由于单幅图像缺失三维信息以及完整的纹理信息，基于单幅图像的真实感三维人体动画合成极具挑战性。针对单幅图像三维信息缺失问题，提出了一种基于SMPL参数模型的三维人体几何重建方法。该方法以单幅图像为输入，先根据输入图像人体轮廓信息变形标准的SMPL参数模型分别生成与目标轮廓一致的正反面的三维几何模型，然后利用基于B样条插值的网格拼接融合算法拼接正反面三维几何，最后为了恢复正确的手部几何，利用基于B样条插值的网格拼接融合算法，将重建后的模型上错误的手部几何用标准SMPL参数模型上正确的手部几何替换。同时，针对单幅图像中纹理缺失的问题，提出了一个称为FBN（front to back network）的对抗生成网络，用于恢复被遮挡的人体背面纹理。实验结果表明，该方法生成的具有完整纹理的人体几何能够由3D运动数据驱动运动，生成具有高度真实感的三维人体动画。

关键词: 人体动画生成, 三维人体重建, 全身纹理生成, 网格融合, 深度学习

XU Jiale, XIAO Xuezhong. Photorealistic Human Body Animation Synthesis Based on Single Image[J]. Computer Engineering and Applications, 2022, 58(22): 229-238.

徐家乐, 肖学中. 基于单幅图像的真实感人体动画合成[J]. 计算机工程与应用, 2022, 58(22): 229-238.

References

[1] HORNUNG A，DEKKERS E，KOBBELT L.Character animation from 2D pictures and 3D motion data[J].ACM Transactions on Graphics，2007，26（1）：1.
[2] MA L，JIA X，SUN Q，et al.Pose guided person image generation[C]//Advances in Neural Information Processing Systems 30，2017：406-416.
[3] ESSER P，SUTTER E，OMMER B.A variational U-Net for conditional appearance and shape generation[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition，2018：8857-8866.
[4] SIAROHIN A，SANGINETO E，LATHUILIERE S，et al.Deformable GANs for pose-based human image generation[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition，2018：3408-3416.
[5] ZHU Z，HUANG T，SHI B，et al.Progressive pose attention transfer for person image generation[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：2347-2356.
[6] YANG L，WANG P，ZHANG X，et al.Region-adaptive texture enhancement for detailed person image synthesis[C]//2020 IEEE International Conference on Multimedia and Expo，2020：1-6.
[7] REN Y，YU X，CHEN J，et al.Deep image spatial transformation for person image generation[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：7690-7699.
[8] LI Y，HUANG C，LOY C C.Dense intrinsic appearance flow for human pose transfer[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：3693-3702.
[9] LIU W，PIAO Z，MIN J，et al.Liquid Warping GAN：a unified framework for human motion imitation，appearance transfer and novel view synthesis[C]//2019 IEEE/CVF International Conference on Computer Vision，2019：5904-5913.
[10] NATSUME R，SAITO S，HUANG Z，et al.SiCloPe：Silhouette-based clothed people[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：4480-4490.
[11] SAITO S，HUANG Z，NATSUME R，et al.PIFu：pixel-aligned implicit function for high-resolution clothed human digitization[C]//2019 IEEE/CVF International Conference on Computer Vision，2019：2304-2314.
[12] SAITO S，SIMON T，SARAGIH J，et al.PIFuHD：multi-level pixel-aligned implicit function for high-resolution 3D human digitization[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：84-93.
[13] ALLDIECK T，PONS-MOLL G，THEOBALT C，et al.Tex2Shape：detailed full human body geometry from a single image[C]//2019 IEEE/CVF International Conference on Computer Vision，2019：2293-2303.
[14] HUANG Z，XU Y，LASSNER C，et al.ARCH：animatable reconstruction of clothed humans[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：3093-3102.
[15] LOPER M，MAHMOOD N，ROMERO J，et al.SMPL：a skinned multi-person linear model[J].ACM Transactions on Graphics，2015，34（6）：1-16.
[16] KOLOTOUROS N，PAVLAKOS G，BLACK M J，et al.Learning to reconstruct 3D human pose and shape via model-fitting in the loop[C]//2019 IEEE/CVF International Conference on Computer Vision，2019：2252-2261.
[17] KANAZAWA A，BLACK M J，JACOBS D W，et al.End-to-end recovery of human shape and pose[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition，2018：7122-7131.
[18] PAVLAKOS G，CHOUTAS V，GHORBANI N，et al.Expressive body capture：3D hands，face，and body from a single image[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：10975-10985.
[19] WENG C Y，CURLESS B，KEMELMACHER-SHLIZERMAN I.Photo Wake-Up：3D character animation from a single photo[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：5908-5917.
[20] MAHMOOD N，GHORBANI N，TROJE N F，et al.AMASS：archive of motion capture as surface shapes[C]//2019 IEEE/CVF International Conference on Computer Vision，2019：5442-5451.
[21] ALLDIECK T，MAGNOR M，XU W，et al.Video based reconstruction of 3D people models[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition，2018：8387-8397.
[22] CHEN L C，ZHU Y，PAPANDREOU G，et al.Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//15th European Conference on Computer Vision，2018：801-818.
[23] CAO Z，HIDALGO G，SIMON T，et al.OpenPose：realtime multi-person 2D pose estimation using part affinity fields[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2019，43（1）：172-186.
[24] FLOATER M S.Mean value coordinates[J].Computer Aided Geometric Design，2003，20（1）：19-27.
[25] RONNEBERGER O，FISCHER P，BROX T.U-Net：convolutional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham：Springer，2015：234-241.
[26] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409.
1556，2014.
[27] ISOLA P，ZHU J Y，ZHOU T，et al.Image-to-image translation with conditional adversarial networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition，2017：1125-1134.
[28] ZHU H，SU H，WANG P，et al.View extrapolation of human body from a single image[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition，2018：4450-4459.
[29] ZHENG Z，YU T，WEI Y，et al.DeepHuman：3D human reconstruction from a single image[C]//2019 IEEE/CVF International Conference on Computer Vision，2019：7739-7749.
[30] GLOROT X，BENGIO Y.Understanding the difficulty of training deep feedforward neural networks[C]//13th International Conference on Artificial Intelligence and Statistics，2010：249-256.
[31] VAROL G，CEYLAN D，RUSSELL B，et al.BodyNet：volumetric inference of 3D human body shapes[C]//15th European Conference on Computer Vision，2018：20-36.