基于单幅图像形状特征的三维漫画人脸重建

doi:10.3778/j.issn.1002-8331.2308-0490

摘要/Abstract

摘要： 针对单幅图像的三维漫画人脸重建存在地标检测准确性差和生成模型还原高频细节能力低的问题，提出了一种多尺度特征融合与高频信息映射的两阶段方法。在第一阶段中，多尺度通道融合地标检测器用于提高检测的准确性。其中多尺度特征由HRNet产生；由通道注意力和Swin Transformer构成的注意力层用于多尺度通道融合特征提取；为了提高生成地标的精度，损失函数由地标损失和热图损失两部分构成。在第二阶段中，傅里叶特征共享层变形网络使生成的三维漫画人脸具有更丰富的高频形状细节。其中傅里叶特征映射提取高维特征，使网络学习更多形状的高频信息；共享层超网络加快了网络的收敛和重建速度。该方法应用于CaricatureFace和3DCaricShop数据集。实验结果表明，该方法中的地标检测器的平均检测误差减少了4.4%；变形网络在形状重建上的均方误差减少了26%，并且平均重建时间减少了18%；最终重建出的三维漫画人脸具有夸张的形状和自然的细节。

关键词: 地标检测, 三维漫画人脸, 人脸重建, 三维形变模型, 深度学习, 自解码器

Abstract: Aiming at the problems of poor landmark detection accuracy and low ability of the generation model to restore high-frequency details in 3D caricature reconstruction from a single image, this paper proposes a two-stage method of multi-scale feature fusion and high-frequency information mapping. In the first stage, a multi-scale channel fusion landmark detector is used to improve the detection accuracy. The multi-scale features are generated by HRNet, and the attention layer composed of channel attention and Swin Transformer is used for multi-scale channel fusion feature extraction. In order to improve the accuracy of generating landmarks, the loss function consists of two parts: landmark loss and heat map loss. In the second stage, the Fourier feature share layer deformable network enables the generated 3D caricature to have richer high-frequency shape details. Among them, the Fourier feature map extracts high-dimensional features, so that the network can learn more high-frequency information of shapes, and the share layer hypernetwork accelerates the convergence and reconstruction speed of the network. The method is applied to the CaricatureFace and 3DCaricShop datasets. Experimental results show that the average detection error of the landmark detector in this method is reduced by 4.4%; the mean square error of the deformation network in shape reconstruction is reduced by 26%, and the average reconstruction time is reduced by 18%; the final reconstructed 3D caricatures have exaggerated shapes and natural details.

Key words: landmark detection, 3D caricatures, face reconstruction, 3D deformable model, deep learning, auto-decoder

孙刘杰, 王佳耀, 王文举. 基于单幅图像形状特征的三维漫画人脸重建[J]. 计算机工程与应用, 2025, 61(1): 282-290.

SUN Liujie, WANG Jiayao, WANG Wenju. 3D Caricature Reconstruction Based on Shape Features of Single Image[J]. Computer Engineering and Applications, 2025, 61(1): 282-290.

参考文献

[1] WU Q, ZHANG J, LAI Y K, et al. Alive caricature from 2D to 3D[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 7336-7345.
[2] KING D E. Dlib-ml: a machine learning toolkit[J]. Journal of Machine Learning Research, 2009, 10: 1755-1758.
[3] CAI H, GUO Y, PENG Z, et al. Landmark detection and 3D face reconstruction for caricature using a nonlinear parametric model[J]. Graphical Models, 2021, 115: 101103.
[4] YE Z, XIA M, SUN Y, et al. 3D-CariGAN: an end-to-end solution to 3D caricature generation from normal face photos[J]. IEEE Transactions on Visualization and Computer Graphics, 2023,?29(4): ?2203-2210.
[5] LIU F, TRAN L, LIU X. 3D face modeling from diverse raw scan data[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 9408-9418.
[6] SAITO S, HUANG Z, NATSUME R, et al. PIFu: pixel-aligned implicit function for high-resolution clothed human digitization[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 2304-2314.
[7] LI S, SU S, LIN J, et al. Deep 3D caricature face generation with identity and structure consistency[J]. Neurocomputing, 2021, 454: 178-188.
[8] HUANG M, DAI J, PAN J, et al. 3D-CariNet: end-to-end 3D caricature generation from natural face images with differentiable renderer[C]//Proceedings of the 29th Pacific Conference on Computer Graphics and Applications, 2021.
[9] QIU Y, XU X, QIU L, et al. 3DCaricShop: a dataset and a baseline method for single-view 3D caricature face reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 10236-10245.
[10] JUNG Y, JANG W, KIM S, et al. Deep deformable 3D caricatures with learned shape control[C]//Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference, 2022: 1-9.
[11] LV J, SHAO X, XING J, et al. A deep regression architecture with two-stage re-initialization for high performance facial landmark detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 3317-3326.
[12] TRIGEORGIS G, SNAPE P, NICOLAOU M A, et al. Mnemonic descent method: a recurrent process applied for end-to-end face alignment[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 4177-4187.
[13] NEWELL A, YANG K, DENG J. Stacked Hourglass networks for human pose estimation[C]//Proceedings of the 14th European Conference on Computer Vision, Amsterdam, The Netherlands, Oct 11-14, 2016. Cham: Springer International Publishing, 2016: 483-499.
[14] SUN K, ZHAO Y, JIANG B, et al. High-resolution representations for labeling pixels and regions[J]. arXiv:1904.04514, 2019.
[15] NIBALI A, HE Z, MORGAN S, et al. Numerical coordinate regression with convolutional neural networks[J]. arXiv:1801.07372, 2018.
[16] HAO-PAN R E N, WEN-MING W, DE-JIAN W E I, et al. Human pose estimation based on high-resolution net[J]. Journal of Graphics, 2021, 42(3): 432.
[17] WANG X, WANG W, LU J, et al. HRST: an improved HRNet for detecting joint points of pigs[J]. Sensors, 2022, 22(19): 7215.
[18] CARION N, MASSA F, SYNNAEVE G, et al. End-to-end object detection with transformers[C]//Proceedings of the European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 213-229.
[19] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv:2010.11929, 2020.
[20] WU W, CAI Y, ZHOU Q. Transmarker: a pure vision transformer for facial landmark detection[C]//Proceedings of the 2022 26th International Conference on Pattern Recognition (ICPR), 2022: 3580-3587.
[21] LI J, JIN H, LIAO S, et al. RePFormer: refinement pyramid transformer for robust facial landmark detection[J]. arXiv:2207.03917, 2022.
[22] LIU Z, LIN Y, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 10012-10022.
[23] PARK J J, FLORENCE P, STRAUB J, et al. DeepSDF: learning continuous signed distance functions for shape representation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 165-174.
[24] MESCHEDER L, OECHSLE M, NIEMEYER M, et al. Occupancy networks: learning 3D reconstruction in function space[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 4460-4470.
[25] LIN C H, WANG C, LUCEY S. SDF-SRN: learning signed distance 3D object reconstruction from static images[C]//Advances in Neural Information Processing Systems, 2020: 11453-11464.
[26] TANCIK M, SRINIVASAN P, MILDENHALL B, et al. Fourier features let networks learn high frequency functions in low dimensional domains[C]//Advances in Neural Information Processing Systems, 2020: 7537-7547.
[27] LANDGRAF Z, HORNUNG A S, CABRAL R S. PINs: progressive implicit networks for multi-scale neural representations[J]. arXiv:2202.04713, 2022.
[28] HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 7132-7141.
[29] HUO J, LI W, SHI Y, et al. WebCaricature: a benchmark for caricature recognition[J]. arXiv:1703.03230, 2017.
[30] LI W, XIONG W, LIAO H, et al. CariGAN: caricature generation through weakly paired adversarial learning[J]. Neural Networks, 2020, 132: 66-74.
[31] ZAFEIRIOU S, TRIGEORGIS G, CHRYSOS G, et al. The menpo facial landmark localisation challenge: a step towards the solution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017: 170-179.
[32] KOWALSKI M, NARUNIEC J, TRZCINSKI T. Deep alignment network: a convolutional neural network for robust face alignment[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017: 88-97.
[33] KAZEMI V, SULLIVAN J. One millisecond face alignment with an ensemble of regression trees[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014: 1867-1874.
[34] WU Y, HASSNER T, KIM K G, et al. Facial landmark detection with tweaked convolutional neural networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(12): 3067-3074.
[35] SINDEL A, MAIER A, CHRISTLEIN V. ArtFacePoints: high-resolution facial landmark detection in paintings and prints[C]//Proceedings of the European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 298-313.