Review of Single-Image 3D Face Reconstruction Methods

doi:10.3778/j.issn.1002-8331.2210-0041

Abstract

Abstract: In recent years, 3D face reconstruction task, as an important part of “digital human” technology, has received great attention from both academia and industry. In particular, 3D face reconstruction task based on a single image has made great progress by fully combining traditional camera model, illumination model, 3D face statistical deformation model with the deep convolutional neural network and deep generative models. This paper focuses on the single-image 3D face reconstruction problem, and divides the existing research works into two categories based on implicit space coding and explicit space regression. The first type of research works optimize the basis coefficient solution and loss function design of the basic 3D face statistical model to improve the reconstruction effect, which has the advantage of robustness in face topology change but lacks detailed features. The second type of research works represent 3D faces in the forms of multiple data in explicit space and regress them directly by deep networks, which can usually obtain more personalized 3D face detail features and have better robustness to interference factors such as illumination and occlusion. Furthermore, based on the commonly used datasets and evaluation metrics, this paper fully explores and compares the advantages and disadvantages of some typical methods of both categories. Finally, it summarizes the whole paper and points out the main challenges and future development trends of the single-image based 3D face reconstruction task.

Key words: 3D face reconstruction, single-image, explicit space regression, implicit space coding

摘要： 近年来，三维人脸重建任务作为“数字人”技术的重要组成部分，受到了学术界和工业界的广泛关注。基于单张图像的三维人脸重建任务在充分结合传统相机模型、光照模型、三维人脸统计形变模型与深度卷积网络、深度生成模型等方面技术之后取得了长足的进步。聚焦单张图像三维人脸重建问题，将现有研究工作分为基于隐空间编码和基于显空间回归两类。第一类研究工作对基础三维人脸统计模型的基系数求解、损失函数设计等进行优化，提升重建效果，在人脸拓扑结构变化方面具备鲁棒性优势，但缺乏细节特征。第二类工作以显空间多种数据形式表示三维人脸并直接通过深度网络进行回归，通常可获得更加个性化的三维人脸细节特征且对光照、遮挡等干扰因素具有较好的鲁棒性。进一步，基于常用数据集和评价指标，充分探讨并比较了两类方法中一些典型方法的优缺点。最后对全文进行总结，并给出了单张图像三维人脸重建任务面临的主要挑战及未来发展趋势。

关键词: 三维人脸重建, 单张图像, 显空间回归, 隐空间编码

WANG Jingting, LI Huibin. Review of Single-Image 3D Face Reconstruction Methods[J]. Computer Engineering and Applications, 2023, 59(17): 1-21.

王静婷, 李慧斌. 单张图像三维人脸重建方法综述[J]. 计算机工程与应用, 2023, 59(17): 1-21.

References

[1] 何嘉玉，黄宏博，张红艳，等.基于深度学习的单幅图像三维人脸重建研究综述[J].计算机科学，2022，49（2）：40-50.
HE J Y，HUANG H B，ZHANG H Y，et al.Review of 3D face reconstruction based on single image[J].Computer Science，2022，49（2）：40-50.
[2] 娄方园，齐梦娜，王竹新，等.元宇宙场域下的教育数字人及其应用[J].图书馆论坛，2023，43（3）：101-108.
LOU F Y，QI M N，WANG Z X，et al.Educational digital human and its application in the field of metaverse[J].Library Tribune，2023，43（3）：101-108.
[3] GREEN R.Spherical harmonic lighting：the gritty details[C]//Game Developers Conference，2003.
[4] SLOAN P P.Stupid spherical harmonics（SH） tricks[C]//Game Developers Conference，2008.
[5] PARKE F I.Measuring three-dimensional surfaces with a two-dimensional data tablet[J].Computers & Graphics，1975，1（1）：5-7.
[6] BLANZ V，VETTER T.A morphable model for the synthesis of 3D faces[C]//Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques，1999：187-194.
[7] PAYSAN P，KNOTHE R，AMBERG B，et al.A 3D face model for pose and illumination invariant face recognition[C]//2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance，2009： 296-301.
[8] CAO C，WENG Y L，ZHOU S，et al.Facewarehouse：a 3D facial expression database for visual computing[J].IEEE Transactions on Visualization and Computer Graphics， 2014，20（3）：413-425.
[9] BOOTH J，ROUSSOS A，PONNIAH A，et al.Large-scale 3D morphable models[J].International Journal of Computer Vision，2017，126（5）：1-22.
[10] RICHARDSON E，SELA M，OREL R，et al.Learning detailed face reconstruction from a single image[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：1259-1268.
[11] GECER B，PLOUMPIS S，KOTSIA I，et al.Ganfit：generative adversarial network fitting for high fidelity 3D face reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：1155-1164.
[12] JACKSON A S，BULAT A，ARGYRIOU V，et al.Large pose 3D face reconstruction from a single image via direct volumetric CNN regression[C]//2017 IEEE International Conference on Computer Vision，2017.
[13] FENG Y，WU F，SHAO X，et al.Joint 3D face reconstruction and dense alignment with position map regression network[C]//Proceedings of the European Conference on Computer Vision，2018：534-551.
[14] ZENG X X，PENG X J，QIAO Y.DF2Net：a dense-fine-finer network for detailed 3D face reconstruction[C]//IEEE International Conference on Computer Vision，2019：2315-2324.
[15] MILDENHALL B，SRINIVASAN P P，TANCIK M，et al.NERF：representing scenes as neural radiance fields for view synthesis[J].Communications of the ACM，2021，65（1）：99-106.
[16] HONG Y，PENG B，XIAO H，et al.HeadNeRF：a real-time NeRF-based parametric head model[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：20374-20384.
[17] VLASIC D，BRAND M，PFISTER H，et al.Face transfer with multilinear models[J].ACM Transactions on Graphics，2005，24（3）：426-435.
[18] YANG H，ZHU H，WANG Y，et al.Facescape：a large-scale high quality 3D face dataset and detailed riggable 3D face prediction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：601-610.
[19] YANG J，ZHANG D，FRANGI A F，et al.Two-dimensional PCA：a new approach to appearance-based face representation and recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2004，26（1）：131-137.
[20] JIN Z，BHANU B.Analysis-by-synthesis[M].[S.l.]：Elsevier Science Inc，2015.
[21] GERIG T，MOREL-FORSTER A，BLUMER C，et al.Morphable face models-an open framework[C]//2018 13th IEEE International Conference on Automatic Face & Gesture Recognition（FG 2018），2018：75-82.
[22] LI T，BOLKART T，BLACK M J，et al.Learning a model of facial shape and expression from 4D scans[J].ACM Trans Graph，2017，36（6）：1-17.
[23] HORN B K P，BROOKS M J.The variational approach to shape from shading[J].Computer Vision，Graphics，and Image Processing，1986，33（2）：174-208.
[24] FAN X，CHENG S，KANG H，et al.Dual neural networks coupling data regression with explicit priors for monocular 3D face reconstruction[J].IEEE Transactions on Multimedia，2020，23：1252-1263.
[25] LUO Y，TU X，XIE M.Learning robust 3D face reconstruction and discriminative identity representation[C]//2019 IEEE 2nd International Conference on Information Communication and Signal Processing（ICICSP），2019：317-321.
[26] DOU P，SHAH S K，KAKADIARIS I A.End-to-end 3D face reconstruction with deep neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：5908-5917.
[27] ZHU W，WU H T，CHEN Z，et al.Reda：reinforced differentiable attribute for 3D face reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：4958-4967.
[28] LIN J，YUAN Y，SHAO T，et al.Towards high-fidelity 3D face reconstruction from in-the-wild images using graph convolutional networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：5891-5900.
[29] SCHROFF F，KALENICHENKO D，PHILBIN J.FaceNet：a unified embedding for face recognition and clustering[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：815-823.
[30] ZHU X，LEI Z，LIU X，et al.Face alignment across large poses：a 3D solution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：146-155.
[31] GUO J，ZHU X，YANG Y，et al.Towards fast，accurate and stable 3D dense face alignment[C]//European Conference on Computer Vision.Cham：Springer，2020：152-168.
[32] ZHOU Y，DENG J，KOTSIA I，et al.Dense 3D face decoding over 2500fps：Joint texture & shape convolutional mesh decoders[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：1097-1106.
[33] YANG X.Feature sharing attention 3D face reconstruction with unsupervised learning from in-the-wild photo collection[J].Journal of Physics：Conference Series，2022，2258（1）：012051.
[34] ZHU X，YU C，HUANG D，et al.Beyond 3DMM：learning to capture high-fidelity 3D face shape[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2022，45（2）：1442-1457.
[35] CAI H，GUO Y，PENG Z，et al.Landmark detection and 3D face reconstruction for caricature using a nonlinear parametric model[J].Graphical Models，2021，115：101-103.
[36] QIU Y，XU X，QIU L，et al.3DcaricShop：a dataset and a baseline method for single-view 3D caricature face reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：10236-10245.
[37] TU X，ZHAO J，XIE M，et al.3D face reconstruction from a single image assisted by 2D face images in the wild[J].IEEE Transactions on Multimedia，2020，23：1160-1172.
[38] TEWARI A，ZOLLHOFER M，KIM H，et al.Mofa：model-based deep convolutional face autoencoder for unsupervised monocular reconstruction[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops，2017：1274-1283.
[39] DENG Y，YANG J，XU S，et al.Accurate 3D face reconstruction with weakly-supervised learning：from single image to image set[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2019.
[40] WU Y，DONG L F.3D face shape and texture reconstruction based on weak supervised learning[J].Application of Computer System，2020，29（11）：183-189.
[41] ZHANG J，LIN L，ZHU J，et al.Weakly-supervised multi-face 3D reconstruction[J].arXiv：2101.02000，2021.
[42] CHEN Z，WANG Y，GUAN T，et al.Transformer-based 3D face reconstruction with end-to-end shape-preserved domain transfer[J].IEEE Transactions on Circuits and Systems for Video Technology，2022，32（12）：8383-8393.
[43] TRAN A T，HASSNER T，MASI I，et al.Regressing robust and discriminative 3D morphable models with a very deep neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：5163-5172.
[44] CHANG F J，TRAN A T，HASSNER T，et al.FacePoseNet：making a case for landmark-free face alignment[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops，2017：1599-1608.
[45] GENOVA K，COLE F，MASCHINOT A，et al.Unsupervised training for 3D morphable model regression[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：8377-8386.
[46] ZIELONKA W，BOLKART T，THIES J.Towards metrical reconstruction of human faces[J].arXiv：2204.06607，2022.
[47] DENG J，GUO J，XUE N，et al.Arcface：additive angular margin loss for deep face recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：4690-4699.
[48] FENG Y，FENG H，BLACK M J，et al.Learning an animatable detailed 3D face model from in-the-wild images[J].ACM Transactions on Graphics（ToG），2021，40（4）：1-13.
[49] DENG Q，LE B H，JIN A，et al.End-to-end 3D face reconstruction with expressions and specular albedos from single in-the-wild images[C]//Proceedings of the 30th ACM International Conference on Multimedia，2022：4694-4703.
[50] CHINAEV N，CHIGORIN A，LAPTEV I.MobileFace：3D face reconstruction with efficient CNN regression[C]//Proceedings of the European Conference on Computer Vision（ECCV） Workshops，2018.
[51] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial networks[J].Communications of the ACM，2020，63（11）：139-144.
[52] ZHU J Y，PARK T，ISOLA P，et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2223-2232.
[53] ALMAHAIRI A，RAJESHWAR S，SORDONI A，et al.Augmented CycleGAN：learning many-to-many mappings from unpaired data[C]//International Conference on Machine Learning，2018：195-204.
[54] KALCHBRENNER N，GREFENSTETTE E，BLUNSOM P.A convolutional neural network for modelling sentences[J].arXiv：1404.2188，2014.
[55] LATTAS A，MOSCHOGLOU S，GECER B，et al.Avatar-Me：realistically renderable 3D facial reconstruction “in-the-wild”[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：760-769.
[56] RANJAN A，BOLKART T，SANYAL S，et al.Generating 3D faces using convolutional mesh autoencoders[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：704-720.
[57] CHENG S，BRONSTEIN M，ZHOU Y，et al.MeshGAN：non-linear 3D morphable models of faces[J].arXiv：1903.10384，2019.
[58] LEE G H，LEE S W.Uncertainty-aware mesh decoder for high fidelity 3D face reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：6100-6109.
[59] PIAO J，SUN K，WANG Q，et al.Inverting generative adversarial renderer for face reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：15619-15628.
[60] 吴永江，汪亚明，郑俊褒，等.基于智能优化算法的非刚体三维运动结构重建[J].工业控制计算机，2014，27（5）：97-98.
WU Y J，WANG Y M，ZHENG J B，et al.Non-rigid 3D motion and shape reconstruction based on intelligent optimization algorithm[J].Industrial Control Computer，2014，27（5）：97-98.
[61] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems，2017.
[62] CHEN Y，WU F，WANG Z，et al.Self-supervised learning of detailed 3D face reconstruction[J].IEEE Transactions on Image Processing，2020，29：8696-8705.
[63] KAO Y，PAN B，XU M，et al.Single-image 3D face reconstruction under perspective projection[J].arXiv：2205. 04126，2022.
[64] LIN Z，LIN J，LI L，et al.High-quality 3D face reconstruction with affine convolutional networks[C]//Proceedings of the 30th ACM International Conference on Multimedia，2022：2495-2503.
[65] 师帅杰.基于体素和点云的单目视觉三维重建技术研究[D].哈尔滨：哈尔滨工业大学，2021.
SHI S J.Research on monocular 3D reconstruction technology based on voxel and point cloud[D].Harbin：Harbin Institute of Technology，2021.
[66] WU S，RUPPRECHT C，VEDALDI A.Unsupervised learning of probably symmetric deformable 3D objects from images in the wild[C]//IEEE International Conference on Computer Vision，2020：1-10.
[67] CHEN A，CHEN Z，ZHANG G，et al.Photo-realistic facial details synthesis from single image[C]//IEEE International Conference on Computer Vision，2019：9428-9438.
[68] SELA M，RICHARDSON E，KIMMEL R.Unrestricted facial geometry reconstruction using image-to-image translation[C]//2017 IEEE International Conference on Computer Vision（ICCV），2017：1585-1594.
[69] ENGEL K，HADWIGER M，KNISS J M，et al.Real-time volume graphics[C]//ACM SIGGRAPH，2004.
[70] PARK K，SINHA U，HEDMAN P，et al.Hypernerf：a higher-dimensional representation for topologically varying neural radiance fields[J].arXiv：2106.13228，2021.
[71] CHAN E R，MONTEIRO M，KELLNHOFER P，et al.piGAN：periodic implicit generative adversarial networks for 3D-aware image synthesis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：5799-5809.
[72] SUN J，WANG X，ZHANG Y，et al.Fenerf：face editing in neural radiance fields[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：7672-7682.
[73] YIN L，WEI X，SUN Y，et al.A 3D facial expression database for facial behavior research[C]//7th International Conference on Automatic Face and Gesture Recognition（FGR06），2006：211-216.
[74] ZHANG X，YIN L，COHN J F，et al.A high-resolution spontaneous 3D dynamic facial expression database[C]//2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition（FG），2013：1-6.
[75] BAGDANOV A D，DEL BIMBO A，MASI I.The florence 2D/3D hybrid face dataset[C]//Proceedings of the 2011 Joint ACM Workshop on Human Gesture and Behavior Understanding，2011：79-80.
[76] PHILLIPS P J，FLYNN P J，SCRUGGS T，et al.Overview of the face recognition grand challenge[C]//Proceedings of CVPR 2005，2005：947-954.
[77] SANYAL S，BOLKART T，FENG H，et al.Learning to regress 3D face shape and expression from an image without 3D supervision[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：7763-7772.
[78] LIU Z，LUO P，WANG X，et al.Deep learning face attributes in the wild[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：3730-3738.
[79] KARRAS T，LAINE S，AILA T.A style-based generator architecture for generative adversarial networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：4401-4410.
[80] STRATOU G，GHOSH A，DEBEVEC P，et al.Effect of illumination on automatic expression recognition：a novel 3D relightable facial database[C]//2011 IEEE International Conference on Automatic Face & Gesture Recognition（FG），2011：611-618.
[81] BESL P J，MCKAY N D.Method for registration of 3D shapes[C]//Sensor Fusion IV：Control Paradigms and Data Structures，1992：586-606.
[82] HORE A，ZIOU D.Image quality metrics：PSNR vs.SSIM[C]//2010 20th International Conference on Pattern Recognition，2010：2366-2369.
[83] 王兴.基于人脸关键点和三维重建的算法研究与实现[D].银川：宁夏大学，2020.
WANG X.Research and implementation of facial landmark detection and 3D face reconstruction[D].Yinchuan：Ningxia University，2020.
[84] 曾小星.深度人脸检测与三维重建方法研究[D].深圳：中国科学院大学（中国科学院深圳先进技术研究院），2021.
ZENG X X.Deep learning methods for face detection and 3D reconstruction[D].Shenzhen：University of Chinese Academy of Sciences（Shenzhen Institute of Advanced Technology，Chinese Academy of Sciences），2021.
[85] YIN B，WANG W，YAO T，et al.Adv-makeup：a new imperceptible and transferable attack on face recognition[J].arXiv：2105.03162，2021.
[86] LI C，MOREL-FORSTER A，VETTER T，et al.To fit or not to fit：model-based face reconstruction and occlusion segmentation from weak supervision[J].arXiv：2106.09614，2021.
[87] RUAN Z，ZOU C，WU L，et al.SADRNet：self-aligned dual face regression networks for robust 3D dense face alignment and reconstruction[J].IEEE Transactions on Image Processing，2021，30：5793-5806.
[88] FRIDOVICH-KEIL S，YU A，TANCIK M，et al.Plenoxels：radiance fields without neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：5501-5510.
[89] BARRON J T，MILDENHALL B，TANCIK M，et al.Mip-nerf：a multiscale representation for anti-aliasing neural radiance fields[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：5855-5864.