Review of Research on Facial Makeup Transfer

doi:10.3778/j.issn.1002-8331.2108-0347

Abstract

Abstract: The goal of facial makeup transfer is to transfer makeup from reference image to source image while maintaining the facial features of source image, which has important theoretical research value and great market application value. At present, the generative adversarial network has become the mainstream technology to solve the facial makeup transfer problems. This paper expounds the main challenges faced by facial makeup transfer, systematically combs the existing facial makeup transfer methods according to different critical problems to be solved, analyzes their advantages and limitations, summarizes the common loss functions of facial makeup transfer network, introduces the common facial makeup transfer datasets and model evaluation methods, and discusses the future development trend in the field of facial makeup transfer.

Key words: facial makeup transfer, generative adversarial networks（GAN）, color matching, pattern transfer, partial transfer

摘要： 人脸妆容迁移的目标是在保持源图像面部特征的同时，将人脸妆容从参考图像迁移到源图像，具有重要的理论研究价值和巨大的市场应用价值。目前，生成对抗网络已成为解决人脸妆容迁移问题的主流技术。阐述了人脸妆容迁移面临的主要挑战，按照重点解决问题的不同系统地梳理了已有的人脸妆容迁移方法并分析了其优点和局限性，总结了人脸妆容迁移网络常用的损失函数，介绍了常用的人脸妆容迁移数据集以及模型评价方法，讨论了人脸妆容迁移领域未来的发展趋势。

关键词: 人脸妆容迁移, 生成对抗网络, 颜色匹配, 图案迁移, 局部迁移

MI Aizhong, ZHANG Wei, QIAO Yingxu, XU Chengjing, HUO Zhanqiang. Review of Research on Facial Makeup Transfer[J]. Computer Engineering and Applications, 2022, 58(2): 15-26.

米爱中, 张伟, 乔应旭, 许成敬, 霍占强. 人脸妆容迁移研究综述[J]. 计算机工程与应用, 2022, 58(2): 15-26.

References

[1] TONG W S，TANG C K，BROWN M，et al.Example-based cosmetic transfer[C]//15th Pacific Conference on Computer Graphics and Applications（PG’07），Maui，HI，USA，Oct 28-Nov 2，2007.Piscataway：IEEE，2007：211-218.
[2] GUO D，SIM T.Digital face makeup by example[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition，Miami，FL，USA，Jun 20-25，2009.Piscataway：IEEE，2009：73-79.
[3] SCHERBAUM K，RITSCHEL T，HULLIN M，et al.Computer-suggested facial makeup[J].Computer Graphics Forum，2011，30（2）：485-492.
[4] BLANZ V，VETTER T.A morphable model for the synthesis of 3D faces[C]//SIGGRAPH99：26th International Conference on Computer Graphics and Interactive Techniques，Los Angeles，California，USA，Aug 8-13，1999.New York：ACM Press/Addison-Wesley Publishing Co，1999：187-194.
[5] LI C，ZHOU K，LIN S.Simulating makeup through physics-based manipulation of intrinsic image layers[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition（CVPR），Boston，MA，USA，Jun 7-12，2015.Piscataway：IEEE，2015：4621-4629.
[6] LIU S，OU X Y，QIAN R H，et al.Makeup like a superstar：deep localized makeup transfer network[C]//Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence，New York，USA，Jul 9-15，2016.New York：AAAI Press，2016：2568-2575.
[7] WANG S Y，FU Y.Face behind makeup[C]//Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence，Phoenix Arizona，Feb 12-17，2016.New York：AAAI Press，2016：58-64.
[8] 王伟光，钱祥利.基于深度学习的人脸妆容迁移算法[J].计算机应用研究，2021，38（5）：1559-1562.
WANG W G，QIAN X L.Face makeup transfer algorithm based on deep learning[J].Application Research of Computers，2021，38（5）：1559-1562.
[9] 黄妍，何泽文，张文生.一种多通路的分区域快速妆容迁移深度网络[J].软件学报，2019，30（11）：3549-3566.
HUANG Y，HE Z W，ZHANG W S.Fast multi-way regional makeup transfer deep network[J].Journal of Software，2019，30（11）：3549-3566.
[10] GOODFELLOW I J，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial networks[J].arXiv：1406.2661，2014.
[11] 王曙燕，金航，孙家泽.GAN图像对抗样本生成方法[J].计算机科学与探索，2021，15（4）：702-711.
WANG S Y，JIN H，SUN J Z.Method for image adversarial samples generating based on GAN[J].Journal of Frontiers of Computer Science and Technology，2021，15（4）：702-711.
[12] 彭晏飞，高艺，杜婷婷，等.生成对抗网络的图像超分辨率重建方法[J].计算机科学与探索，2020，14（9）：1612-1620.
PENG Y F，GAO Y，DU T T，et al.Single image super-resolution reconstruction method for generative adversarial network[J].Journal of Frontiers of Computer Science and Technology，2020，14（9）：1612-1620.
[13] LI M Y，CHEN L，WEI W H，et al.An improved generative adversarial network for micro-expressions based on multi-label learning from action units[C]//4th International Conference on Image and Graphics Processing（ICIGP 2021），Sanya，China，Aug 6-8，2021.New York，USA：Association for Computing Machinery，2021：59-64.
[14] ALANAZI Y，SATO N，LIU T B，et al.Simulation of electron-proton scattering events by a feature-augmented and transformed generative adversarial network（FAT-GAN）[C]//Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence（IJCAI 2021），Montreal，Canada，Aug 21-26，2021.
[15] ZHU J Y，PARK T.Unpaired image-to-image translation using cycle-consistent adversarial networks[J].arXiv：1703.
10593，2017.
[16] LI T T，QIAN R H，DONG C et al.BeautyGAN：instance-level facial makeup transfer with deep generative adversarial network[C]//Proceedings of the 26th ACM International Conference on Multimedia，Seoul Republic of Korea，Oct 22-26，2018.New York：Association for Computing Machinery，2018：645-653.
[17] CHANG H W，LU J W，YU F，et al.PairedCycleGAN：asymmetric style transfer for applying and removing makeup[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition，Salt Lake City，UT，USA，Jun 18-23，2018.Piscataway：IEEE，2018：40-48.
[18] GU Q，WANG G Z，CHIU M T，et al.LADN：local adversarial disentangling network for facial makeup and de-makeup[C]//2019 IEEE/CVF International Conference on Computer Vision，Seoul，Korea，Oct 27-Nov 2，2019.Piscataway：IEEE，2019：10480-10489.
[19] JIANG W T，LIU S，GAO J，et al.PSGAN：pose and expression robust spatial-aware GAN for customizable makeup transfer[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition，Seattle，WA，USA，Jun 13-19，2020.Piscataway：IEEE，2020：5193-5201.
[20] WAN Z，CHEN H，ZHANG J，et al.Facial attribute transformers for precise and robust makeup transfer[J].arXiv：2104.02894，2021.
[21] LIU S，JIANG W T，GAO C，et al.PSGAN++：robust detail-preserving makeup transfer and removal[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2021.
[22] NGUYEN T，TRAN A，HOAI M.Lipstick ain’t enough：beyond color matching for in-the-wild makeup transfer[C]//2021 IEEE Conference on Computer Vision and Pattern Recognition，Jun 19-25，2021.
[23] KIPS R，GORI P，PERROT M，et al.CA-GAN：weakly supervised color aware GAN for controllable makeup transfer[C]//Computer Vision-ECCV 2020 Workshops，Glasgow，UK，Aug 23-28，2020.Berlin，Heidelberg：Springer，2020.
[24] ZHANG H，CHEN W，HE H，et al.Disentangled makeup transfer with generative adversarial network[J].arXiv：1907.01144，2019.
[25] HORITA D，AIZAWA K.SLGAN：style- and latent-guided generative adversarial network for desirable makeup transfer and removal[J].arXiv：2009.07557，2020.
[26] LYU Y M，DONG J，PENG B，et al.SOGAN：3d-aware shadow and occlusion robust GAN for makeup transfer[J].arXiv：2104.10567，2021.
[27] HOSHEN D.MakeupBag：disentangling makeup extraction and application[J].arXiv：2012.02157，2020.
[28] ZHU F，CAO H，FENG Z，et al.Semi-supervised eye makeup transfer by swapping learned representation[C]//2019 IEEE/CVF International Conference on Computer Vision Workshop，Seoul，Korea，Oct 27-28，2019.Piscataway：IEEE，2019：3858-3867.
[29] CHEN H J，HUI K M，WANG S Y，et al.Beautyglow：on-demand makeup transfer framework with reversible generative network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Long Beach，CA，USA，Jun 15-20，2019.Piscataway：IEEE，2019：10034-10042.
[30] HUANG Z，ZHENG Z，YAN C，et al.Real-world automatic makeup via identity preservation makeup net[C]//Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence，Yokohama，Japen，Jan 7-15，2020.
[31] 包仁达，庾涵，朱德发，等.基于区域敏感生成对抗网络的自动上妆算法[J].软件学报，2019，30（4）：896-913.
BAO R D，YU H，ZHU D F，et al.Automatic makeup with region sensitive generative adversarial networks[J].Journal of Software，2019，30（4）：896-913.
[32] DENG H，HAN C，CAI H M，et al.Spatially-invariant style-codes controlled makeup transfer[C]//2021 IEEE Conference on Computer Vision and Pattern Recognition，Jun 19-25，2021.
[33] VASWANI A，SHAZEER N.Attention is all you need[J].arXiv：1706.03762，2017.
[34] FENG Y，WU F，SHAO X H，et al.Joint 3d face reconstruction and dense alignment with position map regression network[C]//Proceedings of the European Conference on Computer Vision，Munich，Germany，Sep 8-14，2018.Berlin，Heidelberg：Springer，2018.
[35] KINGMA D P，DHARIWAL P.Glow：generative flow with invertible 1x1 convolutions[J].arXiv：1807.03039，2018.
[36] HEUSEL M，RAMSAUER H.GANs trained by a two time-scale update rule converge to a local nash equilibrium[C]//NIPS’17：Proceedings of the 31st International Conference on Neural Information Processing Systems，California，USA，Dec 2017.NY，USA：Curran Associates Inc，2017：6629-6640.
[37] ZHANG R，ISOLA P.The unreasonable effectiveness of deep features as a perceptual metric[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition，Salt Lake City，UT，USA，Jun 18-23，2018.Piscataway：IEEE，2018：586-595.
[38] DAMTCHEVA A.Can facial cosmetics affect the matching accuracy of face recognition systems?[C]//2012 IEEE Fifth International Conference on Biometrics：Theory，Applications and Systems，Arlington，VA，USA，Sep 23-27，2012.Piscataway：IEEE，2012：391-398.
[39] CHEN C，DANRCHEVA A，ROSS A.Automatic facial makeup detection with application in face recognition[C]//2013 International Conference on Biometrics，Madrid，Spain，Jun 4-7，2013.Piscataway：IEEE，2013：1-8.
[40] CHEN C，DANRCHEVA A，ROSS A.Spoofing faces using makeup：an investigative study[C]//2017 IEEE International Conference on Identity，Security and Behavior Analysis，New Delhi，India，Feb 22-24，2017.Piscataway：IEEE，2017：1-8.
[41] SARFRAZ M，SEIBOLD C.Content and colour distillation for learning image translations with the spatial profile loss[J].arXiv：1908.00274，2019.
[42] BARRATT S，SHARAMA R.A note on the inception score[J].arXiv：1801.01973，2018.
[43] WANG Z，BOVIK A C，SHEIKH H R，et al.Image quality assessment：from error visibility to structural similarity[J].IEEE Transactions on Image Processing，2004，13（4）：600-612.