Survey of Two-Dimensional Image Virtual Try-On Technology

doi:10.3778/j.issn.1002-8331.2209-0454

Abstract

Abstract: As a hot topic in deep learning research in recent years, virtual try-on technology enables users to intuitively and conveniently check the effect of trying on their favorite clothes, provides more convenience for online shopping, and also has important applications in clothing design, games, animation and other fields. In the virtual try-on technology, the 2D images-based virtual try-on methods can abandon the expensive hardware cost like 3D scanner and time cost of 3D virtual try-on methods, has received more attention from researchers because of its simple and economical characteristics. This paper focuses on the two-dimensional virtual try-on technology, and summarizes it from the following aspects：firstly, the basic principle and process of the two-dimensional virtual try-on technology; the two types of 2D virtual try-on technologies are summarized in a unified way; thirdly, the commonly used data sets, evaluation indicators and loss functions of 2D virtual try-on technology are summarized, and this paper proposes the shortcomings of two-dimensional virtual try-on technologies and the future development trend in the direction of high quality, diversification, multimodality, and integration.

Key words: virtual try-on, network structure, human features, clothing deformation, U-Net

摘要： 虚拟试衣技术作为近年来深度学习研究中的热点课题，旨在使用户能直观方便查看自己试穿心仪服装的效果，为网购提供更多便利，且在服装设计、游戏、动画等领域也有着重要的应用。其中基于二维图像的虚拟试衣技术可以摒弃三维虚拟试衣所需昂贵的如三维扫描仪等设备的成本和时间代价，以其简单经济的特点受到研究者们更多关注。该文聚焦于二维虚拟试衣技术，从以下几方面进行总结：第一，阐述二维虚拟试衣技术的基本原理与流程；第二，在技术改进方面，以时间顺序和改进方向两者统一对二维虚拟试衣技术进行总结；第三，对二维虚拟试衣技术常用数据集、评价指标和损失函数进行总结，并提出二维虚拟试衣技术的不足和未来在高质量、多样化、多模态、集成化等方向的发展趋势。

关键词: 虚拟试衣, 网络结构, 人体特征, 服装变形, U-Net

TAN Zelin, BAI Jing. Survey of Two-Dimensional Image Virtual Try-On Technology[J]. Computer Engineering and Applications, 2023, 59(15): 17-26.

谭泽霖, 白静. 二维图像虚拟试衣技术综述[J]. 计算机工程与应用, 2023, 59(15): 17-26.

References

[1] 刘洋，李琪，殷猛.网络购物节氛围对消费者冲动购物行为的刺激作用[J].商业研究，2018（7）：18-23.
LIU Y，LI Q，YIN M.The influence of internet shopping festival atmosphere on consumer impulse buying[J].Commercial Research，2018（7）：18-23.
[2] 雷启然，尚笑梅.虚拟试衣技术的发展与应用[J].现代丝绸科学与技术，2018，33（6）：37-40.
LEI Q R，SHANG X M.Application of virtual fitting technology in apparel industry[J].Modern Silk Science & Technology，2018，33（6）：37-40.
[3] 尹喆，尚笑梅.三维虚拟试衣技术在服装行业的发展及应用综述[J].现代丝绸科学与技术，2019，34（1）：38-40.
YIN Z，SHANG X M.The development and application of 3D virtual dressing technology in garment industry[J].Modern Silk Science & Technology，2019，34（1）：38-40.
[4] MENG Y，MOK P Y，JIN X.Interactive virtual try-on clothing design systems[J].Computer-Aided Design，2010，42（4）：310-321.
[5] WU N，DENG Z，HUANG Y，et al.A fast garment fitting algorithm using skeleton?based error metric[J].Computer Animation and Virtual Worlds，2018，29（3/4）：1811.
[6] WU N，CHAO Q，CHEN Y，et al.AgentDress：realtime clothing synthesis for virtual agents using plausible deformations[J].IEEE Transactions on Visualization and Computer Graphics，2021，27（11）：4107-4118.
[7] PAN X，MAI J，JIANG X，et al.Predicting loose-fitting garment deformations using bone-driven motion networks[C]//Proceedings of ACM SIGGRAPH 2022 Conference，2022：1-10.
[8] 赵娟，魏雪霞，徐增波.基于深度学习的2D虚拟试衣技术研究进展[J].丝绸，2021，58（9）：48-52.
ZHAO J，WEI X X，XU Z B.Research progress of 2D virtual fitting technology based on deep learning[J].Journal of Silk，2021，58（9）：48-52.
[9] GU X L，GAO F，TAN M，et al.Fashion analysis and understanding with artificial intelligence[J].Information Processing and Management，2020，57（5）.
[10] 王鹏，方志军，赵晓丽，等.基于深度学习的人体图像分割算法[J].武汉大学学报（理学版），2017，63（5）：466-470.
WANG P，FANG Z J，ZHAO X L，et al.Human segmentation based on deep learning[J].Journal of Wuhan University（Natural Science Edition），2017，63（5）：466-470.
[11] GONG K，LIANG X，ZHANG D，et al.Look into person：self-supervised structure-sensitive learning and a new benchmark for human parsing[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：932-940.
[12] RUAN T，LIU T，HUANG Z，et al.Devil in the details：towards accurate single and multiple human parsing[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2019：4814-4821.
[13] CAO Z，SIMON T，WEI S E，et al.Realtime multi-person 2d pose estimation using part affinity fields[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：7291-7299.
[14] HE K，GKIOXARI G，DOLLáR P，et al.Mask R-CNN[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2961-2969.
[15] 罗述谦，阎华.基于薄板样条的MRI图像与脑图谱的配准方法[J].中国生物医学工程学报，2004（6）：479-485.
LUO S Q，YAN H.Registration between MR image and brain atlas based on thin palte splines[J].Chinese Journal of Biomedical Engineering，2004（6）：479-485.
[16] HAN X T，WU Z X，WU Z，et al.Viton：an image-based virtual try-on network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：7543-7552.
[17] RONNEBERGER O，FISCHER P，BROX T.U-net：convolutional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham：Springer，2015：234-241.
[18] JETCHEV N，BERGMANN U.The conditional analogy gan：swapping fashion articles on people images[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops，2017：2287-2292.
[19] WANG B，ZHENG H，LIANG X，et al.Toward characteristic-preserving image-based virtual try-on network[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：589-604.
[20] LEE H J，LEE R，KANG M，et al.LA-VITON：a network for looking-attractive virtual try-on[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops，2019.
[21] FINCATO M，LANDI F，CORNIA M，et al.VITON-GT：an image-based virtual try-on model with geometric transformations[C]//2020 25th International Conference on Pattern Recognition（ICPR），2021：7669-7676.
[22] HE S，SONG Y Z，XIANG T.Style-based global appearance flow for virtual try-on[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：3470-3479.
[23] KARRAS T，LAINE S，AILA T.A style-based generator architecture for generative adversarial networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：4401-4410.
[24] SONG D，LI T，MAO Z，et al.SP-VITON：shape-preserving image-based virtual try-on network[J].Multimedia Tools and Applications，2020，79（45）：33757-33769.
[25] YU R，WANG X，XIE X.Vtnfp：an image-based virtual try-on network with body and clothing feature preservation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：10511-10520.
[26] HAN X，HU X，HUANG W，et al.Clothflow：a flow-based model for clothed person generation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：10471-10480.
[27] PHAM D L，NGYUEN N T，CHUNG S T.Keypoints-based 2D virtual try-on network system[J].Journal of Korea Multimedia Society，2020，23（2）：186-203.
[28] MINAR M R，TUAN T T，AHN H，et al.CP-VTON+：clothing shape and texture preserving image-based virtual try-on[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020.
[29] YANG H，ZHANG R，GUO X，et al.Towards photo-realistic virtual try-on by adaptively generating-preserving image content[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：7850-7859.
[30] JANDIAL S，CHOPRA A，AYUSH K，et al.Sievenet：a unified framework for robust image-based virtual try-on[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision，2020：2182-2190.
[31] RAFFIEE A H，SOLLAMI M.Garmentgan：photo-realistic adversarial fashion transfer[C]//2020 25th International Conference on Pattern Recognition（ICPR），2021：3923-3930.
[32] TAN Z L，BAI J，ZHANG S M，et al.NL-VTON：a non-local virtual try-on network with feature preserving of body and clothes[J].Scientific Reports，2021，11（1）：1-13.
[33] CHOI S，PARK S，LEE M，et al.Viton-HD：high-resolution virtual try-on via misalignment-aware normalization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：14131-14140.
[34] HONDA S.Viton-gan：virtual try-on image generator trained with adversarial loss[J].arXiv：1911.07926，2019.
[35] WANG J，ZHANG W，LIU W，et al.Down to the last detail：virtual try-on with detail carving[J].arXiv：1912.06324，2019.
[36] ISSENHUTH T，MARY J，CALAUZèNES C.End-to-end learning of geometric deformations of feature maps for virtual try-on[J].arXiv：1906.01347，2019.
[37] ISSENHUTH T，MARY J，CALAUZèNES C.Do not mask what you do not need to mask：a parser-free virtual try-on[C]//European Conference on Computer Vision.Cham：Springer，2020：619-635.
[38] GE Y，SONG Y，ZHANG R，et al.Parser-free virtual try-on via distilling appearance flows[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：8485-8493.
[39] MORELLI D，FINCATO M，CORNIA M，et al.Dress code：high-resolution multi-category virtual try-on[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：2231-2235.
[40] PANDEY N，SAVAKIS A.Poly-GAN：multi-conditioned GAN for fashion synthesis[J].Neurocomputing，2020，414：356-364.
[41] GE C，SONG Y，GE Y，et al.Disentangled cycle consistency for highly-realistic virtual try-on[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：16928-16937.
[42] REN B，TANG H，MENG F，et al.Cloth interactive transformer for virtual try-on[J].arXiv：2104.05519，2021.
[43] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial networks[J].Communications of the ACM，2020，63（11）：139-144.
[44] WANG Z，BOVIK A C，SHEIKH H R，et al.Image quality assessment：from error visibility to structural similarity[J].IEEE Transactions on Image Processing，2004，13（4）：600-612.
[45] ZHANG R，ISOLA P，EFROS A A，et al.The unreasonable effectiveness of deep features as a perceptual metric[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：586-595.
[46] SALIMANS T，GOODFELLOW I，ZAREMBA W，et al.Improved techniques for training GANs[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems，2016.
[47] HEUSEL M，RAMSAUER H，UNTERTHINER T，et al.Gans trained by a two time-scale update rule converge to a local nash equilibrium[J].arXiv：1706.08500v6，2017.