Application and Progress of Deep Learning in 2D Virtual Try-on Technology

doi:10.3778/j.issn.1002-8331.2209-0352

Abstract

Abstract: Virtual try-on technology has wide application and research value in promoting the informatization and intelligence of the clothing industry, and is one of the research hotspots of artificial intelligence in the field of intelligent clothing manufacturing. At present, virtual try-on mainly focuses on two-dimensional virtual try-on based on image generation. This paper provides a comprehensive overview of two-dimensional virtual try-on technology. It introduces and analyzes traditional virtual try-on, and categorizes and summarizes the main tasks, types, development process, and models of existing two-dimensional virtual try-on technology. It also discusses in detail the principles and related improvements of representative algorithms of various types. It summarizes the application of traditional virtual try-on and two-dimensional virtual try-on technology, and discusses the extension technology of two-dimensional virtual try-on technology. It summarizes and compares the benefits and drawbacks of using traditional and current virtual try-on technology, as well as summarizes and forecasts the field’s future development.

Key words: virtual try-on, deep learning, clothing intelligent manufacturing, human parsing and understanding

摘要： 虚拟试衣技术对于促进服装产业的信息化和智能化有着广泛的应用研究价值，是人工智能在服装智能制造领域的研究热点之一。目前虚拟试衣主要是基于图像生成的二维虚拟试衣研究，对二维虚拟试衣技术进行全面概述。介绍和分析了传统的虚拟试衣，对现有二维虚拟试衣技术进行了主要任务、类型、发展过程、模型等方面的分类整理，并详细探讨了各类型代表算法的原理以及相关改进。总结了传统虚拟试衣与二维虚拟试衣技术的应用，并讨论了二维虚拟试衣技术的扩展技术。对传统与当前的虚拟试衣技术的应用与优缺点进行了梳理和小结，对该领域的未来发展进行了总结与展望。

关键词: 虚拟试衣, 深度学习, 服装智能制造, 人体解析和理解

HUA Ailing, YU Feng, CHEN Ziyi, WANG Hua, JIANG Minghua. Application and Progress of Deep Learning in 2D Virtual Try-on Technology[J]. Computer Engineering and Applications, 2023, 59(11): 37-45.

花爱玲, 余锋, 陈子宜, 王画, 姜明华. 深度学习在二维虚拟试衣技术的应用与进展[J]. 计算机工程与应用, 2023, 59(11): 37-45.

References

[1] YU J，LIN Z，YANG J，et al.Generative image inpainting with contextual attention[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：5505-5514.
[2] NEALEN A，MüLLER M，KEISER R，et al.Physically based deformable models in computer graphics[J].Computer Graphics Forum，2006，25（4）：809-836.
[3] NARAIN R，SAMII A，O’BRIEN J F.Adaptive anisotropic remeshing for cloth simulation[J].ACM Transactions on Graphics（TOG），2012，31（6）：1-10.
[4] CIRIO G，LOPEZ-MORENO J，MIRAUT D，et al.Yarn-level simulation of woven cloth[J].ACM Transactions on Graphics（TOG），2014，33（6）：1-11.
[5] WANG L，LI H，XIAO Q，et al.Automatic pose and wrinkle transfer for aesthetic garment display[J].Computer Aided Geometric Design，2021，89：102020.
[6] WANG T Y，CEYLAN D，POPOVIC J，et al.Learning a shared shape space for multimodal garment design[J].arXiv：1806.11335，2018.
[7] SANTESTEBAN I，OTADUY M A，DAN C.Learning‐based animation of clothing for virtual try‐on[J].Computer Graphics Forum，2019，38（2）：355-366.
[8] PATEL C，LIAO Z，PONS-MOLL G.TailorNet：predicting clothing in 3D as a function of human pose，shape and garment style[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：7365-7375.
[9] MA Q，YANG J，RANJAN A，et al.Learning to dress 3D people in generative clothing[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：6469-6478.
[10] PAN X，MAI J，JIANG X，et al.Predicting loose-fitting garment deformations using bone-driven motion networks[C]//ACM SIGGRAPH 2022 Conference Proceedings，2022：1-10.
[11] LOPER M，MAHMOOD N，ROMERO J，et al.SMPL：a skinned multi-person linear model[J].ACM Transactions on Graphics（TOG），2015，34（6）：1-16.
[12] WU N，DENG Z，HUANG Y，et al.A fast garment fitting algorithm using skeleton‐based error metric[J].Computer Animation and Virtual Worlds，2018，29（3/4）：e1811.
[13] REED S，AKATA Z，YAN X，et al.Generative adversarial text to image synthesis[C]//International Conference on Machine Learning，2016：1060-1069.
[14] LASSNER C，PONS-MOLL G，GEHLER P V.A generative model of people in clothing[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：853-862.
[15] ISOLA P，ZHU J Y，ZHOU T，et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：1125-1134.
[16] ZHU J Y，PARK T，ISOLA P，et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2223-2232.
[17] CHOI Y，CHOI M，KIM M，et al.StarGAN：unified generative adversarial networks for multi-domain image-to-image translation[C]//2018 IEEE CVF Conference on Computer Vision and Pattern Recognition，2018：8789-8797.
[18] WANG T C，LIU M Y，ZHU J Y，et al.High-resolution image synthesis and semantic manipulation with conditional GANs[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：8798-8807.
[19] LIANG X，ZHANG H，XING E P.Generative semantic manipulation with contrasting gan[J].arXiv：1708.00315，2017.
[20] ZHU J Y，KR?HENBüHL P，SHECHTMAN E，et al.Generative visual manipulation on the natural image manifold[C]//European Conference on Computer Vision.Cham：Springer，2016：597-613.
[21] MIRZA M，OSINDERO S.Conditional generative adversarial nets[J].arXiv：1411.1784，2014.
[22] RADFORD A，METZ L，CHINTALA S.Unsupervised representation learning with deep convolutional generative adversarial networks[J].arXiv：1511.06434，2015.
[23] ZHU S，URTASUN R，FIDLER S，et al.Be your own Prada：fashion synthesis with structural coherence[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：1680-1688.
[24] JETCHEV N，BERGMANN U.The conditional analogy GAN：swapping fashion articles on people images[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops，2017：2287-2292.
[25] PANDEY N，SAVAKIS A.Poly-GAN：multi-conditioned GAN for fashion synthesis[J].Neurocomputing，2020，414：356-364.
[26] HAN X，WU Z，WU Z，et al.VITON：an image-based virtual try-on network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：7543-7552.
[27] LIU Y，ZHAO M，ZHANG Z，et al.Arbitrary virtual try-on network：characteristics preservation and trade-off between body and clothing[J].arXiv：2111.12346，2021.
[28] CHANG Y，PENG T，HE R，et al.DP-VTON：toward detail-preserving image-based virtual try-on network[C]//ICASSP 2021 IEEE International Conference on Acoustics，Speech and Signal Processing，2021：2295-2299.
[29] FELE B，LAMPE A，PEER P，et al.C-VTON：context-driven image-based virtual try-on network[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision，2022：3144-3153.
[30] JANDIAL S，CHOPRA A，AYUSH K，et al.SieveNet：a unified framework for robust image-based virtual try-on[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision，2020：2182-2190.
[31] FINCATO M，LANDI F，CORNIA M，et al.VITON-GT：an image-based virtual try-on model with geometric transformations[C]//2020 25th International Conference on Pattern Recognition（ICPR），2021：7669-7676.
[32] BELONGIE S，MALIK J，PUZICHA J.Shape matching and object recognition using shape contexts[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2002，24（4）：509-522.
[33] WANG B，ZHENG H，LIANG X，et al.Toward characteristic-preserving image-based virtual try-on network[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：589-604.
[34] LEE H J，LEE R，KANG M，et al.LA-VITON：a network for looking-attractive virtual try-on[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops，2019.
[35] YANG H，ZHANG R，GUO X，et al.Towards photo-realistic virtual try-on by adaptively generating-preserving image content[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：7850-7859.
[36] HONDA S.VITON-GAN：virtual try-on image generator trained with adversarial loss[J].arXiv：1911.07926，2019.
[37] YU R，WANG X，XIE X.VTNFP：an image-based virtual try-on network with body and clothing feature preservation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：10511-10520.
[38] MINAR M R，TUAN T T，AHN H，et al.CP-VTON+：clothing shape and texture preserving image-based virtual try-on[C]//CVPR Workshops，2020.
[39] XIE Z，LAI J，XIE X.LG-VTON：fashion landmark meets image-based virtual try-on[C]//Chinese Conference on Pattern Recognition and Computer Vision（PRCV）.Cham：Springer，2020：286-297.
[40] LIU G，SONG D，TONG R，et al.Toward realistic virtual try-on through landmark guided shape matching[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2021：2118-2126.
[41] CHOI S，PARK S，LEE M，et al.VITON-HD：high-resolution virtual try-on via misalignment-aware normalization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：14131-14140.
[42] HAN X，HU X，HUANG W，et al.Clothflow：a flow-based model for clothed person generation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：10471-10480.
[43] CHOPRA A，JAIN R，HEMANI M，et al.Zflow：gated appearance flow-based virtual try-on with 3D priors[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：5433-5442.
[44] HE S，SONG Y Z，XIANG T.Style-based global appearance flow for virtual try-on[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：3470-3479.
[45] DU C，YU F，JIANG M，et al.VTON-SCFA：a virtual try-on network based on the semantic constraints and flow alignment[J].IEEE Transactions on Multimedia，2022，25：777-791.
[46] ISSENHUTH T，MARY J，CALAUZèNES C.Do not mask what you do not need to mask：a parser-free virtual try-on[C]//European Conference on Computer Vision.Cham：Springer，2020：619-635.
[47] LIN C，LI Z，ZHOU S，et al.RMGN：a regional mask guided network for parser-free virtual try-on[J].arXiv：2204.11258，2022.
[48] GE Y，SONG Y，ZHANG R，et al.Parser-free virtual try-on via distilling appearance flows[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：8485-8493.
[49] WU Z，LIN G，TAO Q，et al.M2E-Try On Net：fashion from model to everyone[C]//Proceedings of the 27th ACM International Conference on Multimedia，2019：293-301.
[50] YU L，ZHONG Y，WANG X.Inpainting-based virtual try-on network for selective garment transfer[J].IEEE Access，2019，7：134125-134136.
[51] NEUBERGER A，BORENSTEIN E，HILLELI B，et al.Image based virtual try-on network from unpaired data[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：5184-5193.
[52] DONG H，LIANG X，SHEN X，et al.Towards multi-pose guided virtual try-on network[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：9026-9035.
[53] WANG J，SHA T，ZHANG W，et al.Down to the last detail：virtual try-on with fine-grained details[C]//Proceedings of the 28th ACM International Conference on Multimedia，2020：466-474.
[54] HU B，LIU P，ZHENG Z，et al.SPG-VTON：semantic prediction guidance for multi-pose virtual try-on[J].IEEE Transactions on Multimedia，2022，24：1233-1246.
[55] HSIEH C W，CHEN C Y，CHOU C L，et al.FashionOn：semantic-guided image-based virtual try-on with detailed human and clothing information[C]//Proceedings of the 27th ACM International Conference on Multimedia，2019：275-283.
[56] DU C，YU F，JIANG M，et al.Multi-pose virtual try-on via self-adaptive feature filtering[C]//2022 IEEE International Conference on Acoustics，Speech and Signal Processing，2022：2544-2548.
[57] DONG H，LIANG X，SHEN X，et al.Fw-GAN：flow-navigated warping GAN for video virtual try-on[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：1161-1170.
[58] 褚乐阳，陈卫东，谭悦，等.重塑体验：扩展现实（XR）技术及其教育应用展望——兼论“教育与新技术融合”的走向[J].远程教育杂志，2019，37（1）：17-31.
CHU Y Y，CHEN W D，TAN Y，et al.Rebuilding the experience：extended reality（XR） technology and its education application outlook—also discuss the trend of “education and new technology integration”[J].Journal of Distance Education，2019，37（1）：17-31.
[59] WANG X，YU K，WU S，et al.ESRGAN：enhanced super-resolution generative adversarial networks[C]//Proceedings of the European Conference on Computer Vision（ECCV） Workshops，2018.
[60] SAJJADI M S M，SCHOLKOPF B，HIRSCH M.Enhance-Net：single image super-resolution through automated texture synthesis[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：4491-4500.
[61] ELGAMMAL A，LIU B，ELHOSEINY M，et al.CAN：creative adversarial networks，generating “art” by learning about styles and deviating from style norms[J].arXiv：1706.07068，2017.
[62] ANTIPOV G，BACCOUCHE M，DUGELAY J L.Face aging with conditional generative adversarial networks[C]//2017 IEEE International Conference on Image Processing（ICIP），2017：2089-2093.
[63] WU N，CHAO Q，CHEN Y，et al.AgentDress：realtime clothing synthesis for virtual agents using plausible deformations[J].IEEE Transactions on Visualization and Computer Graphics，2021，27（11）：4107-4118.
[64] MIR A，ALLDIECK T，PONS-MOLL G.Learning to transfer texture from clothing images to 3D humans[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：7023-7034.
[65] ZHAO F，XIE Z，KAMPFFMEYER M，et al.M3D-VTON：a monocular-to-3D virtual try-on network[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：13239-13249.