内容感知的图像重定向方法综述

doi:10.3778/j.issn.1002-8331.2109-0145

摘要/Abstract

摘要： 随着各种智能设备的不断涌现，同一图像往往需要改变纵横比和大小来适应不同显示屏幕。基于内容感知的图像重定向方法通过研究图像重要内容的保持，使得图像自动适应到不同大小的显示设备上。主要对近年来基于内容感知的图像重定向方法的研究现状进行总结。按照重要度图的获取以及基于重要度图的重定向方法两方面回顾了手工特征的图像重定向；从基于深度神经网络的重定向算法、基于深度强化学习的多操作算法、基于美学感知的裁剪算法三个类别总结了现有的图像重定向的深度学习方法；在介绍已有的重定向数据集以及评价方法基础上，对主流的图像重定向算法进行分析比较，并按照类别总结了各个算法的实现原理和优缺点；针对现阶段所存在的问题和挑战，提出了该领域未来的研究方向。该研究旨在为感兴趣的研究人员提供有意义的帮助，以便推动该领域的进一步发展。

关键词: 图像重定向, 深度学习, 卷积神经网络, 重定向数据集, 评价方法

Abstract: With the emergence of various smart devices, images often need to change the aspect ratio and sizes to adapt to different display screens. The content-aware image retargeting methods make the image automatically adapt to different display devices by researching how to keep image important content. This paper mainly summarizes the research status of content-based image retargeting methods in recent years. Firstly, the image retargeting methods based on handcrafted features are reviewed according to the acquisition of importance map and the retargeting method based on importance map. Secondly, the existing image retargeting methods based on deep learning are summarized from three categories：the image retargeting algorithm based on deep neural network, multi-operation algorithm based on deep reinforcement learning and cropping algorithm based on aesthetic perception. Thirdly, based on the introduction of the existing retargeting datasets and evaluation methods, the state-of-the-art image retargeting algorithms are analyzed and compared, and the implementation principles, advantages and disadvantages of each algorithm are summarized according to the categories. Finally, according to the existing problems and challenges at this stage, the future research direction in this field is put forward. This paper aims to provide interested researchers with meaningful help in order to promote the further development of this field.

Key words: image retargeting, deep learning, convolutional neural networks, retargeting datasets, evaluation metrics

郭迎春, 张萌, 郝小可. 内容感知的图像重定向方法综述[J]. 计算机工程与应用, 2022, 58(4): 22-39.

GUO Yingchun, ZHANG Meng, HAO Xiaoke. Review on Content-Aware Image Retargeting Methods[J]. Computer Engineering and Applications, 2022, 58(4): 22-39.

参考文献

[1] ZHOU Y，ZHANG L，ZHANG C，et al.Perceptually aware image retargeting for mobile devices[J].IEEE Transactions on Image Processing，2017，27（5）：2301-2313.
[2] THéVENAZ P，BLU T，UNSER M.Interpolation revisited medical images application[J].IEEE Transactions on Medical Imaging，2000，19（7）：739-758.
[3] SURESHA D，PRAKASH H N.Single picture super resolution of natural images using N-neighbor adaptive bilinear interpolation and absolute asymmetry based wavelet hard thresholding[C]//2016 2nd International Conference on Applied and Theoretical Computing and Communication Technology，Bengaluru，Jul 21-23，2016.Piscataway：IEEE，2016：387-393.
[4] CHEN Y L，KLOPP J，SUN M，et al.Learning to compose with professional photographs on the web[C]//25th ACM International Conference on Multimedia，California，Oct 23-27，2017.New York：ACM，2017：37-45.
[5] CHEN J，BAI G，LIANG S，et al.Automatic image cropping：a computational complexity study[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition，Las Vegas，Jun 26-Jul 1，2016.Piscataway：IEEE，2016：507-515.
[6] WANG W，SHEN J.Deep cropping via attention box prediction and aesthetics assessment[C]//2017 IEEE International Conference on Computer Vision，Venice，Oct 22-29，2017.Piscataway：IEEE，2017：2186-2194.
[7] LU P，ZHANG H，PENG X J，et al.Aesthetic guided deep regression network for image cropping[J].Signal Processing：Image Communication，2019，77：1-10.
[8] LU P，ZHANG H，PENG X J，et al.Learning the relation between interested objects and aesthetic region for image cropping[J].IEEE Transactions on Multimedia，2021，23：3618-3630.
[9] WEI Z，ZHANG J，SHEN X，et al.Good view hunting：learning photo composition from dense view pairs[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition，Salt Lake City，Jun 18-22，2018.Piscataway：IEEE，2018：5437-5446.
[10] LI D，WU H，ZHANG J，et al.A2-RL：aesthetics aware reinforcement learning for image cropping[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition，Salt Lake City，Jun 18-22，2018.Piscataway：IEEE，2018：8193-8201.
[11] LI D，WU H，ZHANG J，et al.Fast A3Rl：aesthetics-aware adversarial reinforcement learning for image cropping[J].IEEE Transactions on Image Processing，2019，28（10）：5105-5120.
[12] AVIDAN S，SHAMIR A.Seam carving for content-aware image resizing[J].ACM Transactions on Graphics，2007，26（3）：10.
[13] 施美玲，徐丹.内容感知图像缩放技术综述[J].中国图象图形学报，2012，17（2）：157-168.
SHI M L，XU D.Survey on content-aware image resizing techniques[J].Journal of Image and Graphics，2012，17（2）：157-168.
[14] KARNI Z，FREEDMAN D，GOTSMAN C.Energy-based image deformation[J].Computer Graphics Forum，2009，28（5）：1257-1268.
[15] ITTI L，KOCH C，NIEBUR E.A model of saliency-based visual attention for rapid scene analysis[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，1998，20（11）：1254-1259.
[16] WANG Y S，TAI C L，SORKINE O，et al.Optimized scale-and-stretch for image resizing[J].ACM Transactions on Graphics，2008，27（5）：1-8.
[17] SHI M，PENG G，YANG L，et al.Optimal bi-directional seam carving for content-aware image resizing[C]//Proceedings of the Entertainment for Education，and 5th International Conference on E-learning and Games，Changchun，Aug 16-18，2010.Berlin，Heidelberg：Springer，2010：456-467.
[18] CHO D，PARK J，OH T H，et al.Weakly-and self-supervised learning for content-aware deep image retargeting[C]//2017 IEEE International Conference on Computer Vision，Venice，Oct 22-29，2017.Piscataway：IEEE，2017：4558-4567.
[19] TAN W，YAN B，LIN C，et al.Cycle-IR：deep cyclic image retargeting[J].IEEE Transactions on Multimedia，2019，22（7）：1730-1743.
[20] ARAR M，DANON D，COHEN-OR D，et al.Image resizing by reconstruction from deep features[J].arXiv：1904.08475，2019.
[21] SONG E，LEE M，LEE S.CarvingNet：content-guided seam carving using deep convolution neural network[J].IEEE Access，2018，7：284-292.
[22] WU J，XIE R，SONG L，et al.Deep feature guided image retargeting[C]//2019 IEEE Visual Communications and Image Processing，Sydney，Dec 1-4，2019.Piscataway：IEEE，2019：1-4.
[23] LIN J，ZHOU T，CHEN Z.DeepIR：a deep semantics driven framework for image retargeting[C]//2019 IEEE International Conference on Multimedia & Expo Workshops，Shanghai，Jul 8-12，2019.Piscataway：IEEE，2019：54-59.
[24] LIU S，WEI Z，SUN Y，et al.Composing semantic collage for image retargeting[J].IEEE Transactions on Image Processing，2018，27（10）：5032-5043.
[25] YAN B，NIU X，BARE B，et al.Semantic segmentation guided pixel fusion for image retargeting[J].IEEE Transactions on Multimedia，2019，22（3）：676-687.
[26] AHMADI M，KARIMI N，SAMAVI S.Context-aware saliency detection for image retargeting using convolutional neural networks[J].Multimedia Tools and Applications，2021，80（8）：11917-11941.
[27] WANG Z，ZHANG W，ZHOU H.Perception-guided multi-channel visual feature fusion for image retargeting[J].Signal Processing：Image Communication，2019，79：63-70.
[28] GUO D，DING J，TANG J，et al.NIF-based seam carving for image resizing[J].Multimedia Systems，2015，21（6）：603-613.
[29] OLIVEIRA S A F，NETO A R R，BEZERRA F N.A novel genetic algorithms and SURF-based approach for image retargeting[J].Expert Systems with Applications，2016，44：332-343.
[30] 郭迎春，梁云鹤，于明，等.基于图像分块和优化累积能量图的线裁剪算法[J].电子与信息学报，2018，40（2）：331-337.
GUO Y C，LIANG Y H，YU M，et al.An improved seam carving algorithm based on image blocking and optimized cumulative energy map[J].Journal of Electronics & Information Technology，2018，40（2）：331-337.
[31] RUBINSTEIN M，SHAMIR A，AVIDAN S.Improved seam carving for video retargeting[J].ACM Transactions on Graphics，2008，27（3）：1-9.
[32] PRITCH Y，KAV-VENAKI E，PELEG S.Shift-map image editing[C]//2009 IEEE 12th International Conference on Computer Vision，Kyoto，Sep 29-Oct 2，2009.Piscataway：IEEE，2009：151-158.
[33] NOH H，HAN B.Seam carving with forward gradient difference maps[C]//20th ACM International Conference on Multimedia，Nara，Oct 29-Nov 2，2012.New York：ACM，2012：709-712.
[34] SHI M，YANG L，PENG G，et al.A content-aware image resizing method with prominent object size adjusted[C]//17th ACM Symposium on Virtual Reality Software and Technology，Hong Kong，China，Nov 22-24，2010.New York：ACM，2010：175-176.
[35] 施美玲，徐丹.主体大小能控的内容感知图像缩放[J].计算机辅助设计与图形学学报，2011，23（5）：915-922.
SHI M L，XU D.A prominent object size adjustable method for content-aware image resizing[J].Journal of Computer-Aided Design & Computer Graphics，2011，23（5）：915-922.
[36] ZHANG G X，CHENG M M，HU S M，et al.A shape-preserving approach to image resizing[J].Computer Graphics Forum，2009，28（7）：1897-1906.
[37] LIN S S，YEH I C，LIN C H，et al.Patch-based image warping for content-aware retargeting[J].IEEE Transactions on Multimedia，2012，15（2）：359-368.
[38] GUO Y，LIU F，SHI J，et al.Image retargeting using mesh parametrization[J].IEEE Transactions on Multimedia，2009，11（5）：856-867.
[39] JIN Y，LIU L，WU Q.Nonhomogeneous scaling optimization for realtime image resizing[J].The Visual Computer，2010，26（6）：769-778.
[40] RAN L，LV N，MENG X.Image retargeting based on spring analogy[C]//2014 IEEE International Conference on Progress in Informatics and Computing，Shanghai，May 16-18，2014.Piscataway：IEEE，2014：250-254.
[41] 时健，郭延文，杜振龙，等.一种基于网格参数化的图像适应方法[J].软件学报，2008，19（S1）：19-30.
SHI J，GUO Y W，DU Z L，et al.A mesh parameterization-based image retargeting method[J].Journal of Software，2008，19（S1）：19-30.
[42] RUBINSTEIN M，SHAMIR A，AVIDAN S.Multi-operator media retargeting[J].ACM Transactions on Graphics，2009，28（3）：1-11.
[43] DONG W，ZHOU N，PAUL J C，et al.Optimized image resizing using seam carving and scaling[J].ACM Transactions on Graphics，2009，28（5）：1-10.
[44] DONG W M，BAO G B，ZHANG X P，et al.Fast multi-operator image resizing and evaluation[J].Journal of Computer Science and Technology，2012，27（1）：121-134.
[45] SU P C，XIANG Z H，WU H W.SCAN：a multi-operator image retargeting scheme[C]//Signal and Information Processing Association Annual Summit and Conference，Angkor Wat，Dec 9-12，2014.Piscataway：IEEE，2014：1-5.
[46] ZHU L，CHEN Z.Fast genetic multi-operator image retargeting[C]//2016 IEEE Visual Communications and Image Processing，Chengdu，Nov 27-30，2016.Piscataway：IEEE，2016：1-4.
[47] ZHU L，CHEN Z，CHEN X，et al.Saliency & structure preserving multi-operator image retargeting[C]//2016 IEEE
International Conference on Acoustics，Speech and Signal Processing，Shanghai，Mar 20-25，2016.Piscataway：IEEE，2016：1706-1710.
[48] 朱鹭伟，陈昭炯.结合美学原则的内容感知图像缩放算法[J].计算机工程与应用，2017，53（4）：189-194.
ZHU L W，CHEN Z J.Content aware image resizing algorithm using aesthetic rules[J].Computer Engineering and Applications，2017，53（4）：189-194.
[49] DU H，LIU Z，JIANG J，et al.Stretchability-aware block scaling for image retargeting[J].Journal of Visual Communication and Image Representation，2013，24（4）：499-508.
[50] 谷香丽，迟静，张彩明.基于弹簧变形模型的图像缩放方法[J].中国图象图形学报，2018，23（5）：756-765.
GU X L，CHI J，ZHANG C M.Image scaling based on the spring deformation model[J].Journal of Image and Graphics，2018，23（5）：756-765.
[51] SHOCHER A，BAGON S，ISOLA P，et al.INGAN：capturing and retargeting the “DNA” of a natural image[C]//2019 IEEE/CVF International Conference on Computer Vision，Seoul，Oct 27-Nov 2，2019.Piscataway：IEEE，2019：4492-4501.
[52] MASTAN I D，RAMAN S.DCIL：deep contextual internal learning for image restoration and image retargeting[C]//2020 IEEE/CVF Winter Conference on Applications of Computer Vision，Snowmass Village，Mar 1-5，2020.Piscataway：IEEE，2020：2366-2375.
[53] LIAO J，YAO Y，YUAN L，et al.Visual attribute transfer through deep image analogy[J].arXiv：1705.01088，2017.
[54] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial nets[J].Advances in Neural Information Processing Systems，2014，27：2672-2680.
[55] YAN B，LI K，YANG X，et al.Seam searching-based pixel fusion for image retargeting[J].IEEE Transactions on Circuits and Systems for Video Technology，2014，25（1）：15-23.
[56] JAIN S D，XIONG B，GRAUMAN K.Pixel objectness[J].arXiv：1701.05349，2017.
[57] LIN G，MILAN A，SHEN C，et al.RefineNet：multi-path refinement networks for high-resolution semantic segmentation[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition，Honolulu，Jul 21-26，2017.Piscataway：IEEE，2017：1925-1934.
[58] ZHAO H，SHI J，QI X，et al.Pyramid scene parsing network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition，Honolulu，Jul 21-26，2017.Piscataway：IEEE，2017：2881-2890.
[59] REYNOLDS D A.Gaussian mixture models[J].Encyclopedia of Biometrics，2009，741：659-663.
[60] ACHANTA R，SHAJI A，SMITH K，et al.SLIC superpixels compared to state-of-the-art superpixel methods[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2012，34（11）：2274-2282.
[61] ZHOU Y，CHEN Z，LI W.Weakly supervised reinforced multi-operator image retargeting[J].IEEE Transactions on Circuits and Systems for Video Technology，2020，31（1）：126-139.
[62] BARNES C，SHECHTMAN E，FINKELSTEIN A，et al.PatchMatch：a randomized correspondence algorithm for structural image editing[J].ACM Transactions on Graphics，2009，28（3）：24.
[63] BARNES C，SHECHTMAN E，GOLDMAN D B，et al.The generalized patchmatch correspondence algorithm[C]//11th European Conference on Computer Vision，Hersonissos，Sep 5-11，2010.Berlin：Springer，2010：29-43.
[64] KAJIURA N，KOSUGI S，WANG X，et al.Self-play reinforcement learning for fast image retargeting[C]//28th ACM International Conference on Multimedia，Seattle，Oct 12-16，2020.New York：ACM，2020：1755-1763.
[65] WANG L，WANG X，YAMASAKI T，et al.Aspect-ratio-preserving multi-patch image aesthetics score prediction[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，California，Jun 16-20，2019.Piscataway：IEEE，2019.
[66] JIN Y，WU Q，LIU L.Aesthetic photo composition by optimal crop-and-warp[J].Computers & Graphics，2012，36（8）：955-965.
[67] SALIMUN C，PURCHASE H C，SIMMONS D R，et al.The effect of aesthetically pleasing composition on visual search performance[C]//6th Nordic Conference on Human-Computer Interaction：Extending Boundaries，Iceland，Oct 16-20，2010.New York：ACM，2010：422-431.
[68] ZHANG M，ZHANG L，SUN Y，et al.Auto cropping for digital photographs[C]//2005 IEEE International Conference on Multimedia and Expo，Amsterdam，Jul 6，2005.Piscataway：IEEE，2005.
[69] CHENG B，NI B，YAN S，et al.Learning to photograph[C]//18th ACM International Conference on Multimedia，Firenze，Oct 25-29，2010.New York：ACM，2010：291-300.
[70] YAN J，LIN S，BING K S，et al.Learning the change for automatic image cropping[C]//2013 IEEE Conference on Computer Vision and Pattern Recognition，Portland，Jun 23-28，2013.Piscataway：IEEE，2013：971-978.
[71] STENTIFORD F.Attention based auto image cropping[C]//2007 ICVS Workshop on Computational Attention & Applications，Bielefeld，Mar 21-24，2007：1-9.
[72] PARK J，LEE J Y，TAI Y W，et al.Modeling photo composition and its application to photo re-arrangement[C]//2012 19th IEEE International Conference on Image Processing，Orlando，Sep 30-Oct 3，2012.Piscataway：IEEE，2012：2741-2744.
[73] LU P，LIU J，PENG X，et al.Weakly supervised real-time image cropping based on aesthetic distributions[C]//28th ACM International Conference on Multimedia，Seattle，Oct 12-16，2020.New York：ACM，2020：120-128.
[74] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409.
1556，2014.
[75] ZHANG T，YU M，GUO Y，et al.Content-aware retargeted image quality assessment[J].Information，2019，10（3）：111.
[76] 曹玉东，刘海燕，贾旭，等.基于深度学习的图像质量评价方法综述[J/OL].计算机工程与应用（2021-08-27）[2021-09-05].http：//kns.cnki.net/kcms/detail/11.2127.TP.20210827.
1430.002.html.
CAO Y D，LIU H Y，JIA X，et al.Overview of image quality assessment method based on deep learning[J/OL].Computer Engineering and Applications（2021-08-27）[2021-09-05].http：//kns.cnki.net/kcms/detail/11.2127.TP.20210827.
1430.002.html.
[77] WANG Z，BOVIK A C，SHEIKH H R，et al.Image quality assessment：from error visibility to structural similarity[J].IEEE Transactions on Image Processing，2004，13（4）：600-612.
[78] LIU C，YUEN J，TORRALBA A.SIFT flow：dense correspondence across scenes and its applications[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2010，33（5）：978-994.
[79] ZHANG Y，LIN W，ZHANG X，et al.Aspect ratio similarity（ARS） for image retargeting quality assessment[C]//2016 IEEE International Conference on Acoustics，Speech and Signal Processing，Shanghai，Mar 20-25，2016.Piscataway：IEEE，2016：1080-1084.
[80] ZHANG Y，LIN W，LI Q，et al.Multiple-level feature-based measure for retargeted image quality[J].IEEE Transactions on Image Processing，2017，27（1）：451-463.
[81] SIMAKOV D，CASPI Y，SHECHTMAN E，et al.Summarizing visual data using bidirectional similarity[C]//2008 IEEE Conference on Computer Vision and Pattern Recognition，Alaska，Jun 24-26，2008.Piscataway：IEEE，2008：1-8.
[82] MANJUNATH B S，OHM J R，VASUDEVAN V V，et al.Color and texture descriptors[J].IEEE Transactions on Circuits and Systems for Video Technology，2001，11（6）：703-715.
[83] KASUTANI E，YAMADA A.The MPEG-7 color layout descriptor：a compact image feature description for high-speed image/video segment retrieval[C]//2001 International Conference on Image Processing，Thessaloniki，Oct 7-10，2001.Piscataway：IEEE，2001，1：674-677.
[84] RUBINSTEIN M，GUTIERREZ D，SORKINE O，et al.A comparative study of image retargeting[J].ACM Transactions on Graphics，2010，29（6）：1-10.
[85] KR?HENBüHL P，LANG M，HORNUNG A，et al.A system for retargeting of streaming video[J].ACM Transactions on Graphics，2009，28（5）：1-10.
[86] WOLF L，GUTTMANN M，COHEN-OR D.Non-homogeneous content-driven video-retargeting[C]//2007 IEEE 11th International Conference on Computer Vision，Rio de Janeiro，Oct 14-21，2007.Piscataway：IEEE，2007：1-6.
[87] BRADLEY R A，TERRY M E.Rank analysis of incomplete block designs I the method of paired comparisons[J].Biometrika，1952，39（3/4）：324-345.
[88] MA L，LIN W，DENG C，et al.Image retargeting quality assessment：a study of subjective scores and objective metrics[J].IEEE Journal of Selected Topics in Signal Processing，2012，6（6）：626-639.
[89] HSU C C，LIN C W，FANG Y，et al.Objective quality assessment for image retargeting based on perceptual geometric distortion and information loss[J].IEEE Journal of Selected Topics in Signal Processing，2014，8（3）：377-389.
[90] LI G，YU Y.Deep contrast learning for salient object detection[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition，Las Vegas，Jun 26-Jul 1，2016.Piscataway：IEEE，2016：478-487.
[91] EVERINGHAM M，VAN GOOL L，WILLIAMS C K I，et al.The PASCAL visual object classes challenge 2007（VOC2007） results[DB/OL].（2007-09-24）[2007-10-15].http：//www.pascal-network.org/challenges/VOC/voc2007/workshop/
index.html.
[92] MURRAY N，MARCHESOTTI L，PERRONNIN F.AVA：a large-scale database for aesthetic visual analysis[C]//2012 IEEE Conference on Computer Vision and Pattern Recognition，Providence，Jun 16-21，2012.Piscataway：IEEE，2012：2408-2415.
[93] CHEN Y L，HUANG T W，CHANG K H，et al.Quantitative analysis of automatic image cropping algorithms：a dataset and comparative study[C]//2017 IEEE Winter Conference on Applications of Computer Vision，Santa Rosa，Mar 27-29，2017.Piscataway：IEEE，2017：226-234.
[94] FANG C，LIN Z，MECH R，et al.Automatic image cropping using visual composition，boundary simplicity and content preservation models[C]//22nd ACM International Conference on Multimedia，Orlando，Nov 3-7，2014.New York：ACM，2014：1105-1108.
[95] MEI Y，GUO X，SUN D，et al.Deep supervised image retargeting[C]//2021 IEEE International Conference on Multimedia and Expo，Shenzhen，Jul 5-9，2021.Piscataway：IEEE，2021：1-6.
[96] LIU N，LI L，ZHAO W，et al.Instance-level relative saliency ranking with graph reasoning[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2021.
[97] 于明，张吉俊，郭迎春，等.基于多层级注意力融合的图像美学重定向[J/OL].激光与光电子学进展（2021-03-01）[2021-09-22].http：//kns.cnki.net/kcms/detail/31.1690.TN.
20210301.1019.038.html.
YU M，ZHANG J J，GUO Y C，et al.Image aesthetics retargeting algorithm based on multi-level attention fusion[J/OL].Laser & Optoelectronics Progress（2021-03-01）[2021-09-22].http：//kns.cnki.net/kcms/detail/31.1690.TN.20210301.
1019.038.html.