Recent Advances on Panoramic Image Quality Assessment Methods

doi:10.3778/j.issn.1002-8331.2206-0157

Abstract

Abstract: With the rapid development of virtual reality（VR） technology, panoramic videos and images have become a new form of media display. Therefore, the quality assessment of panoramic image（PIQA） has great practical significance for improvement of VR technology. The researches of PIQA methods in recent five years are reviewed. The existing objective measures of panoramic image are analyzed, including the improved method based on peak signal-to-noise ratio and the improved method based on structural similarity respectively. The methods of deep learning for panoramic image quality assessment are summarized. The quality assessment methods of panoramic images are studied for two unique distortions, the stitching distortion and the projection distortion. The commonly used public datasets and some self-built datasets for PIQA are summarized. By using the pearson linear correlation coefficient, Spearman rank order correlation coefficient and root mean squared error as the evaluation performance measures, the degree of correlation between objective assessment of PIQA methods and subjective perception of human being is compared. The research trend in the future of PIQA methods is prospected.

Key words: panoramic image, image quality assessment, objective assessment, deep learning, stitching distortion, projection distortion

摘要： 随着虚拟现实技术的快速发展，全景视频图像已成为一种新的媒体展示形式被投入使用。全景图像的质量评价方法对促进虚拟现实技术具有重要的现实意义。研究了近五年来全景图像质量评价新方法。分析了全景图像客观评价指标，包括基于峰值信噪比的改进方法和基于结构相似性的改进方法。归纳总结了基于深度学习的全景图像质量评价方法以及其特有的缝合失真和投影失真的评价方法。搜集整理了常用的全景图像质量评价公共数据集与专有自建数据集。采用皮尔森相关系数、斯皮尔曼秩相关系数与均方根误差作为评价指标，根据客观评分与人眼主观感受的相关程度对现有全景图像质量评价方法进行了对比分析。对全景图像质量评价方法的发展方向进行展望。

关键词: 全景图像, 图像质量评价, 客观评价, 深度学习, 投影失真, 特殊失真

AI Da, BAI Yansong, YU Kexin, YUAN Hui, LIU Ying. Recent Advances on Panoramic Image Quality Assessment Methods[J]. Computer Engineering and Applications, 2022, 58(24): 1-11.

艾达, 白岩松, 于可欣, 元辉, 刘颖. 全景图像质量评价方法最新进展[J]. 计算机工程与应用, 2022, 58(24): 1-11.

References

[1] PATIL S B，PATIL S R.Survey on approaches used for image quality assessment[C]//2017 International Conference on Energy，Communication，Data Analytics and Soft Computing（ICECDS），Chennai，Aug 1-2，2017.Piscataway：IEEE，2017：987-991.
[2] 鄢杰斌，方玉明，刘学林.图像质量评价研究综述——从失真的角度[J].中国图象图形学报，2022，27（5）：1430-1466.
YAN J B，FANG Y M，LIU X L.The review of distortion-related image quality assessment[J].Journal of Image and Graphics，2022，27（5）：1430-1466.
[3] 张淑芳，张聪，张涛，等.通用型无参考图像质量评价算法综述[J].计算机工程与应用，2015，51（19）：13-23.
ZHANG S F，ZHANG C，ZHANG T，et al.Review on universal no-reference image quality assessment algorithm[J].Computer Engineering and Applications，2015，51（19）：13-23.
[4] AHMED I T，DER C S，JAMIL N，et al.Analysis of probability density functions in existing no-reference image quality assessment algorithm for contrast-distorted images[C]//2019 IEEE 10th Control and System Graduate Research Colloquium（ICSGRC），Shah Alam，Aug 2-3，2019.Piscataway：IEEE，2019：133-137.
[5] DONG H，SHEN A D，KONG Y Y，et al.No-reference defocused image quality assessment based on human visual system[C]//2019 IEEE International Conference on Signal，Information and Data Processing（ICSIDP），Chongqing，Dec 11-13，2019.Piscataway：IEEE，2019：1-6.
[6] JOSHI P，PRAKASH S.Continuous wavelet transform based no-reference image quality assessment for blur and noise distortions[J].IEEE Access，2018，6：33871-33882.
[7] ZHOU B Z，SHAO F，MENG X C，et al.No-reference quality assessment for pansharpened images via opinion-unaware learning[J].IEEE Access，2019，7：40388-40401.
[8] 曹玉东，刘海燕，贾旭，等.基于深度学习的图像质量评价方法综述[J].计算机工程与应用，2021，57（23）：27-36.
CAO Y D，LIU H Y，JIA X，et al.Overview of image quality assessment method based on deep learning[J].Computer Engineering and Applications，2021，57（23）：27-36.
[9] Information Technology.Coded representation of immersive media-part 2：omnidirectional media format（first edition）：ISO/IEC 23090-2：2019[S].IX-ISO，2019.
[10] YU M，LAKSHMAN H，GIROD B，et al.A framework to evaluate omnidirectional video coding schemes[C]//2015 IEEE International Symposium on Mixed and Augmented Reality，Fukuoka，Sep 29-Oct 3，2015.Piscataway：IEEE，2015：31-36.
[11] SUN Y L，LU A，YU L，et al.Weighted-to-spherically-uniform quality evaluation for omnidirectional video[J].IEEE Signal Processing Letters，2017，24（9）：1408-1412.
[12] ZAKHARCHENKO V，CHOI K P，PARK J H，et al.Quality metric for spherical panoramic video[C]//Optics and Photonics for Information Processing X，San Diego，Aug 28-Sep 1，2016.Bellingham：SPIE，2016.
[13] CHEN S J，ZHANG Y X，LI Y M，et al.Spherical structural similarity index for objective omnidirectional video quality assessment[C]//2018 IEEE International Conference on Multimedia and Expo（ICME），San Diego，Jul 23-27，2018.Piscataway：IEEE，2018：1-6.
[14] ZHOU Y F，YU M，MA H L，et al.Weighted-to-spherically-uniform SSIM objective quality evaluation for panoramic video[C]//2018 14th IEEE International Conference on Signal Processing（ICSP），Beijing，Aug 12-16，2018.Piscataway：IEEE，2018：54-57.
[15] LIM H T，KIM H G，YONG M R，et al.VR IQA NET：deep virtual reality image quality assessment using adversarial learning[J].IEEE，2018：6737-6741.
[16] TRUONG T Q，TRAN H，THANG T C，et al.Non-reference quality assessment model using deep learning for omnidirectional images[C]//2019 IEEE 10th International Conference on Awareness Science and Technology（iCAST），Morioka，Oct 23-25，2019.Piscataway：IEEE，2019：1-5.
[17] HOU J W，LIN W S，ZHAO B Q.Content-dependency reduction with multi-task learning in blind stitched panoramic image quality assessment[C]//2020 IEEE International Conference on Image Processing（ICIP），Abu Dhabi，Oct 25-28，2020.Piscataway：IEEE，2020：3463-3467.
[18] YANG L，XU M，DENG X，et al.Spatial attention-based non-reference perceptual quality prediction network for omnidirectional images[C]//2021 IEEE International Conference on Multimedia and Expo（ICME），Shenzhen，Jul 5-9，2021.Piscataway：IEEE，2021：1-6.
[19] ZHOU Y，GONG W K，SUN Y J，et al.Pyramid feature aggregation for hierarchical quality prediction of stitched panoramic images[J].IEEE Transactions on Multimedia（Early Access），2022.
[20] SUN W，MIN X K，ZHAI G T，et al.MC360IQA：a multi-channel CNN for blind 360-degree image quality assessment[J].IEEE Journal of Selected Topics in Signal Processing，2020，14（1）：64-77.
[21] XU J H，ZHOU W，CHEN Z B.Blind omnidirectional image quality assessment with viewport oriented graph convolutional networks[J].IEEE Transactions on Circuits and Systems for Video Technology，2020，31（5）：1724-1737.
[22] TIAN C Z，CHAI X L，CHEN G，et al.VSOIQE：a novel viewport-based stitched 360° omnidirectional image quality evaluator[J].IEEE Transactions on Circuits and Systems for Video Technology（Early Access），2022.
[23] ZHOU Y，SUN Y J，LI L D，et al.Omnidirectional image quality assessment by distortion discrimination assisted multi-stream network[J].IEEE Transactions on Circuits and Systems for Video Technology，2022，32（4）：1767-1777.
[24] LING S Y，CHEUNG G，CALLET P L，et al.No-reference quality assessment for stitched panoramic images using convolutional sparse coding and compound feature selection[C]//2018 IEEE International Conference on Multimedia and Expo（ICME），San Diego，Jul 23-27，2018.Piscataway：IEEE，2018：1-6.
[25] ZHU S D，ZHANG Y Z，TAO L，et al.A novel method for quality assessment of image stitching based on the Gabor filtering[C]//2018 IEEE International Conference on Information and Automation（ICIA），Wuyishan，Aug 11-13，2018.Piscataway：IEEE，2018：1605-1610.
[26] LI J，ZHAO Y F，YE W H，et al.Attentive deep stitching and quality assessment for 360° omnidirectional images[J].IEEE Journal of Selected Topics in Signal Processing，2019，14（1）：209-221.
[27] LIU Z G，MO Z.Combining local and global features for quality assessment of stitched images in virtual reality[C]//2021 The 9th International Conference on Information Technology：IoT and Smart City（ICIT 2021），Guangzhou，Dec 22-25，2021.New York：Association for Computing Machinery，2021：7-10.
[28] CUI Y L，JIANG G Y，YU M，et al.Local visual and global deep features based blind stitched panoramic image quality evaluation using ensemble learning[J].IEEE Transactions on Emerging Topics in Computational Intelligence（Early Access），2022：1-15.
[29] ZHENG X L，JIANG G Y，YU M，et al.Segmented spherical projection based blind omnidirectional image quality assessment[J].IEEE Access，2020，8：31647-31659.
[30] JIANG H，JIANG G，YU M，et al.Cubemap-based perception-driven blind quality assessment for 360-degree images[J].IEEE Transactions on Image Processing，2021，30：2364-2377.
[31] JIANG H，JIANG G Y，YU M，et al.Multi-angle projection based blind omnidirectional image quality assessment[J].IEEE Transactions on Circuits and Systems for Video Technology，2022，32（7）：4211-4223.
[32] JABAR F，ASCENSO J，QUELUZ M P，et al.Perceptual analysis of perspective projection for viewport rendering in 360° images[C]//2017 IEEE International Symposium on Multimedia（ISM），Taichung，Dec 11-13，2017.Piscataway：IEEE，2017：53-60.
[33] JABAR F，ASCENSO J，QUELUZ M P，et al.Field-of-view effect on the perceived quality of omnidirectional images[C]//2020 IEEE International Conference on Multimedia & Expo Workshops（ICMEW），London，Jul 6-10，2020.Piscataway：IEEE，2020：1-6.
[34] CHEUNG G，YANG L Y，TAN Z G，et al.A content-aware metric for stitched panoramic image quality assessment[C]//2017 IEEE International Conference on Computer Vision Workshops（ICCVW），Venice，Oct 22-29，2017.Piscataway：IEEE，2017：2487-2494.
[35] XIAO J，EHINGER K A，OLIVA A，et al.Recognizing scene viewpoint using panoramic place representation[C]//2012 IEEE Conference on Computer Vision and Pattern Recognition，Providence，Jun 16-21，2012.Piscataway：IEEE，2012：2695-2702.
[36] WALLACE G K.The JPEG still picture compression standard[J].IEEE Transactions on Consumer Electronics，1992，38（1）.
[37] SKODRAS A，CHRISTOPOULOS C，EBRAHIMI T，et al.The JPEG 2000 still image compression standard[J].IEEE Signal Processing Magazine，2001，18（5）：36-58.
[38] SULLIVAN G J，OHM J R，HAN W J，et al.Overview of the high efficiency video coding （HEVC） standard[J].IEEE Transactions on Circuits & Systems for Video Technology，2013，22（12）：1649-1668.
[39] DUAN H Y，ZHAI G T，MIN X K，et al.Perceptual quality assessment of omnidirectional images[C]//2018 IEEE International Symposium on Circuits and Systems（ISCAS），Florence，May 27-30，2018.Piscataway：IEEE，2018：1-5.
[40] SUN W，GU K，MA S W，et al.A large-scale compressed 360-degree spherical image database：from subjective quality evaluation to objective model comparison[C]//2018 IEEE 20th International Workshop on Multimedia Signal Processing（MMSP），Vancouver，Aug 29-31，2018.Piscataway：IEEE，2018：1-6.
[41] Multi-distortions visual attention quality dataset （MVAQD）[DB/OL].[2019-10-03].https：//github.com/Jianghao2019/MVAQD.
[42] MADHUSUDANA P C，SOUNDARARAJAN R.Subjective and objective quality assessment of stitched images for virtual reality[J].IEEE Transactions on Image Processing，2019，28（11）：5620-5635.
[43] LI J，YU K W，ZHAO Y F.Cross-reference stitching quality assessment for 360° omnidirectional images[C]//Proceedings of the 27th ACM International Conference on Multimedia，Nice，Oct 21-25，2019.New York：Association for Computing Machinery，2019：2360-2368.
[44] BRUNNSTROM K，HANDS D，SPERANZA F，et al.VQEG validation and ITU standardization of objective perceptual video quality metrics [Standards in a Nutshell][J].IEEE Signal Processing Magazine，2009，26（3）：96-101.