Overview of Image Quality Assessment Method Based on Deep Learning

doi:10.3778/j.issn.1002-8331.2106-0228

Abstract

Abstract:

Image quality evaluation is a measurement of the visual quality of an image or video. The researches on image quality evaluation algorithms in the past 10 years are reviewed. First, the measurement indicators of image quality evaluation algorithm and image quality evaluation datasets are introduced. Then, the different classification of image quality evaluation methods are analyzed, and image quality evaluation algorithms with deep learning technology are focused on, basic model of which is deep convolutional network, deep generative adversarial network and transformer. The performance of algorithms with deep learning is often higher than that of traditional image quality assessment algorithms. Subsequently, the principle of image quality assessment with deep learning is described in detail. A specific no-reference image quality evaluation algorithm based on deep generative adversarial network is introduced, which improves the reliability of simulated reference images through enhanced confrontation learning. Deep learning technology requires massive data support. Data enhancement methods are elaborated to improve the performance of the model. Finally, the future research trend of digital image quality evaluation is summarized.

Key words: deep learning, image quality assessment, generative adversarial network, Convolutional Neural Network（CNN）, data enhancement

摘要：

图像质量评价是对图像或视频的视觉质量的一种度量，主要分析了最近10年图像质量评价算法的研究情况。介绍了图像质量评价算法的衡量指标以及常用的图像质量评价数据集，对图像质量评价方法的分类做了阐述，重点分析了基于深度学习技术的图像质量评价算法。目前，该类算法的基础模型主要包括深度卷积神经网络、深度生成对抗网络和变换器，其性能通常高于传统的图像质量评价算法。描述了基于深度学习技术的图像质量评价算法的原理，重点介绍了基于深度生成对抗网络的无参考图像质量评价算法，通过增强对抗学习强度提高模拟参考图的可靠性。深度学习技术需要海量训练数据的支持，探讨和总结数据集增强的方法，对数字图像质量评价方法的未来研究进行展望。

关键词: 深度学习, 图像质量评价, 生成对抗网络, 卷积神经网络, 数据增强

CAO Yudong, LIU Haiyan, JIA Xu, LI Xiaohui. Overview of Image Quality Assessment Method Based on Deep Learning[J]. Computer Engineering and Applications, 2021, 57(23): 27-36.

曹玉东，刘海燕，贾旭，李晓会. 基于深度学习的图像质量评价方法综述[J]. 计算机工程与应用, 2021, 57(23): 27-36.

References

[1] 王志明.无参考图像质量评价综述[J].自动化学报，2015，41（6）：1062-1079.
WANG Z M.Review of no-reference image quality assessment[J].Acta Automatica Sinica，2015，41（6）：1062-1079.
[2] 张淑芳，张聪，张涛，等.通用型无参考图像质量评价算法综述[J].计算机工程与应用，2015，51（19）：13-23.
ZHANG S F，ZHANG C，ZHANG T，et al.Review on universal no-reference image quality assessment algorithm[J].Computer Engineering and Applications，2015，51（19）：13-23.
[3] 褚江，陈强，杨曦晨.全参考图像质量评价综述[J].计算机应用研究，2014，31（1）：13-22.
CHU J，CHEN Q，YANG X C.Review on full reference image quality assessment algorithms[J].Application Research of Computers，2014，31（1）：13-22.
[4] RAPPORTEUR ITU-T Q21/9.Final report from the video quality experts group on the validation of objective models of video quality assessment，Phase II（FR-TV2） VQEG[R].Geneva，Swiss：ITU Telecommunication Standardization Sector，2003.
[5] MA K，DUANMU Z，WU Q，et al.Waterloo exploration database：new challenges for image quality assessment models[J].IEEE Transactions on Image Processing，2017，26（99）：1004-1016.
[6] SHEIKH H R，SABIR M F，BOVIK A C.A statistical evaluation of recent full reference quality assessment algorithms[J].IEEE Transactions on Image Processing，2006，15（11）：3440-3451.
[7] SHEIKH H R，WANG Z，CORMACK L，et al.LIVE image quality assessment database release 2[EB/OL].[2021-04-10].http：//live.ece.utexas.edu/research/quality.
[8] PONOMARENKO N，LUKIN V，ZELENSKY A，et al.TID2008-a database for evaluation of full-reference visual quality assessment metrics[J].Advances of Modern Radioelectronics，2009（10）：30-45.
[9] PONOMARENKO N，IEREMEIEV O，LUKIN V，et al.Color image database TID2013：peculiarities and preliminary results[C]//Proceedings of 4th Europian Workshop on Visual Information Processing，Paris，France，2013：106-111.
[10] PONOMARENKO N，JIN L，IEREMEIEV O，et al.Image database TID2013：peculiarities，results and perspectives[J].Signal Processing：Image Communication，2015（30）：57-77.
[11] LARSON E C，CHANDLER D M.Most apparent distortion：full-reference image quality assessment and the role of strategy[J].Journal of Electronic Imaging，2010，19（1）：1-21.
[12] LIN H，HOSU V，SAUPE D.KADID-10k：a large-scale artificially distorted IQA database[C]//2019 Eleventh International Conference on Quality of Multimedia Experience（QoMEX），2019：1-3.
[13] HOSU V，LIN H，SZIRANYI T，et al.KonIQ-10k：an ecologically valid database for deep learning of blind image quality assessment[J].IEEE Transactions on Image Processing，2020，29：4041-4056.
[14] 姚旺，刘云鹏，朱昌波.基于人眼视觉特性的深度学习全参考图像质量评估方法[J].红外与激光工程，2018，47（7）：39-46.
YAO W，LIU Y P，ZHU C B.Deep learning of full-reference image quality assessment based on human visual properties[J].Infrared and Laser Engineering，2018，47（7）：39-46.
[15] WANG Z，BOVIK A C，SHEIKH H R.Image quality assessment：from error visibility to structural similarity[J].IEEE Transactions on Image Processing，2004，13（4）：600-612.
[16] WANG Z，BOVIK A C，SHEIKH H R.Quality-aware images[J].IEEE Transactions on Image Processing，2006，15（6）：1680-1689.
[17] ZHANG L，MOU X.FSIM：a feature similarity index for image quality assessment[J].IEEE Transactions on Image Processing，2011，20（8）：2378.
[18] ZHOU W，QIANG L.Information content weighting for perceptual image quality assessment[J].IEEE Transactions on Image Processing，2011，20（5）：1185-1198.
[19] SHEIKH H R，BOVIK A C，VECIANA G.An information fidelity criterion for image quality assessment using natural scene statistics[J].IEEE Transactions on Image Processing，2005，14（12）：2117-2128.
[20] SHEIKH H R，BOVIK A C.Image information and visual quality[J].IEEE Transactions on Image Processing，2006，15（2）：430-444.
[21] 王同罕，贾惠珍，舒华忠.基于梯度幅度和梯度方向直方图的全参考图像质量评价算法[J].东南大学学报（自然科学版），2018，48（2）：276-281.
WANG T H，JIA H Z，SHU H Z.Full-reference image quality assessment algorithm based on gradient magnitude and histogram of oriented gradient[J].Journal of Southeast University（Natural Science Edition），2018，48（2）：276-281.
[22] YE P，KUMAR J，KANG L，et al.Unsupervised feature learning framework for no-reference image quality assessment[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition，2012：1098-1105.
[23] LIU Y，GU K，ZHANG Y，et al.Unsupervised blind image quality evaluation via statistical measurements of structure，naturalness and perception[J].IEEE Transactions on Circuits and Systems for Video Technology，2020，30（4）：929-943.
[24] KANG L，YE P，LI Y，et al.Simultaneous estimation of image quality and distortion via multi-task convolutional neural networks[C]//Proceedings of IEEE International Conference on Image Processing，2015：2791-2795.
[25] BOSSE S，MANIRY D，MULLER K R.Deep neural networks for no-reference and full-reference image quality assessment[J].IEEE Transaction on Image Processing，2018，27（1）：206-219.
[26] KIM A，NGUYEN，LEE S.Deep CNN-based blind image quality predictor[J].IEEE Transactions on Neural Networks and Learning Systems，2019，30（1）：11-24.
[27] 高方远，何立火.基于深度网络和视觉特性的无参考图像质量评价方法[J].南京师大学报（自然科学版），2019，42（3）：20-26.
GAO F Y，HE L H.No-reference image quality assessment based on deep network and visual characteristics[J].Journal of Nanjing Normal University（Natural Science Edition），2019，42（3）：20-26.
[28] LIN K Y，WANG G.Hallucinated-IQA：no-reference image quality assessment via adversarial learning[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），2018：732-741.
[29] ZHU H.Image quality assessment model based on multi-feature fusion of energy internet of things[J].Future Generation Computer Systems，2020，112：501-506.
[30] 朱秀昌，唐贵进.可视质量评价方法综述[J].南京邮电大学学报（自然科学版），2018，38（4）：1-11.
ZHU X C，TANG G J.Overview of visual quality assessment methods[J].Journal of Nanjing University of Posts and Telecommunications（Natural Science Edition），2018，38（4）：1-11.
[31] 周景超，戴汝为，肖柏华.图像质量评价研究综述[J].计算机科学，2008（7）：1-4.
ZHOU J C，DAI R W，XIAO B H.Overview of image quality assessment research[J].Computer Science，2008（7）：1-4.
[32] 杨璐，王辉，魏敏.基于机器学习的无参考图像质量评价综述[J].计算机工程与应用，2018，54（19）：34-42.
YANG L，WANG H，WEI M.Review of no-reference image quality assessment based on machine learning[J].Computer Engineering and Applications，2018，54（19）：34-42.
[33] 佟雨兵，胡薇薇，杨东凯，等.视频质量评价方法综述[J].计算机辅助设计与图形学学报，2006（5）：735-741.
TONG Y B，HU W W，YANG D K，et al.A review on the video quality assessment methods[J].Journal of Computer-Aided Design & Computer Graphics，2006（5）：735-741.
[34] 曹玉东，蔡希彪.基于并行小规模卷积神经网络的图像质量评价[J/OL].计算机工程与科学[2020-09-26].http：//kns.cnki.net/kcms/detail/43.1258.TP.20200917.1923.002.html.
CAO Y D，CAI X B.Image quality evaluation algorithm with parallel small CNN[J/OL].Computer Engineering & Scinence[2020-09-26].http：//kns.cnki.net/kcms/detail/43.1258.TP.20200917.1923.002.html.
[35] MA Y，CAI X，SUN F.Towards no-reference image quality assessment based on multi-scale convolutional neural network[J].Computer Modeling in Engineering and Sciences，2020，123（1）：201-216.
[36] MA Y，CAI X B，SUN F M，et al.No-reference image quality assessment based on multi?task generative adversarial network[J].IEEE Access，2019，7：146893-146902.
[37] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[J].arXiv：1706.03762，2017.
[38] KHAN S，NASEER M，HAYAT M，et al.Transformers in vision：a survey[J].arXiv：2101.01169，2021.
[39] KANG L，YE P，LI Y.Convolutional neural networks for no-reference image quality assessment[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington，DC：IEEE Computer Society，2014：1733-1740.
[40] HOU W L，GAO X B，TAO D C，et al.Blind image quality assessment via deep learning[J].IEEE Transactions on Neural Networks & Learning Systems，2017，26（6）：1275-1286.
[41] SIMONCELLI E P，OLSHAUSEN B A.Natural image statistics and neural representation[J].Annu Rev Neurosci，2001，24（1）：1193-1216.
[42] HINTON G E，OSINDERO S，TEH Y W.A fast learning algorithm for deep belief nets[J].Neural Comput，2006，18（7）：1527-1554.
[43] LIU X，WEIJER D，BAGDANOV A D.RankIQA：learning from rankings for no-reference image quality assessment[C]//Proceedings of 2017 IEEE International Conference on Computer Vision.Washington，DC：IEEE Computer Society，2017：1040-1049.
[44] MA K，LIU W，ZHANG K，et al.End-to-end blind image quality assessment using deep neural networks[J].IEEE Transactions on Image Processing，2018，27（3）：1202-1213.
[45] REN H，CHEN D，WANG Y.RAN4IQA：restorative adversarial nets for no-reference image quality assessment[J].arXiv：1712.05444，2017.
[46] 曹玉东，蔡希彪.基于增强型对抗学习的无参考图像质量评价算法[J].计算机应用，2020，40（11）：3166-3171.
CAO Y D，CAI X B.No-reference image quality assessment algorithm with enhanced adversarial learning[J].Journal of Computer Applications，2020，40（11）：3166-3171.
[47] YOU J，KORHONEN J.Transformer for image quality assessment[J].arXiv：2101.01097，2021.
[48] CHEON M，YOON S J，KANG B，et al.Perceptual image quality assessment with transformers[J].arXiv：2104.14730，2021.
[49] GU J，CAI H，DONG C，et al.NTIRE 2021 challenge on perceptual image quality assessment[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop，2021.
[50] RAMACHANDRAN P，PARMAR N，VASWANI A，et al.Stand-alone self-attention in vision models[J].arXiv：1906.05909，2019.
[51] CORDONNIER J B，LOUKAS A，JAGGI M.On the relationship between self-attention and convolutional layers[J].arXiv：1911.03584，2019.
[52] YE P，KUMAR J，DOERMANN D.Beyond human opinion scores：blind image quality assessment based on synthetic scores[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington，DC：IEEE Computer Society，2014：4241-4248.
[53] MITTAL A，SOUNDARARAJAN R，BOVIK A C.Making a completely blind image quality analyzer[J].IEEE Signal Process Letters，2013，20（3）：209-212.
[54] XUE W，ZHANG L，MOU X，et al.Gradient magnitude similarity deviation：a highly efficient perceptual image quality index[J].IEEE Transactions on Image Processing，2014，23（2）：684-695.
[55] XUE W，ZHANG L，MOU X.Learning without human scores for blind image quality assessment[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition（CVPR），2013：995-1002.
[56] GAO F，WANG Y，LI P，et al.Deep sim：deep similarity for image quality assessment[J].Neurocomputing，2017，257（27）：104-114.
[57] MA K，LIU W T，LIU T L，et al.dipIQ：blind image quality assessment by learning-to-rank discriminable image pairs[J].IEEE Transactions on Image Processing，2017，26（8）：3951-3964.
[58] KIM J，LEE S.Fully deep blind image quality predictor[J].IEEE Journal of Selected Topics in Signal Processing，2017，11（1）：206-220.
[59] 冯天鹏.基于稀疏表达联合迁移学习的盲图像质量评价[D].武汉：武汉大学，2017.
FENG T P.Blind image quality assessment based on sparse representation fused with transfer learning[D].Wuhan：Wuhan University，2017.