Survey of Research on Compressed Image Enhancement Methods

doi:10.3778/j.issn.1002-8331.2304-0248

Abstract

Abstract: Nowadays efficient image compression has become a necessary means for effective storage and transmission of digital images. However, compressed images inevitably suffer from some issues such as block artifacts, ringing artifacts, image blurring, etc. As an important way to improve image coding efficiency, compressed-image enhancement technology not only can improve compressed-image quality, but also is widely used in the pre-processing stage of computer vision tasks such as detection, recognition, segmentation, etc. This paper provides an overview of the research on compressed-image enhancement methods from the following aspects. Firstly, this paper introduces the development and classification of image enhancement technique, and compares their advantages and disadvantages, starting with traditional compressed-image enhancement approaches and compressed-image enhancement methods based on deep learning. Secondly, several key techniques of compressed-image enhancement are introduced and analyzed, such as contrastive learning, reinforcement learning, curriculum learning, knowledge distillation, adversarial learning, and network architecture search. Finally, this paper summarizes and gives some prospects on the future development direction of compressed-image enhancement technique.

Key words: image compression, compression distortion, image enhancement, deep learning, neural network

摘要： 现在高效的图像压缩已经成为数字图像有效存储和传输的必要手段。经过压缩之后的图像难免存在块伪影、震荡伪影、图像模糊等问题。压缩图像增强技术作为图像编码效率提升的重要方式不仅能够提升压缩图像的质量，而且被广泛应用到计算机视觉任务如检测、识别、分割等的预处理阶段。从以下几个方面对压缩图像增强方法研究进行综述。从传统的压缩图像增强方法和基于深度学习的压缩图像增强方法入手，介绍图像增强技术的发展与分类，同时比较它们的优缺点。介绍并分析压缩图像增强的几种关键性技术如对比学习、强化学习、课程学习、知识蒸馏、对抗学习和网络架构搜索。总结全文并且对压缩图像增强技术的未来发展方向进行展望。

关键词: 图像压缩, 压缩失真, 图像增强, 深度学习, 神经网络

ZHAO Lijun, CAO Congying, ZHANG Jinjing, ZHAO Jie, CHEN Bintao, WANG Anhong. Survey of Research on Compressed Image Enhancement Methods[J]. Computer Engineering and Applications, 2023, 59(21): 26-38.

赵利军, 曹聪颖, 张晋京, 赵杰, 陈彬涛, 王安红. 压缩图像增强方法研究综述[J]. 计算机工程与应用, 2023, 59(21): 26-38.

References

[1] 颜兵，王金鹤，赵静.基于均值滤波和小波变换的图像去噪技术研究[J].计算机技术与发展，2011，21（2）：51-53.
YAN B，WANG J H，ZHAO J.Research of image denoising technology based on mean filtering and wavelet transformation[J].Computer Technology and Development，2011，21（2）：51-53.
[2] 刘广迪.基于Kinect的案件现场三维重建方法研究[D].北京：中国人民公安大学，2017.
LIU G D.Research on Kinect based 3D reconstruction method for case scenes[D].Beijing：People’s Public Security University of China，2017.
[3] 李鸿林，张忠民，羿宗琪.中值滤波技术在图像处理中的应用[J].信息技术，2004，28（7）：26-27.
LI H L，ZHANG Z M，YI Z Q.The application of median filtering on image processing[J].Information Technology，2004，28（7）：26-27.
[4] TOMASI C，MANDUCHI R.Bilateral filtering for gray and color images[C]//International Conference on Computer Vision.Bombay：IEEE，2002.
[5] ZHANG J，XIONG R，ZHAO C，et al.CONCOLOR：constrained non-convex low-rank model for image deblocking[J].IEEE Transactions on Image Processing，2016，25（3）：1246-1259.
[6] CHANG H，NG M，ZENG T.Reducing artifacts in JPEG decompression via a learned dictionary[J].IEEE Transactions on Signal Processing，2013，62（3）：718-728.
[7] BUADES A，COLL B，MOREL J M.A non-local algorithm for image denoising[C]//IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2005：60-65.
[8] MA X，ZOU J，LI W，et al.Miniature spectrometer based on a Fourier transform spectrometer chip and a commercial photodetector array[J].Chinese Optics Letters，2019，17（12）：123001.
[9] HERRREO B A，LI J，KHAZAEI M，et al.On-chip Fourier-transform spectrometers and machine learning：a new route to smart photonic sensors[J].Optics Letters，2019，44（23）：5840-5843.
[10] 付华，李楠，高楠.数字信号处理[M].北京：电子工业出版社，2018：231-240.
FU H，LI N，GAO N，et al.Digital signal processing[M].Beijing：Publishing House of Electronics Industry，2018：231-240.
[11] COIFMAN R，WICKERHAUER V.Entropy-based algorithms for best basis selection[J].IEEE Transactions on Information Theory，1992，38（2）：713-718.
[12] MICHAL A，MICHAL E，ALFRED B.K-SVD：an algorithm for designing overcomplete dictionaries for sparse representation[J].IEEE Transactions on Signal Processing：A Publication of the IEEE Signal Processing Society，2006，54（11）：4311-4322.
[13] DABOV K，FOI A，KATKOVNIK V，et al.Image denoising by sparse 3D-transform domain collaborative filtering[J].IEEE Transactions on Image Processing，2007，16（8）：2080-2095.
[14] DONG C，DENG Y，CHEN C L，et al.Compression artifacts reduction by a deep convolutional network[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：576-584.
[15] ZHANG K，ZUO W，CHEN Y，et al.Beyond a Gaussian denoiser：residual learning of deep CNN for image denoising[J].IEEE Transactions on Image Processing，2017，26（7）：3142-3155.
[16] JIN Z，MUHAMMAD Z I，BOBKOV D，et al.A flexible deep CNN framework for image restoration[J].IEEE Transactions on Multimedia，2019，22（4）：1055-1068.
[17] TODERICI G，O’MALLEY S M，HWANG S J，et al.Variable rate image compression with recurrent neural networks[J].arXiv：1511.06085，2015.
[18] TODERICI G，VINCENT D，JOHNSTON N，et al.Full resolution image compression with recurrent neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：5306-5314.
[19] RIPPEL O，BOURDEV L.Real-time adaptive image compression[C]//Proceedings of the 34th International Conference on Machine Learning，2017：2922-2930.
[20] WANG Z，CUN X，BAO J，et al.Uformer：a general U-shaped transformer for image restoration[J].arXiv：2106.03106，2021.
[21] LIANG J，CAO J，SUN G，et al.SwinIR：image restoration using swin transformer[J].arXiv：2108.10257，2021.
[22] WANG Z，LIU D，CHANG S，et al.D3：deep dual-domain based fast restoration of JPEG-compressed images[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：2764-2772.
[23] LIU P，ZHANG P，ZHANG K，et al.Multi-level wavelet-CNN for image restoration[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops，2018：773-782.
[24] CHEN H，HE X，QING L，et al.DPW-SDNet：dual pixel-wavelet domain deep CNNs for soft decoding of JPEG-compressed images[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops，2018：711-720.
[25] KIRMEMIS O，BAKAR G，TEKALP A M.Learned compression artifact removal by deep residual networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops，2018：2602-2605.
[26] ZHANG X，YANG W，HU Y，et al.DMCNN：dual-domain multi-scale convolutional neural network for compression artifacts removal[C]//The 25th IEEE International Conference on Image Processing（ICIP），2018：390-394.
[27] ZHANG B，CHEN Y，TIAN X，et al.Implicit dual-domain convolutional network for robust color image compression artifact reduction[J].IEEE Transactions on Circuits and Systems for Video Technology，2020，30：3982-3994.
[28] ZHANG K，ZUO W，ZHANG L.Learning a single convolutional super-resolution network for multiple degradations[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：3262-3271.
[29] BAI Y，YANG X，LIU X，et al.Towards end-to-end image compression and analysis with transformers[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2022：104-112.
[30] GUO J，CHAO H.One-to-many network for visually pleasing compression artifacts reduction[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：3038-3047.
[31] FAN Q，CHEN D，YUAN L，et al.A general decoupled learning framework for parameterized image operators[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2019，43（1）：33-47.
[32] HE J，DONG C，QIAO Y.Interactive multi-dimension modulation with dynamic controllable residual learning for image restoration[C]//Proceedings 16th European Conference on Computer Vision，Glasgow，UK，August 23-28，2020.[S.l.]：Springer International Publishing，2020：53-68.
[33] WANG W，GUO R，TIAN Y，et al.CFSNet：toward a controllable feature space for image restoration[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：4140-4149.
[34] XU J，YUAN M，YAN D M，et al.Deep unfolding multi-scale regularizer network for image denoising[J].Computational Visual Media，2023，9（2）：335-350.
[35] NING Q，DONG W，SHI G，et al.Accurate and lightweight image super-resolution with model-guided deep unfolding network[J].IEEE Journal of Selected Topics in Signal Processing，2020，15（2）：240-252
[36] YAN K，ZHOU M，ZHANG L，et al.Memory-augmented model-driven network for pansharpening[C]//Proceedings 17th European Conference on Computer Vision，2022：306-322.
[37] YANG G，ZHANG L，ZHOU M，et al.Model-guided multi-contrast deep unfolding network for MRI super-resolution reconstruction[C]//Proceedings of the 30th ACM International Conference on Multimedia，2022：3974-3982.
[38] FU X，WANG M，CAO X，et al.A model-driven deep unfolding method for JPEG artifacts removal[J].IEEE Transactions on Neural Networks and Learning Systems，2021，33（11）：6802-6816.
[39] YOU D，XIE J，ZHANG J.ISTA-Net++：flexible deep unfolding network for compressive sensing[C]//2021 IEEE International Conference on Multimedia and Expo（ICME），2021：1-6.
[40] SONG J，CHEN B，ZHANG J.Memory-augmented deep unfolding network for compressive sensing[C]//Proceedings of the 29th ACM International Conference on Multimedia，2021：4249-4258.
[41] YOU D，ZHANG J，XIE J，et al.COAST：controllable arbitrary-sampling network for compressive sensing[J].IEEE Transactions on Image Processing，2021，30：6066-6080.
[42] ZHANG J，ZHAO C，GAO W.Optimization-inspired compact deep compressive sensing[J].IEEE Journal of Selected Topics in Signal Processing，2020，14（4）：765-774.
[43] MOU C，WANG Q，ZHANG J.Deep generalized unfolding networks for image restoration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：17399-17410.
[44] WANG H，LI Y，ZHANG H，et al.InDuDoNet+：a model-driven interpretable dual domain network for metal artifact reduction in CT images[J].arXiv：2112.12660，2021.
[45] ZHENG C，SHI D，SHI W.Adaptive unfolding total variation network for low-light image enhancement[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：4439-4448.
[46] HUANG Y，LI S，WANG L，et al.Unfolding the alternating optimization for blind super resolution[C]//Advances in Neural Information Processing Systems，2020：5632-5643.
[47] WU W，WENG J，ZHANG P，et al.URetinex-Net：retinex-based deep unfolding network for low-light image enhancement[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：5901-5910.
[48] REN C，HE X，QING L，et al.Deep deblocker driven adaptive iteration scheme for compressed image recovery[C]//2021 IEEE International Conference on Multimedia and Expo（ICME），2021：1-6.
[49] AGHABIGLOU A，EKSIOGLU E M.Deep unfolding architecture for MRI reconstruction enhanced by adaptive noise maps[J].Biomedical Signal Processing and Control，2022，78：104016.
[50] ZHOU M，YAN K，PAN J，et al.Memory-augmented deep unfolding network for guided image super-resolution[J].International Journal of Computer Vision，2023，131（1）：215-242.
[51] WANG H，XIE Q，ZHAO Q，et al.RCDNet：an interpretable rain convolutional dictionary network for single image deraining[J].IEEE Transactions on Neural Networks and Learning Systems，2023.
[52] ZHANG K，ZUO W，GU S，et al.Learning deep CNN denoiser prior for image restoration[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：3929-3938.
[53] WANG Z，BOVIK A C，SHEIKH H R，et al.Image quality assessment：from error visibility to structural similarity[J].IEEE Transactions on Image Processing，2004，13（4）：600-612.
[54] CANDèS E J，TAO T.The power of convex relaxation：near-optimal matrix completion[J].IEEE Transactions on Information Theory，2010，56（5）：2053-2080.
[55] PEARSON K.LIII.On lines and planes of closest fit to systems of points in space[J].The London，Edinburgh，and Dublin Philosophical Magazine and Journal of Science，1901，2（11）：559-572.
[56] 邸云霞，孔慧华，牛晓伟.基于主成分分析的多能谱CT图像分析方法研究[J].CT理论与应用研究，2022，31（6）：749-760.
DI Y X，KONG H H，NIU X W.Research on image analysis method of spectral CT based on principal component analysis[J].CT Theory and Applications，2022，31（6）：749-760.
[57] 肖汉，孙陆鹏，李彩林，等.面向GPU的直方图统计图像增强并行算法[J].计算机科学与探索，2022，16（10）：2273-2285.
XIAO H，SUN L P，LI C L，et al.GPU-oriented parallel algorithm for histogram statistical image enhancement[J].Journal of Frontiers of Computer Science and Technology，2022，16（10）：2273-2285.
[58] 林成创，单纯，赵淦森，等.机器视觉应用中的图像数据增广综述[J].计算机科学与探索，2021，15（4）：583-611.
LIN C C，SHAN C，ZHAO G S，et al.Review of image data augmentation in computer vision[J].Journal of Frontiers of Computer Science and Technology，2021，15（4）：583-611.
[59] GUTMANN M，HYV?RINEN A.Noise-contrastive estimation：a new estimation principle for unnormalized statistical models[C]//Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics，2010：297-304.
[60] KINGMA D P，WELLING M.Stochastic gradient VB and the variational auto-encoder[C]//Second International Conference on Learning Representations，2014：121.
[61] CARON M，MISRA I，MAIRAL J，et al.Unsupervised learning of visual features by contrasting cluster assignments[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems，2020：9912-9924.
[62] HE K，FAN H，WU Y，et al.Momentum contrast for unsupervised visual representation learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：9729-9738.
[63] GRILL J B，STRUB F，ALTCHé F，et al.Bootstrap your own latent-a new approach to self-supervised learning[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems，2020：21271-21284.
[64] WU G，JIANG J，LIU X，et al.A practical contrastive learning framework for single image super-resolution[J].arXiv：2111.13924，2021.
[65] JI H，FENG X，PEI W，et al.U2-former：a nested U-shaped transformer for image restoration[J].arXiv：2112.02279，2021.
[66] WU H，QU Y，LIN S，et al.Contrastive learning for compact single image dehazing[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：10551-10560.
[67] LI F，SHEN L，MI Y，et al.DRCNet：dynamic image restoration contrastive network[C]//Proceedings 17th European Conference on Computer Vision，Tel Aviv，Israel，October 23-27，2022：514-532.
[68] LI B，LIU X，HU P，et al.All-in-one image restoration for unknown corruption[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：17452-17462.
[69] KAELBLING L P，LITTMAN M L，MOORE A W.Reinforcement learning：a survey[J].Journal of Artificial Intelligence Research，1996，4：237-285.
[70] RODERICK M，MACGLASHAN J，TELLEX S.Implementing the deep q-network[J].arXiv：1711.07478，2017.
[71] CASAS N.Deep deterministic policy gradient for urban traffic light control[J].arXiv：1703.09035，2017.
[72] LILLICRAP T P，HUNT J J，PRITZEL A，et al.Continuous control with deep reinforcement learning[J].arXiv：1509.02971，2015.
[73] YU K，DONG C，LIN L，et al.Crafting a toolchain for image restoration by deep reinforcement learning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：2443-2452.
[74] ZHANG J，ZHANG Q，ZHAO X，et al.Boosting denoisers with reinforcement learning for image restoration[J].Soft Computing，2022，26（7）：3261-3272.
[75] YU K，WANG X，DONG C，et al.Path-restore：learning network path selection for image restoration[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2021，44（10）：7078-7092.
[76] FURUTA R，INOUE N，YAMASAKI T.PixelRL：fully convolutional network with reinforcement learning for image processing[J].IEEE Transactions on Multimedia，2019，22（7）：1704-1719.
[77] BENGIO Y，LOURADOUR J，COLLOBERT R，et al.Curriculum learning[C]//Proceedings of the 26th Annual International Conference on Machine Learning，2009：41-48.
[78] KOCMI T，BOJAR O.Curriculum learning and minibatch bucketing in neural machine translation[J].arXiv：1707.09533，2017.
[79] KUMAR M，PACKER B，KOLLER D.Self-paced learning for latent variable models[C]//Proceedings of the 23rd International Conference on Neural Information Processing Systems，2010：1189-1197.
[80] CHANG Y，CHEN M，YU C，et al.Direction and residual awareness curriculum learning network for rain streaks removal[J].IEEE Transactions on Neural Networks and Learning Systems，2023：1-15.
[81] SHU J，XIE C，GAO Z.Blind restoration of atmospheric turbulence-degraded images based on curriculum learning[J].Remote Sensing，2022，14（19）：4797.
[82] HINTON G，VINYALS O，DEAN J.Distilling the knowledge in a neural network[J].arXiv：1503.02531，2015.
[83] HUANG Z，WANG N.Like what you like：knowledge distill via neuron selectivity transfer[J].arXiv：1707. 01219，2017.
[84] PASSALIS N，TEFAS A.Learning deep representations with probabilistic knowledge transfer[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：268-284.
[85] LEE S H，KIM D H，SONG B C.Self-supervised knowledge distillation using singular value decomposition[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：335-350.
[86] ZHU H，CHEN Z，LIU S.Learning knowledge representation with meta knowledge distillation for single image super-resolution[J].arXiv：2207.08356，2022.
[87] XIA B，ZHANG Y，WANG Y，et al.Knowledge distillation based degradation estimation for blind super-resolution[J].arXiv：2211.16928，2022.
[88] LI J，YANG H，YI Q，et al.Multiple degradation and reconstruction network for single image denoising via knowledge distillation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：558-567.
[89] CUI X，WANG C，REN D，et al.Semi-supervised image deraining using knowledge distillation[J].IEEE Transactions on Circuits and Systems for Video Technology，2022，32（12）：8327-8341.
[90] GALTERI L，SEIDENARI L，BERTINI M，et al.Deep universal generative adversarial compression artifact removal[J].IEEE Transactions on Multimedia，2019，21（8）：2131-2145.
[91] 张雪峰，许华文.一种基于条件生成对抗网络的高感知图像压缩方法[J].东北大学学报（自然科学版），2022，43（6）：783-791.
ZHANG X F，XU H W.High perceptual image compression based on conditional GAN[J].Journal of Northeastern University（Natural Science），2022，43（6）：783-791.
[92] MA H，LIU D，WU F.Rectified wasserstein generative adversarial networks for perceptual image restoration[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2022，45（3）：3648-3663.
[93] ZHANG B W，GU S Y，ZHANG B，et al.StyleSwin：transformer-based GAN for high-resolution image generation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：11304-11314.
[94] YU S，CHEN B，XU Y，et al.HEVC compression artifact reduction with generative adversarial networks[C]//2019 11th International Conference on Wireless Communications and Signal Processing（WCSP），2019：1-6.
[95] NAYMAN N，NOY A，RIDNIK T，et al.XNAS：Neural architecture search with expert advice[J].arXiv：1906. 08031，2019.
[96] CHEN Y，MENG G，ZHANG Q，et al.RENAS：reinforced evolutionary neural architecture search[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：4787-4796.
[97] LEE B，KO K，HONG J，et al.Domain-agnostic single-image super-resolution via a meta-transfer neural architecture search[J].Neurocomputing，2023，524：59-68.
[98] CHENG G，MATSUNE A，DU H，et al.Exploring more diverse network architectures for single image super-resolution[J].Knowledge-Based Systems，2022，235：107648.
[99] WU Y，GONG Y，ZHAO P，et al.Compiler-aware neural architecture search for on-mobile real-time super-resolution[C]//17th European Conference on Computer Vision，Tel Aviv，Israel，October 23-27，2022：92-111.
[100] CAI L，FU Y，HUO W，et al.Multi-scale attentive image de-raining networks via neural architecture search[J].arXiv：2207.00728，2022.
[101] NING Q，DONG W，LI X，et al.Searching efficient model-guided deep network for image denoising[J].IEEE Transactions on Image Processing，2022，32：668-681.
[102] ZHANG X，ZENG H，ZHANG L.Efficient hardware-aware neural architecture search for image super-resolution on Mobile Devices[C]//Proceedings of the Asian Conference on Computer Vision，2022：721-738.