单幅RGB图像计算光谱成像的深度学习研究综述

doi:10.3778/j.issn.1002-8331.2211-0082

摘要/Abstract

摘要： 为解决传统的光谱成像方法成本高、图像采集时间较长的问题，深度学习被引入计算光谱成像来研究如何从单幅RGB图像中重建光谱，为各种计算机视觉应用提供辅助信息。当前对基于深度学习的单幅RGB图像计算光谱成像方法还未有全面、系统的深入认识与研究。为此针对计算光谱成像所使用的深度学习算法和网络模型进行了系统的归纳、分析和对比。基于CNN（convolutional neural networks）、GAN（generative adversarial networks）、注意力和Transformer四个类别详细梳理了近几年重建性能优异的有监督学习方法；基于自编码器和领域自适应两类别分析、探讨、比较了热度较高的无监督学习方法。同时列举了算法常用数据集和评估指标，对未来的研究趋势和发展方向进行了展望。

关键词: 计算机视觉, 深度学习, 光谱图像, 计算光谱成像

Abstract: Deep learning is introduced into computational spectral imaging to address the high cost and long image acquisition time of traditional spectral imaging methods to investigate how spectral information can be recovered from a single RGB image to provide assistance for various computer vision applications. Computational spectral imaging methods for single RGB images based on deep learning lack comprehensive and systematic research. Deep learning algorithms and network models used for computational spectral imaging are summarised, analyzed and compared. The four categories of CNN（convolutional neural networks）, GAN（generative adversarial networks）, Attention and Transformer are used to sort out supervised learning methods with excellent reconstruction performance in recent years. The unsupervised learning methods are discussed in terms of both self-encoders and domain adaptation. Datasets and evaluation metrics commonly used for the algorithms are listed, and future research trends and development directions are given.

Key words: computer vision, deep learning, spectral image, computational spectral imaging

姜中敏, 张婉言, 王文举. 单幅RGB图像计算光谱成像的深度学习研究综述[J]. 计算机工程与应用, 2023, 59(10): 22-34.

JIANG Zhongmin, ZHANG Wanyan, WANG Wenju. Research of Deep Learning-Based Computational Spectral Imaging for Single RGB Image[J]. Computer Engineering and Applications, 2023, 59(10): 22-34.

参考文献

[1] HU Z，FANG C，LI B，et al.First-in-human liver-tumour surgery guided by multispectral fluorescence imaging in the visible and near-infrared-i/ii windows[J].Nature Biomedical Engineering，2020，4（3）：259-271.
[2] SEGARRA J，BUCHAILLOT M L，ARAUS J L，et al.Remote sensing for precision agriculture：sentinel-2 improved features and applications：5[J].Agronomy，2020，10（5）：641.
[3] SHI N，LI G，LEI E，et al.Hyperspectral imaging to Chinese paintings at the palace museum[J].Sciences of Conservation and Archaeology，2017，29（3）：23-29.
[4] SHIMONI M，HAELTERMAN R，PERNEEL C.Hypersectral imaging for military and security applications：combining myriad processing and sensing techniques[J].IEEE Geoscience and Remote Sensing Magazine，2019，7（2）：101-117.
[5] GREEN R O，EASTWOOD M L，SARTURE C M，et al.Imaging spectroscopy and the airborne visible/infrared imaging spectrometer（aviris）[J].Remote Sensing of Environment，1998，65（3）：227-248.
[6] JAMES J.Spectrograph design fundamentals[M].Cambridge：Cambridge University Press，2007.
[7] 张春光.基于超光谱成像系统的声光可调滤波技术研究[D].哈尔滨：哈尔滨工业大学，2008.
ZHANG C H.The technology of the acousto-optical tunable filter based on the hyperspectral imaging system[D].Harbin：Harbin Institute of Technology，2008.
[8] CAO X，DU H，TONG X，et al.A prism-mask system for multispectral video acquisition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2011，33（12）：2423-2435.
[9] WAGADARIKAR A，JOHN R，WILLETT R，et al.Single disperser design for coded aperture snapshot spectral imaging[J].Applied Optics，2008，47（10）：44-51.
[10] DONOHO D L.Compressed sensing[J].IEEE Transactions on Information Theory，2006，52（4）：1289-1306.
[11] ABED F M，AMIRSHAHI S H，ABED M R M.Reconstruction of reflectance data using an interpolation technique[J].Journal of the Optical Society of America-Optics Image Science and Vision，2009，26（3）：613-624.
[12] ZHU Y，LI B，XU X.Spectral reconstruction and accuracy appraisal based on pseudo inverse method[C]//2012 Symposium on Photonics and Optoelectronics，2012：1-3.
[13] ELRIFAI I，MAHGOUB H，MAGDY M，et al.Enhanced spectral reflectance reconstruction using pseudo-inverse estimation method[J].International Journal of Image Processing，2013，7（3）：278-285.
[14] ARAD B，BEN-SHAHAR O.Sparse recovery of hyperspectral signal from natural RGB images[C]//14th European Conference on Computer Vision.Cham：Springer，2016：19-34.
[15] WANG L，XIONG Z，SHI G，et al.Adaptive nonlocal sparse representation for dual-camera compressive hyperspectral imaging[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，39（10）：2104-2111.
[16] ZHANG S，WANG L，FU Y，et al.Computational hyperspectral imaging based on dimension-discriminative low-rank tensor recovery[C]//2019 IEEE/CVF International Conference on Computer Vision，2019：10182-10191.
[17] ARAD B，BEN-SHAHAR O，TIMOFTE R，et al.NTIRE 2018 challenge on spectral reconstruction from RGB images[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2018：1042-104209.
[18] ARAD B，TIMOFTE R，BEN-SHAHAR O，et al.NTIRE 2020 challenge on spectral reconstruction from an RGB image[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2020：1806-1822.
[19] ARAD B，TIMOFTE R，YAHEL R，et al.NTIRE 2022 spectral recovery challenge and data set[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2022：862-880.
[20] RONNEBERGER O，FISCHER P，BROX T.U-Net：convolutional networks for biomedical image segmentation[C]//18th International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham：Springer，2015：234-241.
[21] SHI W，CABALLERO J，HUSZáR F，et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition，2016：1874-1883.
[22] MEI Y，FAN Y，ZHOU Y，et al.Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：5689-5698.
[23] ZHANG J，SU R，FU Q，et al.A survey on computational spectral reconstruction methods from RGB to hyperspectral imaging[J].Scientific Reports，2022，12：11905.
[24] YASUMA F，MITSUNAGA T，ISO D，et al.Generalized assorted pixel camera：postcapture control of resolution，dynamic range，and spectrum[J].IEEE Transactions on Image Processing，2010，19（9）：2241-2253.
[25] CHAKRABARTI A，ZICKLER T.Statistics of real-world hyperspectral images[C]//2011 IEEE Conference on Computer Vision and Pattern Recognition，2011：193-200.
[26] NGUYEN R M H，PRASAD D K，BROWN M S.Training-based spectral reconstruction from a single RGB image[C]//13th European Conference on Computer Vision.Cham：Springer，2014：186-201.
[27] CHOI I，JEON D S，NAM G，et al.High-quality hyperspectral reconstruction using a spectral prior[J].ACM Transactions on Graphics，2017，36（6）：218.
[28] SHRESTHA R，PILLAY R，GEORGE S，et al.Quality evaluation of spectral imaging：quality factors and metrics[J].Journal of the International Colour Association，2014，12：22-35.
[29] YUHAS R H，GOETZ A F H，BOARDMAN J W.Discrimination among semi-arid landscape endmembers using the spectral angle mapper （SAM） algorithm[C]//Summaries of the Third Annual JPL Airborne Geoscience Workshop，1992：147-149.
[30] ROMERO J，GARCíA-BELTRáN A，HERNáNDEZ-ANDRéS J.Linear bases for representation of natural and artificial illuminants[J].Journal of The Optical Society of America A，1997，14（5）：1007-1014.
[31] GALLIANI S，LANARAS C，MARMANIS D，et al.Learned spectral super-resolution[J].arXiv：1703.09470，2017.
[32] KOUNDINYA S，SHARMA H，SHARMA M，et al.2D-3D CNN based architectures for spectral reconstruction from RGB images[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2018：957-964.
[33] ZHANG L，LANG Z，WANG P，et al.Pixel-aware deep function-mixture network for spectral super-resolution[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence，2020：12821-12828.
[34] STIEBEL T，KOPPERS S，SELTSAM P，et al.Reconstructing spectral images from RGB-images using a convolutional neural network[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2018：948-953.
[35] FUBARA B J，SEDKY M，DYKE D.RGB to spectral reconstruction via learned basis functions and weights[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2020：1984-1993.
[36] ZHU Z，LIU H，HOU J，et al.Deep amended gradient descent for efficient spectral reconstruction from single RGB images[J].IEEE Transactions on Computational Imaging，2021，7：1176-1188.
[37] YAN L，WANG X，ZHAO M，et al.Reconstruction of hyperspectral data from RGB images with prior category information[J].IEEE Transactions on Computational Imaging，2020，6：1070-1081.
[38] CAN Y B，TIMOFTE R.An efficient CNN for spectral reconstruction from RGB images[J].arXiv：1804.04647，2018.
[39] XIONG Z，SHI Z，LI H，et al.HSCNN：CNN-based hyperspectral image recovery from spectrally undersampled projections[C]//2017 IEEE International Conference on Computer Vision Workshops，2017：518-525.
[40] SHI Z，CHEN C，XIONG Z W，et al.HSCNN+：advanced CNN-based hyperspectral recovery from RGB images[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2018：1052-1058.
[41] NG P C，VERDIE Y，LU J，et al.Efficient hyperspectral reconstruction from RGB images with line-pixel deconvolution[C]//2022 IEEE 14th Image，Video，and Multidimensional Signal Processing Workshops，2022：1-5.
[42] LI J，DU S，WU C，et al.DRCR net：dense residual channel re-calibration network with non-local purification for spectral super resolution[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2022：1258-1267.
[43] ALVAREZ GILA A，V AN DE WEIJER J，GARROTE E，et al.Adversarial networks for spatial context-aware spectral image reconstruction from RGB[C]//2017 16th IEEE International Conference on Computer Vision，2017：480-490.
[44] MIAO X，Y UAN X，PU Y，et al.Lambda-Net：reconstruct hyperspectral images from a snapshot measurement[C]//2019 IEEE/CVF International Conference on Computer Vision，2019：4058-4068.
[45] LIU X，GHERBI A，WEI Z，et al.Multispectral image reconstruction from color images using enhanced variational autoencoder and generative adversarial network[J].IEEE Access，2021，9：1666-1679.
[46] LIU L，LEI S，SHI Z，et al.Hyperspectral remote sensing imagery generation from RGB images based on joint discrimination[J].IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing，2021，14：7624-7636.
[47] ISOLA P，ZHU J Y，ZHOU T，et al.Image-to-image translation with conditional adversarial networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition，2017：5967-5976.
[48] WANG X，GIRSHICK R，GUPTA A，et al.Non-local neural networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2018：7794-7803.
[49] LIU P，ZHAO H.Adversarial networks for scale feature-attention spectral image reconstruction from a single RGB[J].Sensors，2020，20（8）：E2426.
[50] BANERJEE A，PALRECHA A.MXR-U-nets for real time hyperspectral reconstruction[J].arXiv：2004.07003，2020.
[51] 宋蓓蓓，马穗娜，何帆，等.Res2-Unet深度学习网络的RGB-高光谱图像重建[J].光学精密工程，2022，30（13）：1606-1619.
SONG P P，MA H N，HE F，et al.Hyperspectral reconstruction from RGB images based on Res2-Unet deep learning network[J].Optical Precision Engineering，2022，30（13）：1606-1619.
[52] ZHAO Y，PO L-M，YAN Q，et al.Hierarchical regression network for spectral reconstruction from RGB images[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2020：1695-1704.
[53] PENG H，CHEN X，ZHAO J.Residual pixel attention network for spectral reconstruction from RGB images[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2020：2012-2020.
[54] KAYA B，CAN Y B，TIMOFTE R.Towards spectral estimation from a single RGB image in the wild[C]//2019 IEEE/CVF International Conference on Computer Vision Workshops，2019：3546-3555.
[55] LI J，WU C，SONG R，et al.Adaptive weighted attention network with camera spectral sensitivity prior for spectral reconstruction from RGB images[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2020：1894-1903.
[56] GAO X，WANG T，YANG J，et al.Deep-learning-based hyperspectral imaging through a RGB camera[J].Journal of Electronic Imaging，2021，30：053014.
[57] KOHEI Y，HAN X-H.Deep residual attention network for hyperspectral image reconstruction[C]//2020 25th International Conference on Pattern Recognition，2021：8547-8553.
[58] NATHAN D S，UMA K，VINOTHINI D S，et al.Light weight residual dense attention net for spectral reconstruction from RGB images[J].arXiv：2004.06930，2020.
[59] WANG W，WANG J.Double ghost convolution attention mechanism network：a framework for hyperspectral reconstruction of a single RGB image：2[J].Sensors，2021，21（2）：666.
[60] HAN K，WANG Y，TIAN Q，et al.GhostNet：more features from cheap operations[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：1577-1586.
[61] FU Y，ZHANG T，WANG L，et al.Coded hyperspectral image reconstruction using deep external and internal learning[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2022，44（7）：3404-3420.
[62] ZHANG T，FU Y，WANG L，et al.Hyperspectral image reconstruction using deep external and internal learning[C]//2019 IEEE/CVF International Conference on Computer Vision，2019：8559-8568.
[63] VASWANI A，SHAZEER N，PARMAR N，et a.Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems，2017：6000-6010.
[64] CHEN C F R，FAN Q，PANDA R.CrossViT：cross-attention multi-scale vision transformer for image classification[C]//2021 IEEE/CVF International Conference on Computer Vision，2021：347-356.
[65] DAI Z，CAI B，LIN Y，et al.UP-DETR：unsupervised pre-training for object detection with transformers[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：1601-1610.
[66] STRUDEL R，GARCIA R，LAPTEV I，et al.Segmenter：transformer for semantic segmentation[C]//2021 IEEE/CVF International Conference on Computer Vision，2021：7242-7252.
[67] LI K，WANG S，ZHANG X，et al.Pose recognition with cascade transformers[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：1944-1953.
[68] YANG S，QUAN Z，NIE M，et al.TransPose：keypoint localization via transformer[C]//2021 IEEE/CVF International Conference on Computer Vision，2021：11782-11792.
[69] CAI Y，LIN J，HU X，et al.Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：17481-17490.
[70] CAI Y，LIN J，HU X，et al.Coarse-to-fine sparse transformer for hyperspectral image reconstruction[C]//17th European Conference on Computer Vision.Cham：Springer，2022：686-704.
[71] CAI Y，LIN J，LIN Z，et al.MST++：multi-stage spectral-wise transformer for efficient spectral reconstruction[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2022：744-754.
[72] 金秋雨.基于卷积神经网络的多光谱图像重建技术研究[D].沈阳：沈阳工业大学，2021.
JIN Q Y.Research on multispectral image reconstruction technology based on convolutional neural network[D].Shenyang：Shenyang University of Technology，2021.
[73] MA L，RATHGEB A，MUBARAK H，et al.Unsupervised super-resolution reconstruction of hyperspectral histology images for whole-slide imaging[J].Journal of Biomedical Optics，2022，27（5）：059801.
[74] SUN Y，YANG Y，LIU Q，et al.Unsupervised spatial-spectral network learning for hyperspectral compressive snapshot reconstruction[J].IEEE Transactions on Geoscience and Remote Sensing，2022，60：5514314.
[75] LIU Z，ZHENG Y，HAN X H.Deep unsupervised fusion learning for hyperspectral image super resolution[J].Sensors，2021，21（7）：2348.
[76] GANIN Y，USTINOVA E，AJAKAN H，et al.Domain-adversarial training of neural networks[J].The Journal of Machine Learning Research，2016，17（l）：59.
[77] 施展.基于深度学习的高光谱图像空谱超分辨率重建[D].合肥：中国科学技术大学，2020.
SHI Z.Deep learning based spatial and spectral super-resolution reconstruction of hyperspectral images[D].Hefei：University of Science and Technology of China，2020.
[78] MEHTA A，SINHA H，MANDAL M，et al.Domain-aware unsupervised hyperspectral reconstruction for aerial image dehazing[C]//2021 IEEE Winter Conference on Applications of Computer Vision，2021：413-422.
[79] ZHU Z，LIU H，HOU J，et al.Semantic-embedded unsupervised spectral reconstruction from single RGB images in the wild[C]//2021 IEEE/CVF International Conference on Computer Vision，2021：2259-2268.
[80] LI D，LI S，ZHU M，et al.Unsupervised data fidelity en-hancement network for spectral CT reconstruction[C]//SPIE 11312，Medical Imaging 2020：Physics of Medical Imaging，2020：1081-1088.
[81] MARTíNEZ E，CASTRO S，BACCA J，et al.Efficient transfer learning for spectral image reconstruction from RGB images[C]//2020 IEEE Colombian Conference on Applications of Computational Intelligence，2020：1-6.
[82] GREFF K，BELLETTI F，BEYER L，et al.Kubric：a scalable dataset generator[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：3739-3751.
[83] MARTINEZ J，SHEWAKRAMANI J，WEI LIU T，et al.Permute，quantize，and fine-tune：efficient compression of neural networks[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：15694-15703.
[84] FAN L，DING Y，FAN D，et al.GrainSpace：a large-scale dataset for fine-grained and domain-adaptive recognition of cereal grains[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：21084-21093.
[85] LIU Y，ZHANG W，XIANG C，et al.Learning to affiliate：mutual centralized learning for few-shot classification[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：14391-14400.
[86] CHEN S，HONG Z，XIE G S，et al.MSDN：mutually semantic distillation network for zero-shot learning[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：7602-7611.
[87] FRANKLE J，CARBIN M.The lottery ticket hypothesis：finding sparse，trainable neural networks[J].arXiv：1803.
03635，2018.
[88] LOU Q，GUO F，KIM M，et al.AutoQ：automated kernel-wise neural network quantization[J].arXiv：1902.05690，2019.
[89] GOU J，YU B，MAYBANK S J，et al.Knowledge distillation：a survey[J].International Journal of Computer Vision，2021，129（6）：1789-1819.
[90] KOREN Y，BELL R，VOLINSKY C.Matrix factorization techniques for recommender systems[J].Computer，2009，42（8）：30-37.
[91] WANG Q，LI D，HUANG X，et al.Optimizing FFT-based convolution on ARMv8 multi-core CPUs[C]//26th International Conference on Parallel and Distributed Computing.Cham：Springer，2020：248-262.
[92] MAZZIA V，SALVETTI F，CHIABERGE M.Efficient-CapsNet：capsule network with self-attention routing：1[J].Scientific Reports，2021，11（1）：14634.
[93] WANG X，LIAN L，YU S X.Unsupervised visual attention and invariance for reinforcement learning[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：6673-6683.
[94] CAO H，TAN C，GAO Z，et al.A survey on generative diffusion model[J].arXiv：2209.02646，2022.