Temporal and Spatial Fusion of Remote Sensing Images：A Review

doi:10.3778/j.issn.1002-8331.2111-0131

Abstract

Abstract: Big data of remote sensing image with high temporal and spatial resolution plays an important role in remote sensing field. However, due to technique and budget constraints, a single satellite sensor cannot acquire remote sensing images with both high spatial resolution and high temporal resolution. Therefore, the temporal and spatial fusion technology of remote sensing image is regarded as one of the effective ways to solve the tradeoff between temporal resolution and spatial resolution. With the wide application of deep learning in various fields, deep learning technology has been proved to be a very effective method to solve image problems. According to the research results of scholars at home and abroad, the classical algorithm of remote sensing image spatiotemporal fusion is comprehensively summarized. Meanwhile, the research results of remote sensing image spatiotemporal fusion algorithm based on deep learning are analyzed, which are replicated on three datasets and the experimental results are analyzed, and the future of remote sensing image spatiotemporal fusion is prospected.

Key words: remote sensing images, high spatial resolution, traditional learning, deep learning, spatiotemporal fusion

摘要： 高时空分辨率的遥感图像大数据在遥感领域发挥着重要作用。然而，由于技术上和预算上的限制等原因，目前单一的卫星传感器无法获取同时具有高空间分辨率和高时间分辨率的遥感影像。因此遥感图像时空融合技术被认为是解决时间分辨率和空间分辨率折衷问题的有效途径之一。随着深度学习在各领域的广泛应用，深度学习技术已经被证实是解决图像问题非常有效的方法。针对国内外学者的研究成果，全面总结遥感图像时空融合的经典算法，同时重点分析基于深度学习的遥感图像时空融合算法的研究成果，在三个数据集上进行复现并分析实验结果，并对未来遥感图像时空融合进行展望。

关键词: 遥感图像, 高空间分辨率, 传统学习, 深度学习, 时空融合

YANG Guangqi, LIU Hui, ZHONG Xiwu, CHEN Long, QIAN Yurong. Temporal and Spatial Fusion of Remote Sensing Images：A Review[J]. Computer Engineering and Applications, 2022, 58(10): 27-40.

杨广奇, 刘慧, 钟锡武, 陈龙, 钱育蓉. 遥感图像时空融合综述[J]. 计算机工程与应用, 2022, 58(10): 27-40.

References

[1] 刘建波，马勇，武易天，等.遥感高时空融合方法的研究进展及应用现状[J].遥感学报，2016，20（5）：1038-1049.
LIU J B，MA Y，WU Y T，et al.Review of methods and applications of high spatiotemporal fusion of remote sensing data[J].Journal of Remote Sensing，2016，20（5）：1038-1049.
[2] 李树涛，李聪妤，康旭东.多源遥感图像融合发展现状与未来展望[J].遥感学报，2021，25（1）：148-166.
LI S T，LI C Y，KANG X D.Development status and future prospects of multi-source remote sensing image fusion[J].Journal of Remote Sensing，2021，25（1）：148-166.
[3] LI W，ZHANG X，PENG Y，et al.Spatiotemporal fusion of remote sensing images using a convolutional neural network with attention and multiscale mechanisms[J].International Journal of Remote Sensing，2021，42（6）：1973-1993.
[4] WALKER J J，DE BEURS K M，WYNNE R H，et al.Evaluation of Landsat and MODIS data fusion products for analysis of dryland forest phenology[J].Remote Sensing of Environment，2012，117：381-393.
[5] JIA D，CHENG C，SONG C，et al.A hybrid deep learning-based spatiotemporal fusion method for combining satellite images with different resolutions[J].Remote Sensing，2021，13（4）：645.
[6] 蒋文杰，罗晓曙，戴沁璇.基于对抗网络遥感图像超分辨率重建研究[J].计算机工程与应用，2020，56（21）：199-203.
JIANG W J，LUO X S，DAI Q X.Research on super-resolution reconstruction of remote sensing image based on improved conditional generative adversarial networks[J].Computer Engineering and Applications，2020，56（21）：199-203.
[7] 张永梅，滑瑞敏，马健喆，等.基于深度学习与超分辨率重建的遥感高时空融合方法[J].计算机工程与科学，2020，42（9）：1578-1586.
ZHANG Y M，HUA R M，MA J Z，et al.A high spatial temporal fusion method based on deep learning and super resolution reconstruction[J].Computer Engineering and Science，2020，42（9）：1578-1586.
[8] 董文全，蒙继华.遥感数据时空融合研究进展及展望[J].国土资源遥感，2018，30（2）：1-11.
DONG W Q，MENG J H.Review of spatiotemporal fusion model of remote sensing data[J].Remote Sensing for Land and Resources，2018，30（2）：1-11.
[9] LI J，LI Y，HE L，et al.Spatio-temporal fusion for remote sensing data：an overview and new benchmark[J].Science China：Information Sciences，2020，63（4）：140301.
[10] CHEN B，HUANG B，XU B.Comparison of spatiotemporal fusion models：a review[J].Remote Sensing，2015，7（2）：1798-1835.
[11] 万亚玲，钟锡武，刘慧，等.卷积神经网络在高光谱图像分类中的应用综述[J].计算机工程与应用，2021，57（4）：1-10.
WAN Y L，ZHONG X W，LIU H，et al.Survey of application of convolutional neural network in classification of hyperspectral images[J].Computer Engineering and Applications，2021，57（4）：1-10.
[12] LIU H，QIAN Y，ZHONG X，et al.Research on super-resolution reconstruction of remote sensing images：a comprehensive review[J].Optical Engineering，2021，60（10）：100901.
[13] MASUOKA E，FLEIG A，WOLFE R E，et al.Key characteristics of MODIS data products[J].IEEE Transactions on Geoscience and Remote Sensing，1998，36（4）：1313-1323.
[14] ZHU X，CAI F，TIAN J，et al.Spatiotemporal fusion of multisource remote sensing data：literature survey，taxonomy，principles，applications，and future directions[J].Remote Sensing，2018，10（4）：527.
[15] ZHUKOV B，OERTEL D，LANZL F，et al.Unmixing-based multisensor multiresolution image fusion[J].IEEE Transactions on Geoscience and Remote Sensing，1999，37（3）：1212-1226.
[16] ZHANG W，LI A，JIN H，et al.An enhanced spatial and temporal data fusion model for fusing Landsat and MODIS surface reflectance to generate high temporal Landsat-like data[J].Remote Sensing，2013，5（10）：5346-5368.
[17] GAO F，MASEK J，SCHWALLER M，et al.On the blending of the Landsat and MODIS surface reflectance：predicting daily Landsat surface reflectance[J].IEEE Transactions on Geoscience and Remote sensing，2006，44（8）：2207-2218.
[18] LI A，BO Y，ZHU Y，et al.Blending multi-resolution satellite sea surface temperature（SST） products using Bayesian maximum entropy method[J].Remote Sensing of Environment，2013，135：52-63.
[19] HUANG B，ZHANG H，SONG H，et al.Unified fusion of remote-sensing imagery：generating simultaneously high-resolution synthetic spatial-temporal-spectral earth observations[J].Remote Sensing Letters，2013，4（6）：561-569.
[20] HUANG B，SONG H.Spatiotemporal reflectance fusion via sparse representation[J].IEEE Transactions on Geoscience and Remote Sensing，2012，50（10）：3707-3716.
[21] SONG H，HUANG B.Spatiotemporal satellite image fusion through one-pair image learning[J].IEEE Transactions on Geoscience and Remote Sensing，2012，51（4）：1883-1896.
[22] WANG S，CAO J，YU P.Deep learning for spatio-temporal data mining：a survey[J].arXiv：1906.04928，2019.
[23] ZHU X，HELMER E H，GAO F，et al.A flexible spatiotemporal method for fusing satellite images with different resolutions[J].Remote Sensing of Environment，2016，172：165-177.
[24] GEVAERT C M，GARCíA-HARO F J.A comparison of STARFM and an unmixing-based algorithm for Landsat and MODIS data fusion[J].Remote Sensing of Environment，2015，156：34-44.
[25] MASELLI F，REMBOLD F.Integration of LAC and GAC NDVI data to improve vegetation monitoring in semi-arid environments[J].International Journal of Remote Sensing，2002，23（12）：2475-2488.
[26] AMORóS-LóPEZ J，GóMEZ-CHOVA L，ALONSO L，et al.Multitemporal fusion of Landsat/TM and ENVISAT/MERIS for crop monitoring[J].International Journal of Applied Earth Observation and Geoinformation，2013，23：132-141.
[27] WU M，NIU Z，WANG C，et al.Use of MODIS and Landsat time series data to generate high-resolution temporal synthetic Landsat data using a spatial and temporal reflectance fusion model[J].Journal of Applied Remote Sensing，2012，6（1）：063507.
[28] WU M，HUANG W，NIU Z，et al.Generating daily synthetic Landsat imagery by combining Landsat and MODIS data[J].Sensors，2015，15（9）：24002-24025.
[29] LU M，CHEN J，TANG H，et al.Land cover change detection by integrating object-based data blending model of Landsat and MODIS[J].Remote Sensing of Environment，2016，184：374-386.
[30] ZURITA-MILLA R，CLEVERS J G P W，SCHAEPMAN M E.Unmixing-based Landsat TM and MERIS FR data fusion[J].IEEE Geoscience and Remote Sensing Letters，2008，5（3）：453-457.
[31] ZHU X，CHEN J，GAO F，et al.An enhanced spatial and temporal adaptive reflectance fusion model for complex heterogeneous regions[J].Remote Sensing of Environment，2010，114（11）：2610-2623.
[32] HILKER T，WULDER M A，COOPS N C，et al.A new data fusion model for high spatial-and temporal-resolution mapping of forest disturbance based on Landsat and MODIS[J].Remote Sensing of Environment，2009，113（8）：1613-1627.
[33] WENG Q，FU P，GAO F.Generating daily land surface temperature at Landsat resolution by fusing Landsat and MODIS data[J].Remote Sensing of Environment，2014，145：55-67.
[34] HWANG T，SONG C，BOLSTAD P V，et al.Downscaling real-time vegetation dynamics by fusing multi-temporal MODIS and Landsat NDVI in topographically complex terrain[J].Remote Sensing of Environment，2011，115（10）：2499-2512.
[35] SHEN H，WU P，LIU Y，et al.A spatial and temporal reflectance fusion model considering sensor observation differences[J].International Journal of Remote Sensing，2013，34（12）：4367-4383.
[36] FU D，CHEN B，WANG J，et al.An improved image fusion approach based on enhanced spatial and temporal the adaptive reflectance fusion model[J].Remote Sensing，2013，5（12）：6346-6360.
[37] ZHANG H，SUN Y，SHI W，et al.An object-based spatiotemporal fusion model for remote sensing images[J].European Journal of Remote Sensing，2021，54（1）：86-101.
[38] LIAO L，SONG J，WANG J，et al.Bayesian method for building frequent Landsat-like NDVI datasets by integrating MODIS and Landsat NDVI[J].Remote Sensing，2016，8（6）：452.
[39] XUE J，LEUNG Y，FUNG T.A Bayesian data fusion approach to spatio-temporal fusion of remotely sensed images[J].Remote Sensing，2017，9（12）：1310.
[40] SHEN H，MENG X，ZHANG L.An integrated framework for the spatio-temporal-spectral fusion of remote sensing images[J].IEEE Transactions on Geoscience and Remote Sensing，2016，54（12）：7135-7148.
[41] WU B，HUANG B，ZHANG L.An error-bound-regularized sparse coding for spatiotemporal reflectance fusion[J].IEEE Transactions on Geoscience and Remote Sensing，2015，53（12）：6791-6803.
[42] LIU X，DENG C，WANG S，et al.Fast and accurate spatiotemporal fusion based upon extreme learning machine[J].IEEE Geoscience and Remote Sensing Letters，2016，13（12）：2039-2043.
[43] MOOSAVI V，TALEBI A，MOKHTARI M H，et al.A wavelet-artificial intelligence fusion approach（WAIFA） for blending Landsat and MODIS surface temperature[J].Remote Sensing of Environment，2015，169：243-254.
[44] WEI J，WANG L，LIU P，et al.Spatiotemporal fusion of remote sensing images with structural sparsity and semi-coupled dictionary learning[J].Remote Sensing，2017，9（1）：21.
[45] LI X，LING F，FOODY G M，et al.Generating a series of fine spatial and temporal resolution land cover maps by fusing coarse spatial resolution remotely sensed images and fine spatial resolution land cover maps[J].Remote Sensing of Environment，2017，196：293-311.
[46] QUAN J，ZHAN W，MA T，et al.An integrated model for generating hourly Landsat-like land surface temperatures over heterogeneous landscapes[J].Remote Sensing of Environment，2018，206：403-423.
[47] SONG H，LIU Q，WANG G，et al.Spatiotemporal satellite image fusion using deep convolutional neural networks[J].IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing，2018，11（3）：821-829.
[48] THOMAS C，RANCHIN T，WALD L，et al.Synthesis of multispectral images to high spatial resolution：a critical review of fusion methods based on remote sensing physics[J].IEEE Transactions on Geoscience and Remote Sensing，2008，46（5）：1301-1312.
[49] TAN Z，YUE P，DI L，et al.Deriving high spatiotemporal remote sensing images using deep convolutional network[J].Remote Sensing，2018，10（7）：1066.
[50] TAN Z，DI L，ZHANG M，et al.An enhanced deep convolutional model for spatiotemporal image fusion[J].Remote Sensing，2019，11（24）：2898.
[51] LIU X，DENG C，CHANUSSOT J，et al.StfNet：a two-stream convolutional neural network for spatiotemporal image fusion[J].IEEE Transactions on Geoscience and Remote Sensing，2019，57（9）：6552-6564.
[52] WANG X F，WANG X Y.Spatiotemporal fusion of remote sensing image based on deep learning[J].Journal of Sensors，2020.DOI：10.1155/2020/8873079.
[53] ZHANG H，SONG Y，HAN C，et al.Remote sensing image spatiotemporal fusion using a generative adversarial network[J].IEEE Transactions on Geoscience and Remote Sensing，2020，59（5）：4273-4286.
[54] LEDIG C，THEIS L，HUSZáR F，et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition，2017：4681-4690.
[55] TAN Z，GAO M，LI X，et al.A flexible reference-insensitive spatiotemporal fusion model for remote sensing images using conditional generative adversarial network[J].IEEE Transactions on Geoscience and Remote Sensing，2021.DOI：10.1109/TGRS.2021.3050551.
[56] LI W，ZHANG X，PENG Y，et al.DMNet：a network architecture using dilated convolution and multiscale mechanisms for spatiotemporal fusion of remote sensing images[J].IEEE Sensors Journal，2020，20（20）：12190-12202.
[57] PENG M，ZHANG L，SUN X，et al.A fast three-dimensional convolutional neural network-based spatiotemporal fusion method（STF3DCNN） using a spatial-temporal-spectral dataset[J].Remote Sensing，2020，12（23）：3888.
[58] LI W，CAO D，PENG Y，et al.MSNet：a multi-stream fusion network for remote sensing spatiotemporal fusion based on transformer and convolution[J].Remote Sensing，2021，13（18）：3724.
[59] HUYNH-THU Q，GHANBARI M.Scope of validity of PSNR in image/video quality assessment[J].Electronics Letters，2008，44（13）：800-801.
[60] CHEN Z，PU H，WANG B，et al.Fusion of hyperspectral and multispectral images：a novel framework based on generalization of pan-sharpening methods[J].IEEE Geoscience and Remote Sensing Letters，2014，11（8）：1418-1422.
[61] WALD L.Quality of high resolution synthesised images：is there a simple criterion?[C]//3rd Conference on Fusion of Earth Data：Merging Point Measurements，Raster Maps and Remotely Sensed Images，2000：99-103.
[62] YUHAS R H，GOETZ A F H，BOARDMAN J W.Discrimination among semi-arid landscape endmembers using the spectral angle mapper（SAM） algorithm[C]//Summaries of the 3rd Annual JPL Airborne Geoscience Workshop，1992，1：147-149.
[63] ALPARONE L，WALD L，CHANUSSOT J，et al.Comparison of pansharpening algorithms：outcome of the 2006 GRS-S data-fusion contest[J].IEEE Transactions on Geoscience and Remote Sensing，2007，45（10）：3012-3021.
[64] ZHANG Z，BLUM R S.A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application[J].Proceedings of the IEEE，1999，87（8）：1315-1326.