Review on Semantic Segmentation of UAV Aerial Images

doi:10.3778/j.issn.1002-8331.2105-0423

Abstract

Abstract:

With the rapid development of Unmanned Aerial Vehicle（UAV） technology, research institutions and industries have attached importance of UAV’s application. Optical images and videos are vital for the UAV to sense the environment, occupying an important position in UAV vision. As a hot spot of the current research of computer vision, semantic segmentation is widely investigated in the fields of unmanned driving and intelligent robot. Semantic segmentation of UAV aerial images is based on the UAV aerial image semantic segmentation technology to enable the UAV to work in complex scenes. First of all, a brief introduction to the semantic segmentation technology and the application development of UAV is given. Meanwhile, the relevant UAV aerial data sets, characteristics of aerial images and commonly used evaluation metrics for semantic segmentation are introduced. Secondly, according to the characteristics of UAV aerial images, it introduces the relevant semantic segmentation methods. In this section, analysis and comparison are made in three aspects including the small object detection, the real-time performance of the models and the multi-scale information integration. Finally, the related applications of semantic segmentation for UAV are reviewed, including line detection, the application of agriculture and building extraction, and analysis of the development trend and challenges in the future is made.

Key words: Unmanned Aerial Vehicle（UAV） imagery, semantic segmentation, computer vision, deep learning, convolution neural network

摘要：

随着无人机技术的快速发展，无人机在研究领域和工业应用方面受到了广泛的关注。图像和视频是无人机感知周围环境的重要途径。图像语义分割是计算机视觉领域的研究热点，在无人驾驶、智能机器人等场景中应用广泛。无人机航拍图像语义分割是在无人机航拍图像的基础上，运用语义分割技术使无人机获得场景目标智能感知能力。介绍了语义分割技术和无人机的应用发展、相关无人机航拍数据集、无人机航拍图像特点和常用语义分割评价指标。针对无人机航拍的特点介绍了相关语义分割方法，包括小目标、模型实时性和多尺度整合等方面。综述无人机语义分割相关应用，包括线检测、农业和建筑物提取等方向，并分析无人机语义分割未来发展趋势和挑战。

关键词: 无人机影像, 语义分割, 计算机视觉, 深度学习, 卷积神经网络

CHENG Qing, FAN Man, LI Yandong, ZHAO Yuan, LI Chenglong. Review on Semantic Segmentation of UAV Aerial Images[J]. Computer Engineering and Applications, 2021, 57(19): 57-69.

程擎，范满，李彦冬，赵远，李诚龙. 无人机航拍图像语义分割研究综述[J]. 计算机工程与应用, 2021, 57(19): 57-69.

References

[1] 郑凯，李建胜.基于深度神经网络的图像语义分割综述[J].测绘与空间地理信息，2020，43（10）：119-125.
ZHENG Kai，LI Jiansheng.An overview of image semantic segmentation based on deep learning[J].Geomatics & Spatial Information Technology，2020，43（10）：119-125.
[2] 陈彦彤，李雨阳，陈伟楠，等.基于深度语义分割的遥感图像海面舰船检测研究[J].仪器仪表学报，2020，41（1）：236-243.
CHEN Yantong，LI Yuyang，CHEN Weinan，et al.Sea surface ship detection based on deep semantic segmentation using remote sensing image[J].Chinese Journal of Scientific Instrument，2020，41（1）：236-243.
[3] 徐辉，祝玉华，甄彤，等.深度神经网络图像语义分割方法综述[J].计算机科学与探索，2021，15（1）：47-59.
XU Hui，ZHU Yuhua，ZHEN Tong，et al.Survey of image semantic segmentation methods based on deep neural network[J].Journal of Frontiers of Computer Science and Technology，2021，15（1）：47-59.
[4] 江波，屈若锟，李彦冬，等.基于深度学习的无人机航拍目标检测研究综述[J].航空学报，2021，42（4）：131-145.
JIANG Bo，QU Ruokun，LI Yandong，et al.Survey of object detection in UAV imagery based on deep learning[J].Acta Aeronautica et Astronautica Sinica，2021，42（4）：131-145.
[5] 王秋生，贺云涛，张晓辉，等.基于无人机航拍图像的车道线检测方法综述[J].无人系统技术，2019，2（5）：9-16.
WANG Qiusheng，HE Yuntao，ZHANG Xiaohui，et al.Review of lane detection methods based on UAV aerial images[J].Unmanned Systems Technology，2019，2（5）：9-16.
[6] 邝辉宇，吴俊君.基于深度学习的图像语义分割技术研究综述[J].计算机工程与应用，2019，55（19）：12-21.
KUANG Huiyu，WU Junjun.Survey of image semantic segmentation based on deep learning[J].Computer Engineering and Applications，2019，55（19）：12-21.
[7] MARDIA K V，HAINSWORTH T J.A spatial thresholding method for image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，1988，10（6）：919-927.
[8] LAKSHMI S，SANKARANARAYANAN D V.A study of edge detection techniques for segmentation computing approaches[J].International Journal of Computer Applications，2010（1）：35-41.
[9] GIANNAKEAS N，KARVELIS P S，EXARCHOS T P，et al.Segmentation of microarray images using pixel classification—Comparison with clustering-based methods[J].Computers in Biology and Medicine，2013，43（6）：705-716.
[10] ADAMS R，BISCHOF L.Seeded region growing[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，1994，16：641-647.
[11] 胡太，杨明.结合目标检测的小目标语义分割算法[J].南京大学学报（自然科学），2019，55（1）：73-84.
HU Tai，YANG Ming.A small object semantic segmentation algorithm combined with object detection[J].Journal of Nanjing University（Natural Science），2019，55（1）：73-84.
[12] GHIASI G，FOWLKES C C.Laplacian reconstruction and refinement for semantic segmentation[C]//Proceedings of European Conference on Computer Vision，2016：519-534.
[13] RAZAVIAN A S，AZIZPOUR H，SULLIVAN J，et al.CNN features off-the-shelf：An astounding baseline for recognition[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops，2014.
[14] 王嫣然，陈清亮，吴俊君.面向复杂环境的图像语义分割方法综述[J].计算机科学，2019，46（9）：36-46.
WANG Yanran，CHEN Qingliang，WU Junjun.Research on image semantic segmentation for complex environments[J].Computer Science，2019，46（9）：36-46.
[15] LECUN Y，BOTTOU L，BENGIO Y，et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE，1998，86（11）：2278-2324.
[16] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.Image-Net classification with deep convolutional neural networks[J].Communacation ACM，2017，60（6）：84-90.
[17] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[C]//Proceedings of International Conference on Learning Representations，2015.
[18] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition（CVPR），2016.
[19] SHELHAMER E，LONG J，DARRELL T.Fully convolutional networks for semantic segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，39（4）：640-651.
[20] BADRINARAYANAN V，KENDALL A，CIPOLLA R.SegNet：A deep convolutional encoder-decoder architecture for image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，39（12）：2481-2495.
[21] CHEN L C，PAPANDREOU G，KOKKINOS I，et al.Semantic image segmentation with deep convolutional nets and fully connected CRFs[J].arXiv：1412.7062，2014.
[22] CHEN L，PAPANDREOU G，KOKKINOS I，et al.DeepLab：Semantic image segmentation with deep convolutional nets，atrous convolution，and fully connected CRFs[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2018，40（4）：834-848.
[23] CHEN L C，PAPANDREOU G，SCHROFF F，et al.Rethinking Atrous Convolution for Semantic Image Segmentation[J].arXiv：1706.05587，2017.
[24] CHEN L C，ZHU Y，PAPANDREOU G，et al.Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//2018 European Conference on Computer Vision（ECCV），2018：833-851.
[25] LIN G，MILAN A，SHEN C，et al.RefineNet：Multi-path refinement networks for high-resolution semantic segmentation[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition（CVPR），2017.
[26] ZHAO H，SHI J，QI X，et al.Pyramid scene parsing network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition（CVPR），2017.
[27] 阿柴本柴.语义分割综述：截止2020，深度学习背景下的语义分割的发展状况[EB/OL].（2020-04-18）[2021-04-15].https：//mp.weixin.qq.com/s/bJ3SPEywXNAS-jKK9fJRrg. ACHAIBENCHAI.Overview of semantic segmentation：By 2020，the development of semantic segmentation in the context of deep learning[EB/OL].（2020-04-18）[2021-04-15].https：//mp.weixin.qq.com/s/bJ3SPEywXNAS-jKK9fJRrg.
[28] LIN T Y，MAIRE M，BELONGIE S，et al.Microsoft COCO：Common objects in context[C]//2014 European Conference on Computer Vision（ECCV），2014：740-755.
[29] CHEN Y，WANG Y，LU P，et al.Large-scale structure from motion with semantic constraints of aerial images[C]//2018 Chinese Conference on Pattern Recognition and Computer Vision（PRCV），2018：347-359.
[30] NIGAM I，HUANG C，RAMANAN D.Ensemble knowledge transfer for semantic segmentation[C]//2018 IEEE Winter Conference on Applications of Computer Vision（WACV），2018：1499-1508.
[31] LYU Y，VOSSELMAN G，XIA G S，et al.UAVid：A semantic segmentation dataset for UAV imagery[J].ISPRS Journal of Photogrammetry and Remote Sensing，2020，165：108-119.
[32] MAGGIORI E，TARABALKA Y，CHARPIAT G，et al.Can semantic labeling methods generalize to any city? the inria aerial image labeling benchmark[C]//2017 IGARSS IEEE International Geoscience & Remote Sensing Symposium，2017.
[33] LI Y，XIAO Z，ZHEN X，et al.Attentional information fusion networks for cross-scene power line detection[J].IEEE Geoscience and Remote Sensing Letters，2019，16（10）：1635-1639.
[34] RONNEBERGER O，FISCHER P，BROX T.U-Net：Convolutional networks for biomedical image segmentation[J].arXiv：1505.04597，2015.
[35] YANG T，WU Y，ZHAO J，et al.Semantic segmentation via highly fused convolutional network with multiple soft cost functions[J].Cognitive Systems Research，2019，53：20-30.
[36] HE K，GKIOXARI G，DOLLáR P，et al.Mask R-CNN[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2020，42（2）：386-397.
[37] WANG L，LEE C Y，TU Z，et al.Training deeper convolutional networks with deep supervision[J].arXiv：1505. 02496，2015.
[38] DUMOULIN V，VISIN F.A guide to convolution arithmetic for deep learning[J].arXiv：1603.07285，2016.
[39] YU F，KOLTUN V.Multi-scale context aggregation by dilated convolutions[J].arXiv：1511.07122，2015.
[40] 刘文，王海荣，周北京.DeepLabv3plus-IRCNet：小目标特征提取的图像语义分割[J].中国图象图形学报，2021，26（2）：391-401.
LIU Wen，WANG Hairong，ZHOU Beijing.DeepLabv3plus-IRCNet：An image semantic segmentation method for small target feature extraction[J].Journal of Image and Graphics，2021，26（2）：391-401.
[41] 王明常，朱春宇，陈学业，等.基于FPN Res-Unet的高分辨率遥感影像建筑物变化检测[J].吉林大学学报（地球科学版），2021，51（1）：296-306.
WANG Mingchang，ZHU Chunyu，CHEN Xueye，et al.Building change detectionin high resolution remote sensing images based on FPN Res-Unet[J].Journal of Jilin University（Earth Science Edition），2021，51（1）：296-306.
[42] 曹春红，段鸿轩，曹玲，等.基于多级特征级联的遥感图像实时语义分割[J].山东大学学报（工学版），2021（2）：19-25.
CAO Chunhong，DUAN Hongxuan，CAO Ling，et al.Real-time semantic segmentation of high-resolution remote sensing image based on multi-level feature cascade[J].Journal of Shandong University（Engineering Science），2021（2）：19-25.
[43] PASZKE A，CHAURASIA A，KIM S，et al.ENet：A deep neural network architecture for real-time semantic segmentation[J].arXiv：1606.02147，2016.
[44] WATANABE S，HORI T，KARITA S，et al.ESPNet：End-to-end speech processing toolkit[C]//Proceedings of the Annual Conference of the International Speech Communication Association，2018：2207-2211.
[45] YU C，WANG J，PENG C，et al.BiSeNet：Bilateral segmentation network for real-time semantic segmentation[C]//Proceedings of European Conference on Computer Vision，2018.
[46] 尹鹏峰，薛彦兵，蔡靖，等.用于实时语义分割的带孔金字塔注意力网络[J].中国科技论文，2021（1）：1-7.
YIN Pengfeng，XUE Yanbing，CAI Jing，et al.Dilated pyramid attention network for real-time semantic segmentation[J].China Sciencepaper，2021（1）：1-7.
[47] 鲍海龙，万敏，刘忠祥，等.基于区域自我注意力的实时分割网络[J].激光与光电子学进展，2021（8）：196-202.
BAO Hailong，WAN Min，LIU Zhongxiang，et al.Design of a real-time semantic segmentation network based on regional self-attention[J].Laser & Optoelectronics Progress，2021（8）：196-202.
[48] ZHENG S，JAYASUMANA S，ROMERA-PAREDES B，et al.Conditional random fields as recurrent neural networks[C]//2015 IEEE International Conference on Computer Vision（ICCV），2015.
[49] RUSSAKOVSKY O，DENG J，SU H，et al.ImageNet large scale visual recognition challenge[J].International Journal of Computer Vision（IJCV），2015，115（3）：211-252.
[50] EVERINGHAM M，GOOL L V，WILLIAMS C，et al.The pascal Visual Object Classes（VOC） challenge[J].International Journal of Computer Vision，2010，88（2）：303-338.
[51] GEIGER A，LENZ P，URTASUN R.Are we ready for autonomous driving？The KITTI vision benchmark suite[C]//Proceedings of IEEE Conference on Computer Vision & Pattern Recognition，2015.
[52] CORDTS M，OMRAN M，RAMOS S，et al.The cityscapes dataset for semantic urban scene understanding[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition（CVPR），2016.
[53] MADAAN R，MATURANA D，SCHERER S.Wire detection using synthetic data and dilated convolutional networks for unmanned aerial vehicles[C]//2017 IEEE/RSJ International Conference on Intelligent Robots and Systems（IROS），2017.
[54] DUBEY G，MADAAN R，SCHERER S.DROAN—disparity-space representation for obstacle avoidance：Enabling wire mapping & avoidance[C]//2018 IEEE/RSJ International Conference on Intelligent Robots and Systems（IROS），2018：6311-6318.
[55] MAURER M，HOFER M，FRAUNDORFER F，et al.Automated inspection of power line corridors to measure vegetation undercut using UAV-based images[C]//ISPRS Annals of the Photogrammetry，Remote Sensing and Spatial Information Sciences，2017：33-40.
[56] CHOI H，KOO G，KIM B J，et al.Weakly supervised power line detection algorithm using a recursive noisy label update with refined broken line segments[J].Expert Systems with Applications，2021，165：113895.
[57] LIAO J，CAO L，LI W，et al.UnetDVH-linear：Linear feature segmentation by dilated convolution with vertical and horizontal kernels[J].Sensors，2020，20：5759.
[58] CANDAMO J，KASTURI R，GOLDGOF D，et al.Detection of thin lines in low quality video[J].IEEE Transactions on Aerospace and Electronic Systems，2009，45（3）：1-25.
[59] AVIZONIS P，BARRON B.Low cost wire detection system[C]//Proceedings of the 18th Digital Avionics Systems Conference，1999.
[60] BOJARSKI M，CHOROMANSKA A，CHOROMANSKI K，et al.VisualBackProp：Visualizing CNNs for autonomous driving[J].arXiv：1611.05418，2016.
[61] YANG T，WU Y，ZHAO J，et al.Semantic segmentation via highly fused convolutional network with multiple soft cost functions[J].Cognitive Systems Research，2018，53：20-30.
[62] WU Y，YANG T，ZHAO J，et al.VH-HFCN based parking slot and lane markings segmentation on panoramic surround view[C]//2018 IEEE Intelligent Vehicles Symposium（IV），2018.
[63] SA I，POPOVIC M，KHANNA R，et al.WeedMap：A large-scale semantic weed mapping framework using aerial multispectral imaging and deep neural network for precision farming[J].Remote Sensing，2018，10（9）：1423.
[64] PERSELLO C，TOLPEKIN V A，BERGADO J R，et al.Delineation of agricultural fields in smallholder farms from satellite images using fully convolutional networks and combinatorial grouping[J].Remote Sensing of Environment，2019，231：111253.
[65] SADEGHI-TEHRAN P，VIRLET N，AMPE E M，et al.DeepCount：In-field automatic quantification of wheat spikes using simple linear iterative clustering and deep convolutional neural networks[J].Frontiers in Plant Science，2019，10：1176.
[66] KERKECH M，HAFIANE A，CANALS R.VddNet：Vine disease detection network based on multispectral images and depth map[J].Remote Sensing，2020，12（20）：3305.
[67] ZHAO X，YUAN Y，SONG M，et al.Use of unmanned aerial vehicle imagery and deep learning UNet to extract rice lodging[J].Sensors，2019，19：3758.
[68] KAMILARIS A，PRENAFETA-BOLDU F X.Deep learning in agriculture：A survey[J].Computers and Electronics in Agriculture，2018，147：70-90.
[69] WILKE N，SIEGMANN B，KLINGBEIL L，et al.Quantifying lodging percentage and lodging severity using a UAV-based canopy height model combined with an objective threshold approach[J].Remote Sensing，2019，11（5）：515.
[70] ZHANG Y，GONG W，SUN J，et al.Web-Net：A novel nest networks with ultra-hierarchical sampling for building extraction from aerial imageries[J].Remote Sensing，2019，11（16）：1897.
[71] IQBAL J，ALI M.Weakly-supervised domain adaptation for built-up region segmentation in aerial and satellite imagery[J].ISPRS Journal of Photogrammetry and Remote Sensing，2020，167：263-275.
[72] WANG Y，WU G，GUO Y，et al.Learn to extract building outline from misaligned annotation through nearest feature selector[J].Remote Sensing，2020，12（17）：2722.
[73] MAJD R D，MOMENI M，MOALLEM P.Transferable object-based framework based on deep convolutional neural networks for building extraction[J].IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing，2019，12（8）：2627-2635.
[74] COTE M，SAEEDI P.Automatic rooftop extraction in nadir aerial imagery of suburban regions using corners and variational level set evolution[J].IEEE Transactions on Geoscience & Remote Sensing，2013，51（1）：313-328.
[75] YI Y，NEWSAM S.Comparing SIFT descriptors and gabor texture features for classification of remote sensed imagery[C]//2008 15th IEEE International Conference on Image Processing，2018.
[76] LOWE D G.Object recognition from local scale-invariant features[C]//Proceedings of IEEE International Conference on Computer Vision，1999.
[77] MOUNTRAKIS G，IM J，OGOLE C.Support vector machines in remote sensing：A review[J].ISPRS Journal of Photogrammetry and Remote Sensing，2011，66（3）：247-259.
[78] JI S P，WEI S Q，LU M，et al.Fully convolutional networks for multisource building extraction from an open aerial and satellite imagery dataset[J].IEEE Transactions on Geoscience and Remote Sensing，2019，57（1）：574-586.
[79] ROTTENSTEINER F，SOHN G，JUNG J，et al.The ISPRS benchmark on urban object classification and 3D building reconstruction[J].ISPRS Annals of Photogrammetry，Remote Sensing and Spatial Information Sciences，2012，1/3：293-298.
[80] MNIH V.Machine learning for aerial image labeling[D].CAN：University of Toronto，2013.
[81] CHEN Y H，CHEN W Y，CHEN Y T，et al.No more discrimination：Cross city adaptation of road scene segmenters[C]//2017 IEEE International Conference on Computer Vision，2017.
[82] TSAI Y H，HUNG W C，SCHULTER S，et al.Learning to adapt structured output space for semantic segmentation[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2018.
[83] WANG S，BAI M，MATTYUS G，et al.Toronto city：Seeing the world with a million eyes[C]//2017 IEEE International Conference on Computer Vision，2017.