Overview of Research on Spatial Registration Algorithms of Underwater Opti-Acoustic Images

doi:10.3778/j.issn.1002-8331.2208-0313

Abstract

Abstract: Underwater opti-acoustic image alignment is a key technology for information fusion in underwater devices. Based on a brief description of the concept and examples of underwater opti-acoustic image alignment, the current relevant algorithms for underwater opti-acoustic image reconstruction and recovery are analysed, the research progress of region and feature-based alignment algorithms for underwater heterogenous images are reviewed in detail, the development status of two research directions with high accuracy based on the similarity of image domain and shape features is focused on, and according to the research hotspots of heterogenous images alignment in other fields. The development trend of underwater opti-acoustic image alignment research is foreseen in terms of increasing structural constraints on imaging models, introducing phase congruency, generative adversarial networks and other algorithms to improve the alignment accuracy.

Key words: underwater opti-acoustic images, image registration, image processing, feature detection, machine vision

摘要： 水下光声图像配准是水下设备实现信息融合的关键技术。在简述了水下光声图像配准的概念及实例的基础上，分析了目前水下光声图像重建与复原的相关算法，详细综述了水下异源图像基于区域和特征的配准算法研究进展，重点论述了基于图像域和形状特征相似度的两个准确度较高的研究方向的发展现状，并根据其他领域的异源图像配准的研究热点，从增加成像模型的结构性约束、引入相位一致性和生成对抗网络等算法提高配准精度，展望了水下声光图像配准研究的发展趋势。

关键词: 水下声光图像, 图像配准, 图像处理, 特征提取, 机器视觉

GUO Yinjing, MA Xinrui, XU Yuecheng, KONG Fang, LYU Wenhong. Overview of Research on Spatial Registration Algorithms of Underwater Opti-Acoustic Images[J]. Computer Engineering and Applications, 2023, 59(5): 14-27.

郭银景, 马新瑞, 许越铖, 孔芳, 吕文红. 水下光声图像空间配准算法研究综述[J]. 计算机工程与应用, 2023, 59(5): 14-27.

References

[1] HOZYŃ S.A review of underwater mine detection and classification in sonar imagery[J].Electronics，2021，10（23）：2943.
[2] WANG Y S，JI Y，WOO H，et al.Acoustic camera-based pose graph SLAM for dense 3-D mapping in underwater environments[J].IEEE Journal of Oceanic Engineering，2021，46（3）：829-847.
[3] YAMAFUNE K，TORRE R，CASTRO F.Multi-image photogrammetry to record and reconstruct underwater shipwreck sites[J].Journal of Archaeological Method and Theory，2017，24：703-725.
[4] KRISHNARAJ N，ELHOSENY M，THENMOZHI M，et al.Deep learning model for real-time image compression in Internet of underwater things（IoUT）[J].Journal of Real-Time Image Processing，2020，17（6）：2097-2111.
[5] JAFFE J S.Underwater optical imaging：the past，the present，and the prospects[J].IEEE Journal of Oceanic Engineering，2015，40（3）：683-700.
[6] LIU Y，WANG H，CAI L，et al.Fundamentals and advancements of topology discovery in underwater acoustic sensor networks：a review[J].IEEE Sensors Journal，2021，21（19）：21159-21174.
[7] TRUCCO A，PALMESE M，REPETTO S.Devising an affordable sonar system for underwater 3-D vision[J].IEEE Transactions on Instrumentation and Measurement，2008，57（10）：2348-2354.
[8] DIAMANT R，LAMPE L.Low probability of detection for underwater acoustic communication：a review[J].IEEE Access，2018，6：19099-19112.
[9] TADDEI T，ZHANG L.Space-time registration-based model reduction of parameterized one-dimensional hyperbolic PDEs[J].EDP Sciences，2021，55（1）：99-130.
[10] CHENG L，CHEN S，LIU X，et al.Registration of laser scanning point clouds：a review[J].Sensors，2018，18（5）：1641.
[11] KULKARNI S C，REGE P P.Pixel level fusion techniques for SAR and optical images：a review[J].Information Fusion，2020，59：19-29.
[12] ZHOU J C，WEI X J，SHI J Y，et al.Underwater image enhancement via two-level wavelet decomposition maxi-mum brightness color restoration and edge refinement histogram stretching[J].Optics Express，2022，30（10）：17290-17306.
[13] ZHANG W，ZHUANG P，SUN H H，et al.Underwater image enhancement via minimal color loss and locally adaptive contrast enhancement[J].IEEE Transactions on Image Processing，2022，31：3997-4010.
[14] ZHOU J，ZHANG D，ZHANG W.Underwater image enhancement method via multi-feature prior fusion[J].Applied Intelligence，2022：1-23.
[15] SAID A B，HADJIDI R，MELKEMI K E，et al.Multispectral image denoising with optimized vector non-local mean filter[J].Digital Signal Processing，2016，58：115-126.
[16] CUI J，HU C，XU H.Adaptive median filtering for forward-looking sonar image registration[C]//INTER-NOISE and NOISE-CON Congress and Conference Proceedings，Hog Kong，China，2017：2534-2540.
[17] LI S，ZHAO J，ZHANG H，et al.A non-local low-rank algorithm for sub-bottom profile sonar image denoising[J].Remote Sensing，2020，12（14）：2336.
[18] SANG E，SHEN Z，BIAN H，et al.Sonar image denoising based on HMT model in morphological wavelet domain[C]//2010 International Conference on Image Analysis and Signal Processing，Zhejiang，2010：214-218.
[19] 贾伟.利用NSCT循环抽样进行水声图像去噪[J].舰船科学技术，2016，38（24）：112-114.
JIA W.The underwater acoustic image denoising based on NSCT cycle sampling[J].Ship Science and Technology，2016，38（24）：112-114.
[20] 刘光宇，曾志勇，曹禹，等.基于密度聚类与灰度变换的NSST域声呐图像去噪[J].湖南大学学报（自然科学版），2022，49（8）：186-195.
LIU G Y，ZENG Z Y，CAO Y，et al.Sonar image denoising based on density clustering and gray scale transformation in NSST domain[J].Journal of Hunan University（Natural Sciences），2022，49（8）：186-195.
[21] CHEN P，CAI X W，ZHAO D D，et al.Despeckling for side-scan sonar images based on adaptive block-matching and 3D filtering[J].Opto-Electronic Engineering，2020，47（7）：1-10.
[22] HUANG Y，LI W，YUAN F.Speckle noise reduction in sonar image based on adaptive redundant dictionary[J].Journal of Marine Science and Engineering，2020，8（10）：761.
[23] TIAN W，CHEN Z，SHEN J，et al.Underwater sonar image denoising through nonconvex total variation regularization and generalized Kullback-Leibler fidelity[J].Journal of Ambient Intelligence and Humanized Computing，2021：1-15.
[24] YE X F，LI P，ZHANG J G，et al.A feature-matching method for side-scan sonar images based on nonlinear scale space[J].Journal of Marine Science and Technology，2016，21（1）：38-47.
[25] LU B，WANG J，ZHANG Q.A variable exponent high-order variational model for noise removal[J].Journal of Computational Information Systems，2015，11（13）：4605-4614.
[26] ZHAO D，CAI T，CHEN P，et al.3-D acoustic image denoising for a sonar system with sparse planar arrays[J].IEEE Transactions on Instrumentation and Measurement，2022，71：1-17.
[27] KIM J，SONG S，YU S C.Denoising auto-encoder based image enhancement for high resolution sonar image[C]//2017 IEEE Underwater Technology（UT），Busan，2017：1-5.
[28] 刘秋逸.基于生成对抗网络的声呐图像生成及盲去噪方法研究[D].哈尔滨：哈尔滨工程大学，2020.
LIU Q Y.Generative adversarial networks based sonar image synthesis and blind denoising methods[D].Harbin：Harbin Engineering University，2020.
[29] PRIYADHARSINI R，SHARMILA T S.An efficient edge preserving interpolation method for underwater acoustic image resolution enhancement[J].Archives of Acoustics，2022，47（2）：267-274.
[30] MUTHURAMAN D L，SANTHANAM S M.Contrast improvement on side scan sonar images using retinex based edge preserved technique[J].Marine Geophysical Research，2022，43（2）：1-19.
[31] YOON K S，KIM W J.Efficient edge‐preserved sonar image enhancement method based on CVT for object recognition[J].IET Image Processing，2019，13（1）：15-23.
[32] SHI P，LU L，FAN X，et al.A novel underwater sonar image enhancement algorithm based on approximation spaces of random sets[J].Multimedia Tools and Applications，2022，81（4）：4569-4584.
[33] LIU J，LI B Y，GUAN W X，et al.A scale-adaptive matching algorithm for underwater acoustic and optical images[J].Sensors，2020，20（15）：1-13.
[34] ABU A，DIAMANT R.Feature set for classification of man-made underwater objects in optical and SAS data[J].Sensors，2022，22（6）：6027-6041.
[35] SENGUPTA D，GUPTA P，BISWAS A.A survey on mutual information based medical image registration algorithms[J].Neurocomputing，2022，486：174-188.
[36] MAJUMDER S，SHEDING S，DURRANT-WHYTE H.Sensor fusion and map building localization for underwater navigation[C]//Proceedings Australian Conference on Robotics and Automation，Honolulu，2000：25-30.
[37] MAJUMDER S，ROSENBLATT J，SHEDING S，et al.Map building and localization for underwater navigation[C]//Lecture Notes in Control and Information Sciences：Experimental Robotics VII，2002：511-520.
[38] MAJUMDER S，SHEDING S，DURRANT-WHYTE H.Multisensor data fusion for underwater navigation[J].Robotics and Autonomous System，2001，35（2）：97-108.
[39] LAGUDI A，BIANCO G，MUZZUPAPPA M，et al.An alignment method for the integration of underwater 3D data captured by a stereovision system and an acoustic camera[J].Sensors，2016，16（4）：536.
[40] 李本源.水下声光图像的尺度自适应匹配方法的研究[D].长春：吉林大学，2021.
LI B Y.Research on scale-adaptive matching algorithm for underwater acoustic and optical images[D].Changchun：Jilin University，2021.
[41] SONG S M，MICHAEL MERRMANN J，SI B L，et al.Two-dimensional forward-looking sonar image registration by maximization of peripheral mutual information[J].International Journal of Advanced Robotic Systems，2017，14（6）：1-17.
[42] PARTON S，DALGLEISH F，BEAUJEAN P P，et al.Acquisition and registration of bathymetric acoustic data and MOFSLI（multiple overlapping field of view serial laser imagery）[C]//2013 MTS/IEEE OCEANS，Bergen，2013：1-8.
[43] HURTóS N，CUF X，PETILLOT Y，et al.Fourier-based registrations for two-dimensional forward-looking sonar image mosaicking[C]//2012 IEEE/RSJ International Conference on Intelligent Robots and Systems，Vilamoura-Algarve，2012：5298-5305.
[44] SAXENA S，POHIT M.Near infrared and visible image registration using whale optimization algorithm[J].International Journal of Applied Metaheuristic Computing，2022，13（1）：1-14.
[45] ZHANG H M，SHAO N，MENG X K，et al.Fast image matching method and its applications in underwater positioning[C]//2010 International Conference on Electrical and Control Engineering，Wuhan，2010：970-973.
[46] YAN H，YANG S W，XUE Q，et al.HR optical and SAR image registration using uniform optimized feature and extend phase congruency[J].International Journal of Remote Sensing，2022，43（1）：52-74.
[47] LIU X Z，AI Y F，TIAN B，et al.Robust and fast registration of infrared and visible images for electro-optical pod[J].Transactions on Industrial Electronics，2019，66（2）：1335-1344.
[48] ZHANG L，LI B，TIAN L F，et al.LPPCO：a novel multi-modal medical image registration using new feature descriptor based on the local phase and phase congruency of different orientations[J].Access，2018，6：71976-71987.
[49] 孔祥山.基于红外与可见光图像的无人机测距技术研究[D].成都：电子科技大学，2018.
KONG X S.Research on UAV ranging technology based on infrared and visible images[D].Chengdu：University of Electronic Science and Technology of China，2018.
[50] FAN J W，YE Y X，LIU G C，et al.Phase congruency order-based local structural feature for SAR and optical image matching[J].Geoscience and Remote Sensing Letters，2022，19：1-5.
[51] LI Z Y，ZHNAG H T，CHEN J Y.Robust optical and SAR image registration based on phase congruency scale space[J].Geoscience and Remote Sensing Letters，2022，19：6515805.
[52] FUNK C J，LEMAIRE I P，SUTTON J L，et al.Apparatus for scanning an underwater area：3775735[P].1973.
[53] MORONI D，PASCALI A M，REGGIANNINI M，et al.Underwater scene understanding by optical and acoustic data integration[C]//Proceedings of Meetings on Acoustics ECUA2012，2012.
[54] ANSARI S.A review on SIFT and SURF for underwater image feature detection and matching[C]//2019 IEEE International Conference on Electrical，Computer and Communication Technologies（ICECCT），Coimbatore，2019：1-4
[55] 陆军，陈坤，范哲君.点邻域尺度差异描述的点云配准算法[J].国防科技大学学报，2021，43（3）：128-134.
LU J，CHEN K，FAN Z J.Point cloud registration algorithm based on scale difference descriptor of point neighborhood[J].Journal of National University of Defense，2021，43（3）：128-134.
[56] BABAEE M，NEGAHDARIPOUR S.3-D object modeling from 2-D occluding contour correspondences by opti-acoustic stereo imaging[J].Computer Vision and Image Understanding，2015，132：56-74.
[57] SPEARS A，HOWARD A M，WEAT M，et al.Acoustic sonar and video sensor fusion for landmark detection in an under-ice environment[C]//2014 Oceans-St.John’s，2014：1-8.
[58] HUANG Y B，ZHANG Z，XU X L.Image stitching algorithm based on region division for underwater dam crack image[C]//2021 IEEE International Conference on Progress in Informatics and Computing（PIC），Shanghai，2021：112-117.
[59] TANG Z J，MA G Q，LU J Q，et al.Sonar image mosaic based on a new feature matching method[J].IET Image Processing，2020，14（10）：2149-2155.
[60] 李海森，高珏，徐超，等.应用TBD的多波束声呐图像序列SIFT特征追踪[J].哈尔滨工程大学学报，2016，37（8）：1009-1014.
LI H S，GAO J，XU C，et al.SIFT feature tracking in multi-beam sonar image sequences using the track-before-detect technique[J].Journal of Harbin Engineering University，2016，37（8）：1009-1014.
[61] ZHANG W Y，ZHOU T，XU C，et al.A SIFT-like feature detector and descriptor for multibeam sonar imaging[J].Journal of Sensors，2021：8845814.
[62] 田嘉禾，王宁，陈廷凯，等.基于自适应深度约束的水下双目图像特征匹配[J].中国舰船研究，2021，16（6）：124-131.
TIAN J H，WANG N，CHEN T K，et al.Adaptive depth constraint-based underwater binocular image feature matching[J].Chinese Journal of Ship Research，2021，16（6）：124-131.
[63] HAO M，JIN J，ZHOU M C，et al.Robust multisource remote sensing image registration method based on scene shape similarity[J].Photogrammetric Engineering and Remote Sensing，2019，85（10）：725-736.
[64] ZHU B，YE Y X，YANG C，et al.Fast and robust registration of aerial images and lidar data based on structural features and 3D phase correlation[J].arXiv：2004. 09811，2020.
[65] WANG Z Z，LI J S，WANG X，et al.Underwater terrain image stitching based on spatial gradient feature block[J].Computers，Materials & Continua，2022，72（2）：4157-4171.
[66] SUHARYANTO S，HASIBUAN Z A，ANDONO P N，et al.Contrast limited adaptive histogram equalization for underwater image matching optimization use SURF[C]//2nd International Conference on Enhanced Research and Industrial Application 2020，Yogyakarta，2020.
[67] GWON D H，SHIN Y S，KIM Y，et al.Nontemporal rela-tive pose estimation for opti-acoustic bundle adjustment[C]//OCEANS 2016 MTS/IEEE，Monterey，2016：1-5.
[68] LI Z L，LAN R S，CHEN Z，et al.Underwater high-precision panoramic 3D image generation[C]//2020 8th International Conference on Digital Home（ICDH），Dalian，2021：39-44.
[69] 刘嘉欣.基于声光图像辅助水下AUV导航与目标探测研究[D].长春：吉林大学，2020.
LIU J X.Research on underwater AUV navigation and target detection based on acousto-optic image[D].Changchun：Jilin University，2020.
[70] YE Y，SHEN L.Hopc：a novel similarity metric based on geometric structural properties for multi-modal remote sensing image matching[J].ISPRS Annals of the Photogrammetry，Remote Sensing and Spatial Information Sciences，2016，3：9.
[71] YE Y，LI S，HAO M，et al.Robust optical-to-SAR image matching based on shape properties[J].IEEE Geoscience and Remote Sensing Letters，2017，14：564-568.
[72] 王蒙蒙，叶沅鑫，朱柏，等.基于空间约束和结构特征的光学与SAR影像配准[J].武汉大学学报（信息科学版），2022，47（1）：141-148.
WANG M M，YE Y X，ZHU B，et al.An automatic registration method for optical and SAR images based on spatial constraint and structure features[J].Geomatics and Information Science of Wuhan University，2022，47（1）：141-148.
[73] XIONG X，XU Q，JIN G W，et al.Rank-based local self-similarity descriptor for optical-to-SAR image matching[J].IEEE Geoscience and Remote Sensing Letters，2020，17（10）：1742-1746.
[74] ZHANG W R，MIAO J M，LI M F，et al.Research on underwater image registration method based on MSRCR and SIFT[C]//International Conference on Computer Vision and Pattern Analysis，Guangzhou，2022：74-80.
[75] YANG L Z，YANG Z L.Underwater image registration based on polar complex exponential transform[C]//Proceedings of 2021 International Conference on Autonomous Unmanned Systems（ICAUS 2021），Singapore，2021：1203-1213.
[76] YANG K C，YU L，XIA M，et al.Nonlinear RANSAC with crossline correction：an algorithm for vision-based curved cable detection system[J].Optics and Lasers in Engineering，2021，141：106417.
[77] 盛明伟，唐松奇，万磊，等.基于改进CNN-RANSAC的水下图像特征配准方法[J].计算机工程与科学，2020，42（5）：859-868.
SHENG M W，TANG S Q，WAN L，et al.An underwater image feature registration method based on improved CNN-RANSAC[J].Computer Engineering & Science，2020，42（5）：859-868.
[78] MACHADO M，ZAFFARI G，RIBEIRO P O，et al.Description and matching of acoustic images using a forward looking sonar：a topological approach[J].IFAC-PapersOnLine，2017，50（1）：2317-2322.
[79] HUANG S D，XU Z L，TSANG I W，et al.Auto-weighted multi-view co-clustering with bipartite graphs[J].Information Sciences，2020，512：18-30.
[80] DOS SANTOS M M，DE GIACOMO G G，DREWS P L J，et al.Satellite and underwater sonar image matching using deep learning[C]//2019 Latin American Robotics Symposium（LARS），2019 Brazilian Symposium on Robotics（SBR） and 2019 Workshop on Robotics in Education（WRE），Rio Grande，2019：109-114.
[81] DOS SANTOS M M，DE GIACOMO G G，DREWS P L J，et al.Cross-view and cross-domain underwater localization based on optical aerial and acoustic underwater images[J].IEEE Robotics and Automation Letters，2022，7（2）：4969-4974.
[82] DOS SANTOS M M，DE GIACOMO G G，DREWS P L J，et al.Matching color aerial images and underwater sonar images using deep learning for underwater localization[J].IEEE Robotics and Automation Letters，2020，5（4）：6365-6370.
[83] 魏泓安，单小军，郑柯，等.基于深度学习的SAR与光学影像配准方法综述[J].无线电工程，2021，51（12）：1363-1372.
WEI H A，SHAN X J，ZHENG K，et al.Review of SAR and optical image registration based on deep learning[J].Radio Engineering，2021，51（12）：1363-1372.
[84] ZHOU X T，YU C L，YUAN X，et al.Nonlinear intensity sonar image matching based on deep convolution features[C]//OCEANS 2022，Chennai，2022：1-5.
[85] YANG W L，FAN S S，XU S X，et al.Autonomous underwater vehicle navigation using sonar image matching based on convolutional neural network[J].IFAC-PapersOnline，2019，52（21）：156-162.
[86] QI Q，ZHANG Y C，TIAN F，et al.Underwater image co-enhancement with correlation feature matching and joint learning[J].Transactions on Circuits and Systems for Video Technology，2022，32（3）：1133-1147.
[87] VALDENEGRO-TORO M.Improving sonar image patch matching via deep learning[C]//2017 European Conference on Mobile Robots，Paris，2017：1-6.
[88] ZAGORUYKO S，KOMODAKIS N.Learning to compare image patches via convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Boston，2015：4353-4361.
[89] FANG Y，HU J，DU C，et al.SAR-optical image matching by integrating siamese U-Net with FFT correlation[J].Geoscience and Remote Sensing Letters，2022，19：1-5.
[90] 陈世伟，夏海，杨小冈，等.基于风格迁移不变特征的SAR与光学图像配准算法[J].系统工程与电子技术，2022，44（5）：1536-1542.
CHEN S W，XIA H，YANG X G，et al.SAR and optical image registration algorithm based on style transfer invariable features[J].Systems Engineering and Electronics，2022，44（5）：1536-1542.
[91] JANG H，LEE Y，KIM G，et al.CNN-based opti-acoustic transformation for underwater feature matching[J].The Journal of Korea Robotics Society，2020，15（1）：1-7.
[92] MERKLE N，AUER S，MüLLER R，et al.Exploring the potential of conditional adversarial networks for optical and SAR image matching[J].Selected Topics in Applied Earth Observations and Remote Sensing，2018，11（6）：1811-1820.
[93] ZHOU X，YU C，YUAN X，et al.A matching algorithm for underwater acoustic and optical images based on image attribute transfer and local features[J].Sensors，2021，21（21）：7043.
[94] CHO Y，JANG H，MALAV R，et al.Underwater image dehazing via unpaired image-to-image translation[J].International Journal of Control，Automation and Systems，2020，18（3）：605-614.
[95] TERAYAMA K，SHIN K，MIZUNO K，et al.Integration of sonar and optical camera images using deep neural networkfor fish monitoring[J].Aquacultural Engineering，2019，86：102000.
[96] HUO G Y，WU Z Y，LI J B.Underwater object classification in sidescan sonar images using deep transfer learning and semisynthetic training data[J].IEEE Access，2020，8：47407-47418.