Graph Neural Network and Its Research Progress in Field of Image Processing

doi:10.3778/j.issn.1002-8331.2205-0503

Abstract

Abstract: Graph neural network （GNN） is a deep learning-based model for processing graph-structured data, which has received much attention from researchers for its good interpretability and powerful nonlinear fitting ability to graph-structured data. With the rise of GNN, GNN has been developed to integrate with image processing techniques and has made breakthroughs in image classification, human body analysis and visual quizzing. Firstly, image processing techniques and the theory of traditional neural networks are introduced, and the principles, characteristics and shortcomings of five major classes of GNNs are analyzed. Secondly, the applications of GNN in the image processing field from five technical levels are analyzed respectively, and the representative models of each class of methods are listed. Thirdly, the common models described in the paper are compared and summarized from the perspective of both datasets and performance evaluation metrics, and nine common public datasets in image processing are introduced in addition. Finally, areas for improvement in GNN in the field of image processingare analyzed in depth, and the prospects of its application in the field of image processing are presented.

Key words: graph neural network （GNN）, , deep learning, , graph structure, , image processing

摘要： 图神经网络（graph neural network，GNN）是一种基于深度学习的图结构数据处理模型，因良好的可解释性和对图结构数据强大的非线性拟合能力而受到研究者广泛关注。随着GNN的逐步优化，GNN与图像处理技术实现融合发展，在图像分类、人体解析和视觉问答等方面取得重大突破。对图像处理技术和传统神经网络理论进行介绍，并对五类GNN的原理、特点和不足进行分析与总结；同时从数据集和性能评估指标两个角度对文中所述的常用模型进行对比与总结，并补充介绍了九种常见的图像处理领域公共数据集；最后深入分析了GNN在图像处理领域中有待改进的方面，并对其应用前景进行展望。

关键词: 图神经网络（GNN）, 深度学习, 图结构, 图像处理

JIANG Yuying, CHEN Xinyu, LI Guangming, WANG Fei, GE Hongyi. Graph Neural Network and Its Research Progress in Field of Image Processing[J]. Computer Engineering and Applications, 2023, 59(7): 15-30.

蒋玉英, 陈心雨, 李广明, 王飞, 葛宏义. 图神经网络及其在图像处理领域的研究进展[J]. 计算机工程与应用, 2023, 59(7): 15-30.

References

[1] BATTAGLIA P W，HAMRICK J B，BAPST V，et al.Relational inductive biases，deep learning，and graph networks[J].arXiv：1806.01261，2018.
[2] SANCHEZ-GONZALEZ A，HEESS N，SPRINGENBERG J T，et al.Graph networks as learnable physics engines for inference and control[C]//Proceedings of International Conference on Machine Learning，2018：4470-4479.
[3] HAMILTON W，YING Z，LESKOVEC J.Inductive representation learning on large graph[C]//Advances in Neural Information Processing Systems，2017：1025-1035.
[4] KIPF T N，WELLING M.Semi-supervised classification with graph convolutional networks[J].arXiv：1609.02907，2016.
[5] SHANG J，XIAO C，MA T，et al.Gamenet：graph augmented memory networks for recommending medication combination[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2019：1126-1133.
[6] PENG H，WANG H，DU B，et al.Spatial temporal incidence dynamic graph neural networks for traffic flow forecasting[J].Information Sciences，2020，521：277-290.
[7] 姜山，丁治明，徐馨润，等.面向路网交通流态势预测的图神经网络模型[J].计算机科学与探索，2021，15（6）：1084-1091.
JIANG S，DING Z M，XU X R，et al.Graph neural network for traffic flow situation prediction[J].Journal of Frontiers of Computer Science and Technology，2021，15（6）：1084-1091.
[8] 何倩倩，孙静宇，曾亚竹.基于邻域感知图神经网络的会话推荐[J].计算机工程与应用，2022，58（9）：107-115.
HE Q Q，SUN J Y，ZENG Y Z.Neighborhood awareness graph neural networks for session-based recommendation[J].Computer Engineering and Applications，2022，58（9）：107-115.
[9] 吴正洋，汤庸，刘海.个性化学习推荐研究综述[J].计算机科学与探索，2022，16（1）：21-40.
WU Z Y，TANG Y，LIU H.Survey of personalized learning recommendation[J].Journal of Frontiers of Computer Science and Technology，2022，16（1）：21-40.
[10] QI X，LIAO R，JIA J，et al.3D graph neural networks for rgbd semantic segmentation[C]//Proceedings of the IEEE International Conference on Computer Vision，2017.
[11] LANDRIEU L，SIMONOVSKY M.Large-scale point cloud semantic segmentation with superpoint graphs[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：4558-4567.
[12] LECUN Y，BOTTOU L，BENGIO Y，et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE，1998，86（11）：2278-2324.
[13] CAI H，ZHENG V W，CHANG K C.A comprehensive survey of graph embedding：problems，techniques，and applications[J].IEEE Transactions on Knowledge and Data Engineering，2018，30（9）：1616-1637.
[14] CHITRADEVI B，SRIMATHI P.An overview on image processing techniques[J].International Journal of Innovative Research in Computer and Communication Engineering，2014，2（11）：6466-6472.
[15] ABDI H.A neural network primer[J].Journal of Biological Systems，1994，2（3）：247-281.
[16] SAZLI M H.A brief review of feed-forward neural networks[J].Communications Faculty of Sciences University of Ankara Series A2-A3 Physical Sciences and Engineering，2006，50（1）.
[17] 杨丽，吴雨茜，王俊丽，等.循环神经网络研究综述[J].计算机应用，2018，38（S2）：1-6.
YANG L，WU Y Q，WANG J L，et al.Research on research on recurrent neural networks[J].Journal of Computer Applications，2018，38（S2）：1-6.
[18] HOCHREITER S，SCHMIDHUBER J.Long short-term memory[J].Neural Computation，1997，9（8）：1735-1780.
[19] CHO K，VAN MERRI?NBOER B，GULCEHRE C，et al.Learning phrase representations using RNN encoder-decoder for statistical machine translation[J].arXiv：1406. 1078，2014.
[20] HOPFIELD J J.Hopfield network[J].Scholarpedia，2007，2（5）：1977.
[21] HINTON G E，SEJNOWSKI T J.Learning and relearning in Boltzmann machines[J].Parallel Distributed Processing：Explorations in the Microstructure of Cognition，1986，282/317：2.
[22] GORI M，MONFARDINI G，SCARSELLI F.A new model for learning in graph domains[C]//Proceedings.2005 IEEE International Joint Conference on Neural Networks，2005：729-734.
[23] HUSH D R，HORNE B G.Progress in supervised neural networks[J].IEEE Signal Processing Magazine，1993，10（1）：8-39.
[24] LECUN Y，BOSER B，DENKER J S，et al.Backpropagation applied to handwritten zip code recognition[J].Neural Computation，1989，1（4）：541-551.
[25] 吴博，梁循，张树森，等.图神经网络前沿进展与应用[J].计算机学报，2022，45（1）：35-68.
WU B，LIANG X，ZHANG S S，et al.Advances and applications in graph neural network[J].Chinese Journal of Computers，2022，45（1）：35-68.
[26] DEFFERRARD M，BRESSON X，VANDERGHEYNST P.Convolutional neural networks on graphs with fast localized spectral filtering[C]//Advances in Neural Information Processing Systems，2016.
[27] 师海忠.无向图语言[J].计算机科学，2011，38（6）：259-261.
SHI H Z.Undirected graph language[J].Computer Science，2011，38（6）：259-261.
[28] 尚亚灵，袁道华，丁莹，等.消息传递机制的比较及其最新发展[J].计算机工程与设计，2007（10）：2265-2269.
SHANG Y L，YUAN D H，DING Y，et al.Comparison and latest development of message-passing mechanisms[J].Computer Engineering and Design，2007（10）：2265-2269.
[29] 马帅，刘建伟，左信.图神经网络综述[J].计算机研究与发展，2022，59（1）：47-80.
MA S，LIU J W，ZUO X.Survey on graph neural network[J].Journal of Computer Research and Development，2022，59（1）：47-80.
[30] GILMER J，SCHOENHOLZ S S，RILEY P F，et al.Neural message passing for quantum chemistry[C]//Proceedings of the International Conference on Machine Learning，2017：1263-1272.
[31] ATWOOD J，TOWSLEY D.Diffusion-convolutional neural networks[C]//Advances in Neural Information Processing Systems，2016.
[32] GAO H，WANG Z，JI S.Large-scale learnable graph convolutional networks[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining，2018：1416-1424.
[33] GAMON J A，RAHMAN A，DUNGAN J，et al.Spectral network（SpecNet）—what is it and why do we need it?[J].Remote Sensing of Environment，2006，103（3）：227-235.
[34] CHEN J，MA T，XIAO C.FastGCN：fast learning with graph convolutional networks via importance sampling[J].arXiv：1801.10247，2018.
[35] VELI?KOVI? P，CUCURULL G，CASANOVA A，et al.Graph attention networks[J].arXiv：1710.10903，2017.
[36] 李硕朋，齐思宇，林绍福，等.图神经网络及其在通信网络领域应用综述[J].北京工业大学学报，2021（8）：971-981.
LI S P，QI S Y，LIN S F，et al.Survey of graph neural network and its applications in communication networks[J].Journal of Beijing University of Technology，2021（8）：971-981.
[37] ZHANG J，SHI X，XIE J，et al.GAAN：Gated attention networks for learning on large and spatiotemporal graphs[J].arXiv：1803.07294，2018.
[38] WANG X，JI H，SHI C，et al.Heterogeneous graph attention network[C]//Proceedings of the World Wide Web Conference，2019：2022-2032.
[39] YANG S，LI G，YU Y.Dynamic graph attention for referring expression comprehension[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：4644-4653.
[40] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[41] HUANG G，LIU Z，VAN DER MAATEN L，et al.Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：4700-4708.
[42] WANG H，BELL D.Extended k-nearest neighbours based on evidence theory[J].The Computer Journal，2004，47（6）：662-672.
[43] LI Y，TARLOW D，BROCKSCHMIDT M，et al.Gated graph sequence neural networks[J].arXiv：1511.05493，2015.
[44] KELLEY J L.General topology[M].[S.l.]：Courier Dover Publications，2017.
[45] KINGMA D P，WELLING M.Auto-encoding variational Bayes[J].arXiv：1312.6114，2013.
[46] BEAL M J.Variational algorithms for approximate Bayesian inference[D].London：University of London，2003.
[47] KIPF T N，WELLING M.Variational graph auto-encoders[J].arXiv：1611.07308，2016.
[48] MA T，CHEN J，XIAO C.Constrained generation of semantically valid graphs via regularizing variational autoencoders[C]//Advances in Neural Information Processing Systems，2018.
[49] ZHU D，CUI P，WANG D，et al.Deep variational network embedding in wasserstein space[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining，2018：2827-2836.
[50] PAN S，HU R，LONG G，et al.Adversarially regularized graph autoencoder for graph embedding[J].arXiv：1802. 04407，2018.
[51] 李方，吴国栋，涂立静，等.图自编码器推荐研究综述[J].计算机工程与科学，2022，44（2）：335-344.
LI F，WU G D，TU L J，et al.A review of graph auto-encoder recommendation[J].Computer Engineering and Science，2022，44（2）：335-344.
[52] LI Y，YU R，SHAHABI C，et al.Diffusion convolutional recurrent neural network：data-driven traffic forecasting[J].arXiv：1707.01926，2017.
[53] JAIN A，ZAMIR A R，SAVARESE S，et al.Structural-RNN：deep learning on spatio-temporal graphs[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：5308-5317.
[54] YU B，YIN H，ZHU Z.Spatio-temporal graph convolutional networks：a deep learning framework for traffic forecasting[J].arXiv：1709.04875，2017.
[55] YAN S，XIONG Y，LIN D.Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence，2018.
[56] 赵宏，翟冬梅，石朝辉.短时交通流预测模型综述[J].都市快轨交通，2019，32（4）：50-54.
ZHAO H，ZHAI D M，SHI C H.Review of short-term traffic flow forecasting models[J].Urban Rapid Rail Transit，2019，32（4）：50-54.
[57] DE CAO N，KIPF T.An implicit generative model for small molecular graphs[J].arXiv：1805.11973，2018.
[58] BOJCHEVSKI A，SHCHUR O，ZüGNER D，et al.Netgan：generating graphs via random walks[C]//Proceedings of the International Conference on Machine Learning，2018：610-619.
[59] MARINO K，SALAKHUTDINOV R，GUPTA A.The more you know：Using knowledge graphs for image classification[J].arXiv：1612.04844，2016.
[60] WANG X，YE Y，GUPTA A.Zero-shot recognition via semantic embeddings and knowledge graphs[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：6857-6866.
[61] WANG Z，CHEN T，REN J，et al.Deep reasoning with knowledge graph for social relationship understanding[J].arXiv：1807.00504，2018.
[62] WANG X，ZHANG L，ROTH H，et al.Interactive 3D segmentation editing and refinement via gated graph neural networks[C]//Proceedings of the International Workshop on Graph Learning in Medical Imaging，2019：9-17.
[63] LIANG X，SHEN X，FENG J，et al.Semantic object parsing with graph LSTM[C]//Proceedings of the European Conference on Computer Vision，2016：125-143.
[64] LIANG X，LIN L，SHEN X，et al.Interpretable structure-evolving LSTM[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：1010-1019.
[65] LI L，GAN Z，CHENG Y，et al.Relation-aware graph attention network for visual question answering[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：10313-10322.
[66] MI L，CHEN Z.Hierarchical graph attention network for visual relationship detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：13886-13895.
[67] XU B，DING S，ZHANG Y.Image classification model based on GAT[J].Journal of Physics（Conference Series），2020，1570（1）：012082.
[68] LIN D，LIN J，ZHAO L，et al.Multilabel aerial image classification with a concept attention graph neural network[J].IEEE Transactions on Geoscience and Remote Sensing，2021，60：1-12.
[69] HE X，LIU Q，YANG Y.MV-GNN：multi-view graph neural network for compression artifacts reduction[J].IEEE Transactions on Image Processing，2020，29：6829-6840.
[70] LI Y，YAO H，DUAN L，et al.Adaptive feature fusion via graph neural network for person re-identification[C]//Proceedings of the 27th ACM International Conference on Multimedia，2019：2115-2123.
[71] SHEN Y，LI H，YI S，et al.Person re-identification with deep similarity-guided graph neural network[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：486-504.
[72] WANG G，DONG G，LI H，et al.Remote sensing image synthesis via graphical generative adversarial networks[C]//Proceedings of the 2019 IEEE International Geoscience and Remote Sensing Symposium，2019：10027-10030.
[73] MNIH V，HEESS N，GRAVES A.Recurrent models of visual attention[C]//Advances in Neural Information Processing Systems，2014.
[74] YANG Z，HE X，GAO J，et al.Stacked attention networks for image question answering[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：21-29.
[75] TENEY D，ANDERSON P，HE X，et al.Tips and tricks for visual question answering：Learnings from the 2017 challenge[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：4223-4232.
[76] YANG Y，NEWSAM S.Bag-of-visual-words and spatial extensions for land-use classification[C]//Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems，2010：270-279.
[77] HUA Y，MOU L，ZHU X X.Label relation inference for multi-label aerial image classification[C]//Proceedings of the 2019 IEEE International Geoscience and Remote Sensing Symposium，2019：5244-5247.
[78] MüLLER K，SCHWARZ H，MARPE D，et al.3D high-efficiency video coding for multi-view video and depth data[J].IEEE Transactions on Image Processing，2013，22（9）：3366-3378.
[79] SMOLIC A，MUELLER K，MERKLE P，et al.3D video and free viewpoint video-technologies，applications and MPEG standards[C]/Proceedings of the 2006 IEEE International Conference on Multimedia and Expo，2006：2161-2164.
[80] FOI A，KATKOVNIK V，EGIAZARIAN K.Pointwise shape-adaptive DCT for high-quality denoising and deblocking of grayscale and color images[J].IEEE Transactions on Image Processing，2007，16（5）：1395-1411.
[81] JUNG C，JIAO L，QI H，et al.Image deblocking via sparse representation[J].Signal Processing：Image Communication，2012，27（6）：663-677.
[82] CHANG H，NG M K，ZENG T.Reducing artifacts in JPEG decompression via a learned dictionary[J].IEEE Transactions on Signal Processing，2013，62（3）：718-728.
[83] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial nets[C]//Advances in Neural Information Processing Systems，2014.
[84] WANG H，WANG J，WANG J，et al.GraphGAN：graph representation learning with generative adversarial nets[J].arXiv：1711.08267，2017.
[85] DING M，TANG J，ZHANG J.Semi-supervised learning on graphs with generative adversarial nets[C]//Proceedings of the 27th ACM International Conference on Information and Knowledge Management，2018：913-922.
[86] DENG J，DONG W，SOCHER R，et al.Imagenet：a large-scale hierarchical image database[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition，2009：248-255.
[87] LIN T-Y，MAIRE M，BELONGIE S J，et al.Microsoft COCO：common objects in context[J].arXiv：1405.0312，2014.
[88] LI J，WONG Y，ZHAO Q，et al.Dual-glance model for deciphering social relationships[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2650-2659.
[89] BILIC P，CHRIST P F，VORONTSOV E，et al.The liver tumor segmentation benchmark（lits）[J].arXiv：1901. 04056，2019.
[90] LIANG X，LIU S，SHEN X，et al.Deep human parsing with active template regression[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2015，37（12）：2402-2414.
[91] LU C，KRISHNA R，BERNSTEIN M，et al.Visual relationship detection with language priors[C]//Proceedings of the European Conference on Computer Visioner，2016：852-869.
[92] RUSANOVSKYY D，MüLLER K，VETRO A.Common test conditions of 3DV core experiments[C]//Joint Collaborative Team on 3D Video Coding Extension Development of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 5th Meeting，2013：1-8.
[93] ZHENG L，SHEN L，TIAN L，et al.Scalable person re-identification：a benchmark[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：1116-1124.
[94] RISTANI E，SOLERA F，ZOU R，et al.Performance measures and a data set for multi-target，multi-camera tracking[C]//Proceedings of the European Conference on Computer Vision，2016：17-35.
[95] LI W，ZHAO R，XIAO T，et al.Deepreid：deep filter pairing neural network for person re-identification[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2014：152-159.
[96] XIA G-S，HU J，HU F，et al.AID：a benchmark data set for performance evaluation of aerial scene classification[J].IEEE Transactions on Geoscience and Remote Sensing，2017，55（7）：3965-3981.
[97] CAMPOS-TABERNER M，ROMERO-SORIANO A，GATTA C，et al.Processing of extremely high-resolution Lidar and RGB data：outcome of the 2015 IEEE GRSS data fusion contest—part a：2-D contest[J].IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing，2016，9（12）：5547-5559.
[98] YANG Z，COHEN W，SALAKHUDINOV R.Revisiting semi-supervised learning with graph embeddings[C]//Proceedings of the International Conference on Machine Learning，2016：40-48.
[99] SONG S，LICHTENBERG S P，XIAO J.Sun RGB-D：a RGB-D scene understanding benchmark suite[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：567-576.
[100] KRISHNA R，ZHU Y，GROTH O，et al.Visual genome：connecting language and vision using crowdsourced dense image annotations[J].International Journal of Computer Vision，2017，123（1）：32-73.
[101] ZHOU B，ZHAO H，PUIG X，et al.Semantic understanding of scenes through the ade20k dataset[J].International Journal of Computer Vision，2019，127（3）：302-321.
[102] EVERINGHAM M，VAN GOOL L，WILLIAMS C，et al.Pascal visual object classes challenge results[J].Available From WWW Pascal-Network Org，2005，1（6）：7.
[103] KRIZHEVSKY A，HINTON G.Learning multiple layers of features from tiny images[D].Toronto：University of Toronto，2009.
[104] HARIHARAN B，ARBEL??EZ P，BOURDEV L，et al.Semantic contours from inverse detectors[C]//Proceedings of the 2011 International Conference on Computer Vision，2011：991-998.
[105] CORDTS M，OMRAN M，RAMOS S，et al.The cityscapes dataset for semantic urban scene understanding[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：3213-3223.
[106] TORRALBA A，FERGUS R，FREEMAN W T.80 million tiny images：a large data set for nonparametric object and scene recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2008，30（11）：1958-1970.
[107] LI Q，HAN Z，WU X M.Deeper insights into graph convolutional networks for semi-supervised learning[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence，2018.
[108] SIEGEL P H.THz technology：an overview[J].International Journal of High Speed Electronics and Systems，2003，13（2）：351-394.
[109] REN S，HE K，GIRSHICK R，et al.Faster R-CNN：towards real-time object detection with region proposal networks[C]//Advances in Neural Information Processing Systems，2015.
[110] 梁延禹，李金宝.多尺度非局部注意力网络的小目标检测算法[J].计算机科学与探索，2020，14（10）：1744-1753.
LIANG Y Y，LI J B.Small objects detection method based on multi-scale non-local attention network[J].Journal of Frontiers of Computer Science and Technology，2020，14（10）：1744-1753.