Graph Convolutional Neural Network and Its Application in Image Recognition

doi:10.3778/j.issn.1002-8331.2302-0273

Abstract

Abstract: Convolutional neural network has found widespread application in the field of image recognition, demonstrating remarkable feature extraction capabilities. However, it is inherently designed for processing structured data in Euclidean space, making it less suitable for handling unstructured data. To address this limitation, graph convolutional neural network leverages spectral and spatial methods to extend the scope of convolutional operations, enabling feature learning in non-Euclidean spaces. GCN possesses translational invariance for graph data, facilitating representation learning for unstructured data. Firstly, the basic principles and improvement work of two types of graph convolutional neural networks based on spectral domain and space domain are explained. Then, around the field of image recognition, the application of graph convolutional neural network in multi-label image recognition, skeleton-based action recognition and hyperspectral image classification is introduced, the research progress is summarized, and the performance comparison and analysis of related models are carried out. Finally, the content of the full text is summarized and the future development direction is looked forward.

Key words: image recognition, graph convolutional neural network, non-Euclidean space, deep learning, artificial intelligence

摘要： 卷积神经网络被广泛应用于图像识别领域并且展现出强大的特征提取能力，但它只能处理欧氏空间的结构化数据，无法适用于非结构化数据的处理。为应对该限制，图卷积神经网络利用谱域和空域方法，拓展了卷积运算的范围，使其能够在非欧几里德空间中进行特征学习，具备图数据的平移不变性，可以实现对非结构化图数据的表征学习。首先阐述了基于频域和空域的两种类型图卷积神经网络的基本原理，并且介绍了相关的改进工作；然后围绕图像识别领域，重点介绍了图卷积神经网络在多标签图像识别、基于骨架的动作识别和高光谱图像分类中的具体应用，总结其研究的最新进展，并对相关的模型进行了性能对比与分析；最后对全文内容进行总结，并对未来的发展方向进行展望。

关键词: 图像识别, 图卷积神经网络, 非欧氏空间, 深度学习, 人工智能

LI Wenjing, BAI Jing, PENG Bin, YANG Zhanyuan. Graph Convolutional Neural Network and Its Application in Image Recognition[J]. Computer Engineering and Applications, 2023, 59(22): 15-35.

李文静, 白静, 彭斌, 杨瞻源. 图卷积神经网络及其在图像识别领域的应用综述[J]. 计算机工程与应用, 2023, 59(22): 15-35.

References

[1] LECUN Y，BOSER B，DENKER J S，et al.Backpropagation applied to handwritten zip code recognition[J].Neural Computation，1989，1（4）：541-551.
[2] LECUN Y，BOTTOU L，BENGIO Y，et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE，1998，86（11）：2278-2324.
[3] CHEN Z M，WEI X S，WANG P，et al.Multi-label image recognition with graph convolutional networks[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：5177-5186.
[4] YAN S，XIONG Y，LIN D.Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence，2018.
[5] QIN A，SHANG Z，TIAN J，et al.Spectral-spatial graph convolutional networks for semi-supervised hyperspectral image classification[J].IEEE Geoscience and Remote Sensing Letters，2018，16（2）：241-245.
[6] 徐冰冰，岑科廷，黄俊杰，等.图卷积神经网络综述[J].计算机学报，2020，43（5）：755-780.
XU B B，CEN K T，HUANG J J，et al.A survey on graph convolutional neural network[J].Chinese Journal of Computers，2020，43（5）：755-780.
[7] ZHANG S，TONG H，XU J，et al.Graph convolutional networks：a comprehensive review[J].Computational Social Networks，2019，6（1）：1-23.
[8] LIANG F，QIAN C，YU W，et al.Survey of graph neural networks and applications[J].Wireless Communications and Mobile Computing，2022.DOI：10.1155/2022/9261537.
[9] CAO P，ZHU Z，WANG Z，et al.Applications of graph convolutional networks in computer vision[J].Neural Computing and Applications，2022，34：13387-13405.
[10] CHEN C，WU Y，DAI Q，et al.A survey on graph neural networks and graph transformers in computer vision：a task-oriented perspective[J].arXiv：2209.13232，2022.
[11] SHUMAN D I，NARANG S K，FROSSARD P，et al.The emerging field of signal processing on graphs[J].IEEE Signal Processing Magazine，2013，30（3）：83-98.
[12] BRUNA J，ZAREMBA W，SZLAM A，et al.Spectral networks and locally connected networks on graphs[J].arXiv：1312.6203，2013.
[13] DEFFERRARD M，BRESSON X，VANDERGHEYNST P.Convolutional neural networks on graphs with fast localized spectral filtering[C]//Advances in Neural Information Processing Systems 29，2016.
[14] KIPF T N，WELLING M.Semi-supervised classification with graph convolutional networks[J].arXiv：1609.02907，2016.
[15] XU B，SHEN H，CAO Q，et al.Graph wavelet neural network[J].arXiv：1904.07785，2019.
[16] INATSUKI H，UTO T.Graph wavelet convolutional network with graph clustering[C]//Proceedings of the 2022 37th International Technical Conference on Circuits/Systems，Computers and Communications，2022：165-168.
[17] LI R，WANG S，ZHU F，et al.Adaptive graph convolutional neural networks[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence，2018.
[18] BO D，SHI C，WANG L，et al.Specformer：spectral graph neural networks meet transformers[J].arXiv：2303.01028，2023.
[19] 公沛良，艾丽华.用于半监督分类的二阶近似谱图卷积模型[J].自动化学报，2021，47（5）：1067-1076.
GONG P H，AI L H.Two-order approximate spectral convolutional model for semi-supervised classification[J].Acta Automatica Sinica，2021，47（5）：1067-1076.
[20] ZHU H，KONIUSZ P.Simple spectral graph convolution[C]//Proceedings of the 9th International Conference on Learning Representations，2021.
[21] BO D，WANG X，SHI C，et al.Beyond low-frequency information in graph convolutional networks[C]//Proceedings of the 35th AAAI Conference on Artificial Intelligence，2021：3950-3957.
[22] BIANCHI F M，GRATTAROLA D，LIVI L，et al.Graph neural networks with convolutional arma filters[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2021，44（7）：3496-3507.
[23] JIANG B，ZHANG Z，LIN D，et al.Graph learning convolutional networks[J].arXiv：1811.09971，2019.
[24] WU F，SOUZA A，ZHANG T，et al.Simplifying graph convolutional networks[C]//Proceedings of the 36th International Conference on Machine Learning，2019：6861-6871.
[25] WANG J，WANG Y，YANG Z，et al.Bi-GCN：binary graph convolutional network[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：1561-1570.
[26] KLICPERA J，BOJCHEVSKI A，GüNNEMANN S.Predict then propagate：graph neural networks meet personalized pagerank[J].arXiv：1810.05997，2019.
[27] MESGARAN M，HAMZA A B.Anisotropic graph convolutional network for semi-supervised learning[J].IEEE Transactions on Multimedia，2020，23：3931-3942.
[28] WANG J，LIANG J，CUI J，et al.Semi-supervised learning with mixed-order graph convolutional networks[J].Information Sciences，2021，573：171-181.
[29] GASTEIGER J，WEI?ENBERGER S，GüNNEMANN S.Diffusion improves graph learning[C]//Advances in Neural Information Processing Systems 32，2019.
[30] WANG X，ZHU M，BO D，et al.AM-GCN：adaptive multichannel graph convolutional networks[C]//Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining，2020：1243-1253.
[31] PAGE L，BRIN S，MOTWANI R，et al.The PageRank citation ranking：bringing order to the web[R].Stanford InfoLab，1999.
[32] WEICKERT J.Anisotropic diffusion in image processing[M].Stuttgart：Teubner，1998.
[33] BLACK M J，SAPIRO G，MARIMONT D H，et al.Robust anisotropic diffusion[J].IEEE Transactions on Image Processing，1998，7（3）：421-432.
[34] ZHANG Y，HAMZA A B.Vertex-based diffusion for 3D mesh denoising[J].IEEE Transactions on Image Processing，2007，16（4）：1036-1045.
[35] MICHELI A.Neural network for graphs：a contextual constructive approach[J].IEEE Transactions on Neural Networks，2009，20（3）：498-511.
[36] NIEPERT M，AHMED M，KUTZKOV K.Learning convolutional neural networks for graphs[C]//Proceedings of the 33rd International Conference on Machine Learning，2016：2014-2023.
[37] HECHTLINGER Y，CHAKRAVARTI P，QIN J.A generalization of convolutional neural networks to graph structured data[J].arXiv：1704.08165，2017.
[38] HAMILTON W，YING Z，LESKOVEC J.Inductive representation learning on large graphs[C]//Advances in Neural Information Processing Systems 30，2017.
[39] CHEN J，ZHU J，SONG L.Stochastic training of graph convolutional networks with variance reduction[J].arXiv：1710.10568，2017.
[40] CHEN J，MA T，XIAO C.FastGCN：fast learning with graph convolutional networks via importance sampling[J].arXiv：1801.10247，2018.
[41] ZOU D，HU Z，WANG Y，et al.Layer-dependent importance sampling for training deep and large graph convolutional networks[C]//Advances in Neural Information Processing Systems 32，2019.
[42] HUANG T，ZHANG Y，WU J，et al.MG-GCN：fast and effective learning with mix-grained aggregators for training large graph convolutional networks[J].arXiv：2011.09900，2020.
[43] HUANG W，ZHANG T，RONG Y，et al.Adaptive sampling towards fast graph representation learning[C]//Advances in Neural Information Processing Systems 31，2018.
[44] CHIANG W L，LIU X，SI S，et al.Cluster-GCN：an efficient algorithm for training deep and large graph convolutional networks[C]//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining，2019：257-266.
[45] ZENG H，ZHOU H，SRIVASTAVA A，et al.GraphSAINT：graph sampling based inductive learning method[J].arXiv：1907.
04931，2019.
[46] ZHANG X，YANG L，ZHANG B，et al.Multi-scale aggregation graph neural networks based on feature similarity for semi-supervised learning[J].Entropy，2021，23（4）：403.
[47] ATWOOD J，TOWSLEY D.Diffusion convolutional neural networks[C]//Advances in Neural Information Processing Systems 29，2016.
[48] VELI?KOVI? P，CUCURULL G，CASANOVA A，et al.Graph attention networks[J].arXiv：1710.10903，2017.
[49] JIANG B，WANG X，LUO B.PH-GCN：person reidentification with part-based hierarchical graph convolutional network[J].arXiv：1907.08822，2019.
[50] YADAV R K，ABHISHEK A，SOURAV S，et al.GCN with clustering coefficients and attention module[C]//Proceedings of the 2020 19th IEEE International Conference on Machine Learning and Applications，2020：185-190.
[51] JI C，WANG R，ZHU R，et al.HopGAT：hop-aware supervision graph attention networks for sparsely labeled graphs[J].arXiv：2004.04333，2020.
[52] VASHISHTH S，YADAV P，BHANDARI M，et al.Confidence-based graph convolutional networks for semi-supervised learning[C]//Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics，2019：1792-1801.
[53] PEI H，WEI B，CHANG K C C，et al.Geom-GCN：geometric graph convolutional networks[J].arXiv：2002.05287，2020.
[54] JIN W，DERR T，WANG Y，et al.Node similarity preserving graph convolutional networks[C]//Proceedings of the 14th ACM International Conference on Web Search and Data Mining，2021：148-156.
[55] ZHOU J，DU Y，ZHANG R，et al.Adaptive depth graph attention networks[J].arXiv：2301.06265，2023.
[56] GILMER J，SCHOENHOLZ S S，RILEY P F，et al.Neural message passing for quantum chemistry[C]//Proceedings of the International Conference on Machine Learning，2017：1263-1272.
[57] XU K，HU W，LESKOVEC J，et al.How powerful are graph neural networks?[J].arXiv：1810.00826，2018.
[58] ZHANG M，CUI Z，NEUMANN M，et al.An end-to-end deep learning architecture for graph classification[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence，2018.
[59] LI Q，QIAO M，BIAN W，et al.Conditional graphical lasso for multi-label image classification[C]//Proceedings of the 2016 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2016：2977-2986.
[60] LEE C W，FANG W，YEH C K，et al.Multi-label zero-shot learning with structured knowledge graphs[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition，2018：1576-1585.
[61] LIN T Y，MAIRE M，BELONGIE S，et al.Microsoft COCO：common objects in context[C]//Proceedings of the 13th European Conference on Computer Vision.Cham：Springer，2014：740-755.
[62] EVERINGHAM M，VAN GOOL L，WILLIAMS C K I，et al.The pascal visual object classes （VOC） challenge[J].International Journal of Computer Vision，2010，88（2）：303-338.
[63] LI Q，PENG X，QIAO Y，et al.Learning label correlations for multi-label image recognition with graph networks[J].Pattern Recognition Letters，2020，138：378-384.
[64] XIE Y，WANG Y，LIU Y，et al.Label graph learning for multi-label image recognition with cross-modal fusion[J].Multimedia Tools and Applications，2022，81（18）：25363-25381.
[65] SUN D，MA L，DING Z，et al.An attention-driven multi-label image classification with semantic embedding and graph convolutional networks[J].Cognitive Computation，2023，15（4）：1308-1319.
[66] WANG Y，XIE Y，FAN L，et al.STMG：swin transformer for multi-label image recognition with graph convolution network[J].Neural Computing and Applications，2022，34（12）：10051-10063.
[67] SINGH I P，GHORBEL E，OYEDOTUN O，et al.Multi-label image classification using adaptive graph convolutional networks：from a single domain to multiple domains[J].arXiv：2301.04494，2023.
[68] YE J，HE J，PENG X，et al.Attention-driven dynamic graph convolutional network for multi-label image recognition[C]//Proceedings of the 16th European Conference on Computer Vision.Cham：Springer，2020：649-665.
[69] CAO P，CHEN P，NIU Q.Multi-label image recognition with two-stream dynamic graph convolution networks[J].Image and Vision Computing，2021，113：104238.
[70] LAN Z，MAEDA K，OGAWA T，et al.Multi-label image recognition based on multi-modal graph convolutional networks using captioning features[C]//Proceedings of the 2021 IEEE 10th Global Conference on Consumer Electronics，2021：273-274.
[71] SHAHROUDY A，LIU J，NG T T，et al.NTU RGB+ D：a large scale dataset for 3D human activity analysis[C]//Proceedings of the 2016 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2016：1010-1019.
[72] LIU J，SHAHROUDY A，PEREZ M，et al.NTU RGB+ D 120：a large-scale benchmark for 3D human activity understanding[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2019，42（10）：2684-2701.
[73] KAY W，CARREIRA J，SIMONYAN K，et al.The Kinetics human action video dataset[J].arXiv：1705.06950，2017.
[74] THAKKAR K，NARAYANAN P J.Part-based graph convolutional network for action recognition[J].arXiv：1809.
04983，2018.
[75] LI M，CHEN S，CHEN X，et al.Actional-structural graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：3595-3603.
[76] SHI L，ZHANG Y，CHENG J，et al.Skeleton-based action recognition with directed graph neural networks[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：7912-7921.
[77] LIU S，BAI X，FANG M，et al.Mixed graph convolution and residual transformation network for skeleton-based action recognition[J].Applied Intelligence，2022，52（2）：1544-1555.
[78] CAI J，JIANG N，HAN X，et al.JOLO-GCN：mining joint-centered light-weight information for skeleton-based action recognition[C]//Proceedings of the 2021 IEEE/CVF Winter Conference on Applications of Computer Vision，2021：2735-2744.
[79] LIU Z，ZHANG H，CHEN Z，et al.Disentangling and unifying graph convolutions for skeleton-based action recognition[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：143-152.
[80] YANG W，ZHANG J，CAI J，et al.HybridNet：integrating GCN and CNN for skeleton-based action recognition[J].Applied Intelligence，2023，53（1）：574-585.
[81] SHI L，ZHANG Y，CHENG J，et al.Two-stream adaptive graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：12026-12035.
[82] CHENG K，ZHANG Y，HE X，et al.Skeleton-based action recognition with shift graph convolutional network[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：183-192.
[83] XU W，WU M，ZHU J，et al.Multi-scale skeleton adaptive weighted GCN for skeleton-based human action recognition in IoT[J].Applied Soft Computing，2021，104：107236.
[84] ALSARHAN T，ALI U，LU H.Enhanced discriminative graph convolutional network with adaptive temporal modelling for skeleton-based action recognition[J].Computer Vision and Image Understanding，2022，216：103348.
[85] SETIAWAN F，YAHYA B N，CHUN S J，et al.Sequential inter-hop graph convolution neural network（SIhGCN） for skeleton-based human action recognition[J].Expert Systems with Applications，2022，195：116566.
[86] 杨清山，穆太江.采用蒸馏训练的时空图卷积动作识别融合模型[J].中国图象图形学报，2022，27（4）：1290-1301.
YANG Q S，MU T J.Action recognition using ensembling of different distillation-trained spatial-temporal graph convolution models[J].Journal of Image and Graphics，2022，27（4）：1290-1301.
[87] MOU L，LU X，LI X，et al.Nonlocal graph convolutional networks for hyperspectral image classification[J].IEEE Transactions on Geoscience and Remote Sensing，2020，58（12）：8246-8257.
[88] BAI J，DING B，XIAO Z，et al.Hyperspectral image classification based on deep attention graph convolutional network[J].IEEE Transactions on Geoscience and Remote Sensing，2021，60：5504316.
[89] XI B，LI J，LI Y，et al.Semi-supervised cross-scale graph prototypical network for hyperspectral image classification[C]//Proceedings of the 2021 IEEE International Geoscience and Remote Sensing Symposium，Brussels，2021：2851-2854.
[90] WAN S，GONG C，ZHONG P，et al.Multiscale dynamic graph convolutional network for hyperspectral image classification[J].IEEE Transactions on Geoscience and Remote Sensing，2019，58（5）：3162-3177.
[91] WAN S，GONG C，ZHONG P，et al.Hyperspectral image classification with context-aware dynamic graph convolutional network[J].IEEE Transactions on Geoscience and Remote Sensing，2020，59（1）：597-612.
[92] DING Y，ZHAO X，ZHANG Z，et al.Semi-supervised locality preserving dense graph neural network with ARMA filters and context-aware learning for hyperspectral image classification[J].IEEE Transactions on Geoscience and Remote Sensing，2021，60：5511812.
[93] WAN S，PAN S，ZHONG P，et al.Dual interactive graph convolutional networks for hyperspectral image classification[J].IEEE Transactions on Geoscience and Remote Sensing，2021，60：5502418.
[94] ZHU W，ZHAO C，QIN B，et al.Short and long range graph convolution network for hyperspectral image classification[C]//Proceedings of the 2022 IEEE International Geoscience and Remote Sensing Symposium，2022：3564-3567.
[95] DING Y，ZHAO X，ZHANG Z，et al.Graph sample and aggregate-attention network for hyperspectral image classification[J].IEEE Geoscience and Remote Sensing Letters，2022，19：5504205.
[96] YU L，PENG J，CHEN N，et al.Two-branch deeper graph convolutional network for hyperspectral image classification[J].IEEE Transactions on Geoscience and Remote Sensing，2023，61：5506514.