Automatic Annotation of Knowledge Points in Picture-Based Educational Resources for Knowledge Scenarios

doi:10.3778/j.issn.1002-8331.2308-0102

Abstract

Abstract: Aiming at the challenge of inconsistency between the visual features of picture resources and the semantics of advanced knowledge, a new automatic annotation algorithm for knowledge points is proposed, called the situational hypergraph convolutional network based on knowledge scenarios (SHGCN), which can efficiently organize and manage picture data, promote knowledge understanding and utilization, and improve education intelligence. The algorithm not only extracts explicit visual features of the picture resources, but also mines knowledge information hidden in fine-grained regions. Faster R-CNN and OCR techniques are utilized to identify knowledge entities such as knowledge objects and coordinate texts, and multi-granularity features are fused to generate knowledge vectors. Then, a dual-screening mechanism is proposed to construct different types of knowledge scenarios, and the knowledge scenarios are used as hyperedges to construct a situational hypergraph to model higher-order knowledge correlations between images containing similar knowledge information. Finally, the hypergraph convolution is used to complete the information aggregation of knowledge-similar pictures, and realize the transformation from “vision-semantic” to “vision-semantic-knowledge”. This paper also constructs a physical picture dataset to train and validate SHGCN. Experimental results show that SHGCN outperforms current state-of-the-art methods by fusing explicit visual features and implicit knowledge information of pictures.

Key words: knowledge point annotation, hypergraph convolutional neural network, knowledge scenarios, situational hypergraph

摘要： 针对图片资源的视觉特征与高级知识语义不一致的挑战，提出一种新的知识点自动标注算法，称为基于知识场景的情境超图卷积网络（SHGCN），以便高效组织管理教育领域中的图片数据，促进知识理解与有效利用，实现教育智能化。该算法在提取图片资源显性视觉特征的同时，又挖掘了隐含在细粒度区域的隐性知识信息。利用Faster R-CNN和OCR技术来识别知识对象和坐标文本等知识实体，这些知识实体特征融合后作为该图片的知识向量；提出双筛选机制来生成不同类型的知识场景，并将知识场景作为超边来构建情境超图，建模蕴含相似情境信息的图片间高阶知识相关性。利用超图卷积实现知识相似图片的情境信息聚合，实现“视觉-语义”到“视觉-语义-知识”的转化。还构建了一个物理学科的图片数据集来训练和验证SHGCN。实验结果表明，SHGCN在提取图片显性视觉信息的基础上，进一步挖掘隐性知识信息，其性能优于基线方法。

关键词: 知识点标注, 超图卷积网络, 知识场景, 情境超图

WANG Jing, DU Xu, LI Hao, HU Zhuang. Automatic Annotation of Knowledge Points in Picture-Based Educational Resources for Knowledge Scenarios[J]. Computer Engineering and Applications, 2024, 60(24): 119-130.

王静, 杜旭, 李浩, 胡壮. 面向知识场景的图片类教育资源知识点自动标注算法[J]. 计算机工程与应用, 2024, 60(24): 119-130.

References

[1] HUANG J L, SANG G Y, HE W J. Motivation to teach and preparedness for teaching among preservice teachers in China: the effect of conscientiousness and constructivist teaching beliefs[J]. Frontiers in Psychology, 2023, 14: 1116321.
[2] ZHENG L, WANG C, CHEN X, et al. Evolutionary machine learning builds smart education big data platform: data-driven higher education[J]. Applied Soft Computing, 2023, 136: 110114.
[3] PU N, ZHONG Z, SEBE N, et al. A memorizing and generalizing framework for lifelong person re-identification[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(11): 2393-2414.
[4] DAVID M, ERIC M, RALPH E. SlideImages: a dataset for educational image classification[C]//Advances in Information Retrieval, 2020: 289-296.
[5] 郭崇慧, 邢小宇, 魏伟. 一种基于二部图的试题知识点标注方法[J]. 运筹与管理, 2021, 30(11): 71-75.
GUO C H, XING X Y, WEI W. A knowledge points labeling method for test questions based on bipartitle graph[J]. Operations Research and Management Science, 2021, 30(11): 71-75.
[6] 魏伟, 郭崇慧, 邢小宇. 基于语义关联规则的试题知识点标注及试题推荐[J]. 数据分析与知识发现, 2020, 4(2/3): 182-191.
WEI W, GUO C H, XING X Y. Annotating knowledge points & recommending questions based on semantic association rules[J]. Data Analysis and Knowledge Discovery, 2020, 4(2/3): 182-191.
[7] PALEKAR V, KUMAR L S. Adaptive optimized residual convolutional image annotation model with bionic feature selection model[J]. Computer Standards & Interfaces, 2024, 87: 103780.
[8] KHATCHATOORIAN A G, JAMZAD M. Suggesting an integration system for image annotation[J]. Multimedia Tools and Applications, 2023, 82: 8323-8343.
[9] ADNAN M M, RAHIM M S M, KHAN A R, et al. An improved automatic image annotation approach using convolutional neural network-slantlet transform[J] IEEE Access, 2022, 10: 7520-7532.
[10] LI K H, HUANG Z H, JIA Z H, et al. RAHG: a role-aware hypergraph neural network for node classification in graphs[J]. IEEE Transactions on Network Science and Engineering, 2023, 10(4): 2098-2108.
[11] TEE YY, HONG X N, CHENG D R, et al. Patch-based adversarial training for error-aware circuit annotation of delayered IC images[J]. IEEE Transactions on Circuits and Systems II: Express Briefs, 2023, 70(9): 3694-3698.
[12] CHEN Y T, LIU L W, TAO J J, et al. The image annotation algorithm using convolutional features from intermediate layer of deep learning[J]. Multimedia Tools and Applications, 2021, 80(3): 4237-4261.
[13] 郭崇慧, 吕征达. 一种基于集成学习的试题多知识点标注方法[J]. 运筹与管理, 2020, 29(2): 129-136.
GUO C H, LV Z D. A multi knowledge points labeling methhod for test questions based on ensemble learning[J]. Operations Research and Management Science, 2020, 29(2): 129-136.
[14] 夏盟. 基于机器解答的物理电路学习资源标注与呈现研究[D]. 武汉: 华中师范大学, 2019.
XIA M. Research on annotation and representation of circuit learning resources based on automatic problem solving[D]. Wuhan: Central China Normal University, 2019.
[15] 邢小宇. 基于文本挖掘的试题知识点和认知动词标注方法研究[D]. 大连: 大连理工大学, 2020.
XING X Y. Research on labeling mthods of knowledge points and cognotive verbs for test questions based on text mining[D]. Dalian: Dalian University of Technology, 2020.
[16] 何鑫宇. 基于多标签的数学试题自动标注研究[D]. 天津: 天津师范大学, 2023.
HE X Y. Research on automatic annotation of mathematical question based on multiple labels[D]. Tianjin: Tianjin Normal University, 2023.
[17] 王晓璐. 面向K12数学的试题知识点自动标注研究[D]. 上海: 华东师范大学, 2022.
WANG X L. Research on automatic labeling of knowledge points for questions in K12 mathematics[D]. Shanghai: East China Normal University, 2022.
[18] HAN Z C, XU A F. Microsystems. ecological evolution path of smart education platform based on deep learning and image detection[J]. Microprocessors and Microsystems, 2021, 80: 103343.
[19] SHI D M, TANG H Y. A new multiface target detection algorithm for students in class based on Bayesian optimized YOLOv3 model[J]. Journal of Electrical and Computer Engineering, 2022(1): 4260543.
[20] DASS S, GARY K, CUNNINGHAM J. Predicting student dropout in self-paced MOOC course using random forest model[J]. Information, 2021, 12(11): 476.
[21] WANG Y X, SONG J M, WEI M J, et al. Predicting potential drug-disease associations based on hypergraph learning with subgraph matching[J]. Interdisciplinary Sciences-Computational Life Sciences, 2023, 15(2): 249-261.
[22] HAN J D, TAO Q, TANG Y F, et al. DH-HGCN: dual homogeneity hypergraph convolutional network for multiple social recommendations[C]//Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022: 2190-2194.
[23] XIAO Q G, LI G Y, CHEN Q C, et al. Complex image classification by feature inference[J]. Neurocomputing, 2023, 544: 126231.
[24] ZHOU Y, ZHENG H X, HUANG X, et al. Graph neural networks: taxonomy, advances, and trends[J]. ACM Transactions on Intelligent Systems and Technology, 2022, 13(1): 1-54.
[25] WU F, JING X Y, WEI P F, et al. Semi-supervised multi-view graph convolutional networks with application to webpage classification[J]. Information Sciences, 2022, 591: 142-154.
[26] FENG Y F, YOU H X, ZHANG Z Z, et al. Hypergraph neural networks[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2019: 3558-3565.
[27] JIANG J W, WEI Y X, FENG Y F, et al. Dynamic hypergraph neural networks[C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, 2019: 2635-2641.
[28] BAI S, ZHANG F H, PHILIP H S T. Hypergraph convolution and hypergraph attention[J]. Pattern Recognition, 2021, 110: 107637.
[29] JI J Z, REN Y T, LEI M L. FC-HAT: hypergraph attention network for functional brain network classification[J]. Information Sciences, 2022, 608: 1301-1316.
[30] LIU Q S, SUN Y B, WANG C T, et al. Elastic net hypergraph learning for image clustering and semi-supervised classification[J]. IEEE Transactions on Image Processing, 2016, 26(1): 452-463.
[31] TANG C, LIU X W, WANG P C, et al. Adaptive hypergraph embedded semi-supervised multi-label image annotation[J]. IEEE Transactions on Multimedia, 2019, 21(11): 2837-2849.
[32] WU X P, CHEN Q C, LI W, et al. AdaHGNN: adaptive hypergraph neural networks for multi-label image classification[J]. ACM Multimedia, 2020: 284-293.
[33] CAI Y M, ZHANG Z J, CAI Z H, et al. Hypergraph-structured autoencoder for unsupervised and semisupervised classification of hyperspectral image[J]. IEEE Geoscience Remote Sensing Letters, 2021, 19: 1-5.
[34] DENG Q W, ZHANG S Y, DING Z. An efficient hypergraph approach to robust point cloud resampling[J]. IEEE Transactions on Image Processing, 2022, 31: 1924-1937.
[35] REN S Q, HE K M, ROSS B, et al. Faster R-CNN: towards real-time object detection with region proposal networks[C]//Advances in Neural Information Processing Systems, 2015: 91-99.
[36] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[C]//Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015.
[37] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
[38] SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 1-9.
[39] HUANG G, LIU Z, LAURENS V D M, et al. Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 4700-4708.
[40] ALEXEY D, LUCAS B, ALEXANDER K, et al. An image is worth 16x16 words: Transformers for image recognition at scale[C]//Proceedings of the International Conference on Learning Representations, 2021.
[41] STEPHANE D, HUGO T, MATTHEW L L, et al. ConViT: improving vision transformers with soft convolutional inductive biases[C]//Proceedings of the International Conference on Machine Learning, 2021: 2286-2296.
[42] HONG D F, GAO L R, WU J, et al. Graph convolutional networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience Remote Sensing, 2021, 59(7): 5966-5978.
[43] FU X Y, XIAO J, ZHU Y R, et al. Continual image deraining with hypergraph convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(8): 9534-9551.