关键词: 卷积神经网络, 主题爬虫, 深度学习, LDA主题模型

Abstract: When the traditional theme crawler calculates the topic similarity, it usually adopts the method based on word frequency, vector space model and semantic similarity, which brings certain bottleneck to the improvement of similarity calculation accuracy. Therefore, a convolutional neural network topic crawler that integrates LDA is proposed, and the subject judgment module is regarded as a text classification problem, and the deep neural network is used to improve the theme crawler performance. After the convolutional layer, the theme features extracted by LDA are spliced to make up for the missing information of the traditional convolutional neural network. The experimental results show that this method can effectively improve the average accuracy of the topic judgment module, and it is more advantageous than other methods in the real crawl environment.

Key words: convolutional neural network, subject crawler, deep learning, LDA topic model