计算机工程与应用 ›› 2016, Vol. 52 ›› Issue (3): 59-65.

• 大数据与云计算 • 上一篇    下一篇

基于爬虫和网站分类的主题信息源发现方法

邓厚平,武  刚   

  1. 北京林业大学 信息学院,北京 100083
  • 出版日期:2016-02-01 发布日期:2016-02-03

Discovery of topic-specific information source based on web crawler and website classification

DENG Houping, WU Gang   

  1. College of Information, Beijing Forestry University, Beijing 100083, China
  • Online:2016-02-01 Published:2016-02-03

摘要: 如何发现主题信息源是主题Web信息整合的前提。提出了一种主题信息源发现方法,将主题信息源发现转化为网站主题分类问题,并利用站外链接发现新的信息源。从网站中提取出能反映网站主题的内容特征词和结构特征词,建立描述网站主题的改进的向量空间模型。以该模型为基础,通过类中心向量法与SVM相结合对网站主题进行分类。提出一种能尽量少爬取网页的网络搜索策略,在发现站外链接的同时爬取最能代表网站主题的页面。将该主题信息源发现方法应用于林业商务信息源,通过实验验证了该方法的有效性。

关键词: 网站主题, 特征描述, 分类, 爬虫, 信息源发现

Abstract: The discovery of topic-specific information source is the premise of Web information integration. A topic-specific information discovery method is presented, changing the problem to website topic classification and discover websites using external links. An improved VSM model is established to describe the website topic, using both content and structure features extracted from websites. Based on the improved VSM model, a classification method combining center-vector algorithm and SVM is presented to classify the topic of websites. A web search strategy aiming to minimize the quantity of crawled web page is presented to find out web pages that best represent the topic of the website. The topic-specific information source discovery method is used to find forestry business website for test and performs well.

Key words: website topic, feature description, classification, crawler, information source discovery