计算机工程与应用 ›› 2009, Vol. 45 ›› Issue (1): 140-143.DOI: 10.3778/j.issn.1002-8331.2009.01.044

• 数据库、信号与信息处理 • 上一篇    下一篇

节点频度和语义距离相结合的网页正文信息抽取

孟 军,刘秋水,王秀坤   

  1. 大连理工大学 计算机科学与工程系,辽宁 大连 116023
  • 收稿日期:2008-07-24 修回日期:2008-10-16 出版日期:2009-01-01 发布日期:2009-01-01
  • 通讯作者: 孟 军

Combing node frequency and semantic feature for webpage informative content extraction

MENG Jun,LIU Qiu-shui,WANG Xiu-kun   

  1. Department of Computer Science and Engineering,Dalian University of Technology,Dalian 116023,China
  • Received:2008-07-24 Revised:2008-10-16 Online:2009-01-01 Published:2009-01-01
  • Contact: MENG Jun

摘要: 提出了一种带有节点频度的扩展DOM树模型—BF-DOM树模型(Block node Frequency-Document Object Module),并基于此模型进行网页正文信息的抽取。该方法通过向DOM树的某些节点上添加频度和相关度属性来构造文中新的模型,再结合语义距离抽取网页正文信息。方法主要基于以下三点考虑:在同源的网页集合内噪音节点的频度值很高;正文信息一般由非链接文字组成;与正文相关的链接和文章标题有较近的语义距离。针对8个网站的实验表明,该方法能有效地抽取正文信息,召回率和准确率都在96%以上,优于基于信息熵的抽取方法。

关键词: 信息提取, 带有节点频度的文档对象模型树, 节点频度, 语义距离

Abstract: A new module named BF-DOM tree is proposed in this paper,which extends the Document Object Module Tree by adding two properties,i.e.,block node frequency and relativity,to some nodes.Using this module combined with semantic distance,this method extracts the primary content accurately from the same source based on three facts:noise nodes always have high node frequency property within a given website;primary content blocks are often made up of few link words and many text words;useful links are contained in a useful content blocks and have a close semantic distance with page titles.Experiment on eight respective websites shows the proposed method can identify the primary content blocks with higher precision and recall rate both above 96% which is better than the entropy based method.The method can reduce the storage requirement for search engines;thus,result in smaller indexes,faster search time,and better user satisfaction.

Key words: information extraction, Block node Frequency-Document Object Module(BF-DOM) tree, node frequency, semantic distance