Abstract: In the homepage retrieval result,the user can obtain the content same redundant page frequently.This article proposes one kind of duplicated news web pages removal algorithm though study news content on elements of news subject.This method basic thought is：First,extracts the time and the place phrase which in the news essential factor the event occurs;Then,through extraction time and place phrase extraction news content;Finally,through calculates their similarity according to the study news content to judge the news homepage the heavy multiplicity.The experimental result indicates that,this method can complete in view of the news content duplicated web pages,and obtains the high recall and the accuracy ratio.

Key words: elements of news subject, fuzzy matching, duplicate removal algorithm

摘要： 网页检索结果中,用户经常会得到内容相同的冗余页面。提出了一种通过新闻主题要素学习新闻内容的新闻网页去重算法。该方法的基本思想是：首先,抽取新闻要素中关于事件发生的时间和地点短语;然后,通过抽取的时间和地点短语抽取新闻的内容;最终,根据学习的新闻内容通过计算它们的相似度来判断新闻网页的重复度。实验结果表明,该方法能够完成针对新闻内容的新闻网页的去重,并得到较高的查全率和查准率。

关键词: 新闻主题要素, 模糊匹配, 去重算法