关键词: 新闻网页, 主题内容抽取, 网页去重, 权值计算

Abstract: In the homepage retrieval result, users often get the redundant page with same content. It has not only wasted the storing resources, but also brought a great deal of inconvenience to information retrieval or other text-processing. We first extract the news title, the subject content and the issue date in this article, then divide group according to data issued on the basis of news fragility and conduct the exploration research to duplicated web pages removal. It greatly reduced the computing time, enhanced the duplicated news webpages deletion accuracy.

Key words: news webpages, theme's extraction, duplicated web pages removal, weight calculating