Abstract: Similar web pages that search engine returns not only waste storage resources but also increase the burden on web users.In this paper，a method based on semantic to detect similar web pages is proposed.This method picks up topic sentence vector of web pages through location of the sentence in the text and importance of chunking.Then it detects the similar web pages by calculating semantic similar degree of topic sentence vector.The experiment results show that not only completely similar web pages are detected accurately but also partly similar web pages are detected exactly.

摘要： 搜索引擎返回的重复网页不但浪费了存储资源，而且加重了用户浏览的负担。针对网页重复的特征，提出了一种基于语义的去重方法。该方法通过句子在文本中的位置和组块的重要度，提取出网页正文的主题句向量，然后对主题句向量进行语义相似度计算，把重复的网页去除。实验证明，该方法对全文重复和部分重复的网页都能进行较准确的检测。