Abstract: The new words identification based on large scale corpora is a basis task in Chinese automatic processing. There are many difficulties because the study needs not only processing large scale corpora rapidly, but also requiring much intellectual methods. Based on lots of surveys and researches, it constructs a framework of new Chinese words identification from large scale network corpora, which includes the repeat extraction algorithm based on hierarchical pruning, the new word detection method based on statistical learning and the POS guessing method based on combined features. Through lots of experiments and analyses, the framework can extract repeats from large scale corpora and construct the set of candidate new words rapidly, and can carry out the task of new words detecting and POS guessing with high efficiency and good results.

Key words: large scale corpora, repeat, hierarchical pruning algorithm, new words detection, combined features

摘要： 基于海量语料的热点新词识别是汉语自动处理领域的一项基础性课题，因要求快速处理大规模语料，且在新词检测中需要更多智力因素，在研究中存在较多困难。构建了一个基于海量语料的网络热点新词识别框架，整合了所提出的基于逐层剪枝算法的重复模式提取，基于统计学习模型的新词检测及基于组合特征的新词词性猜测等3个重要算法，用以提高新词识别的处理能力和识别效果。实验和数据分析表明，该框架能高效可靠地从大规模语料中提取重复模式，构造候选新词集合，并能有效实施新词检测和新词属性识别任务，处理效果达到了目前的较好水平。

关键词: 海量语料, 重复模式, 逐层剪枝算法, 新词检测, 组合特征