计算机工程与应用 ›› 2012, Vol. 48 ›› Issue (23): 139-142.

• 数据库、信号与信息处理 • 上一篇    下一篇

中文分词词典结构的研究与改进

叶继平,张桂珠   

  1. 江南大学 物联网工程学院,江苏 无锡 214122
  • 出版日期:2012-08-11 发布日期:2012-08-21

Research and improvement of Chinese word segmentation dictionary

YE Jiping, ZHANG Guizhu   

  1. School of IOT Engineering, Jiangnan University, Wuxi, Jiangsu 214122, China
  • Online:2012-08-11 Published:2012-08-21

摘要: 汉语分词词典是中文信息处理系统的重要基础,词典算法设计的优劣直接关系着分词的速度和效率。分析了三种典型的分词词典结构,提出了一种具有三级索引的新词典结构,并提出了最大正向匹配的改进型匹配算法,从而降低了匹配过程的时间复杂度。最后通过实验,比较了三种典型词典结构与新词典结构的时间效率。实验结果表明,新词典结构具有更高的词典查询速度和分词速度,可以有效满足中文处理系统的需求。

关键词: 中文分词, 正向最大匹配, 词典, 三级索引

Abstract: Chinese word segmentation dictionary is the important base of Chinese information processing system. The arithmetic of the dictionary influences the speed and efficiency of segmentation. This paper analyzes the three typical dictionary structures, and then presents a new dictionary structure using three-level index. On the basis of this new structure, this paper also improves the positive matching method, reduces the complexity of matching process. Then a comparison is given between typical dictionary structures. The result proves that the new dictionary structure is better in many aspects such as the efficiency of retrieving and the speed of Chinese word segmentation and it can fulfill the need of Chinese processing system.

Key words: Chinese word segmentation, positive matching method, word dictionary, three-level index