计算机工程与应用 ›› 2014, Vol. 50 ›› Issue (2): 124-128.

• 数据库、数据挖掘、机器学习 • 上一篇    下一篇

基于改进最大匹配算法的中文分词粗分方法

周  俊1,3,郑中华2,张  炜3   

  1. 1.华中科技大学 模具技术国家重点实验室,武汉 430074
    2.中国人民大学 教育学院,北京 100872
    3.安徽博约信息科技有限责任公司,合肥 230088
  • 出版日期:2014-01-15 发布日期:2014-01-26

Method of Chinese words rough segmentation based on improving maximum match algorithm

ZHOU Jun1,3, ZHENG Zhonghua2, ZHANG Wei3   

  1. 1.State Key Lab of Mold Technology, Huazhong University of Science and Technology, Wuhan 430074, China
    2.School of Education, Renmin University of China, Beijing 100872, China
    3.Anhui Boryou Information Technology Co.Ltd, Hefei 230088, China
  • Online:2014-01-15 Published:2014-01-26

摘要: 中文粗分和歧义消解是中文分词的两大基本过程。通过引入广义词条和诱导词集,在最大匹配算法基础上提出一种中文分词的粗分方法,以最长广义词匹配为原则进行中文分词,利用诱导词集实现交叉型歧义识别。在保证快速准确切分无歧义汉语语句的同时,100%检测并标记有歧义汉语语句中的交叉型歧义,最大程度上简化后续歧义消解过程。通过对含有160万汉字1998年1月人民日报语料测试的结果证明了算法速度、歧义词准确率以及粗分召回率的有效性。

关键词: 中文分词, 最大匹配, 广义词, 诱导词集

Abstract: Chinese words rough segmentation and ambiguity resolution are two fundamental processes of Chinese word segmentation. Under the introduction of generalized term and induced word set, a method used for Chinese words rough segmentation is proposed based on maximum matching method. It executes Chinese word segmentation under the principle of the longest generalized term matching, and recognizes the overlapping ambiguities by utilizing induced word set. It segments Chinese sentences without any ambiguity rapidly and accurately, detects and marks ambiguities by 100 percent in those sentences which have ambiguities, which will simplify the process of ambiguity resolution to the maximum extent. The result of the experiment on People’s Daily corpus in January 1998 which contains 1.6 million Chinese characters shows the method is effective both in speed and accuracy.

Key words: Chinese words segmentation, maximum match, generalized term, induced word set