Research on combinational ambiguity strings in Chinese word segmentation

Computer Engineering and Applications ›› 2011, Vol. 47 ›› Issue (31): 125-127.

• 数据库、信号与信息处理 • Previous Articles Next Articles

Research on combinational ambiguity strings in Chinese word segmentation

YOU Huili，YAN Li，YANG Xiaodong

School of Computer Science and Telecommunication Engineering，Jiangsu University，Zhenjiang，Jiangsu 212013，China

Received:1900-01-01 Revised:1900-01-01 Online:2011-11-01 Published:2011-11-01

中文分词中组合型切分歧义的消解研究

尤慧丽，晏立，杨晓东

江苏大学计算机科学与通信工程学院，江苏镇江 212013

Abstract

Abstract: Combinational ambiguity is one of the most difficult problems in Chinese word segmentation.The paper discusses a new segmentation algorithm to solve the combinational ambiguity.The algorithm automatically extracts contextual information of the combinational ambiguity to establish rules，then uses the C-SVM model and these rules to solve the combinational ambiguity.The People Daily corpus of January 1998 is used in training and testing，and the average accuracy is 89.33%.

Key words: Chinese word segmentation, combinational ambiguity, contextual information, C-Support Vector Machine（C-SVM）

摘要： 针对中文自动分词中组合型歧义消解难的问题，提出了一种新的切分算法来对组合型歧义字段进行消歧。该算法首先自动从训练语料中提取歧义字段的上下文信息来建立规则库，然后利用C-SVM模型结合规则对组合型歧义字段进行歧义消解。最后以1998年1月《人民日报》语料中出现的组合型歧义字段进行训练和测试，实验显示消歧的平均准确率达89.33%。

关键词: 中文自动分词, 组合型歧义, 上下文信息, C-支持向量机

YOU Huili，YAN Li，YANG Xiaodong. Research on combinational ambiguity strings in Chinese word segmentation[J]. Computer Engineering and Applications, 2011, 47(31): 125-127.

尤慧丽，晏立，杨晓东. 中文分词中组合型切分歧义的消解研究[J]. 计算机工程与应用, 2011, 47(31): 125-127.

[1]	TU Wenbo, YUAN Zhenming, YU Kai. Convolutional Neural Networks Without Pooling Layer for Chinese Word Segmentation [J]. Computer Engineering and Applications, 2020, 56(2): 120-126.
[2]	SUN Baoshan, LI Wei. Recurrent Neural Network for Chinese Word Segmentation with Peephole-Connections [J]. Computer Engineering and Applications, 2019, 55(19): 160-165.
[3]	CHENG Yusi1, SHI Yuntao2. Domain specific Chinese word segmentation [J]. Computer Engineering and Applications, 2018, 54(17): 30-34.
[4]	ZHAO Weifeng1，2, ZHANG Qin1. Automatic identification of address description in unstructured Chinese natural language [J]. Computer Engineering and Applications, 2016, 52(23): 19-24.
[5]	ZHU Yanhui, LIU Jing, XU Yeqiang, TIAN Hailong, MA Jin. Chinese word segmentation research based on Conditional Random Field [J]. Computer Engineering and Applications, 2016, 52(15): 97-100.
[6]	ZHANG Sifa, MA Yongge. Design and implementation of vertical search engine for field of geosciences [J]. Computer Engineering and Applications, 2012, 48(33): 85-88.
[7]	YE Jiping, ZHANG Guizhu. Research and improvement of Chinese word segmentation dictionary [J]. Computer Engineering and Applications, 2012, 48(23): 139-142.
[8]	ZHAO Youqiao1, ZHANG Shanshan1, LU Songfeng1, WU Zhijie2. COX：Chinese-oriented XML compressor with high compression ratio [J]. Computer Engineering and Applications, 2012, 48(17): 143-147.
[9]	LI Guohe1，2，3, LIU Guangsheng1，2，3, QIN Bobo1，2，3, WU Weijiang1，2，3, LI Hongqi1，2，3. Method of Chinese word rough segmentation by maximum match and ambiguity detection algorithms [J]. Computer Engineering and Applications, 2012, 48(14): 139-142.
[10]	YU Jiangde¹，WANG Xijie¹，FAN Xiaozhong². Comparing of importance of above-context versus below-context for Chinese word segmentation [J]. Computer Engineering and Applications, 2011, 47(4): 117-120.
[11]	LIU Ronghui^1，2，ZHENG Jianguo¹. Clustering algorithm in Deep Web based on Chinese word segmentation [J]. Computer Engineering and Applications, 2011, 47(4): 138-140.
[12]	ZHANG Jin-song，YUAN Jian. Backtracking matching Chinese segmentation method [J]. Computer Engineering and Applications, 2009, 45(22): 132-134.
[13]	ZHANG Qing-yang,CHAI Sheng. Chinese word segmentation dictionary using two-level index [J]. Computer Engineering and Applications, 2009, 45(19): 139-141.
[14]	LIU Dan,FANG Wei-guo,ZHOU Hong. Performance of smoothing algorithm in Chinese word segmentation by bigram [J]. Computer Engineering and Applications, 2009, 45(17): 33-36.
[15]	ZHOU Sheng^1,2,HU Xiao-feng¹,LUO Pi¹. Research on automatization of virtual news system [J]. Computer Engineering and Applications, 2008, 44(36): 20-23.

Research on combinational ambiguity strings in Chinese word segmentation

中文分词中组合型切分歧义的消解研究

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics