Method of Chinese word rough segmentation by maximum match and ambiguity detection algorithms

Computer Engineering and Applications ›› 2012, Vol. 48 ›› Issue (14): 139-142.

Previous Articles Next Articles

Method of Chinese word rough segmentation by maximum match and ambiguity detection algorithms

LI Guohe1，2，3, LIU Guangsheng1，2，3, QIN Bobo1，2，3, WU Weijiang1，2，3, LI Hongqi1，2，3

1.College of Geophysics and Information Engineering, China University of Petroleum, Beijing 102249, China
2.The State Key Lab of Petroleum Resource and Prospecting, China University of Petroleum, Beijing 102249, China
3.PanPass Institute of Digital Identification Management and Internet of Things, Beijing 100029, China

Online:2012-05-11 Published:2012-05-14

综合最大匹配和歧义检测的中文分词粗分方法

李国和1，2，3，刘光胜1，2，3，秦波波1，2，3，吴卫江1，2，3，李洪奇1，2，3

1.中国石油大学地球物理与信息工程学院，北京 102249
2.中国石油大学（北京）油气资源与探测国家重点实验室，北京 102249
3.石大兆信数字身份管理与物联网技术研究院，北京 100029

Abstract

Abstract: Segmentation of words in Chinese text is very important preprocessing in Chinese information processing. In present, for some demerits such as low accuracy of Chinese word segmentation and big set of Chinese word rough segmentation, a method, CWRS, based on maximal match algorithm is proposed along with omni-segmentation algorithm. It greatly improves the accuracy and reduces the set of rough segmentation according to combination of ambiguity detection and cross ambiguity detection, which lays the foundation for precise segmentation of words in Chinese text. All the experiments are good effects by comparison of CWRS with other algorithms on the same data set of common Chinese texts.

Key words: Chinese word segmentation, rough segmentation, maximum match algorithm, omni-segmentation algorithm, ambiguity detection

摘要： 中文分词是中文文本信息处理的重要预处理。针对目前中文分词中存在的准确率低和粗分结果集大的问题，在最大匹配算法基础上，采用文本切分时的组合歧义检测和交叉歧义检测以及全切分算法，提高了文本粗分的准确率，并减小了粗分结果集的规模，为进一步正确分词奠定基础。通过公共语料库数据集的实验对比，取得很好的效果。

关键词: 中文分词, 粗分, 最大匹配算法, 全切分算法, 歧义检测

LI Guohe1，2，3, LIU Guangsheng1，2，3, QIN Bobo1，2，3, WU Weijiang1，2，3, LI Hongqi1，2，3. Method of Chinese word rough segmentation by maximum match and ambiguity detection algorithms[J]. Computer Engineering and Applications, 2012, 48(14): 139-142.

李国和1，2，3，刘光胜1，2，3，秦波波1，2，3，吴卫江1，2，3，李洪奇1，2，3. 综合最大匹配和歧义检测的中文分词粗分方法[J]. 计算机工程与应用, 2012, 48(14): 139-142.

[1]	TU Wenbo, YUAN Zhenming, YU Kai. Convolutional Neural Networks Without Pooling Layer for Chinese Word Segmentation [J]. Computer Engineering and Applications, 2020, 56(2): 120-126.
[2]	SUN Baoshan, LI Wei. Recurrent Neural Network for Chinese Word Segmentation with Peephole-Connections [J]. Computer Engineering and Applications, 2019, 55(19): 160-165.
[3]	CHENG Yusi1, SHI Yuntao2. Domain specific Chinese word segmentation [J]. Computer Engineering and Applications, 2018, 54(17): 30-34.
[4]	ZHAO Weifeng1，2, ZHANG Qin1. Automatic identification of address description in unstructured Chinese natural language [J]. Computer Engineering and Applications, 2016, 52(23): 19-24.
[5]	ZHU Yanhui, LIU Jing, XU Yeqiang, TIAN Hailong, MA Jin. Chinese word segmentation research based on Conditional Random Field [J]. Computer Engineering and Applications, 2016, 52(15): 97-100.
[6]	ZHANG Sifa, MA Yongge. Design and implementation of vertical search engine for field of geosciences [J]. Computer Engineering and Applications, 2012, 48(33): 85-88.
[7]	YE Jiping, ZHANG Guizhu. Research and improvement of Chinese word segmentation dictionary [J]. Computer Engineering and Applications, 2012, 48(23): 139-142.
[8]	ZHAO Youqiao1, ZHANG Shanshan1, LU Songfeng1, WU Zhijie2. COX：Chinese-oriented XML compressor with high compression ratio [J]. Computer Engineering and Applications, 2012, 48(17): 143-147.
[9]	YU Jiangde¹，WANG Xijie¹，FAN Xiaozhong². Comparing of importance of above-context versus below-context for Chinese word segmentation [J]. Computer Engineering and Applications, 2011, 47(4): 117-120.
[10]	LIU Ronghui^1，2，ZHENG Jianguo¹. Clustering algorithm in Deep Web based on Chinese word segmentation [J]. Computer Engineering and Applications, 2011, 47(4): 138-140.
[11]	YOU Huili，YAN Li，YANG Xiaodong. Research on combinational ambiguity strings in Chinese word segmentation [J]. Computer Engineering and Applications, 2011, 47(31): 125-127.
[12]	ZHANG Jin-song，YUAN Jian. Backtracking matching Chinese segmentation method [J]. Computer Engineering and Applications, 2009, 45(22): 132-134.
[13]	ZHANG Qing-yang,CHAI Sheng. Chinese word segmentation dictionary using two-level index [J]. Computer Engineering and Applications, 2009, 45(19): 139-141.
[14]	LIU Dan,FANG Wei-guo,ZHOU Hong. Performance of smoothing algorithm in Chinese word segmentation by bigram [J]. Computer Engineering and Applications, 2009, 45(17): 33-36.
[15]	ZHOU Sheng^1,2,HU Xiao-feng¹,LUO Pi¹. Research on automatization of virtual news system [J]. Computer Engineering and Applications, 2008, 44(36): 20-23.

Method of Chinese word rough segmentation by maximum match and ambiguity detection algorithms

综合最大匹配和歧义检测的中文分词粗分方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics