Performance of smoothing algorithm in Chinese word segmentation by bigram

doi:10.3778/j.issn.1002-8331.2009.17.010

Computer Engineering and Applications ›› 2009, Vol. 45 ›› Issue (17): 33-36.DOI: 10.3778/j.issn.1002-8331.2009.17.010

• 博士论坛 • Previous Articles Next Articles

Performance of smoothing algorithm in Chinese word segmentation by bigram

LIU Dan,FANG Wei-guo,ZHOU Hong

School of Economy and Management，Beihang University，Beijing 100191，China

Received:2009-02-02 Revised:2009-03-09 Online:2009-06-11 Published:2009-06-11
Contact: LIU Dan

二元语法中文分词数据平滑算法性能研究

刘丹,方卫国,周泓

北京航空航天大学经济管理学院，北京 100191

通讯作者: 刘丹

Abstract

Abstract: This paper discusses the relationships between complexity and real performance based on the corpus of People’s Daily of January，1998，compares the performance of multiple smoothing algorithms.The result reveals that additive smoothing is the best with 99.68% on precision，99.7% on recall in close test，and 98.64% on precision，98.74% on recall in open test.

Key words: smoothing, Chinese word segmentation, bigram

摘要： 将多种平滑算法应用于基于二元语法的中文分词，在1998年1月人民日报语料库的基础上，讨论了困惑度和实际分词性能之间的关系，对比分析各平滑算法的实际性能，结果表明，简单的加值平滑算法性能最优，封闭精度、召回率分别为99.68%、99.7%，开放精度、召回率为98.64%、98.74%。

关键词: 数据平滑, 中文分词, 二元语法

LIU Dan,FANG Wei-guo,ZHOU Hong. Performance of smoothing algorithm in Chinese word segmentation by bigram[J]. Computer Engineering and Applications, 2009, 45(17): 33-36.

刘丹,方卫国,周泓. 二元语法中文分词数据平滑算法性能研究[J]. 计算机工程与应用, 2009, 45(17): 33-36.

[1]	ZHANG Hanyu, WU Zhihao, XU Yong, CHEN Bin. Face Forensics Detection Method Based on Enhanced Convolutional Neural Networks [J]. Computer Engineering and Applications, 2021, 57(8): 220-224.
[2]	PENG Haoyue, QIN Xiaolin,HOU Yu, ZHANG Lige. Adaptive B-Spline Algorithm for Multi-UAV Path Planning [J]. Computer Engineering and Applications, 2020, 56(9): 260-266.
[3]	YUAN Liangyou, ZHOU Hang, HAN Dan, XU Guoliang. Improved Skeleton Extraction Algorithm with Smoothing Iterations [J]. Computer Engineering and Applications, 2020, 56(24): 188-193.
[4]	TU Wenbo, YUAN Zhenming, YU Kai. Convolutional Neural Networks Without Pooling Layer for Chinese Word Segmentation [J]. Computer Engineering and Applications, 2020, 56(2): 120-126.
[5]	YANG-ZHAO Qilin, PENG Dingtao, TANG Qi, LUO Xiaomin. Smoothing Quasi-Newton Algorithm for [lp] Regularization of Sparse Optimization [J]. Computer Engineering and Applications, 2019, 55(22): 163-171.
[6]	SUN Baoshan, LI Wei. Recurrent Neural Network for Chinese Word Segmentation with Peephole-Connections [J]. Computer Engineering and Applications, 2019, 55(19): 160-165.
[7]	REN Zhen1, LI Jiying1, WU Hao2. Single-Observer Tracking Algorithm Based on M-Estimation Robust Backward-Smoothing CKF [J]. Computer Engineering and Applications, 2019, 55(11): 74-79.
[8]	CHENG Yusi1, SHI Yuntao2. Domain specific Chinese word segmentation [J]. Computer Engineering and Applications, 2018, 54(17): 30-34.
[9]	WANG Bin, CHEN Lin, HOU Xiangyu, LI Weimin, SHENG Jinfang. Analysis and prediction for characteristics of user access behavior in transparent computing [J]. Computer Engineering and Applications, 2018, 54(16): 49-54.
[10]	ZHAO Yafei, WEI Guangmei. Study on finite element stress smothing based on BP neural network [J]. Computer Engineering and Applications, 2018, 54(13): 67-72.
[11]	XU Guangmei1, LIU Hongzhe2, ZHANG Jingzun1, WANG Jinhua1. Improving multi-relational Naive Bayesian classifier using smoothing methods [J]. Computer Engineering and Applications, 2017, 53(5): 69-72.
[12]	LIU Di1，2, ZHU Ligu1，2, ZHANG Lei1，2, FENG Dongyu1，2. Research of load-balancing model based on SSAWF mechanism [J]. Computer Engineering and Applications, 2017, 53(10): 90-95.
[13]	LI Juntao, WU Xiaokai. Dynamic consolidation of virtual machines based on Brown’s exponential smoothing [J]. Computer Engineering and Applications, 2016, 52(7): 56-61.
[14]	ZHAO Weifeng1，2, ZHANG Qin1. Automatic identification of address description in unstructured Chinese natural language [J]. Computer Engineering and Applications, 2016, 52(23): 19-24.
[15]	LI Pengfei, WU Hai’e, JING Junfeng, LI Renzhong. Noise classification denoising algorithm for point cloud model [J]. Computer Engineering and Applications, 2016, 52(20): 188-192.

Performance of smoothing algorithm in Chinese word segmentation by bigram

二元语法中文分词数据平滑算法性能研究

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics