计算机工程与应用 ›› 2010, Vol. 46 ›› Issue (24): 129-131.DOI: 10.3778/j.issn.1002-8331.2010.24.039

• 数据库、信号与信息处理 • 上一篇    下一篇

一种基于独立性统计的子串归并算法

周 浪1,2,冯 冲2,黄河燕2,王平尧3   

  1. 1.南京理工大学 计算机科学与技术学院,南京 210094
    2.中国科学院 计算机语言信息工程研究中心,北京 100097
    3.宁波职业技术学院 计算机系,浙江 宁波 315800
  • 收稿日期:2009-02-10 修回日期:2009-04-01 出版日期:2010-08-21 发布日期:2010-08-21
  • 通讯作者: 周 浪

Substring reduction algorithm based on independence statistic

ZHOU Lang1,2,FENG Chong2,HUANG He-yan2,WANG Ping-yao3   

  1. 1.School of Computer Science and Technology,Nanjing University of Science and Technology,Nanjing 210094,China
    2.Research Center of Computer & Language Information Engineering,Chinese Academy of Sciences,Beijing 100097,China
    3.Department of Computer,Ningbo Polytechnic,Ningbo,Zhejiang 315800,China
  • Received:2009-02-10 Revised:2009-04-01 Online:2010-08-21 Published:2010-08-21
  • Contact: ZHOU Lang

摘要: 现行的子串归并算法都是采用一对一的方式针对同频子串提出的。但是在使用词法分析工具对文本进行切分时,不可避免地会产生很多的分词碎片,这直接导致了很多无意义子串的产生。通过分析这些无意义子串和众多父串之间的这种一对多关系,提出了一种基于独立性统计的子串归并算法。最后将该子串归并算法应用在中文术语抽取系统中,使得系统的准确率从91.3%提升到了93.32%。

Abstract: The substring reduction algorithm applied in most cases is mainly focusing on the substrings having the same frequency with the parent string in one to one mode.After being processed by the morphological analysis tool,it’s unavoidable to product many segment fragments which compose many meaningless substrings.According to the analysis of the one to multiple relationship between the meaningless substring and its parent strings,a substring reduction algorithm based on independence statistic is proposed to filter these meaningless substrings.Finally,this substring reduction algorithm is applied in the Chinese multi-words terminology extraction system,and the precision of the term extraction results is improved from 91.3% to 93.32%.

中图分类号: