计算机工程与应用 ›› 2008, Vol. 44 ›› Issue (20): 145-147.DOI: 10.3778/j.issn.1002-8331.2008.20.044

• 数据库、信号与信息处理 • 上一篇    下一篇

基于类信息的文本特征选择与加权算法研究

吕震宇1,林永民1,赵 爽1,陈景年2,朱卫东2   

  1. 1.河北理工大学 经济管理学院,河北 唐山 063009
    2.北京交通大学 计算机与信息技术学院,北京 100044
  • 收稿日期:2007-10-09 修回日期:2007-12-24 出版日期:2008-07-11 发布日期:2008-07-11
  • 通讯作者: 吕震宇

Research on algorithm of text feature selection and weighting based on category

LV Zheng-yu1,LIN Yong-min1,ZHAO Shuang1,CHEN Jing-nian2,ZHU Wei-dong2   

  1. 1.College of Economics and Management,Hebei Polytechnic University,Tangshan,Hebei 063009,China
    2.School of Computer and Information Technology,Beijing Jiaotong University,Beijing 100044,China
  • Received:2007-10-09 Revised:2007-12-24 Online:2008-07-11 Published:2008-07-11
  • Contact: LV Zheng-yu

摘要: 文本自动分类中特征选择和加权的目的是为了降低文本特征空间维数、去除噪音和提高分类精度。传统的特征选择方案筛选出的特征往往偏爱类分布不均匀文档集中的大类,而常用的TF·IDF特征加权方案仅考虑了特征与文档的关系,缺乏对特征与类别关系的考虑。针对上述问题,提出了基于类别信息的特征选择与加权方法,在两个不同的语料集上进行比较和分析实验,结果显示基于类别信息的特征选择与加权方法比传统方法在处理类分布不均匀的文档集时能有效提高分类精度,并且降维程度有所提高。

关键词: 文本分类, 特征选择, 特征加权, 基尼指数

Abstract: The aim of feature selection and weighting in automatic text categorization is to reduce the dimension of feature space,remove noise features and improve classification precision.The features selected by traditional feature selection methods always bias common category,and the commonly used weighting method TF*IDF only considers the relationship between features and documents and ignores the relationship between features and categories.According to the above problem,this paper presents a text feature selection and weighting method based on category.Experiments on skewed category distribution corpus of two different languages show that the method can improve categorization precision effectively,and comparing with traditional method,the feature space dimension is also reduced to a certain degree.

Key words: text categorization, feature selection, feature weighting, Gini-Index