Feature selection method combined on optimized document frequency with LSA

doi:10.3778/j.issn.1002-8331.2009.34.037

Computer Engineering and Applications ›› 2009, Vol. 45 ›› Issue (34): 121-123.DOI: 10.3778/j.issn.1002-8331.2009.34.037

• 数据库、信号与信息处理 • Previous Articles Next Articles

Feature selection method combined on optimized document frequency with LSA

ZHU Hao-dong^1，2，ZHONG Yong^1，2

1.Chengdu Institute of Computer Application，Chinese Academy of Sciences，Chengdu 610041，China
2.Graduate University of Chinese Academy of Sciences，Beijing 100039，China

Received:2008-12-09 Revised:2009-02-27 Online:2009-12-01 Published:2009-12-01
Contact: ZHU Hao-dong

结合优化的文档频和LSA的特征选择方法

朱颢东^1，2，钟勇^1，2

1.中国科学院成都计算机应用研究所，成都 610041
2.中国科学院研究生院，北京 100039

通讯作者: 朱颢东

Abstract

Abstract: In order to improve efficiency and accuracy of text categorization algorithms，feature selection algorithm must be used.However，a number of feature selection algorithms selected features by means of weights and do not take into consideration features of hidden relationship，so selected feature subset has some redundancy and is not better representative.This paper presents document frequency method based on minimum word frequency and uses this method to filter out some terms to reduce the sparsity of text matrix，then LSA method is used to analyze semanteme among words and to eliminate the influence of synonyms and polysemous words．The combined method raises the speed and accuracy of text categorization．The experimental results show that the combined method is promising.

Key words: text categorization, word frequency, document frequency, Latent Semantic Analysis（LSA）

摘要： 为了提高文本分类算法的效率和精度，必须使用特征选择算法来降低特征空间的维数。然而许多常用特征选择算法在选择属性时，只是利用特征的权重而并没有考虑特征之间的隐含关系，使得得到的特征集存在一定的冗余，并不具备较好的代表性。首先给出了一个基于最小词频的文档频方法，并用它过滤掉一些词条以降低文本矩阵的稀疏性，然后使用LSA进行词语间的语义分析，消除同义词和多义词的影响，提高了文本分类的速度与精确度。实验结果表明此种特征选择方法效果良好。

关键词: 文本分类, 词频, 文档频, 潜在语义分析

CLC Number:

TP301

ZHU Hao-dong^1，2，ZHONG Yong^1，2. Feature selection method combined on optimized document frequency with LSA[J]. Computer Engineering and Applications, 2009, 45(34): 121-123.

朱颢东^1，2，钟勇^1，2. 结合优化的文档频和LSA的特征选择方法[J]. 计算机工程与应用, 2009, 45(34): 121-123.

[1]	SHEN Yanguang, JIA Yaoqing. Text Categorization Method Based on Word Co-occurrence and Graph Convolution [J]. Computer Engineering and Applications, 2021, 57(11): 173-178.
[2]	CAO Junbo，YE Xia，XU Feixiang，YIN Liedong. Improved CBOW Emotional Information Acquisition Research [J]. Computer Engineering and Applications, 2020, 56(9): 142-147.
[3]	ZHANG Hang, SHENG Zhiwei, ZHANG Shibin, YANG Min. Application of Simhash Algorithm in Text Deduplication [J]. Computer Engineering and Applications, 2020, 56(11): 246-251.
[4]	YE Xuemei1，2, MAO Xuemin1，2, XIA Jinchun1，2, WANG Bo1，2. Improved Approach to TF-IDF Algorithm in Text Classification [J]. Computer Engineering and Applications, 2019, 55(2): 104-109.
[5]	LIU Haifeng, LIU Shousheng, SONG Aling. Improved method of IG feature selection based on word frequency distribution [J]. Computer Engineering and Applications, 2017, 53(4): 113-117.
[6]	HAN Junbing, Halidan·Abudureyimu, Gulnur·Arken, HE Yan. Improved information gain algorithm based on Uyghur feature selection [J]. Computer Engineering and Applications, 2017, 53(23): 34-38.
[7]	WEI Wen1, YANG Huihua1，2, LI Lingqiao1，2, YANG Hao1, HE Shengtao3. Feature generation and selection method for short text of urban management cases and its application [J]. Computer Engineering and Applications, 2017, 53(18): 115-120.
[8]	SHI Qingwei, CONG Shiyuan. Research on text categorization based on mRMR and LDA [J]. Computer Engineering and Applications, 2016, 52(5): 127-133.
[9]	ZHENG Cheng1，2, WU Wenxiu1，2, DAI Ning1，2. Improved short text classification method based on BTM topic features [J]. Computer Engineering and Applications, 2016, 52(13): 95-100.
[10]	SHI Wenjuan, LONG Shun, YUN Fei. Iterative?text?classification?framework based?on?background learning [J]. Computer Engineering and Applications, 2015, 51(9): 129-134.
[11]	ZHENG Anyi. Improved term weighting algorithm for text sentiment analysis [J]. Computer Engineering and Applications, 2015, 51(21): 30-35.
[12]	LIU Haifeng, LIU Shousheng, SU Zhan. Sample cutting and weighting method in text classification based on position [J]. Computer Engineering and Applications, 2015, 51(2): 131-135.
[13]	FAN Xiaochao1，2, ZHANG Chongyang1, DENG Xiongwei1. Text feature weighting method based on mutual information [J]. Computer Engineering and Applications, 2015, 51(13): 145-148.
[14]	XU Chenkai, GAO Maoting. Improved ART2 neural network for text clustering based on LSA [J]. Computer Engineering and Applications, 2014, 50(24): 133-138.
[15]	MA Wenwen1, WEI Wenhan1, DEGN Yigui1，2. Micro-blog topic detection method based on Latent Semantic Analysis [J]. Computer Engineering and Applications, 2014, 50(1): 96-100.

Feature selection method combined on optimized document frequency with LSA

结合优化的文档频和LSA的特征选择方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics