Multi-document summary using LDA and spectral clustering

Abstract

Abstract: Automatic summarization aims to compress lengthy document into a few short paragraphs, offers comprehensive and concise information to the users and improves the efficiency and accuracy of the information. A summarization method based on Latent Dirichlet Allocation（LDA） is proposed, using Gibbs sampling to estimate the word probability on topics and topic probability on sentences, combing with the LDA parameters and spectral clustering algorithm to extract multi-document summarization. The proposed approach uses a linear formula to integrate the sentence weights, extracting 400-words multi-document summarization. The experimental results show that the proposed method can improve the quality of summary effectively with the automatic summarization evaluation toolkit ROUGE on DUC2002.

Key words: Latent Dirichlet Allocation（LDA）, Gibbs sampling, spectral clustering, multi-document summary

摘要： 自动文摘技术的目标是致力于将冗长的文档内容压缩成较为简短的几段话，将信息全面、简洁地呈现给用户，提高用户获取信息的效率和准确率。所提出的方法在LDA（Latent Dirichlet Allocation）的基础上，使用Gibbs抽样估计主题在单词上的概率分布和句子在主题上的概率分布，结合LDA参数和谱聚类算法提取多文档摘要。该方法使用线性公式来整合句子权重，提取出字数为400字的多文档摘要。使用ROUGE自动摘要评测工具包对DUC2002数据集评测摘要质量，结果表明，该方法能有效地提高摘要的质量。

关键词: Latent Dirichlet Allocation（LDA）, Gibbs抽样, 谱聚类, 多文档摘要

FU Ling, ZHANG Hui. Multi-document summary using LDA and spectral clustering[J]. Computer Engineering and Applications, 2013, 49(16): 142-145.

付玲，张晖. 结合LDA和谱聚类的多文档摘要[J]. 计算机工程与应用, 2013, 49(16): 142-145.

[1]	ZHAO Fan, ZHANG Lin, WEN Zhiquan, YANG Linlin, LIN Guangfeng. Direct and Efficient Natural Scene Chinese Character Approaching Spotting Method [J]. Computer Engineering and Applications, 2021, 57(6): 159-167.
[2]	BAI Lu, ZHAO Xin, KONG Yuting, ZHANG Zhenghang, SHAO Jinxin, QIAN Yurong. Survey of Spectral Clustering Algorithms [J]. Computer Engineering and Applications, 2021, 57(14): 15-26.
[3]	HU Can, CUI Xiaohui. Research on SNS Users’ Posting Behavior and Interest Prediction [J]. Computer Engineering and Applications, 2020, 56(9): 99-105.
[4]	WANG Yusi, LU Deyang, LI Haiyang. Sparse Subspace Clustering Method Based on Fractional Function Constraints [J]. Computer Engineering and Applications, 2020, 56(7): 39-47.
[5]	ZHANG Weiwei, HU Yaqi, ZHAI Guangyu, LIU Zhipeng. Academic Abstract Clustering Method Based on LDA Model and Doc2vec [J]. Computer Engineering and Applications, 2020, 56(6): 180-185.
[6]	ZHU Dan, CHEN Xiaohong, WU Qingyuan, LI Shunming. Subspace Clustering Induced by Adaptive Graph Learning [J]. Computer Engineering and Applications, 2020, 56(21): 30-37.
[7]	YANG Jingya, SUN Linfu, WU Qishi. After-Sales Customer Segmentation Based on Semi-Supervised Spectral Clustering Ensemble [J]. Computer Engineering and Applications, 2020, 56(2): 266-271.
[8]	CHEN Huan, HUANG Bo, ZHU Yimin, YU Lei, YU Yuxin. Short Text Emotion Classification Method Combining LDA and Self-Attention [J]. Computer Engineering and Applications, 2020, 56(18): 165-170.
[9]	YANG Lu, SONG Huansheng, ZHANG Zhaoyang. Highway Vehicle Detection Based on Sparse Trajectory Clustering [J]. Computer Engineering and Applications, 2020, 56(15): 251-258.
[10]	JIANG Yirui, PEI Yang, CHEN Lei, WANG Wenle, DAI Jiangyan, YI Yugen. Multiple Locality-Constrained Self-Representation for Spectral Clustering [J]. Computer Engineering and Applications, 2020, 56(11): 172-178.
[11]	QIN Xu, YANG Wenzhong, WANG Xueying, MA Guoxiang, WANG Qingpeng. Multi-source Topic Fusion Model Based on Co-occurrence Relation [J]. Computer Engineering and Applications, 2020, 56(10): 157-162.
[12]	ZHU Hongzhen1, CHEN Pinghua1, CAI Guilan2. Research on Application of Relationship Mining into Red Wine Data Based on LDA Model [J]. Computer Engineering and Applications, 2019, 55(4): 148-153.
[13]	WANG Xiaoyu1, DING Shifei1，2. Pairwise Constrained Spectral Clustering Algorithm Based on Shared Nearest Neighborhood [J]. Computer Engineering and Applications, 2019, 55(2): 142-147.
[14]	ZHOU Xiaoyu, ZHANG Longbo, WANG Lei, LI Xinxiang. Medical Image Segmentation Combined with Spectral Clustering and Improved RSF Model [J]. Computer Engineering and Applications, 2019, 55(15): 193-197.
[15]	DU Tingting, WEN Guoqiu, WU Lin, TONG Tao, TAN Malong. Spectral Clustering Algorithm Based on Local Covariance Matrix [J]. Computer Engineering and Applications, 2019, 55(14): 148-154.

Multi-document summary using LDA and spectral clustering

结合LDA和谱聚类的多文档摘要

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics