Video Summarization Generation Based on Self-attention Mechanism and Random Forest Regression

doi:10.3778/j.issn.1002-8331.2104-0388

Abstract

Abstract: Video summarization is compressed by generating key frames or fragments, which can greatly shorten the viewing time on the basis of summarizing the main content of the video, and is widely used in the field of video quick browsing and retrieval. Most existing methods only explore based on image content, ignoring the time-series feature of the video and the poor learning ability of the model to wave data, which leads to the lack of time coherence and representativeness of the generated summarization. This paper proposes a video summarization network based on encoder-decoder framework. In particular, the coding part extracts characteristics by the convolution neural network, uses the attention mechanism to improve the weight of key characteristics. And the decoding part is formed by fusing the random forest and bi-directional long short-term memory network, by adjusting the proportion of random forest and bi-directional long short-term memory network in the loss function, the model has strong stability and prediction accuracy. Compared with the other seven methods on two datasets, the experimental results show that the proposed method is effective and feasible. This paper proposes the self-attention mechanism and random forest regression video summarization network to optimize the features by using the self-attention mechanism, and combines the bi-directional long short-term memory network with random forest to improve the stability and generalization of the model, effectively reduces the loss value, and makes the generated video summarization more consistent with the visual characteristics of users.

Key words: computer vision, video summarization, self-attention mechanism, long short-term memory, random forest regression

摘要： 视频摘要是通过生成关键帧或片段来达到压缩视频的效果，能够在概括视频主要内容的基础上极大缩短观看时间，在视频快速浏览与检索领域应用广泛。现有方法大多只基于图像内容进行探索，忽略了视频具有时序的特点，且模型对波动数据学习能力较差，导致生成的摘要缺乏时间连贯性和代表性。提出了一个以编码器-解码器为框架的视频摘要网络。具体来说，编码部分由卷积神经网络提取特征，通过自注意力机制提升对关键特征的权重，而解码部分由融合了随机森林的双向长短期记忆网络构成，通过调整随机森林和双向长短期记忆网络在损失函数中所占比例，使模型具有较强的稳定性和预测准确率。实验在两个数据集上与其他七种方法进行了比较，综合实验结果证明了方法的有效性与可行性。提出了自注意力机制和随机森林回归的视频摘要网络，利用自注意力机制完成对特征的优化，将双向长短期记忆网络与随机森林结合，提升模型的稳定性与泛化性，有效降低损失值，使得生成的视频摘要更符合用户视觉特性。

关键词: 计算机视觉, 视频摘要, 自注意力机制, 长短期记忆网络, 随机森林回归

LI Leiting, WU Guangli, GUO Zhenzhou. Video Summarization Generation Based on Self-attention Mechanism and Random Forest Regression[J]. Computer Engineering and Applications, 2022, 58(4): 198-205.

李雷霆, 武光利, 郭振洲. 自注意力机制和随机森林回归的视频摘要生成[J]. 计算机工程与应用, 2022, 58(4): 198-205.

References

[1] WU G L，LI L T，YAO Y P，et al.Video summarization based on listnet scoring mechanism[C]//Proceedings of the 2020 IEEE 5th International Conference on Computer and Communication Systems，Shanghai，Feb 22-24，2020.Piscataway：IEEE，2020：281-285.
[2] 冀中，江俊杰.基于解码器注意力机制的视频摘要[J].天津大学学报，2018，51（10）：1023-1030.
JI Z，JIANG J J.Video summarization based on decoder attention mechanism[J].Transactions of Tianjin University，2018，51（10）：1023-1030.
[3] 费梦娟.基于用户兴趣和内容重要性学习的视频摘要技术研究[D].杭州：浙江大学，2019.
FEI M J.Research on video summary technology based on user interest and importance of content learning[D].Hangzhou：Zhejiang University，2019.
[4] GONG B，CHAO W L，GRAUMAN K，et al.Diverse sequential subset selection for supervised video summarization[C]//Advances in Neural Information Processing Systems，Canada，Dec 8-13，2014.Cambridge：MIT Press，2014：2069-2077.
[5] MAHASSENI B，LAM M，TODOROVIC S.Unsupervised video summarization with adversarial LSTM networks[C]//Proceedings of the 2017 IEEE/CVF Conference on Computer Vision and Pattern Recognition，Honolulu，Jul 21-26，2017.Piscataway：IEEE，2017：202-211.
[6] GYGLI M，GRABNER H，RIEMENSCHNEIDER H.Creating summaries from user videos[C]//Proceedings of the 13th European Conference on Computer Vision，Switzerland，Sep 6-12，2014.Cham：Springer，2014：505-520.
[7] NGO C W，MA Y F，ZHANG H J.Video summarization and scene detection by graph modeling[J].IEEE Transactions on Circuits and Systems for Video Technology，2005，15（2）：296-305.
[8] POTAPOV D，DOUZE D.Harchaoui category-specific video summarization[C]//Proceedings of the 13th European Conference on Computer Vision，Switzerland，Sep 6-12，2014.Cham：Springer，2014：540-555.
[9] ZHANG K，CHAO W L，SHA F.Video summarization with long short-term memory[C]//Proceedings of the 14th European Conference on Computer Vision，Amsterdam，Oct 8-16，2016.Cham：Springer，2016：766-782.
[10] ZHAO B，LI X L，LU X Q.HAS-RNN：hierarchical structure-adaptive RNN for video summarization[C]//Proceedings of the 31st IEEE Conference on Computer Vision and Pattern Recognition，Salt Lake City，Jun 18-22，2018.Washington：IEEE Computer Society，2018：7405-7414.
[11] HUANG S Y，LI X，ZHANG Z F，et al.User-ranking video summarization with multi-stage spatio-temporal representation[J].IEEE Transactions on Image Processing，2019，28（6）：2654-2664.
[12] JI Z，XIONG K L，PANG Y W.Video summarization with attention-based encoder-decoder networks[J].IEEE Transactions on Circuits and Systems for Video Technology，2020，30（6）：1709-1717.
[13] VASWANI A，SHAZEER N，PARMAR N.Attention is all you need[C]//Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems，Los Angeles，Dec 4-9，2017：5998-6008.
[14] ZHOU K Y，YU Q，TAO X.Deep reinforcement learning for unsupervised video summarization with diversity-representativeness reward[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence，New Orleans，Feb 2-7，2018.Palo Alto：AAAI，2018：7582-7589.
[15] 李依依，王继龙.自注意力机制的视频摘要模型[J].计算机辅助设计与图形学学报，2020，32（4）：652-659.
LI Y Y，WANG J L.Self-attention based video summarization[J].Journal of Computer-Aided Design & Computer Graphics，2020，32（4）：652-659.
[16] FAJTL J，SOKEH H S，ARGYRIOU V.Summarizing videos with attention[C]//Proceedings of Asian Conference on Computer Vision，Australia，Dec 2-6，2018.Cham：Springer，2018：39-54.
[17] SONG Y，VALLMITJANA J，STENT A.TvSum：summarizing web videos using titles[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition，Boston，Jun 8-10，2015.Piscataway：IEEE，2015：5179-5187.
[18] WEI H，NI B，YAN Y.Video summarization via semantic attended networks[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence，New Orleans，Feb 2-7，2018.Palo Alto：AAAI，2018：216-223.
[19] JUNG Y，CHO D，KIM D.Discriminative feature learning for unsupervised video summarization[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence，Hawaii，Jan 29-33，2019.Palo Alto：AAAI，2018：8537-8544.