Video Memorability Prediction Based on Multi-Modal Features Fusion

doi:10.3778/j.issn.1002-8331.2012-0122

Abstract

Abstract: With the explosive growth of online videos, video memorability has become a research hotspot. Video memorability is a metric to describe that how memorable the video is, designing calculation models for automatically predicting video memorability has a wide range of applications and prospects. Most of the current researches on video memorability prediction focused on the common visual features or semantic factors, while didn’t consider the influence of depth features on video memorability. This paper focuses on exploring the depth features of the video. After the video is preprocessed, the depth estimation model is used to extract the depth map. The original video images and the depth maps are input into the pre-trained ResNet152 network to extract the depth features；the TF-IDF algorithm is used to extract semantic features of the video, and different weights are assigned to words that have an impact on video memorability; finally, depth features, semantic features, and C3D spatiotemporal features extracted from video content are late fused. A fusion multi-modal video memorability prediction model is proposed. Experiments are conducted on the large public dataset （VideoMem） provided by the MediaEval 2019 conference. The experimenal tresults achieve a Spearman’s rank correlation of 0.545 （respectively 0.240）for short-term （resp. long-term） memorability prediction, which proves the effectiveness of the model.

Key words: video memorability, multi-modal, features fusion

摘要： 随着网络视频的爆炸式增长，视频记忆度成为热点研究方向。视频记忆度是衡量一个视频令人难忘的程度指标，设计自动预测视频记忆度的计算模型有广泛的应用和前景。当前对视频记忆度预测的研究多集中于普遍的视觉特征或语义因素，没有考虑深度特征对视频记忆度的影响。着重探索了视频的深度特征，在视频预处理后利用现有的深度估计模型提取深度图，将视频原始图像和深度图一起输入预训练的ResNet152网络来提取深度特征；使用TF-IDF算法提取视频的语义特征，并对视频记忆度有影响的单词赋予不同的权重；将深度特征、语义特征和从视频内容中提取的C3D时空特征进行后期融合，提出了一个融合多模态的视频记忆度预测模型。在MediaEval 2019会议提供的大型公开数据集（VideoMem）上进行实验，在视频的短期记忆度预测任务中达到了0.545（长期记忆度预测任务：0.240）的Spearman相关性，证明了该模型的有效性。

关键词: 视频记忆度, 多模态, 特征融合

CHANG Shiying, HU Yan. Video Memorability Prediction Based on Multi-Modal Features Fusion[J]. Computer Engineering and Applications, 2022, 58(14): 219-226.

常诗颖, 胡燕. 多模态特征融合的视频记忆度预测[J]. 计算机工程与应用, 2022, 58(14): 219-226.

References

[1] KAR A，MAVIN P，GHATURLE Y，et al.What makes a video memorable?[C]//2017 IEEE International Conference on Data Science and Advanced Analytics（DSAA），2017：373-381.
[2] ISOLA P，XIAO J，PARIKH D，et al.What makes a photograph memorable?[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2013，36（7）：1469-1482.
[3] ISOLA P，XIAO J，TORRALBA A，et al.What makes an image memorable?[C]//Proceedings of CVPR 2011，2011：145-152.
[4] HAN J，CHEN C，SHAO L，et al.Learning computational models of video memorability from fMRI brain imaging[J].IEEE Transactions on Cybernetics，2014，45（8）：1692-1703.
[5] SHEKHAR S，SINGAL D，SINGH H，et al.Show and recall：learning what makes videos memorable[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops，2017：2730-2739.
[6] TRAN D，BOURDEV L，FERGUS R，et al.Learning spatiotemporal features with 3d convolutional networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：4489-4497.
[7] COHENDET R，YADATI K，DUONG N Q K，et al.Annotating，understanding，and predicting long-term video memorability[C]//Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval，2018：178-186.
[8] COHENDET R，DEMARTY C H，DUONG N Q K，et al.VideoMem：constructing，analyzing，predicting short-term and long-term video memorability[C]//Proceedings of the IEEE International Conference on Computer Vision，2019：2531-2540.
[9] SUN W，ZHANG X.Video memorability prediction with recurrent neural networks and video titles at the 2018 MediaEval predicting media memorability task[C]//Proceedings of MediaEval，2018.
[10] COHENDET R，DEMARTY C H，DUONG N Q K.Transfer learning for video memorability prediction[C]//Proceedings of MediaEval，2018.
[11] GUPTA R，MOTWANI K.Linear models for video memorability prediction using visual and semantic features[C]//Proceedings of MediaEval，2018.
[12] AZCONA D，MOREU E，HU F，et al.Predicting media memorability using ensemble models[C]//CEUR Workshop Proceedings，2020.
[13] LEYVA R，DOCTOR F，SECO DE HERRERA A G，et al.Multimodal deep features fusion for video memorability prediction[C]//Proceedings of MediaEval，2019.
[14] BASAVARAJU S，MITTAL P，SUR A.Image memorability：the role of depth and motion[C]//2018 25th IEEE International Conference on Image Processing（ICIP），2018：699-703.
[15] LAINA I，RUPPRECHT C，BELAGIANNIS V，et al.Deeper depth prediction with fully convolutional residual networks[C]//2016 Fourth International Conference on 3D Vision（3DV），2016：239-248.
[16] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[17] DESINGH K，KRISHNA K M，RAJAN D，et al.Depth really matters：improving visual salient region detection with depth[C]//Proceedings of BMVC，2013.
[18] LU W，QI J，LIU Q，et al.Depth estimation for image dehazing of surveillance on education[J].Journal of Intelligent & Fuzzy Systems，2016，31（5）：2629-2636.
[19] SHEN L，FANG R，YAO Y，et al.No-reference stereoscopic image quality assessment based on image distortion and stereo perceptual information[J].IEEE Transactions on Emerging Topics in Computational Intelligence，2018，3（1）：59-72.
[20] BASAVARAJU S，SUR A.Image memorability prediction using depth and motion cues[J].IEEE Transactions on Computational Social Systems，2020，7（3）：600-609.
[21] KHOSLA A，RAJU A S，TORRALBA A，et al.Understanding and predicting image memorability at a large scale[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：2390-2398.
[22] CONSTANTIN M G，IONESCU B，DEMARTY C H，et al.Predicting media memorability task at MediaEval 2019[C]//Proc of MediaEval 2019 Workshop，Sophia Antipolis，France，2019.
[23] 叶雪梅，毛雪岷，夏锦春，等.文本分类TF-IDF算法的改进研究[J].计算机工程与应用，2019，55（2）：104-109.
YE X M，MAO X M，XIA J C，et al.Improved approach to TF-IDF algorithm in text classification[J].Computer Engineering and Applications，2019，55（2）：104-109.
[24] 王帅，王维莹，陈师哲，等.基于全局和局部信息的视频记忆度预测[J].软件学报，2020，31（7）：1969-1979.
WANG S，WANG W Y，CHEN S Z，et al.Video memorability prediction based on global and local information[J].Journal of Software，2020，31（7）：1969-1979.
[25] SPEER R，CHIN J，HAVASI C.Conceptnet 5.5：an open multilingual graph of general knowledge[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2017.
[26] WEISS A，SANG B，YOON S.Predicting memorability via early fusion deep neural network[C]//Proceedings of Media-
Eval，2018.