News Short Video Classification Model Fusing Multimodal Feature

doi:10.3778/j.issn.1002-8331.2208-0299

Abstract

Abstract: Video classification is an important part of understanding, summarizing and retrieving video data. News short video has the feature that audio information can describe news events more completely than image information, while traditional video classification models often only consider image information or fuse multimodal information of audio and image, which do not consider the primary-secondary relationship between modal information. To address the above problems, a news short video classification model fusing multimodal feature is proposed. It is designed with the fusion mechanism of audio modality as the main and image modality as the auxiliary. In order to make further use of the audio-dominated feature, a two-stage training mode is adopted. Firstly, the audio mode is trained separately, and then the audio and image modes are trained jointly. The image information is used to correct the classification results, so as to improve the accuracy of news short video classification. For the purpose of the model in training and evaluation, 10 304 news broadcast short videos have been collected as experimental dataset, with a total time of about 240 hours. The experimental results show that the classification effect of the proposed model is better than the traditional news short video classification model.

Key words: audio-visual relationship, multimodal feature fusion, news short video classification

摘要： 视频分类是理解、归纳和检索视频数据的一个重要环节。新闻短视频具有音频信息比图像信息更能完整地描述新闻事件的特点，但传统视频分类模型常常只考虑图像信息或融合了音频和图像的多模态信息，并没有考虑模态信息之间的主辅关系。针对上述问题，采用以音频模态为主，图像模态为辅的融合机制，提出了融合多模态特征的新闻短视频分类模型。为进一步利用音频为主的特点，采用两阶段训练方式，使用音频模态单独训练，音频和图像模态联合训练，利用图像信息修正分类结果，提升新闻短视频分类的准确率。为训练和评价模型，采集了10?304个新闻联播短视频作为实验数据集，总时长约为240?h。实验结果表明，所提模型的分类效果优于传统的新闻短视频分类模型。

关键词: 音画关系, 多模态特征融合, 新闻短视频分类

ZENG Xiangjiu, LIU Dawei, LIU Yifan, ZHAO Zhibin, LIU Xiumei, REN Yougui. News Short Video Classification Model Fusing Multimodal Feature[J]. Computer Engineering and Applications, 2023, 59(14): 107-113.

曾祥玖, 刘达维, 刘逸凡, 赵志滨, 柳秀梅, 任酉贵. 融合多模态特征的新闻短视频分类模型[J]. 计算机工程与应用, 2023, 59(14): 107-113.

References

[1] SZEGEDY C，LIU W，JIA Y，et al.Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：1-9.
[2] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[3] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409.
1556，2014.
[4] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.Imagenet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems，2012：1097-1105.
[5] KARPATHY A，TODERICI G，SHETTY S，et al.Large-scale video classification with convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2014：1725-1732.
[6] ZHA S，LUISIER F，ANDREWS W，et al.Exploiting image-trained CNN architectures for unconstrained video classification[J].arXiv：1503.04144，2015.
[7] TRAN D，BOURDEV L，FERGUS R，et al.Learning spatiotemporal features with 3d convolutional networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：4489-4497.
[8] CARREIRA J，ZISSERMAN A.Quo vadis，action recognition? a new model and the kinetics dataset[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：6299-6308.
[9] WANG L，XIONG Y，WANG Z，et al.Temporal segment networks：towards good practices for deep action recognition[C]//European Conference on Computer Vision.Cham：Springer，2016：20-36.
[10] FEICHTENHOFER C，PINZ A，ZISSERMAN A.Convolutional two-stream network fusion for video action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：1933-1941.
[11] LUO C，YUILLE A L.Grouped spatial-temporal aggregation for efficient action recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：5512-5521.
[12] LIN J，GAN C，HAN S.TSM：temporal shift module for efficient video understanding[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：7083-7093.
[13] FEICHTENHOFER C，FAN H，MALIK J，et al.Slowfast networks for video recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：6202-6211.
[14] OGAWA T，SASAKA Y，MAEDA K，et al.Favorite video classificaion based on multimodal bidirectional LSTM[J].IEEE Access，2018，6：61401-61409.
[15] WANG Z，KUAN K，RAVAUT M，et al.Truly multi-modal youtube-8m video classification with video，audio，and text[J].arXiv：1706.05461，2017.
[16] SUN C，MYERS A，VONDRICK C，et al.Videobert：a joint model for video and language representation learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：7464-7473.
[17] PARMAR P，REDDY J，MORRIS B.Piano skills assessment[J].arXiv：2101.04884，2021.
[18] ZHU W，LU J，LI J，et al.Dsnet：a flexible detect-to-summarize network for video summarization[J].IEEE Transactions on Image Processing，2020，30：948-962.
[19] RAVANELLI M，PARCOLLET T，BENGIO Y.The Pytorch-Kaldi speech recognition toolkit[C]//2019 IEEE International Conference on Acoustics，Speech and Signal Processing（ICASSP），2019：6465-6469.
[20] DEVLIN J，CHANG M W，LEE K，et al.BERT：pre-training of deep bidirectional transformers for language understanding[J].arXiv：1810.04805，2018.
[21] CUI Y，CHE W，LIU T，et al.Pre-training with whole word masking for Chinese BERT[J].IEEE/ACM Transactions on Audio，Speech，and Language Processing，2021，29：3504-3514.