Research on Temporal None Padding Network Video Action Recognition Algorithm

doi:10.3778/j.issn.1002-8331.2106-0376

Abstract

Abstract: Video action recognition is a basic problem in the field of image and vision. For the action recognition model based on deep learning, 2D convolution method has few model parameters, but its accuracy is not high. 3D method improves the accuracy but will produce more parameters and computation. In order to reduce the amount of parameters of 3D convolutional neural network action recognition model and reduce the consumption on the premise of maintaining the accuracy, a temporal none padding convolutional network algorithm is proposed. When 3D convolution is performed on video, additional data will not fill in the time dimension, so as to ensure the integrity of temporal information. In order to make full use of the time information, a network structure suitable for this filling method is designed. Firstly, 3D convolution is used to extract the space-time information without padding in the time dimension, and then 3D convolution is transformed into 2D convolution to further extract features by using the network reorganization structure. The experimental results show that the parameters of this network are 10.385×106, and the accuracy is 60.28% on UCF101 dataset without using pretrained weights. Compared with other 3D convolution network action recognition methods, this network has obvious advantages in resource occupation and accuracy.

Key words: action recognition, machine vision, deep learning, 3D convolutional neural network

摘要： 视频行为识别是图像和视觉领域的一个基础问题，在基于深度学习的行为识别模型中，2D卷积方法模型参数较少，但是准确率不高；3D卷积方法在一定程度上提高了准确率，但会产生较多的参数和计算量。为了在保持准确率的前提下降低3D卷积神经网络行为识别模型的参数量，减少计算资源消耗，提出了时域零填充卷积网络行为识别算法，对视频进行3D卷积时不在时间维度上填充额外数据，以此来保证时域信息的完整性。为了充分利用有限的时间信息，设计了适合此填充方式的网络结构：先以时域不填充的方式使用3D卷积提取时空信息，然后利网络重组结构将3D卷积变为2D卷积来进一步提取特征。实验表明，该网络的参数量为10.385×106，不使用预训练权重的情况下在UCF101数据集上准确率为60.28%，与其他3D卷积网络行为识别方法相比在资源占用和准确率上都有明显优势。

关键词: 行为识别, 机器视觉, 深度学习, 3D卷积神经网络

LIU Zhao, YANG Fan, SI Yazhong. Research on Temporal None Padding Network Video Action Recognition Algorithm[J]. Computer Engineering and Applications, 2023, 59(1): 162-168.

刘钊, 杨帆, 司亚中. 时域非填充网络视频行为识别算法研究[J]. 计算机工程与应用, 2023, 59(1): 162-168.

References

[1] 鹿天然，于凤芹，杨慧中，等.基于显著性检测和稠密轨迹的人体行为识别[J].计算机工程与应用，2018，54（14）：163-167.
LU T R，YU F Q，YANG H Z，et al.Human action recognition based on dense trajectories with saliency detection[J].Computer Engineering and Applications，2018，54（14）：163-167.
[2] KUEHNE H，JHUANG H，GARROTE E，et al.HMDB：A large video database for human motion recognition[C]//Proceedings of 2011 International Conference on Computer Vision，2011：2556-2563.
[3] SOOMRO K，ZAMIR A R，SHAH M.UCF101：A dataset of 101 human actions classes from videos in the wild[J].arXiv：1212.0402，2012.
[4] CABA HEILBRON F，ESCORCIA V，GHANEM B，et al.Activitynet：A large-scale video benchmark for human activity understanding[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：961-970.
[5] KARPATHY A，TODERICI G，SHETTY S，et al.Large-scale video classification with convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2014：1725-1732.
[6] KAY W，CARREIRA J，SIMONYAN K，et al.The kinetics human action video dataset[J].arXiv：1705.06950，2017.
[7] ABU-EL-HAIJA S，KOTHARI N，LEE J，et al.Youtube-8m：A large-scale video classification benchmark[J].arXiv：1609.08675，2016.
[8] GOYAL R，EBRAHIMI KAHOU S，MICHALSKI V，et al.The “something something” video database for learning and evaluating visual common sense[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：5842-5850.
[9] JI S，XU W，YANG M，et al.3D convolutional neural networks for human action recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2012，35（1）：221-231.
[10] TRAN D，BOURDEV L，FERGUS R，et al.Learning spatiotemporal features with 3D convolutional networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：4489-4497.
[11] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409. 1556，2014.
[12] HARA K，KATAOKA H，SATOH Y.Towards good practice for action recognition with spatiotemporal 3D convolutions[C]//Proceedings of the 24th International Conference on Pattern Recognition（ICPR），2018：2516-2521.
[13] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[14] DENG J，DONG W，SOCHER R，et al.Imagenet：A large-scale hierarchical image database[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition，2009：248-255.
[15] VAROL G，LAPTEV I，SCHMID C.Long-term temporal convolutions for action recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，40（6）：1510-1517.
[16] SUN L，JIA K，YEUNG D Y，et al.Human action recognition using factorized spatio-temporal convolutional networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：4597-4605.
[17] QIU Z，YAO T，MEI T.Learning spatio-temporal representation with pseudo-3d residual networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：5533-5541.
[18] TRAN D，WANG H，TORRESANI L，et al.A closer look at spatiotemporal convolutions for action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：6450-6459.
[19] LIN J，GAN C，HAN S.Tsm：Temporal shift module for efficient video understanding[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：7083-7093.
[20] JIANG B，WANG M M，GAN W，et al.STM：Spatiotemporal and motion encoding for action recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：2000-2009.
[21] LI Y，JI B，SHI X，et al.TEA：Temporal excitation and aggregation for action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：909-918.
[22] WANG L，XIONG Y，WANG Z，et al.Temporal segment networks：Towards good practices for deep action recognition[C]//Proceedings of the European Conference on Computer Vision，2016：20-36.
[23] 高大鹏，朱建刚.多维度自适应3D卷积神经网络原子行为识别[J].计算机工程与应用，2018，54（4）：174-178.
GAO D P，ZHU J G.Atom action recognition by multi-dimensional adaptive 3D convolutional neural networks[J].Computer Engineering and Applications，2018，54（4）：174-178.
[24] ZHU Y，LI X，LIU C，et al.A Comprehensive study of deep video action recognition[J].arXiv：2012.06567，2020.
[25] TRAN D，RAY J，SHOU Z，et al.Convnet architecture search for spatiotemporal feature learning[J].arXiv：1708.05038，2017.