Improved MobileViT Algorithm for Small Samples

doi:10.3778/j.issn.1002-8331.2307-0250

Abstract

Abstract: To improve the classification ability, training speed, convergence, and inference speed of the MobileViT algorithm based on Transformer on small-sample data, two modules are proposed and inserted into the MobileViT algorithm：convolutional maxpooling downsampling (CMP) and multi-branch residual feature fusion (MR-FF). These modules are respectively used to reduce model parameters, reduce feature redundancy and prevent input feature loss. Taking the results of the MobileViT with the minimum number of parameters as an example, comparative experiments are conducted on the Oxford Flower102 and Mini-ImageNet small-sample datasets. The MobileViT with the inserted CMP and MR-FF modules achieves a 12.9 and 9.4 percentage points increase in test accuracy, a 17% increase in training speed, and a 0.31?ms increase in inference speed. Furthermore, it is found that when only the CMP module is inserted into MobileViT, higher classification accuracy and shorter inference time can be achieved on small-sample datasets with fewer than 60?000 images. Finally, a comparison is made with 5 advanced image classification algorithms, and the improved MobileViT achieves the best test results on small-sample data.

Key words: maxpooling, small-sample, Transformer, image classification

摘要： 为了提高基于Transformer的MobileViT算法在小样本数据上的分类能力，加快算法的训练、收敛以及提高推理速度，提出了卷积池化下采样（convolutional maxpooling downsampling，CMP）和多分支残差特征融合（multi-branch residual feature fusion，MR-FF）两个模块，并插入到MobileViT算法的模块中，分别用于降低模型的参数量，减少特征冗余和防止输入特征丢失。以最小参数量的MobileViT实验结果为例，在Oxford Flower102、Mini-ImageNet小样本数据集上进行了对比实验，插入以上两个模块的MobileViT在测试准确率上分别提升了12.9、9.4个百分点，在训练速度上提升了17%，在推理速度上提升了0.31?ms。当在MobileViT中只插入CMP模块时，可以在小于60?000张图像的小样本数据集上获得更高的分类准确度、最短的推理时间。最后与五种先进的图像分类算法进行了比较，改进的MobileVIT在小样本分类数据上取得了最优的测试结果。

关键词: 最大池化, 小样本, Transformer, 图像分类

ZHANG Bushi, FAN Hong. Improved MobileViT Algorithm for Small Samples[J]. Computer Engineering and Applications, 2024, 60(22): 251-260.

张埠石, 范红. 针对小样本改进的MobileViT算法[J]. 计算机工程与应用, 2024, 60(22): 251-260.

References

[1] 陈龙, 张建林, 彭昊, 等. 多尺度注意力与领域自适应的小样本图像识别[J]. 光电工程, 2023, 50(4): 66-80.
CHEN L, ZHANG J L, PENG H, et al. Few-shot image classification via multi-scale attention and domain adaptation[J]. Opto-Electronic Engineering, 2023, 50(4): 66-80.
[2] 赵春梅, 陈忠碧, 张建林. 基于深度学习的飞机目标跟踪应用研究[J]. 光电工程, 2019, 46(9): 1-10.
ZHAO C M, CHEN Z B, ZHANG J L. Application of aircraft target tracking based on deep learning[J]. Opto-Electronic Engineering, 2019, 46(9): 1-10.
[3] 石超, 陈恩庆, 齐林. 红外视频中的舰船检测[J]. 光电工程, 2018, 45(6): 86-91.
SHI C, CHEN E Q, QI L. Ship detection from infrared video[J]. Opto-Electronic Engineering, 2018, 45(6): 86-91.
[4] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: transformers for image recognition at scale[J]. arXiv:2010.11929, 2020.
[5] TOUVRON H, CORD M, DOUZE M, et al. Training data-efficient image transformers & distillation through attention[C]//Proceedings of the 38th International Conference on Machine Learning, 2021: 10347-10357.
[6] LIU Z, LIN Y, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, 2021: 10012-10022.
[7] LIU Z, HU H, LIN Y, et al. Swin transformer v2: scaling up capacity and resolution[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 12009-12019.
[8] WANG W, XIE E, LI X, et al. Pyramid vision transformer: a versatile backbone for dense prediction without convolutions[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, 2021: 568-578.
[9] WU H, XIAO B, CODELLA N, et al. CVT: introducing convolutions to vision transformers[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, 2021: 22-31.
[10] PAN Z, ZHUANG B, LIU J, et al. Scalable vision transformers with hierarchical pooling[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, 2021: 377-386.
[11] GUO J, HAN K, WU H, et al. CMT: convolutional neural networks meet vision transformers[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 12175-12185.
[12] YUAN K, GUO S, LIU Z, et al. Incorporating convolution designs into visual transformers[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, 2021: 579-588.
[13] CHEN Y, DAI X, CHEN D, et al. Mobile-former: bridging MobileNet and transformer[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 5270-5279.
[14] MEHTA S, RASTEGARI M. MobileViT: light-weight, general-purpose, and mobile-friendly vision transformer[J]. arXiv:2110.02178, 2021.
[15] ZHANG W, HUANG Z, LUO G, et al. TopFormer: token pyramid transformer for mobile semantic segmentation[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 12083-12093.
[16] SANDLER M, HOWARD A, ZHU M, et al. Inverted residuals and linear bottlenecks: mobile networks for classification, detection and segmentation[J]. arXiv:1801.04381, 2018.
[17] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
[18] TAN M, LE Q. EfficientNet: rethinking model scaling for convolutional neural networks[C]//Proceedings of the 36th International Conference on Machine Learning, 2019: 6105-6114.
[19] FRAN C. Deep learning with depth wise separable convolutions[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 1800-1807.
[20] WANG C Y, BOCHKOVSKIY A, LIAO H Y M. YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 7464-7475.
[21] IOFFE S, SZEGEDY C. Batch normalization: accelerating deep network training by reducing internal covariate shift[C]//Proceedings of the 32nd International Conference on Machine Learning, 2015: 448-456.
[22] VINYALS O, BLUNDELL C, LILLICRAP T, et al. Matching networks for one shot learning[C]//Advances in Neural Information Processing Systems 29, 2016.