Few-Shot Learning Image Classification Based on Adaptive Feature Fusion and Transformation

doi:10.3778/j.issn.1002-8331.2105-0290

Abstract

Abstract: The constant variation of data sampling in few shot learning makes the feature extraction of the model inadequate, and it is difficult for the model to manipulate the extracted features. Furthermore, the variation in the data distribution also affects the performance of the few shot model. To address these problems, a few shot image classification model based on adaptive weighted multi-branch is proposed. The multi-channel feature processing module performs the feature extraction and fusion on the input data so that a small data can be fully utilized, and the adaptive branch weights make the feature signals scale with the features. In addition, the feature conversion module adaptively changes to variable data distributions to better aggregate the same classes and improve the classification performance. The proposed model is tested on the Caltech-UCSD Birds-200-2011 dataset and mini-ImageNet dataset for classification performance on different scenes. The classification performance of the proposed model is tested on the Caltech-UCSD Birds-200-2011 dataset and the mini-ImageNet dataset under different scenarios. The experimental results demonstrate that the accuracy of the proposed model is improved by 9.81, 8.16 percentage points and 9.16, 9.21 percentage points in the 5-Way 1-Shot and 5-Way 5-Shot tasks, respectively, compared with baseline. This also shows the effectiveness of the model.

Key words: image classification, few shot learning, feature fusion, feature conversion, adaptive weighted multi-branch

摘要： 小样本学习中数据采样不断变化的特点使得模型特征提取不充分，同时，模型对提取的特征也难以进行相应操作；数据分布的变化也影响着小样本模型的性能。针对这些问题，提出一种基于自适应加权多路分支小样本图像分类模型。多路特征处理模块对输入数据进行特征提取和融合，以便充分利用少量数据；自适应的支路权重使得特征信号随特征进行相应的放缩；特征转换模块对多变的数据分布进行适应性变化，以便更好地聚合同类，提高分类效果。通过使用Caltech-UCSD Birds-200-2011数据集和mini-ImageNet数据集，对所提模型在不同场景下进行分类效果测试。实验结果表明，所提模型在5-Way 1-Shot和5-Way 5-Shot任务中的准确率分别比baseline相比分别提升9.81、8.16个百分点和9.16、9.21个百分点，验证了模型的有效性。

关键词: 图像分类, 小样本学习, 特征融合, 特征转换, 自适应加权多路分支

XU Dong, YANG Guan, LIU Xiaoming, LIU Yang, LIU Jizong, CHEN Jing, GUO Qingyu. Few-Shot Learning Image Classification Based on Adaptive Feature Fusion and Transformation[J]. Computer Engineering and Applications, 2022, 58(24): 223-232.

许栋, 杨关, 刘小明, 刘阳, 刘济宗, 陈静, 郭清宇. 基于自适应特征融合与转换的小样本图像分类[J]. 计算机工程与应用, 2022, 58(24): 223-232.

References

[1] 孙涛，李东升.基于非凸的全变分和低秩混合正则化的图像去模糊模型和算法[J].计算机学报，2020，43（4）：643-652.
SUN T，LI D S.Nonconvex low-rank and total-variation regularized model and algorithm for image deblurring[J].Chinese Journal of Computers，2020，43（4）：643-652.
[2] 党宏社，陶亚凡，张选德.基于混合注意力与强化学习的文本摘要生成[J].计算机工程与应用，2020，56（1）：185-190.
DANG H S，TAO Y F，ZHANG X D.Abstractive summarization model based on mixture attention and reinforcement learning[J].Computer Engineering and Applications，2020，56（1）：185-190.
[3] 金璐，刘士建，王霄，等.基于改进关系网络的小样本红外空中目标分类方法[J].光学学报，2020，40（8）：87-96.
JIN L，LIU S J，WANG X，et al.Infrared aircraft classification method with small samples based on improved relation network[J].Acta Optica Sinica，2020，40（8）：87-96.
[4] 刘靖祎，史彩娟，涂冬景，等.零样本图像分类综述[J].计算机科学与探索，2021，15（5）：812-824.
LIU J Y，SHI C J，TU D J，et al.Survey of zero-shot image classification[J].Journal of Frontiers of Computer Science and Technology，2021，15（5）：812-824.
[5] QIN Y X，ZHANG W G，ZHAO C X，et al.Prior-knowledge and attention based meta-learning for few-shot learning[J].arXiv：1812.04955，2018.
[6] 祝钧桃，姚光乐，张葛祥，等.深度神经网络的小样本学习综述[J].计算机工程与应用，2021，57（7）：22-33.
ZHU J T，YAO G L，ZHANG G X，et al.Survey of few shot learning of deep neural network[J].Computer Engineering and Applications，2021，57（7）：22-33.
[7] MOHRI M，ROSTAMIZADEH A，TALWALKAR A.Foundations of machine learning[M].[S.l.]：MIT Press，2018.
[8] ORIOL V，CHARLES B，TIM L，et al.Matching networks for one shot learning[C]//Advances in Neural Information Processing Systems 29：Annual Conference Neural Information Processing Systems，Barcelona，Spain，2016：3630-3638.
[9] JAKE S，KEVIN S，RICHARD S Z.Prototypical networks for few-shot learning[C]//Advances in Neural Information Processing Systems 30：Annual Conference on Neural Information Processing Systems，Long Beach，CA，USA，2017：4077-4087.
[10] FLOOD S，YANG Y X，ZHANG L，et al.Learning to compare：relation network for few-shot learning[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition，Salt Lake City，UT，USA，2018：1199-1208.
[11] HUANG H X，ZHANG J J，ZHANG J，et al.PTN：a poisson transfer network for semi-supervised few-shot learning[C]//AAAI Conference on Artificial Intelligence，Vancouver，British Columbia，2021：1602-1609.
[12] YE H J，HU H X，ZHAN D C，et al.Few-shot learning via embedding adaptation with set-to-set functions[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition，Seattle，WA，USA，June 13-19，2020：8805-8814.
[13] YOON S W，KIM D Y，SEO J，et al.XtarNet：learning to extract task-adaptive representation for incremental few-shot learning[C]//Proceedings of the 37th International Conference on Machine Learning，2020：10852-10860.
[14] HUANG G，LIU Z，MAATEN L，et al.Weinberger：densely connected convolutional networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition，Honolulu，HI，USA，2017：2261-2269.
[15] CHEN W Y，LIU Y C，ZSOLT K，et al.A closer look at few-shot classification[C]//7th International Conference on Learning Representations，New Orleans，LA，USA，2019.
[16] 马岽奡，唐娉，赵理君，等.深度学习图像数据增广方法研究综述[J].中国图象图形学报，2021，26（3）：487-502.
MA D A，TANG P，ZHAO L J，et al.Review of data augmentation for image in deep learning[J].Journal of Image and Graphics，2021，26（3）：487-502.
[17] WANG Y Q，YAO Q M，JAMES T.K，et al.Generalizing from a few examples：a survey on few-shot learning[J].ACM Computing Surveys，2020，53（3）：63.
[18] WANG Y K，XU C M，LIU C，et al.Instance credibility inference for few-shot learning[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition，Seattle，WA，USA，2020：12833-12842.
[19] JIA Y，SHELHAMER E，DONAHUE J，et al.Caffe：convolutional architecture for fast feature embedding[C]//ACM International Conference on Multimedia，2014：675-678.
[20] TSENG H Y，LEE H Y，HUANG J B，et al.Cross-domain few-shot classification via learned feature-wise transformation[C]//8th International Conference on Learning Representations，Addis Ababa，Ethiopia，2020.
[21] YAROSLAV G，EVGENIYA U，HANA A，et al.Domain-adversarial training of neural networks.[J].arXiv：1505.07818，2015.
[22] 赵小强，蒋红梅.基于特征和类别对齐的领域适应算法[J].控制与决策，2022（5）：1203-1210.
ZHAO X Q，JIANG H M.Domain adaptation based on feature-level and class-level alignment[J].Control and Decision，2022（5）：1203-1210.
[23] CHRISTIAN S，LIU W，JIA Y Q，et al.Going deeper with convolutions[C]//IEEE Conference on Computer Vision and Pattern Recognition，Boston，MA，USA，2015：1-9.
[24] HU J，SHEN L，SUN G.Squeeze-and-excitation networks[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition，Salt Lake City，UT，USA，2018：7132-7141.
[25] ZHANG C，CAI Y J，LIN G S，et al.DeepEMD：few-shot image classification with differentiable earth mover’s distance and structured classifiers[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition，Seattle，WA，USA，2020：12200-12210.
[26] WANG Y，CHAO W L，KILIAN Q W，et al.SimpleShot：revisiting nearest-neighbor classification for few-shot learning[J].arXiv：1911.04623，2019.
[27] CHEN D，CHEN Y，LI Y，et al.Self-supervised learning for few-shot image classification[C]//2021 IEEE International Conference on Acoustics，Speech and Signal Processing（ICASSP），2021：1745-1749.
[28] SIMON C，KONIUSZ P，NOCK R，et al.Adaptive subspaces for few-shot learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：4136-4145.
[29] MA N N，ZHANG X Y，ZHENG H T，et al.ShuffleNet V2：practical guidelines for efficient CNN architecture design[C]//15th European Conference on Computer Vision，Munich，Germany，2018：122-138.