Commonsense Oriented Fine-Grained Data Augmentation

doi:10.3778/j.issn.1002-8331.2210-0361

Abstract

Abstract: The representative researches on data augmentation are mainly carried out on common classification benchmark datasets such as ImageNet. Considering intra-class and inter-class relation in fine-grained visual classification(FGVC) datasets is so different from ordinary classification datasets, data augmentation methods for FGVC need to be further studied. Therefore, this paper proposes a fine-grained semantic image patch mixing method by commonsense(ComSipmix), starting from the fine-grained recognition task and the special properties of the dataset. The proposed method exploits common sense knowledge to mine potential associations between sample labels, and designs a multi-branch convolutional neural network structure for structured image mixing strategy based on this, so that the image mixing process pays more attention to the subtle differences of targets. Through extensive performance tests, it can be verified that the performance of the proposed method is significantly better than the mainstream image mixing-based data augmentation methods. At the same time, through experimental verification, the common sense knowledge proposed in this paper helps to improve the performance of various data augmentation models based on mixed image classes.

Key words: data augmentation, commonsense map, multi-branch convolutional neural network

摘要： 数据增强的代表性研究主要针对ImageNet等常规分类数据集展开。考虑到细粒度识别数据集中测试样本类内及类间关联性与常规分类数据集差异明显，因此针对细粒度识别的数据增强方法尚待深入研究。为此，从细粒度识别任务以及数据集的特殊属性入手提出基于常识辅助的细粒度语义图块混合策略。所提方法利用常识知识挖掘样本标签间潜在关联，以此为基础设计结构化图像混合策略的多支路卷积神经网络结构，使图像混合过程更关注目标的细微差异。通过大量性能测试可验证所提方法的性能明显优于主流的基于图像混合的数据增强方法。同时，通过实验验证，所提出的常识知识有助于多种基于混合图像类的数据增强模型性能提升。

关键词: 数据增强, 常识图谱, 多支路卷积神经网络

LI Huachao, KANG Bin, WANG Lei. Commonsense Oriented Fine-Grained Data Augmentation[J]. Computer Engineering and Applications, 2024, 60(6): 214-221.

李华超, 康彬, 王磊. 常识辅助细粒度数据增强方法[J]. 计算机工程与应用, 2024, 60(6): 214-221.

References

[1] ALBAWI S, MOHAMMED T A, AL-ZAWI S. Unders tanding of a convolutional neural network[C]//Proceedings of the 2017 International Conference on Engineering and Technology (ICET). Antalya, Turkey: IEEE, 2017: 1-6.
[2] MIKO?AJCZYK A, GROCHOWSKI M. Data augmentation for improving deep learning in image classification problem[C]//Proceedings of the 2018 International Interdisciplinary PhD Workshop (IIPhDW). Poland: IEEE, 2018: 117-122.
[3] ZHANG H Y, CISSE M, DAUPHIN Y N, et al. Mixup: beyond empirical risk minimization[J]. arXiv:1710.09412, 2017.
[4] YUN S, HAN D, CHUN S, et al. CutMix: regularization strategy to train strong classifiers with localizable features[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE, 2019: 6022-6031.
[5] HUANG S L, WANG X Z, TAO D C. SnapMix: semantically proportional mixing for augmenting fine-grained data[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2021: 1628-1636.
[6] ZHANG N, DONAHUE J, GIRSHICK R, et al. Part-based R-CNNs for fine-grained category detection[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2014: 834-849.
[7] LIN T Y, ROYCHOWDHURY A, MAJI S. Bilinear CNN models for fine-grained visual recognition[C]//Proceedings of the 2014 IEEE International Conference on Computer Vision (ICCV). Santiago, Chile: IEEE, 2014: 1449-1457.
[8] ZHANG X P, XIONG H K, ZHOU W G, et al. Picking deep filter responses for fine-grained image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, 2016: 1134-1142.
[9] GAO Y, HAN X T, WANG X, et al. Channel interaction networks for fine-grained image categorization[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020: 10818-10825.
[10] WANG Z H, WANG S J, LI H J, et al. Graph-propagation based correlation learning for weakly supervised fine-grained image classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020: 12289-12296.
[11] BEHERA A, WHARTON Z, HEWAGE P, et al. Context-aware attentional pooling (CAP) for fine-grained visual classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2021: 929-937.
[12] ZHAO Y F, YAN K, HUANG F Y, et al. Graph-based high-order relation discovery for fine-grained recognition[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, TN, USA: IEEE, 2021: 15074-15083.
[13] INOUE H. Data augmentation by pairing samples for images classification[J]. arXiv:1801.02929, 2018.
[14] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Proceedings of the Conference and Workshop on Neural Information Processing Systems (NIPS), 2014: 2672-2680.
[15] SUZUKI T. TeachAugment: data augmentation optimization using teacher knowledge[J]. arXiv:2202.12513, 2022.
[16] CUBUK E D, ZOPH B, MANE D, et al. AutoAugment: learning augmentation policies from data[J]. arXiv:1805. 09501, 2018.
[17] RAME A, SUN R, CORD M. MixMo: mixing multiple inputs for multiple outputs via deep subnetworks[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, QC, Canada: IEEE, 2021: 803-813.
[18] CHEN X L, LI L J, LI F F, et al. Iterative visual reasoning beyond convolutions[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, UT, USA:IEEE, 2018: 7239-7248.
[19] SINGH K K, DIVVALA S, FARHADI A, et al. DOCK: detecting objects by transferring common-sense knowledge[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 492-508.
[20] QI M S, WANG Y H, QIN J, et al. KE-GAN: knowledge embedded generative adversarial networks for semi-supervised scene parsing[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA: IEEE, 2019: 5232-5241.
[21] KATO K, LI Y, GUPTA A. Compositional learning for human object interaction[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 247-264.
[22] YU R C, LI A, MORARIU V I, et al. Visual relationship detection with internal and external linguistic knowledge distillation[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV). Venice, Italy: IEEE, 2017: 1068-1076.
[23] ZAREIAN A, WANG Z, YOU H, et al. Learning visual commonsense for robust scene graph generation[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2020: 642-657.
[24] NARASIMHAN M, SCHWING A G. Straight to the facts: learning knowledge base retrieval for factual visual question answering[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 460-477.
[25] BROWN T, MANN B, RYDER N, et al. Language models are few-shot learners[C]//Advances in Neural Information Processing Systems, 2020: 1877-1901.
[26] FU J, ZHENG H, MEI T. Look closer to see better: recurrent attention convolutional neural network for fine-grained image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 4438-4446.
[27] WANG Y M, MORARIU V I, DAVIS L S. Learning a discriminative filter bank within a CNN for fine-grained recognition[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, UT, USA: IEEE, 2018: 4148-4157.
[28] DING Y, ZHOU Y Z, ZHU Y, et al. Selective sparse sampling for fine-grained image recognition[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE, 2019: 6598-6607.
[29] ZHANG L B, HUANG S L, LIU W, et al. Learning a mixture of granularity-specific experts for fine-grained categorization[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE, 2019: 8330-8339.
[30] CHEN Y, BAI Y L, ZHANG W, et al. Destruction and construction learning for fine-grained image recognition[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, CA, USA: IEEE, 2019: 5152-5161.
[31] ZHUANG P Q, WANG Y L, QIAO Y. Learning attentive pairwise interaction for fine-grained classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020: 13130-13137.
[32] DU R Y, CHANG D L, BHUNIA A K, et al. Fine-grained visual classification via progressive multi-granularity training of jigsaw patches[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2020: 153-168.
[33] SUN G L, CHOLAKKAL H, KHAN S, et al. Fine-grained recognition: accounting for subtle differences between similar classes[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020: 12047-12054.
[34] LUO W, YANG X T, MO X J, et al. Cross-X learning for fine-grained visual categorization[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea (South): IEEE, 2019: 8241-8250.
[35] RAO Y M, CHEN G Y, LU J W, et al. Counterfactual attention learning for fine-grained visual categorization and re-identification[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, QC, Canada: IEEE, 2021: 1005-1014.