Application of Label-Bias Network in Datasets with Noisy Labels

doi:10.3778/j.issn.1002-8331.2109-0502

Abstract

Abstract: Noisy labels are common in real-world datasets, which will affect the learning performance of deep neural networks（DNNs） seriously. In view of this phenomenon, a method of noisy label recognition and re-labeling based on label-bias learning is proposed. Two strategies of pseudo-label generation are designed to generate the artificial noisy dataset by the clean datasets from basic network, and label-bias vectors and label-bias matrixes of the artificial noisy dataset are calculated. After that, by using the full connection layers and single-row convolution kernels to strengthen the relevance between similar classes, the noise probabilities of sample data are learned directly by the noise learning networks of label-bias vector network and label-bias matrix network. Thresholds that are linearly related to the noise ratio are designed to distinguish the clean data and noisy data. The factors affecting the network performances, including pseudo-label generation strategy, network structure, and training iteration, are analyzed by experiments. Experiments on a public dataset show in the situations of varies noise ratio distributions, under the premise of stable precision and recall of clean data, the proposed method can significantly improve the precision and recall of noisy data, with the maximum improvement of 16.45% and 21.01% respectively.

Key words: noisy dataset, noisy label, label bias, noise learning, deep learning

摘要： 噪声标签在实际数据集中普遍存在，这将严重影响深度神经网络的学习效果。针对此问题，提出了一种基于标签差学习的噪声标签数据识别与数据再标记方法。该方法设计两种不同的伪标签生成策略，利用基础网络所识别的干净数据生成人工噪声数据集，并计算该数据集的标签差向量或标签差矩阵；以强化相似类别间的关联性为目标，利用全连接层与单行卷积核，设计标签差向量网络与标签差矩阵网络等两种噪声学习网络直接学习样本数据的噪声概率；设计与噪声率线性相关的阈值，对干净数据与噪声数据进行判断。通过设计实验，对包括伪标签生成策略、网络结构、训练迭代次数等影响网络识别性能的因素进行分析。在公开数据集上的测试表明，在多种噪声分布情况中，该算法在保持干净数据的准确率与召回率基本稳定的前提下，能显著提高噪声数据的准确率与召回率，提高幅度最大为16.45%及21.01%。

关键词: 噪声数据集, 噪声标签, 标签差, 噪声学习, 深度学习

JIANG Qianyin, YU Zhi, LI Xiying. Application of Label-Bias Network in Datasets with Noisy Labels[J]. Computer Engineering and Applications, 2023, 59(6): 92-100.

江倩殷, 余志, 李熙莹. 标签差网络在噪声标签数据集中的应用[J]. 计算机工程与应用, 2023, 59(6): 92-100.

References

[1] ZHANG C，BENGIO S，HARDT M，et al.Understanding deep learning requires rethinking generalization[C]//International Conference on Learning Representations（ICLR），France，2017.
[2] KIM Y，YUN J，SHOM H，et at.Joint negative and positive learning for noisy labels[C]//IEEE Conference on Computer Vision and Pattern Recognition（CVPR），2021.
[3] KIM Y，YIM J，YUN J，et at.NLNL：negative learning for noisy labels[C]//2019 IEEE/CVF International Conference on Computer Vision（ICCV 2019），Seoul，2019：101-110.
[4] WANG Y，MA X，CHEN Z，et al.Symmetric cross entropy for robust learning with noisy label[C]//2019 IEEE/CVF International Conference on Computer Vision（ICCV 2019），Seoul，2019：322-330.
[5] ZHANG Z，SABUNCU M R.Generalized cross entropy loss for training deep neural networks with noisy labels[C]//32nd Conference on Neural Information Processing Systems（NIPS），Montreal，2018.
[6] MALACH E，SHALEV-SHWARTZ S.Decoupling "when to update" from "how to update"[C]//31st Annual Conference on Neural Information Processing Systems（NIPS），Long Beach，2017.
[7] HAN B，YAO Q，YU X，et al.Co-teaching：robust training of deep neural networks with extremely noisy labels[C]//32nd Conference on Neural Information Processing Systems（NIPS），Montreal，2018.
[8] CHEN P，LIAO B，CHEN G，et al.Understanding and utilizing deep neural networks trained with noisy labels[C]//Thirty-sixth International Conference on Machine Learning（ICML），Long Beach，2019.
[9] WEI H，FENG L，CHEN X，et al.Combating noisy labels by agreement：a joint training method with co-regularization[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），2020.
[10] JIANG L，ZHOU Z，LEUNG T，et al.MentorNet：learning data-driven curriculum for very deep neural networks on corrupted labels[C]//Thirty-fifth International Conference on Machine Learning（ICML），Stockholmsm?ssan，2018.
[11] NGUYEN D T，MUMMADI C K，NGO T P N，et al.SELF：learning to filter noisy labels with self-ensembling[C]//International Conference on Learning Representations（ICLR），Addis Ababa，2020.
[12] ZHANG Z，ZHANG H，ARIK S O，et al.Distilling effective supervision from severe label noise[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），2020.
[13] LI J，SOCHER R，HOI S C H.DivideMix：learning with noisy labels as semi-supervised learning[C]//International Conference on Learning Representations（ICLR），Addis Ababa，2020.
[14] NISHI K，DING Y，RICH A，et al.Augmentation strategies for learning with noisy labels[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），2021.
[15] BERTHELOT D，CARLINI N，GOODFELLOW I，et al.MixMatch：a holistic approach to semi-supervised learning[C]//33rd Conference on Neural Information Processing Systems（NeurIPS），Vancouver，2019.
[16] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[C]//2015 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），Boston，2015.
[17] KRIZHEVSKY A，HINTON G.Learning multiple layers of features from tiny images[R].Citeseer，2009.