Generative Adversarial Network with Dual Discriminator and Mixed Attention

doi:10.3778/j.issn.1002-8331.2211-0196

Abstract

Abstract: In image generation tasks, how to improve the quality of generated images is a key problem. Currently, the multi-layer convolutional structure adopted by GAN has the problem of local induction bias, which cannot focus on key information, resulting in losing image features during training process. In this paper, a model of generative adversarial network with dual discriminator and mixed attention, termed as DDMA-GAN, is proposed. Firstly, DDMA-GAN designs a mixed attention mechanism, which utilizes channel attention and spatial attention to fully capture image feature information. Secondly, to solve the problem of discrimination error of single discriminator, a dual discriminator structure is proposed. The fusion coefficient is used to fuse the judgment results to make the returned parameters more objective, and the data augmentation module is embedded to further improve the robustness of the model. Finally, the hinge loss is used as loss function to maximize the distance between true and fake samples. The model is verified on public datasets LSUN and CelebA. Experimental results show that images generated by DDMA-GAN on classical datasets are more realistic. FID and MMD of DDMA-GAN are significantly reduced, which fully indicate validity of model.

Key words: image generation, convolutional neural network (CNN), mixed attention, dual discriminator, data augmentation, generative adversarial network (GAN)

摘要： 图像生成任务中，如何提升生成图片的质量是一个关键问题。当前，生成对抗网络采用的多层卷积结构存在局部性归纳偏置的问题，无法准确聚焦关键信息，导致图像特征丢失严重，生成图像效果较差。为此，提出了结合混合注意力的双判别生成对抗网络（DDMA-GAN）。设计了一种混合注意力机制，利用通道注意力和空间注意力模块，从两个维度充分捕获图像特征信息；为解决单判别器存在判别误差的问题，提出一种双判别器结构，使用融合系数将判定结果融合，使回传参数更具客观性，并嵌入数据增强模块，进一步提升模型鲁棒性；采用铰链损失作为模型损失函数，最大化真假样本间的距离，明确决策边界。模型在公开数据集LSUN和CelebA上进行验证，实验结果表明，DDMA-GAN生成的图像更加真实，纹理细节更加丰富，其FID和MMD值均显著降低且优于其他常见模型，证明了模型的有效性。

关键词: 图像生成, 卷积神经网络, 混合注意力, 双判别器, 数据增强, 生成对抗网络

WANG Lei, YANG Jun, ZHANG Chiyu, DAI Zaiyan. Generative Adversarial Network with Dual Discriminator and Mixed Attention[J]. Computer Engineering and Applications, 2024, 60(7): 212-221.

王磊, 杨军, 张驰宇, 代在燕. 结合混合注意力的双判别生成对抗网络[J]. 计算机工程与应用, 2024, 60(7): 212-221.

References

[1] LIU Z, HU H, LIN Y, et al. Swin transformer v2: scaling up capacity and resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), 2022: 12009-12019.
[2] DAI Z, LIU H, LE Q V, et al. Coatnet: marrying convolution and attention for all data sizes[C]//Advances in Neural Information Processing Systems, 2021: 3965-3977.
[3] KOTOVENKO D, WRIGHT M, HEIMBRECHT A, et al. Rethinking style transfer: from pixels to parameterized brushstrokes[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021: 12196-12205.
[4] SALAKHUTDINOV R, MNIH A, HINTON G. Restricted Boltzmann machines for collaborative filtering[C]//Proceedings of the 24th International Conference on Machine Learning, 2007: 791-798.
[5] KINGMA D P, WELLING M. Auto-encoding variational bayes[J]. arXiv:1312.6114, 2013.
[6] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems, 2014: 2672-2680.
[7] RADFORD A, METZ L, CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks[J]. arXiv:1511.06434, 2015.
[8] ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein GAN[J]. arXiv:1701.07875, 2017.
[9] GULRAJANI I, AHMED F, ARJOVSKY M, et al. Improved training of Wasserstein GANs[C]//Proceedings of the Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, Long Beach, Dec 4-9, 2017: 5769-5779.
[10] ZHANG H, GOODFELLOW I, METAXAS D, et al. Self-attention generative adversarial networks[C]//International Conference on Machine Learning, 2019: 7354-7363.
[11] 武随烁, 杨金福, 单义, 等. 使用孪生注意力机制的生成对抗网络的研究[J]. 计算机科学与探索, 2020, 14(5): 833-840.
WU S S, YANG J F, SHAN Y, et al. Research on generative adversarial networks using twins attention mechanism[J]. Journal of Frontiers of Computer Science and Technology, 2020, 14(5): 833-840.
[12] YU F, SEFF A, ZHANG Y, et al. LSUN: construction of a large-scale image dataset using deep learning with humans in the loop[J]. arXiv:1506.03365, 2015.
[13] LIU Z, LUO P, WANG X, et al. Deep learning face attributes in the wild[C]//Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015: 3730-3738.
[14] WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 3-19.
[15] HEUSEL M, RAMSAUER H, UNTERTHINER T, et al. GANs trained by a two time-scale update rule converge to a local Nash equilibrium[C]//Proceedings of the Annual Conference on Neural Information Processing Systems 2017, Long Beach, Dec 4-9, 2017: 6626-6637.
[16] B?HM V, SELJAK U. Probabilistic auto-encoder[J]. arXiv:2006.05479, 2020.
[17] WANG X, GIRSHICK R, GUPTA A, et al. Non-local neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018: 7794-7803.
[18] HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018: 7132-7141.
[19] HOU Q, ZHOU D, FENG J. Coordinate attention for efficient mobile network design[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021: 13713-13722.