Automatic Coloring Method for Gray Image Based on Convolutional Network

doi:10.3778/j.issn.1002-8331.2009-0420

Abstract

Abstract: With the emergence of deep learning, gray image automatic coloring has been developed rapidly, and one-hot coding method with excellent color effect has emerged to avoid the problem of dull color effect based on L2 loss function. This paper uses the basic idea of one-hot coding method, and proposes a convolution network based gray image automatic coloring method. In this method, the encoder and decoder are improved and redesigned, and the neural network structure is redesigned. By using the traditional convolution of Gaussian function and reducing the number of adjacent categories selected by each pixel, the process of one-hot decoding and coding is simplified, the frequent selection of parameters is avoided, and the speed of image processing is improved. By stacking dilated convolution of different sizes, and adding Dropout method, a relatively small and fast gray image automatic coloring network based on deep learning is constructed. Experimental results show that the proposed method has good image coloring effect, and has great advantages in network volume and performance.

Key words: deep learning, image colorization, one-hot coding, Gaussian convolution

摘要： 随着深度学习的出现，灰度图像自动上色得到了快速的发展，出现了上色效果优秀的基于one-hot编码（独热编码）的方法，避免基于[L2]损失函数上色效果平淡的问题。沿用one-hot编码方法的基础思路，提出了一种卷积网络的灰度图像自动上色方法。该方法对编码器与解码器进行了改进和重新设计，重新设计了神经网络结构。通过使用高斯函数的传统卷积，减少每一个像素点选取邻近分类的数量，简化了one-hot解码与编码的流程，避免了频繁选取参数，提升了处理图片的速度。通过堆叠不同尺寸的空洞卷积以及添加Dropout等方法，构建了一个体积相对较小、速度较快的基于深度学习的灰度图自动上色网络。实验证明了提出的方法图像上色效果良好，且在网络体积、上色效果上有较大的优势。

关键词: 深度学习, 图像上色, 独热编码, 高斯卷积

ZHANG Meiyu, LIU Yuehui, HOU Xianghui, QIN Xujia. Automatic Coloring Method for Gray Image Based on Convolutional Network[J]. Computer Engineering and Applications, 2022, 58(7): 229-236.

张美玉, 刘跃辉, 侯向辉, 秦绪佳. 基于卷积网络的灰度图像自动上色方法[J]. 计算机工程与应用, 2022, 58(7): 229-236.

References

[1] ZHANG L，LI C，WONG T T，et al.Two-stage sketch colorization[J].ACM Transactions on Graphics（TOG），2018，37（6）：1-14.
[2] DESHPANDE A，ROCK J，FORSYTH D.Learning large-scale automatic image colorization[C]//Proceedings of International Conference on Computer Vision，2015：567-575.
[3] 杨小军.想象的彩色时空——照片手工上色[J].中国摄影家，2011（2）：126-129.
YANG X J.Imaginative color space time[J].Photo China，2011（2）：126-129.
[4] YU F，KOLTUN V，FUNKHOUSER T.Dilated residual networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition，2017：636-644.
[5] 李维，赵晓乐，段彦隆，等.基于CNN多层面二阶特征融合的肺结节分类[J].计算机科学与探索，2020，14（9）：1590-1601.
LI W，ZHAO X L，DUAN Y L，et.al.Classification of pulmonary nodules based on CNN multi-level second-order feature fusion[J].Journal of Frontiers of Computer Science and Technology，2020，14（9）：1590-1601.
[6] IIZUKA S，SIMO-SERRA E，ISHIKAWA H.Let there be color!：Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification[J].ACM Transactions on Graphics，2016，35（4）：1-11.
[7] CHENG Z，YANG Q，SHENG B.Deep colorization[C]// Proceedings of International Conference on Computer Vision，2015：415-423.
[8] RADFORD A，METZ L，CHINTALA S.Unsupervised representation learning with deep convolutional generative adversarial networks[C]//Proceedings of International Conference on Learning Representations，2016.
[9] GOODFELLOW I J，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial networks[C]//Proceedings of International Conference on Neural Information Processing Systems，2014：2672-2680.
[10] 王海涌，梁红珠.基于改进的GAN的局部遮挡人脸表情识别[J].计算机工程与应用，2020，56（5）：141-146.
WANG H Y，LIANG H Z.Recognition of local occluded facial expressions based on improved generative adversarial network[J].Computer Engineering and Applications，2020，56（5）：141-146.
[11] 孙晓，丁小龙.基于生成对抗网络的人脸表情数据增强方法[J].计算机工程与应用，2020，56（4）：115-121.
SUN X，DING X L.Data augmentation method based on generative adversarial networks for facial expression recognition sets[J].Computer Engineering and Applications，2020，56（4）：115-121.
[12] ZHANG R，ISOLA P，EFROS A A.Colorful image colorization[C]//Proceedings of European Conference on Computer Vision，2016：649-666.
[13] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[C]//Proceedings of International Conference on Learning Representations，2015.
[14] RUSSAKOVSKY O，DENG J，SU H，et al.ImageNet large scale visual recognition challenge[J].International Journal of Computer Vision，2015，115（3）：211-252.
[15] YOO S，BAHNG H，CHUNG S，et al.Coloring with limited data：Few-shot colorization via memory augmented networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2019：11283-11292.
[16] SRIVASTAVA N，HINTON G E，KRIZHEVSKY A，et al.Dropout：A simple way to prevent neural networks from overfitting[J].Journal of Machine Learning Research，2014，15（1）：1929-1958.
[17] LIN M，CHEN Q，YAN S.Network in network[C]//Proceedings of International Conference on Learning Representations，2014.
[18] LONG J，SHELHAMER E，DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of Computer Vision and Pattern Recognition，2015：3431-3440.
[19] 陈丽雪，陈昭炯.基于Lab空间的图像检索算法[J].计算机工程，2008，34（13）：224-226.
CHEN L X，CHEN Z J.Image retrieval algorithms based on Lab space[J].Computer Engineering，2008，34（13）：224-226.
[20] KINGMA D P，BA J.Adam：A method for stochastic optimization[C]//Proceedings of International Conference on Learning Representations，2015.
[21] DUMOULIN V，VISIN F.A guide to convolution arithmetic for deep learning[J].arXiv：1603.07285，2016.
[22] SUN R.Optimization for deep learning：Theory and algorithms[J].arXiv：1912.08957，2019.
[23] STEINER B，DEVITO Z，CHINTALA S，et al.PyTorch：An imperative style，high-performance deep learning library[C]//Proceedings of International Conference on Neural Information Processing Systems，2019：8026-8037.