Mural Color Restoration via Multi-Stage Optimization

doi:10.3778/j.issn.1002-8331.2208-0275

Abstract

Abstract: Dunhuang murals are one of the most valuable and non-renewable heritages of China. The color restoration of mural images is very important for the digital protection and display of Dunhuang murals. This paper proposes a color restoration method based on multi-stage optimization in order to solve the problems of edge artifacts and color aliasing in faded mural restoration. The method obtains the multi-scale representation of the mural image using the Gaussian kernel function. Meanwhile, three encoder-decoder based transfer subnets are constructed to learn the semantic features of multi-scale representation of mural images. The transfer subnet establishes the semantic association between reference mural and faded mural to restore the mural’s color. The proposed multi-stage model adopts a coarse-to-fine optimization strategy. The cross-scale feature fusion module is constructed to realize the feature fusion of image multi-scale representation and establish the interdependency between different stages. Mural color restoration is realized through the gradual optimization of the multi-stage model. The experiments based on copying and real murals demonstrate that the proposed method can effectively eliminate the effect of noise and keep the texture information of the fading mural image while restoring the color of the mural image.

Key words: mural image, color restoration, multi-stage approach, attention mechanism, encoder-decoder network

摘要： 敦煌壁画是中国最有价值和不可再生的文化遗产之一。而壁画图像的色彩复原对敦煌壁画的数字化保护和展示具有重要意义。为了解决褪色壁画图像色彩还原过程中出现的边缘伪影和色彩混叠问题，提出一种基于多阶段优化的壁画图像色彩还原方法。该方法利用高斯核函数得到壁画图像的多尺度表示。同时，构建三个基于编码器-解码器的迁移子网来学习壁画图像多尺度表示的语义特征，在参考壁画和褪色壁画之间建立语义关联来恢复壁画颜色。采用由粗到细的优化策略，在各个阶段间构建跨尺度特征融合模块实现图像多尺度表示的特征融合，建立不同阶段间的特征依赖关系。通过多阶段逐步优化，实现壁画图像的色彩还原。通过对临摹壁画和真实壁画的实验表明，该方法能够较有效地消除噪声影响，在还原壁画色彩的同时能较好地保持褪变色壁画图像的边缘纹理信息。

关键词: 壁画图像, 色彩还原, 多阶段方法, 注意力机制, 编码器-解码器网络

XU Zhigang, CHEN Shicheng, ZHU Honglei. Mural Color Restoration via Multi-Stage Optimization[J]. Computer Engineering and Applications, 2024, 60(2): 162-170.

徐志刚, 陈士成, 朱红蕾. 基于多阶段优化的壁画图像色彩还原[J]. 计算机工程与应用, 2024, 60(2): 162-170.

References

[1] 魏宝刚, 潘云鹤. 基于框架和规则的古代壁画色彩复原[J]. 模式识别与人工智能, 1999, 12(4): 473-479.
WEI B, PAN Y H. Color restoration of ancient murals based on framework and rules[J]. Pattern Recognition and Artificial Intelligence, 1999, 12(4): 473-479.
[2] REINHARD E, ADHIKHMIN M, GOOCH B, et al. Color transfer between images[J]. IEEE Computer Graphics and Applications, 2001, 21(5): 34-41.
[3] 赵国英, 向世明, 李华. 高阶矩在颜色传输中的应用[J]. 计算机辅助设计与图形学学报, 2004(1): 62-66.
ZHAO G, XIANG S M, LI H. Application of higher order moments in color transmission[J]. Journal of Computer-Aided Design and Computer Graphics, 2004(1): 62-66.
[4] 林怡. 壁画图像智能色彩处理技术及其应用[D]. 杭州: 浙江大学, 2002.
LIN Y. Intelligent color processing technology of mural image and its application[D]. Hangzhou: Zhejiang University, 2002.
[5] 李宣妮. 古壁画虚拟修复技术研究[D]. 西安: 西安建筑科技大学, 2014.
LI X N. Research on virtual restoration of the ancient mural[D]. Xi’an: Xi’an University of Architecture and Technology, 2014.
[6] ZHANG R, ISOLA P, EFROS A A. Colorful image colorization[C]//Proceedings of the 14th European Conference on Computer Vision, 2016: 649-666.
[7] GATYS L A, ECKER A S, BETHGE M. Image style transfer using convolutional neural networks[C]//Procddings of 29th IEEE Conference on Computer Vision and Pattern Recognition, 2016: 2414-2423.
[8] 徐志刚, 尹文玉, 朱旭锋. 基于双重约束CNN的壁画图像色彩还原[J]. 华中科技大学学报(自然科学版), 2020, 48(6): 6-12.
XU Z G, YIN W Y, ZHU X F. Color restoration of mural image based on double constrained convolutional neural network[J]. Journal of Huazhong University of Science and Technology(Natural Science Edition), 2020, 48(6): 6-12.
[9] HE M M, CHEN D D, LIAO J, et al. Deep exemplar-based colorization[J]. ACM Transactions on Graphics, 2018, 37(4): 1-16.
[10] XU Z, WANG T, FANG F, et al. Stylization-based architecture for fast deep exemplar colorization[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 9360-9369.
[11] HUANG X, BELONGIE S. Arbitrary style transfer in real-time with adaptive instance normalization[C]//5th International Conference on Learning Representations, 2019.
[12] BLANCH M G, MRAK M, SMEATON A F, et al. End-to-end conditional GAN-based architectures for image colourisation[C]//IEEE 21st International Workshop on Multimedia Signal Processing, 2019: 1-6.
[13] PARK D Y, LEE K H. Arbitrary style transfer with style-attentional networks[C]//Proceedings of 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 5873-5881.
[14] SU J W, CHU H K, HUANG J B. Instance-aware image colorization[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 7965-7974.
[15] CAO J, ZHANG Z, ZHAO A, et al. Ancient mural restoration based on a modified generative adversarial network[J]. Heritage Science, 2020, 8 (1): 1-14.
[16] WANG H L, HAN P H, CHEN Y M, et al. Dunhuang mural restoration using deep learning[C]//SIGGRAPH Asia 2018 Technical Briefs, 2018: 1-4.
[17] WANG X, GIRSHICK R, GUPTA A, et al. Non-local neural networks[C]//Proceedings of 31st Meeting of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 7794-7803.
[18] KIM H, PAPAMAKARIOS G, MNIH A. The Lipschitz constant of self-attention[C]//Proceedings of the 38th International Conference on Machine Learning, 2021: 5562-5571.
[19] PIRRONE R, CANNELLA V, GAMBINO O, et al. WikiArt: an ontology-based information retrieval system for arts[C]//Proceedings of 9th International Conference on Intelligent Systems Design and Applications, 2009: 913-918.
[20] 高山. 再现敦煌: 壁画艺术临摹集[M]. 南京: 江苏凤凰美术出版社, 2018.
GAO S. Reproduction of Dunhuang: a copy collection of mural art[M]. Nanjing: Jiangsu Phoenix Art Publishing House, 2018.
[21] 中国敦煌壁画全集编辑委员会. 中国敦煌壁画全集[M]. 天津: 天津人民美术出版社, 2006.
Editorial Committee of Complete Collection of Dunhuang Murals in China[M]. Tianjin: Tianjin People’s Fine Arts Publishing House, 2006.
[22] MITTAL A, SOUNDARARAJAN R, BOVIK A C. Making a'completely blind' image quality analyzer[J]. IEEE Signal Processing Letters, 2013, 20 (3): 209-212.
[23] MITTAL A, MOORTHY A K, BOVIK A C. No-reference image quality assessment in the spatial domain[J]. IEEE Transactions on Image Processing, 2012, 21 (12): 4695-4708.
[24] WU X, HU Z, SHENG L, et al. StyleFormer: real-time arbitrary style transfer via parametric style composition[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision, 2021: 14598-14607.