Global Structure-Aware and Local Enhancement Network for Face Super-Resolution

doi:10.3778/j.issn.1002-8331.2406-0451

Abstract

Abstract: Aiming at the limitations on the generalization and practicality of face super-resolution methods caused by the reliance on accurate facial key point detection and alignment based on prior knowledge, this paper proposes a global structure-aware and local enhancement network(GSLEN). The global feature path models the overall face structure using multi-scale residual learning and attention mechanisms. The local enhancement path utilizes an estimated face parsing map to enhance reconstruction quality and reduce the network’s dependency on facial priors. Finally, the adaptive feature fusion unit generates clear and detail-rich face images. The improved algorithm is applied to the CelebA and Helen datasets. The experimental evaluations demonstrate that the proposed network outperforms other state-of-the-art methods in face super-resolution results.

Key words: face super-resolution, face illusions, face priors, attention mechanism

摘要： 针对人脸超分辨率在基于先验引导的背景下对精确的人脸关键点检测和面部对齐的依赖，这一依赖对方法的泛化性和实用性带来了一定限制。为了克服这一问题，提出了一种全局结构感知与局部增强网络。该方法通过双路径网络结构和特征融合，实现了全局人脸表示和局部细节表示的有效结合。全局特征路径基于多尺度残差学习和注意力机制对整体人脸结构进行建模，局部增强路径使用估计的人脸解析图来提高重建质量，并减少网络对人脸先验的依赖。自适应特征融合单元能够生成清晰且细节丰富的人脸图像。在CelebA和Helen数据集上的实验评估显示，所提出的网络在人脸超分辨率结果上优于其他先进方法。

关键词: 人脸超分辨率, 面部幻觉, 面部先验, 注意力机制

TAN Shuqiu, LING Zhihao, PAN Jiahao, LIU Yahui. Global Structure-Aware and Local Enhancement Network for Face Super-Resolution[J]. Computer Engineering and Applications, 2025, 61(5): 222-232.

谭暑秋, 凌志豪, 潘嘉豪, 刘亚辉. 面向人脸超分辨率的全局结构感知与局部增强网络[J]. 计算机工程与应用, 2025, 61(5): 222-232.

References

[1] BAKER S, KANADE T. Hallucinating faces[C]//Proceedings of the 4th IEEE International Conference on Automatic Face and Gesture Recognition, 2000: 83-88.
[2] LIU C, SHUM H Y, ZHANG C S. A two-step approach to hallucinating faces: global parametric model and local nonparametric model[C]//Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001.
[3] SIU W C, HUNG K W. Review of image interpolation and super-resolution[C]//Proceedings of the 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference, 2012: 1-10.
[4] CHANG H, YEUNG D Y, XIONG Y. Super-resolution through neighbor embedding[C]//Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004.
[5] WANG X, TANG X. Hallucinating face by eigentransformation[J]. IEEE Transactions on Systems, Man, and Cybernetics, 2005, 35(3): 425-434.
[6] CHUDASAMA V, NIGHANIA K, UPLA K, et al. E-ComSup
ResNet: enhanced face super-resolution through compact network[J]. IEEE Transactions on Biometrics, Behavior, and Identity Science, 2021, 3(2): 166-179.
[7] JIANG K, WANG Z, YI P, et al. Dual-path deep fusion network for face image hallucination[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 33(1): 378-391.
[8] CHEN C, LI X, YANG L, et al. Progressive semantic-aware style transformation for blind face restoration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 11896-11905.
[9] CHEN Y, TAI Y, LIU X, et al. FSRNet: end-to-end learning face super-resolution with facial priors[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 2492-2501.
[10] WANG C, ZHONG Z, JIANG J, et al. Parsing map guided multi-scale attention network for face hallucination[C]//Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing , 2020: 2518-2522.
[11] LI X, CHEN C, ZHOU S, et al. Blind face restoration via deep multi-scale component dictionaries[C]//Proceedings of the European Conference on Computer Vision, 2020: 399-415.
[12] LI X, LI W, REN D, et al. Enhanced blind face restoration with multi-exemplar images and adaptive spatial feature fusion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 2706-2715.
[13] WANG K, ORAMAS J, TUYTELAARS T. Multiple exemplars-based hallucination for face super-resolution and editing[C]//Proceedings of the Asian Conference on Computer Vision, 2020: 258-273.
[14] ZHU S, LIU S, LOY C C, et al. Deep cascaded bi-network for face hallucination[C]//Proceedings of the 14th European Conference on Computer Vision, 2016: 614-630.
[15] BULAT A, TZIMIROPOULOS G. Super-FAN: integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with GANs[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 109-117.
[16] MA C, JIANG Z, RAO Y, et al. Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 5569-5578.
[17] YU X, FERNANDO B, GHANEM B, et al. Face super-resolution guided by facial component heatmaps[C]//Proceedings of the European Conference on Computer Vision, 2018: 217-233.
[18] ZHOU E, FAN H, CAO Z, et al. Learning face hallucination in the wild[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2015: 3871-3877.
[19] CHEN C, GONG D, WANG H, et al. Learning spatial attention for face super-resolution[J]. IEEE Transactions on Image Processing, 2020, 30: 1219-1231.
[20] ZENG K, WANG Z, LU T, et al. Self-attention learning network for face super-resolution[J]. Neural Networks, 2023, 160: 164-174.
[21] LU T, WANG Y, ZHANG Y, et al. Face hallucination via split-attention in split-attention network[C]//Proceedings of the 29th ACM International Conference on Multimedia, 2021: 5501-5509.
[22] WANG C, JIANG J, ZHONG Z, et al. Super-resolving face image by facial parsing information[J]. IEEE Transactions on Biometrics, Behavior, and Identity Science, 2023, 5(4): 435-448.
[23] ZHANG C, LIU Z. Face super-resolution with progressive embedding of multi-scale face priors[C]//Proceedings of the 2022 IEEE International Joint Conference on Biometrics, 2022: 1-8.
[24] ZHUANG C, LI M, ZHANG K, et al. Multi-level landmarguided deep network for face super-resolution[J]. NeuralNetworks, 2022, 152: 276-286.
[25] ZHANG Y, WU Y, CHEN L. MSFSR: a multi-stage face super-resolution with accurate facial representation via enhanced facial boundaries[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020: 504-505.
[26] HU X, REN W, YANG J, et al. Face restoration via plug-and-play 3D facial priors[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(12): 8910-8926.
[27] KALAROT R, LI T, PORIKLI F. Component attention guided face super-resolution network: Cagface[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020: 370-380.
[28] SHEN Z, LAI W S, XU T, et al. Exploiting semantics for face image deblurring[J]. International Journal of Computer Vision, 2020, 128(7): 1829-1846.
[29] YASARLA R, PERAZZI F, PATEL V M. Deblurring face images using uncertainty guided multi-stream semantic networks[J]. IEEE Transactions on Image Processing, 2020, 29: 6251-6263.
[30] YU Y, ZHANG P, ZHANG K, et al. Multiprior learning via neural architecture search for blind face restoration[J]. arXiv:2206.13962, 2023.
[31] WANG C, JIANG J, ZHONG Z, et al. Propagating facial prior knowledge for multitask learning in face super-resolution[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(11): 7317-7331.
[32] MEI Y, FAN Y, ZHANG Y, et al. Pyramid attention network for image restoration[J]. International Journal of Computer Vision, 2023, 131(12): 3207-3225.
[33] WANG Q, WU B, ZHU P, et al. ECA-Net: efficient channel attention for deep convolutional neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 11534-11542.
[34] YU C, WANG J, PENG C, et al. BiseNet: bilateral segmentation network for real-time semantic segmentation[C]//Proceedings of the European Conference on Computer Vision, 2018: 325-341.
[35] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
[36] LIU Y, JIA Q, FAN X, et al. Cross-SRN: structure-preserving super-resolution network with cross convolution[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 32(8): 4927-4939.
[37] WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of the European Conference on Computer Vision, 2018: 3-19.
[38] LIU Z, LUO P, WANG X, et al. Deep learning face attributes in the wild[C]//Proceedings of the IEEE International Conference on Computer Vision, 2015: 3730-3738.
[39] LE V, BRANDT J, LIN Z, et al. Interactive facial feature localization[C]//Proceedings of the 12th European Conference on Computer Vision, 2012: 679-692.
[40] BALTRU?AITIS T, ROBINSON P, MORENCY L P. Openface: an open source facial behavior analysis toolkit[C]//Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision, 2016: 1-10.
[41] ZHANG Y, LI K, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the European Conference on Computer Vision, 2018: 286-301.
[42] DONG C, LOY C C, HE K, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38(2): 295-307.