Parallel Decoder Image Inpainting Method for Edge-Terminal Collaboration

doi:10.3778/j.issn.1002-8331.2101-0178

Abstract

Abstract: Focusing on the problems of poor effect, long response time and high energy consumption of existing neural network image inpainting methods for mobile terminal equipment, this paper puts forward an edge-terminal collaboration-oriented image inpainting method for parallel decoder and calculating offloading strategy. Firstly, based on the edge-terminal collaboration characteristics of mobile edge computing（MEC） technology, the edge-terminal collaboration-oriented, gated convolutional image inpainting network of parallel decoder, ETG-Net（edge-terminal gated convolution network）, is proposed. Secondly, through the edge-terminal sharing of weights, the efficiency of image inpainting and training is improved, and the independent working ability of the mobile terminals is retained. Finally, based on the calculating offloading strategy, some computing tasks of image inpainting will be selectively offloaded to the edge cloud, so as to further reduce the calculation delay and energy consumption of the terminal nodes. Experimental results show that, compared with the advanced models in recent years, the proposed model not only ensures the quality of image inpainting, but also solves the problems of deploying image inpainting models on mobile terminal devices, and reduces the time delay of task response.

Key words: image inpainting, deep learning, generative adversarial network, mobile edge computing, gated convolution, weight sharing

摘要： 针对现有神经网络图像修复方法在移动终端设备上部署存在效果差、响应时间长、高能耗的问题，提出了一种面向边-端协同的并行解码器图像修复方法及计算卸载策略。结合移动边缘计算（mobile edge computing，MEC）技术边-端协同的特性，提出一种面向边-端协同的并行解码器门控卷积图像修复网络ETG-Net（edge-terminal gated convolution network）。通过边-端共享权值的方式，提升图像修复及训练效率，并保留移动终端的独立工作能力。基于计算卸载决策，将图像修复部分计算任务有选择地卸载至边缘云，进一步降低终端节点的计算时延和能耗。实验结果表明，与近年来先进的模型相比，所提模型在保证图像修复质量的同时，解决了移动终端设备上部署图像修复模型存在的问题，降低了任务的响应时延。

关键词: 图像修复, 深度学习, 生成对抗网络, 移动边缘计算, 门控卷积, 权值共享

HUO Xiangzuo, ZHANG Wendong, TIAN Shengwei, HOU Shuxiang. Parallel Decoder Image Inpainting Method for Edge-Terminal Collaboration[J]. Computer Engineering and Applications, 2022, 58(16): 257-264.

霍相佐, 张文东, 田生伟, 侯树祥. 面向边-端协同的并行解码器图像修复方法[J]. 计算机工程与应用, 2022, 58(16): 257-264.

References

[1] MACH P，BECVAR Z.Mobile edge computing：a survey on architecture and computation offloading[J].IEEE Communications Surveys & Tutorials，2017，19（3）：1628-1656.
[2] 谢人超，廉晓飞，贾庆民，等.移动边缘计算卸载技术综述[J].通信学报，2018，39（11）：138-155.
XIE R C，LIAN X F，JIA Q M，et al.Survey on computation offloading in mobile edge computing[J].Journal on Communications，2018，39（11）：138-155.
[3] BERTALMIO M，SAPIRO G，CASELLES V，et al.Image inpainting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques，2000：417-424.
[4] CRIMINISI A，PéREZ P，TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing，2004，13（9）：1200-1212.
[5] 师晓波，蔺素珍，桑新亚.利用结构连续块和多尺度变换的缺损图像修复[J].计算机工程与应用，2019，55（8）：189-194.
SHI X B，LIN S Z，SANG X Y.Restoring method of defects based on continuous patch of structure and multiscale transform[J].Computer Engineering and Applications，2019，55（8）：189-194.
[6] SIMAKOV D，CASPI Y，SHECHTMAN E，et al.Summarizing visual data using bidirectional similarity[C]//Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition，2008：1-8.
[7] BARNES C，SHECHTMAN E，FINKELSTEIN A，et al.PatchMatch：a randomized correspondence algorithm for structural image editing[J].ACM Transactions on Graphics，2009，28（3）：24.
[8] PATHAK D，KRAHENBUHL P，DONAHUE J，et al.Context encoders：feature learning by inpainting[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition，2016：2536-2544.
[9] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems，2014：2672-2680.
[10] 刘昱，刘厚泉.基于对抗训练和卷积神经网络的面部图像修复[J].计算机工程与应用，2019，55（2）：110-115.
LIU Y，LIU H Q.Facial image restoration based on adversarial training and convolutional neural network[J].Computer Engineering and Applications，2019，55（2）：110-115.
[11] IIZUKA S，SIMO-SERRA E，ISHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Graphics，2017，36（4）：107.
[12] YU J，LIN Z，YANG J，et al.Generative image inpainting with contextual attention[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition，2018：5505-5514.
[13] LIU G，REDA F A，SHIH K J，et al.Image inpainting for irregular holes using partial convolutions[C]//Proceedings of the 15th European Conference on Computer Vision，2018：85-100.
[14] ZENG Y，FU J，CHAO H，et al.Learning pyramid-context encoder network for high-quality image inpainting[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition，2019：1486-1494.
[15] SAGONG M，SHIN Y，KIM S，et al.PEPSI：fast image inpainting with parallel decoding network[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition，2019：11360-11368.
[16] NAZERI K，NG E，JOSEPH T，et al.Edgeconnect：generative image inpainting with adversarial edge learning[J].arXiv：1901.00212，2019.
[17] YU J，LIN Z，YANG J，et al.Free-form image inpainting with gated convolution[C]//Proceedings of the 2019 IEEE International Conference on Computer Vision，2019：4471-4480.
[18] SHI W，CAO J，ZHANG Q，et al.Edge computing：vision and challenges[J].IEEE Internet of Things Journal，2016，3（5）：637-646.
[19] 张展，张宪琦，左德承，等.面向边缘计算的目标追踪应用部署策略研究[J].软件学报，2020，31（9）：71-88.
ZHANG Z，ZHANG X Q，ZUO D C，et al.Research on target tracking application deployment strategy for edge computing[J].Journal of Software，2020，31（9）：71-88.
[20] 吴柳青，朱晓荣.基于边-端协同的任务卸载资源分配联合优化算法[J].电信科学，2020，36（3）：46-56.
WU L Q，ZHU X R.Joint optimization algorithm for task offloading resource allocation based on edge-end collaboration[J].Telecommunications Science，2020，36（3）：46-56.
[21] LIM W Y B，LUONG N C，HOANG D T，et al.Federated learning in mobile edge networks：a comprehensive survey[J].IEEE Communications Surveys & Tutorials，2020，22（3）：2031-2063.
[22] LIU Z，LUO P，WANG X，et al.Deep learning face attributes in the wild[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision，2015：3730-3738.
[23] ZHOU B，LAPEDRIZA A，KHOSLA A，et al.Places：a 10 million image database for scene recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，40（6）：1452-1464.
[24] YU F，KOLTUN V.Multi-scale context aggregation by dilated convolutions[J].arXiv：1511.07122，2015.
[25] NOH H，HONG S，HAN B.Learning deconvolution network for semantic segmentation[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision，2015：1520-1528.
[26] MIYATO T，KATAOKA T，KOYAMA M，et al.Spectral normalization for generative adversarial networks[J].arXiv：1802.05957，2018.
[27] 董思岐，李海龙，屈毓锛，等.移动边缘计算中的计算卸载策略研究综述[J].计算机科学，2019，46（11）：32-40.
DONG S Q，LI H L，QU Y B，et al.Survey of research on computation unloading strategy in mobile edge computing[J].Computer Science，2019，46（11）：32-40.
[28] HEUSEL M，RAMSAUER H，UNTERTHINER T，et al.GANs trained by a two time-scale update rule converge to a local Nash equilibrium[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems，2017：6626-6637.