Stereo Image Super-Resolution Network for Loop Structure and PAM

doi:10.3778/j.issn.1002-8331.2101-0509

Abstract

Abstract: The second viewpoint of stereo image provides more detail information for image super-resolution reconstruction network. In order to make full use of complementary information of stereo image, a stereo image super-resolution reconstruction network for deep learning loop structure and parallax attention module（PAM） is proposed. Firstly, the network feature extraction module is composed of loop structure composed of MJR-ASPP+ and extended residual block. In loop structure, the mixed jumping residual（MJR） can aggregate the information of different depths in the network. The improved atrous space pyramid pooling block（ASPP+） is used to extract multi-scale features of the image, and the extended residual block can fuse multi-level features and denoise effectively. Then, the parallax attention module is introduced to obtain the global correspondence in the stereo image, and the useful information of the image pair is integrated. Finally, the super-resolution left（right） image is reconstructed through sub-pixel layer, and FReLU is used in the whole network to improve the spatial correlation efficiency. The network has achieved excellent results in Middlebury, KITTI 2012, KITTI 2015 and Flickr1024 public data sets. The experimental results show that the network has better super-resolution performance.

Key words: stereo image super-resolution reconstruction, deep learning, loop structure, parallax attention module, mixed jumping residual, atrous space pyramid pooling block

摘要： 双目图像第二视点为图像超分辨率重建网络提供更多的细节信息，为更充分利用双目图像的互补信息，提出一种基于深度学习的回环结构与视差注意力模块（PAM）相结合的双目图像超分辨率重建网络。该网络特征提取模块由MJR-ASPP+构成的回环结构与扩张残差块交替级联而成，回环结构中混合跳跃式残差（MJR）能聚合网络中不同深度的信息，改进空洞空间金字塔池化块（ASPP+）用于提取图像多尺度特征，扩张残差块融合多级特征的同时有效去噪；引入视差注意力模块获取双目图像中的全局对应关系，集成图像对的有用信息；通过亚像素层重建出超分辨率左（右）图，并将FReLU用于整个网络中提高捕获空间相关性效率。该网络在Middlebury、KITTI2012、KITTI2015和Flickr1024四个公开数据集中都取得了优异结果，实验结果表明该网络具有更好的超分辨率性能。

关键词: 双目图像超分辨率重建, 深度学习, 回环结构, 视差注意力模块, 混合跳跃式残差, 空洞空间金字塔池化

LI Xue, ZHANG Hongying, WU Yadong, LIAN Weiwen. Stereo Image Super-Resolution Network for Loop Structure and PAM[J]. Computer Engineering and Applications, 2022, 58(17): 239-248.

李雪, 张红英, 吴亚东, 廉炜雯. 回环结构与PAM结合的双目图像超分辨率网络[J]. 计算机工程与应用, 2022, 58(17): 239-248.

References

[1] 谢海平，谢凯利，杨海涛.图像超分辨率方法研究进展[J].计算机工程与应用，2020，56（19）：34-41.
XIE H P，XIE K L，YANG H T.Research progress of image super-resolution methods[J].Computer Engineering and Applications，2020，56（19）：34-41.
[2] 王冲，尚晓清.基于多字典的单幅图像超分辨率重建[J].计算机工程与应用，2019，55（5）：197-202.
WANG C，SHANG X Q.Single image super-resolution by multi-dictionary learning[J].Computer Engineering and Applications，2019，55（5）：197-202.
[3] BATZ M，EICHENSEER A，SEILER J，et al.Hybird super-resolution combining example-based single-image and interpolation-based multi-image reconstruction approaches[C]//2015 IEEE International Conference on Image Processing（ICIP），2015：58-62.
[4] KIM K I，KWON Y.Single-image super-resolution using sparse regression and natural image prior[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2010，32（6）：1127-1133.
[5] IRANI M，PELEG S.Improving resolution by image regi-stration[J].CVGIP：Graphical Models and Image Processing，1991，53（3）：231-239.
[6] TOM B C，KATSAGGELOS A K.Iterative algorithm for improving the resolution of video sequences[C]//Proceedings of Visual Communications and Image Processing，1996：1430-1438.
[7] LIAN Q S，ZHANG W.Super-resolution reconstruction algorithm based on classification of sparse representations of image blocks[J].Journal of Electronics，2012，40（5）：920-925.
[8] 李云红，王珍，张凯兵，等.基于学习的图像超分辨重建方法综述[J].计算机工程与应用，2018，54（15）：13-21.
LI Y H，WANG Z，ZHANG K B，et al.Survey on example learning based single image super-resolution technique[J].Computer Engineering and Applications，2018，54（15）：13-21.
[9] DONG C，LOY C C，HE K，et al.Image super-resolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2014，38（2）：295-307.
[10] DONG C，CHEN C L，TANG X.Accelerating the super-resolution convolutional neural network[C]//European Conference on Computer Vision.Cham：Springer，2016：391-407.
[11] SHI W，CABALLERO J，HUSZáR F，et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//IEEE Conference on Computer Vision and Pattern Recognition，2016：1874-1883.
[12] TAI Y，YANG J，LIU X.Image super-resolution via deep recursive residual network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：3147-3155.
[13] HUANG G，LIU Z，VAN DER MAATEN L，et al.Densely connected convolutional networks[C]//Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition，2017：2261-2269.
[14] KIM J，KWON LEE J，MU LEE K.Accurate image super-resolution using very deepconvolutional networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：1646-1654.
[15] KIM J，LEE J K，LEE K M.Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Las Vegas，NV，USA，2016：1637-1645.
[16] LEDIG C，THEIS L，HUSZáR F，et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition，Honolulu，USA，2017：4681-4690.
[17] JEON D S，BAEK S H，CHOI I，et al.Enhancing the spatial resolution of stereo images using a parallax prior[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），2018：1721-1730.
[18] WANG L，WANG Y，LIANG Z，et al.Learning parallax attention for stereo image super-resolution[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），2019：12250-12259.
[19] YING X，WANG Y，WANG L，et al.A stereo attention module for stereo image super-resolution[J].IEEE Signal Processing Letters，2020，27（99）：496-500.
[20] CHEN L C，PAPANDREOU G，SCHROFF F，et al.Rethinking atrous convolution for semantic image segmentation[J].arXiv：1706.05587，2017.
[21] MA N，ZHANG X，SUN J.Funnel activation for visual recognition[C]//Proceedings of the European Conference on Computer Vision（ECCV），2020：351-368.