Video Super-Resolution Reconstruction Algorithm Based on Optical Flow Residual

doi:10.3778/j.issn.1002-8331.2012-0409

Abstract

Abstract: With the development of convolutional neural network, video super resolution algorithm has achieved remarkable success. Because of the complex dependence between frames, traditional methods lack the ability to model the complex dependence, which makes it difficult to accurately estimate and compensate the motion in the process of video super-resolution reconstruction. A reconstruction based on optical flow residual network is put forward, adjacent video frame of the complementary information by using dense residual network in the low spatial resolution is obtained, again through the pyramid structure, the flow of high resolution video frame is predicted, and then through the subpixel convolution layer will be low resolution video frames into a high-resolution video frame, and the high resolution video frames and prediction of high resolution optical flow motion compensation. Finally, it is input into the super-resolution fusion network to get better results, and a new loss function training network is proposed, which can better constrain the network. Experimental results on open data sets show that the reconstruction effect is improved in terms of peak signal-to-noise ratio, structural similarity and subjective visual effect.

Key words: video super resolution, optical flow estimation, dense residual block

摘要： 随着卷积神经网络的发展，视频超分辨率算法取得了显著的成功。因为帧与帧之间的依赖关系比较复杂，所以传统方法缺乏对复杂的依赖关系进行建模的能力，难以对视频超分辨率重建的过程进行精确地运动估计和补偿。因此提出一个基于光流残差的重建网络，在低分辨率空间使用密集残差网络得到相邻视频帧的互补信息，通过金字塔的结构来预测高分辨率视频帧的光流，通过亚像素卷积层将低分辨率的视频帧变成高分辨率视频帧，并将高分辨率的视频帧与预测的高分辨率光流进行运动补偿，将其输入到超分辨率融合网络来得到更好的效果，提出新的损失函数训练网络，能够更好地对网络进行约束。在公开数据集上的实验结果表明，重建效果在峰值信噪比、结构相似度、主观视觉的效果上均有提升。

关键词: 视频超分辨率, 光流估计, 密集残差块

WU Hao, LAI Huicheng, QIAN Xuze, CHEN Hao. Video Super-Resolution Reconstruction Algorithm Based on Optical Flow Residual[J]. Computer Engineering and Applications, 2022, 58(15): 220-228.

吴昊, 赖惠成, 钱绪泽, 陈豪. 基于光流残差的视频超分辨率重建算法[J]. 计算机工程与应用, 2022, 58(15): 220-228.

References

[1] GARCIA D C，DOREA C，DE QUEIROZ R L.Super resolution for multiview images using depth information[J].IEEE Transactions on Circuits and Systems for Video Technology，2012，22（9）：1249-1256.
[2] WANG N，TAO D，GAO X，et al.A comprehensive survey to face hallucination[J].International Journal of Computer Vision，2014，106（1）：9-30.
[3] AFONSO M，ZHANG F，BULL D R.Video compression based on spatio-temporal resolution adaptation[J].IEEE Transactions on Circuits and Systems for Video Technology，2019，29（1）：275-280.
[4] ZHONG Y，ZHANG L.Remote sensing image subpixel mapping based on adaptive differential evolution[J].IEEE Transactions on Systems Man & Cybernetics Part B Cybernetics A Publication of the IEEE Systems Man & Cybernetics Society，2012，42（5）：1306.
[5] ZHANG L，ZHANG H，SHEN H，et al.A super-resolution reconstruction algorithm for surveillance images[J].Signal Processing，2010，90（3）：848-859.
[6] YUE L，SHEN H，LI J.Image super-resolution：the techniques，applications，and future[J].Signal Processing，2016，128（11）：389-408.
[7] K?HLER T，HUANG X，SCHEBESCH F.Robust multiframe super-resolution employing iteratively re-weighted minimization[J].IEEE Transactions on Computational Imaging，2016，2（1）：42-58.
[8] LIU C，SUN D.A Bayesian approach to adaptive video super resolution[C]//2011 IEEE Conference on Computer Vision and Pattern Recognition（CVPR），2011：209-216.
[9] HUANG Y，WANG W.Bidi rectional recurrent convolutional networks for multi-frame super-resolution[C]//Advances in Neural Information Processing Systems（NIPS），2015：235-243.
[10] KONG D，HAN M，XU W，et al.A conditional random field model for video super-resolution[C]//Advances in International Conference on Pattern Recognition（ICPR），2006：619-622.
[11] NASROLLAHI K，MOESLUND T B.Extracting a good quality frontal face image from a low-resolution video sequence[J].IEEE Transactions on Circuits & Systems for Video Technology，2011，21（10）：1353-1362.
[12] DONG C，LOY C C，HE K，et al.Learning a deep convolutional network for image super-resolution[C]//2014 IEEE?European Conference on Computer Vision（ECCV），2014：184-199.
[13] KIM J，LEE J K，LEE K M.Accurate image super-resolution using very deep convolutional networks[C]// 2016 IEEE Conference on Computer Vision & Pattern Recognition（CVPR），2016：1646-1654.
[14] LIM B，SON S，KIM H，et al.Enhanced deep residual networks for single image super-resolution[C]//2017 IEEE Conference on Computer Vision & Pattern Recognition（CVPR），2017：1132-1140.
[15] LEDIG C，THEIS L，HUSZAR F，et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of the Computer Vision and Pattern Recognition（CVPR），2017：105-114.
[16] 彭晏飞，高艺，杜婷婷，等.生成对抗网络的单图像超分辨率重建方法[J].计算机科学与探索，2020，14（9）：1612-1620.
PENG Y F，GAO Y，DU T T，et al.Single image super-resolution reconstruction method for generative adversarial network[J].Journal of Frontiers of Computer Science and Technology，2020，14（9）：1612-1620.
[17] 唐家军，刘辉，胡雪影.功能型复合深度网络的图像超分辨率重建[J].计算机科学与探索，2020，14（8）：1368-1379.
TANG J J，LIU H，HU X Y.Image super-resolution reconstruction of functional composite deep network[J].Journal of Frontiers of Computer Science and Technology，2020，14（8）：1368-1379.
[18] DAI Q Q，YOO S H，KAPPELER A，et al.Dictionary-based multiple frame video super-resolution[C]//2015 IEEE International Conference on Image Processing（ICIP），2015：83-87.
[19] YANG G S，RAMANAN D.Volumetric correspondence networks for optical flow[C]//Advances in Neural Information Processing Systems（NIPS），2019：793-803.
[20] SAJJADI M S M，VEMULAPALLI R，BROWN M.Frame-recurrent video super-resolution[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition（CVPR），2018：6626-6634.
[21] KAPPELER A，YOO S，DAI Q，et al.Video super-resolution with convolutional neural networks[J].IEEE Transactions on Computational Imaging，2016，2（2）：109-122.
[22] CABALLERO J，LEDIG C，ANDREW A，et al.Real-time video super-resolution with spatio-temporal networks and motion compensation[C]//Advances in Computer Vision and Pattern Recognition（CVPR），2017：2848-2857.
[23] LIAO R，TAO X，LI R，et al.Video super-resolution via deep draft-ensemble learning[C]//Proceedings of the IEEE International Conference on Computer Vision（ICCV），2015：531-539.
[24] JEAN-YVES B.Pyramidal implementation of the lucas kanade feature tracker description of the algorithm[R].USA：Intel Corporation Microprocessor Research Labs，2000.
[25] ZHAO H，GALLO O，FROSIO I，et al.Loss functions for image restoration with neural networks[J].IEEE Transactions on Computational Imaging，2017，3（1）：47-57.
[26] TAO X，GAO H，LIAO R，et al.Detail-revealing deep video super-resolutionin[C]//Advances in Computer Vision and Pattern Recognition（CVPR），2017：4482-4490.
[27] ZHANG Y L，LI K P，LI K，et al.Image super-resolution using very deep residual channel attention networks[C]//European Conference on Computer Vision（ECCV），2018：294-310.
[28] WANG L，GUO Y.Learning for video super-resolution through hr optical flow estimation[C]//Advances in Asian Conference on Computer Vision（ACCV），2018：514-529.
[29] YING X，WANG L，WANG Y，et al.Deformable 3D Convolution for Video Super-Resolution[J].IEEE Signal Processing Letters，2020，27（4）：1500-1504.
[30] YING X Y，WANG L G.Zooming slow-mo：fast and accurate one-stage space-time video super-resolution[C]//Advances in Computer Vision and Pattern Recognition（CVPR），2020：3367-3376.