从放缩到重构的Swin Transformer图像矩形化网络

doi:10.3778/j.issn.1002-8331.2403-0222

摘要/Abstract

摘要： 图像矩形化旨在解决拼接后的图像所存在的边界不规则问题。现有的矩形化方法通过多步扭曲处理来实现图像的矩形化处理。但是这些方法仍然存在一定的内容失真及边界破损等问题。为了解决这些问题，提出了一种单步扭曲处理的图像矩形化解决方案（IRFormer）。具体来说，结合了尺度放缩策略，构建了一个基于Swin Transformer架构的低分辨率单步扭曲分支；结合轻量化策略，构建了一个高分辨率重建及边界修复的分支。通过广泛的实验，验证了IRFormer在多种场景中均具有良好的矩形化表现，具有较高的内容保真性和边界完整性。在定性和定量比较中，IRFormer均展现出了最先进的矩形化性能。

关键词: 图像矩形化, 单级网格预测, 尺度放缩, Swin Transformer, 超分辨率重建

Abstract: Image rectangling aims to resolve the boundary irregularities present in the stitched images. Existing deskew methods achieve deskew of images by multi-step distortion processing. However, these methods still have certain problems such as content distortion and boundary breakage. To solve these problems, a single-step distortion processing solution for image rectangling (IRFormer) is proposed. Specifically, a low-resolution single-step warping branch based on the Swin Transformer architecture is constructed by combining the scale deflation strategy, and a high-resolution reconstruction and boundary restoration branch is constructed by combining the lightweighting strategy. Through extensive experiments, it is verified that IRFormer has good rectangling performance with high content fidelity and boundary integrity in a wide range of scenes. IRFormer demonstrates state-of-the-art rectangularisation performance in both qualitative and quantitative comparisons.

Key words: image rectangling, single-stage mesh prediction, scale scaling, Swin Transformer, super-resolution reconstruction

杨利春, 党建武, 王梦思, 张天胤, 田彬. 从放缩到重构的Swin Transformer图像矩形化网络[J]. 计算机工程与应用, 2025, 61(12): 258-266.

YANG Lichun, DANG Jianwu, WANG Mengsi, ZHANG Tianyin, TIAN Bin. Swin Transformer Image Rectangling Network from Scaling to Reconstruction[J]. Computer Engineering and Applications, 2025, 61(12): 258-266.

参考文献

[1] NIE L, LIN C Y, LIAO K, et al. A view-free image stitching network based on global homography[J]. Journal of Visual Communication and Image Representation, 2020, 73: 102950.
[2] NIE L, LIN C Y, LIAO K, et al. Unsupervised deep image stitching: reconstructing stitched features to images[J]. IEEE Transactions on Image Processing, 2021, 30: 6184-6197.
[3] 朱凯, 李理, 张彤, 等. 视觉Transformer在低级视觉领域的研究综述[J]. 计算机工程与应用, 2024, 60(4): 39-56.
ZHU K, LI L, ZHANG T, et al. Survey of vision Transformer in low-level computer vision[J]. Computer Engineering and Applications, 2024, 60(4): 39-56.
[4] 彭斌, 白静, 李文静, 等. 面向图像分类的视觉Transformer研究进展[J]. 计算机科学与探索, 2024, 18(2): 320-344.
PENG B, BAI J, LI W J, et al. Survey on visual transformer for image classification[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(2): 320-344.
[5] WEXLER Y, SHECHTMAN E, IRANI M. Space-time completion of video[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(3): 463-476.
[6] BALLESTER C, BERTALMIO M, CASELLES V, et al. Filling-in by joint interpolation of vector fields and gray levels[J]. IEEE Transactions on Image Processing, 2001, 10(8): 1200-1211.
[7] KRISHNAN D, TETERWAK P, SARNA A, et al. Boundless: generative adversarial networks for image extension[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2019: 10520-10529.
[8] SUVOROV R, LOGACHEVA E, MASHIKHIN A, et al. Resolution-robust large mask inpainting with Fourier convolutions[C]//Proceedings of the 2022 IEEE/CVF Winter Conference on Applications of Computer Vision. Piscataway: IEEE, 2022: 3172-3182.
[9] HE K M, CHANG H W, SUN J. Rectangling panoramic images via warping[J]. ACM Transactions on Graphics, 2013, 32(4): 1-10.
[10] NIE L, LIN C Y, LIAO K, et al. Deep rectangling for image stitching: a learning baseline[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 5730-5738.
[11] LI J, DENG B S, TANG R F, et al. Local-adaptive image alignment based on triangular facet approximation[J]. IEEE Transactions on Image Processing, 2019, 29: 2356-2369.
[12] BROWN M, LOWE D G. Automatic panoramic image stitching using invariant features[J]. International Journal of Computer Vision, 2007, 74(1): 59-73.
[13] GAO J H, KIM S J, BROWN M S. Constructing image panoramas using dual-homography warping[C]//Proceedings of the CVPR 2011. Piscataway: IEEE, 2011: 49-56.
[14] DU P, NING J F, CUI J G, et al. Geometric structure preserving warp for natural image stitching[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 3678-3686.
[15] LIAO T L, LI N. Natural image stitching using depth maps[J]. arXiv:2202.06276, 2022.
[16] LI A C, GUO J, GUO Y W. Image stitching based on semantic planar region consensus[J]. IEEE Transactions on Image Processing, 2021, 30: 5545-5558.
[17] KWATRA V, SCH?DL A, ESSA I, et al. Graphcut textures: image and video synthesis using graph cuts[J]. ACM Transactions on Graphics, 2003, 22(3): 277-286.
[18] DAI Q Y, FANG F M, LI J C, et al. Edge-guided composition network for image stitching[J]. Pattern Recognition, 2021, 118: 108019.
[19] EDEN A, UYTTENDAELE M, SZELISKI R. Seamless image stitching of scenes with large motions and exposure differences[C]//Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2006: 2498-2505.
[20] LI N, LIAO T L, WANG C. Perception-based seam cutting for image stitching[J]. Signal, Image and Video Processing, 2018, 12(5): 967-974.
[21] LIAO T L, CHEN J, XU Y F. Quality evaluation-based iterative seam estimation for image stitching[J]. Signal, Image and Video Processing, 2019, 13(6): 1199-1206.
[22] CHENG S M, YANG F, CHEN Z, et al. Deep seam prediction for image stitching based on selection consistency loss[J]. arXiv:2302.05027, 2023.
[23] NIE L, LIN C, LIAO K, et al. Learning thin-plate spline motion and seamless composition for parallax-tolerant unsupervised deep image stitching[J]. arXiv:2302.08207, 2023.
[24] LI D P, HE K M, SUN J, et al. A geodesic-preserving method for image warping[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2015: 213-221.
[25] ZHANG Y, LAI Y K, ZHANG F L. Content-preserving image stitching with piecewise rectangular boundary constraints[J]. IEEE Transactions on Visualization and Computer Graphics, 2021, 27(7): 3198-3212.
[26] LIAO K, LIN C Y, WEI Y C, et al. Towards complete scene and regular shape for distortion rectification by curve-aware extrapolation[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 14549-14558.
[27] SHI H, LI Y, YANG K L, et al. FishDreamer: towards fisheye semantic completion via unified image outpainting and segmentation[J] arXiv:2303.13842, 2023.
[28] LIAO K, NIE L, LIN C Y, et al. RecRecNet: rectangling rectified wide-angle images by thin-plate spline model and DoF-based curriculum learning[C]//Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2023: 10766-10775.
[29] HARTLEY R, ZISSERMAN A. Multiple view geometry in computer vision[M]. Cambridge, UK: Cambridge University Press, 2004.
[30] WANG Y F, PERAZZI F, MCWILLIAMS B, et al. A fully progressive approach to single-image super-resolution[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway: IEEE, 2018: 977-97709.
[31] LAI W S, HUANG J B, AHUJA N, et al. Fast and accurate image super-resolution with deep Laplacian pyramid networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(11): 2599-2613.
[32] HAASE D, AMTHOR M. Rethinking depthwise separable convolutions: how intra-kernel correlations lead to improved MobileNets[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 14588-14597.
[33] HENDRYCKS D, GIMPEL K. Gaussian error linear units (GELUs)[J]. arXiv:1606.08415, 2016.
[34] ZHANG T, QI G J, XIAO B, et al. Interleaved group convolutions[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2017: 4383-4392.
[35] LIU Y Q, WANG S Q, ZHANG J, et al. Iterative network for image super-resolution[J]. IEEE Transactions on Multimedia, 2021, 24: 2259-2272.
[36] SHI W Z, CABALLERO J, HUSZáR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 1874-1883.
[37] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv: 1409. 1556, 2014.
[38] JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of the 14th European Conference on Computer Vision. Cham: Springer International Publishing, 2016: 694-711.
[39] MA J Y, LE Z L, TIAN X, et al. SMFuse: multi-focus image fusion via self-supervised mask-optimization[J]. IEEE Transactions on Computational Imaging, 2021, 7: 309-320.
[40] WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[41] FAN Y B, MAO S J, LI M, et al. Rectification for stitched images with deformable meshes and residual networks[J]. Applied Sciences, 2024, 14(7): 2821.
[42] WANG M, MEI Y, YANG L, et al. SDR: stepwise deep rec-tangling model for stitched images[J]. The Visual Computer, 2024, 41: 1197-1211.
[43] TONG T, LI G, LIU X J, et al. Image super-resolution using dense skip connections[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2017: 4809-4817.
[44] HAN W, CHANG S Y, LIU D, et al. Image super-resolution via dual?state recurrent networks[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 1654-1663.
[45] CHOLLET F. Xception: deep learning with depthwise separable convolutions[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 1800-1807.