1.昆明理工大学 信息工程与自动化学院,云南 昆明 650500
2.云南警官学院 云南警用无人系统创新研究院, 云南 昆明 650223
扫 描 看 全 文
WEN Jian, SHAO Jianfei, LIU Jie, et al. Multidimensional attention mechanism and selective feature fusion for image super-resolution reconstruction. [J]. Optics and Precision Engineering 31(17):2584-2597(2023)
WEN Jian, SHAO Jianfei, LIU Jie, et al. Multidimensional attention mechanism and selective feature fusion for image super-resolution reconstruction. [J]. Optics and Precision Engineering 31(17):2584-2597(2023) DOI: 10.37188/OPE.20233117.2584.
针对图像超分辨率重建过程中提取低分辨率特征效果较差,大量高频信息丢失导致的边缘模糊和伪影问题,提出了融合多维注意力机制与选择性特征融合作为图像特征提取模块的图像超分辨率重建方法。网络由若干个基本块和残差操作构建模型的特征提取结构,其核心是一种提取图像特征的异构组卷积块,该模块的对称组卷积块以并行的方式进行卷积提取不同通道间的内部信息特征并进行选择性特征融合,互补卷积块通过全维度动态卷积从空域、输入输出维度和核维度捕捉遗漏的上下文信息,对称组卷积块和互补卷积块连接后的特征采用特征增强残差块去除冗余造成干扰的无用信息。模型通过5种消融实验证明其设计的合理性,在Set5,Set14,BSDS100和Urban100测试集上与其他主流的超分辨率重建方法进行对比,峰值信噪比(PSNR)和结构相似性(SSIM)定量数据均有提升,尤其在放大因子为3的Set5数据集上比次优算法CARN-M均提升0.06 dB,结果表明提出模型具有更优的性能指标和更好的视觉效果。
To address the problems of poor extraction of low-resolution features and blurred edges and artifacts caused by the high loss of high-frequency information in an image super-resolution reconstruction process, this paper proposes an image super-resolution reconstruction method that combines multidimensional attention and selective feature fusion (SKFF) as an image feature extraction module. The network comprises several basic blocks and residual operations to construct the feature extraction structure of the model, the core of which is a heterogeneous group convolution block for extracting image features. The symmetric group convolution block of this module performs convolution in a parallel manner to extract the internal information between different feature channels and performs selective feature fusion. The complementary convolution block captures the missed contextual information from the null domain, input–output dimension, and kernel dimension by full-dimensional dynamic convolution (ODconv). The features obtained after the symmetric group convolution and complementary convolution block processes are connected via a feature-enhanced residual block to remove useless information causing interference by redundancy. The rationality of the model design is demonstrated through five ablation experiments. Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) quantitative data comparison with other mainstream super-resolution reconstruction methods on the Set5, Set14, BSDS100, and Urban100 test sets are improved, especially on the Set5 dataset with an amplification factor of 3, showing a 0.06 dB improvement over the CARN-M algorithm. The experimental results demonstrate that the proposed model has better performance indexes and visual effects.
超分辨率重建多维注意力机制特征融合残差网络
super-resolution reconstructionmultidimensional attention mechanismfeature fusionresidual network
蔡体健, 彭潇雨, 石亚鹏, 等. 通道注意力与残差级联的图像超分辨率重建[J]. 光学 精密工程, 2021, 29(1): 142-151. doi: 10.37188/OPE.20212901.0142http://dx.doi.org/10.37188/OPE.20212901.0142
CAI T J, PENG X Y, SHI Y P, et al. Channel attention and residual concatenation network for image super-resolution[J]. Opt. Precision Eng., 2021, 29(1): 142-151.(in Chinese). doi: 10.37188/OPE.20212901.0142http://dx.doi.org/10.37188/OPE.20212901.0142
ZOU W W, YUEN P C. Very low resolution face recognition problem[J]. IEEE Transactions on Image Processing, 2012, 21(1): 327-340. doi: 10.1109/tip.2011.2162423http://dx.doi.org/10.1109/tip.2011.2162423
SHI W, CABALLERO J, LEDIG C, et al. Cardiac image super -resolution with global correspondence using multi -atlas patchmatch[C].International Conference on Medical Image Computing and Computer-Assisted Intervention. Berlin: Springer, 2013, 8151: 9-16. doi: 10.1007/978-3-642-40760-4_2http://dx.doi.org/10.1007/978-3-642-40760-4_2
朱福珍, 刘越, 黄鑫, 等. 改进的稀疏表示遥感图像超分辨重建[J]. 光学 精密工程, 2019, 27(3): 718-725. doi: 10.13482/j.issn1001-7011.2019.05.004http://dx.doi.org/10.13482/j.issn1001-7011.2019.05.004
ZHU F ZH, LIU Y, HUANG X, et al. Remote sensing image super-resolution based on improved sparse representation[J]. Opt. Precision Eng., 2019, 27(3): 718-725.(in Chinese). doi: 10.13482/j.issn1001-7011.2019.05.004http://dx.doi.org/10.13482/j.issn1001-7011.2019.05.004
CHIANG M C, BOULT T E. Efficient image warping and super-resolution[C].Proceedings Third IEEE Workshop on Applications of Computer Vision. WACV'96.24,1996, Sarasota, FL, USA. IEEE, 2002: 56-61.
HA V K, REN J C, XU X Y, et al. Deep learning based single image super-resolution: a survey[J].International Journal of Automation and Computing, 2019, 16(4): 413-426. doi: 10.1007/s11633-019-1183-xhttp://dx.doi.org/10.1007/s11633-019-1183-x
孙玉宝, 费选, 韦志辉, 等. 基于前向后向算子分裂的稀疏性正则化图像超分辨率算法[J]. 自动化学报, 2010, 36(9): 1232-1238. doi: 10.3724/sp.j.1004.2010.01232http://dx.doi.org/10.3724/sp.j.1004.2010.01232
SUN Y B, FEI X, WEI ZH H, et al. Sparsity regularized image super-resolution model via forward-backward operator splitting method[J]. Acta Automatica Sinica, 2010, 36(9): 1232-1238.(in Chinese). doi: 10.3724/sp.j.1004.2010.01232http://dx.doi.org/10.3724/sp.j.1004.2010.01232
潘宗序, 禹晶, 胡少兴, 等. 基于多尺度结构自相似性的单幅图像超分辨率算法[J]. 自动化学报, 2014, 40(4):594-603. doi: 10.3724/SP.J.1004.2014.02233http://dx.doi.org/10.3724/SP.J.1004.2014.02233
PAN Z X, YU J, HU SH X, et al. Single image super resolution based on multi-scale structural self-similarity[J]. Acta Automatica Sinica, 2014, 40(4):594-603.(in Chinese). doi: 10.3724/SP.J.1004.2014.02233http://dx.doi.org/10.3724/SP.J.1004.2014.02233
DONG C, LOY C C, HE K M, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307. doi: 10.1109/tpami.2015.2439281http://dx.doi.org/10.1109/tpami.2015.2439281
DONG C, LOY C C, TANG X O. Accelerating the Super-resolution Convolutional Neural Network[M].Computer Vision-ECCV 2016. Cham: Springer International Publishing, 2016: 391-407. doi: 10.1007/978-3-319-46475-6_25http://dx.doi.org/10.1007/978-3-319-46475-6_25
HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C].2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2730,2016, Las Vegas, NV, USA. IEEE, 2016: 770-778. doi: 10.1109/cvpr.2016.90http://dx.doi.org/10.1109/cvpr.2016.90
KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C].2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2730,2016, Las Vegas, NV, USA. IEEE, 2016: 1646-1654. doi: 10.1109/cvpr.2016.182http://dx.doi.org/10.1109/cvpr.2016.182
KIM J, LEE J K, LEE K M. Deeply-recursive convolutional network for image super-resolution[C].2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2730,2016, Las Vegas, NV, USA. IEEE, 2016: 1637-1645. doi: 10.1109/cvpr.2016.181http://dx.doi.org/10.1109/cvpr.2016.181
TAI Y, YANG J, LIU X M, et al. MemNet: a persistent memory network for image restoration[C].2017 IEEE International Conference on Computer Vision (ICCV).2229,2017, Venice, Italy. IEEE, 2017: 4549-4557. doi: 10.1109/iccv.2017.486http://dx.doi.org/10.1109/iccv.2017.486
LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C].2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).2126,2017, Honolulu, HI, USA. IEEE, 2017: 1132-1140. doi: 10.1109/cvprw.2017.151http://dx.doi.org/10.1109/cvprw.2017.151
ZHANG Y L, LI K P, LI K, et al. Image Super-resolution using Very Deep Residual Channel Attention Networks[M].Computer Vision-ECCV 2018. Cham: Springer International Publishing, 2018: 294-310. doi: 10.1007/978-3-030-01234-2_18http://dx.doi.org/10.1007/978-3-030-01234-2_18
程德强, 赵佳敏, 寇旗旗, 等. 多尺度密集特征融合的图像超分辨率重建[J]. 光学 精密工程, 2022, 30(20): 2489-2500. doi: 10.37188/OPE.20223020.2489http://dx.doi.org/10.37188/OPE.20223020.2489
CHENG D Q, ZHAO J M, KOU Q Q, et al. Multi-scale dense feature fusion network for image super-resolution[J]. Opt. Precision Eng., 2022, 30(20): 2489-2500.(in Chinese). doi: 10.37188/OPE.20223020.2489http://dx.doi.org/10.37188/OPE.20223020.2489
LI J C, FANG F M, MEI K F, et al. Multi-scale Residual Network for Image Super-resolution[M].Computer Vision-ECCV 2018. Cham: Springer International Publishing, 2018: 527-542. doi: 10.1007/978-3-030-01237-3_32http://dx.doi.org/10.1007/978-3-030-01237-3_32
AHN N, KANG B, SOHN K A. Fast, Accurate, and Lightweight Super-resolution with Cascading Residual Network[M].Computer Vision-ECCV 2018. Cham: Springer International Publishing, 2018: 256-272. doi: 10.1007/978-3-030-01249-6_16http://dx.doi.org/10.1007/978-3-030-01249-6_16
LEDIG C, THEIS L, HUSZÁR F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C].2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2126,2017, Honolulu, HI, USA. IEEE, 2017: 105-114. doi: 10.1109/cvpr.2017.19http://dx.doi.org/10.1109/cvpr.2017.19
WANG X T, YU K, WU S X, et al. ESRGAN: Enhanced Super-resolution Generative Adversarial Networks[M].Lecture Notes in Computer Science. Cham: Springer International Publishing, 2019: 63-79. doi: 10.1007/978-3-030-11021-5_5http://dx.doi.org/10.1007/978-3-030-11021-5_5
TIAN C W, ZHUGE R B, WU Z H, et al. Lightweight image super-resolution with enhanced CNN[J]. Knowledge-Based Systems, 2020, 205: 106235. doi: 10.1016/j.knosys.2020.106235http://dx.doi.org/10.1016/j.knosys.2020.106235
LI X, WANG W H, HU X L, et al. Selective kernel networks[C].2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).1520,2019, Long Beach, CA, USA. IEEE, 2020: 510-519. doi: 10.1109/cvpr.2019.00060http://dx.doi.org/10.1109/cvpr.2019.00060
LI C, ZHOU A, YAO A. Omni-dimensional dynamic convolution[EB/OL]. 2022: arXiv: 2209.07947. https://arxiv.org/abs/2209.07947https://arxiv.org/abs/2209.07947.
TIMOFTE R, AGUSTSSON E, VAN G L, et al. NTIRE 2017 challenge on single image super-resolution: methods and results[C].Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2126,2017, Honolulu, HI, USA. IEEE, 2017: 1110-1121. doi: 10.1109/cvprw.2017.150http://dx.doi.org/10.1109/cvprw.2017.150
BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C].Proceedings ofthe British Machine Vision Conference 2012. Surrey. British Machine Vision Association, 2012: 135.1-135.10. doi: 10.5244/c.26.135http://dx.doi.org/10.5244/c.26.135
ZEYDE R, ELAD M, PROTTER M. On Single Image Scale-up using Sparse-Representations[M].Curves and Surfaces. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012: 711-730. doi: 10.1007/978-3-642-27413-8_47http://dx.doi.org/10.1007/978-3-642-27413-8_47
MARTIN D, FOWLKES C, TAL D, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C].Proceedings Eighth IEEE International Conference on Computer Vision. ICCV.714,2001, Vancouver, BC, Canada. IEEE, 2002: 416-423.
HUANG J B, SINGH A, AHUJA N. Single image super-resolution from transformed self-exemplars[C].2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).712,2015, Boston, MA, USA. IEEE, 2015: 5197-5206. doi: 10.1109/cvpr.2015.7299156http://dx.doi.org/10.1109/cvpr.2015.7299156
ZHANG K, ZUO W M, ZHANG L. Learning a single convolutional super-resolution network for multiple degradations[C].2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1823,2018, Salt Lake City, UT, USA. IEEE, 2018: 3262-3271. doi: 10.1109/cvpr.2018.00344http://dx.doi.org/10.1109/cvpr.2018.00344
WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J].IEEE Transactions on Image Processing: a Publication of the IEEE Signal Processing Society, 2004, 13(4): 600-612. doi: 10.1109/tip.2003.819861http://dx.doi.org/10.1109/tip.2003.819861
SUN J, XU Z B, SHUM H Y. Image super-resolution using gradient profile prior[C].2008 IEEE Conference on Computer Vision and Pattern Recognition. 2328,2008, Anchorage, AK. IEEE, 2008: 1-8. doi: 10.1109/cvpr.2008.4587659http://dx.doi.org/10.1109/cvpr.2008.4587659
TIMOFTE R, DE S V, VAN G L. A+: Adjusted Anchored Neighborhood Regression for Fast Super-resolution[M].Computer Vision-ACCV 2014. Cham: Springer International Publishing, 2015: 111-126. doi: 10.1007/978-3-319-16817-3_8http://dx.doi.org/10.1007/978-3-319-16817-3_8
SCHULTER S, LEISTNER C, BISCHOF H. Fast and accurate image upscaling with super-resolution forests[C]. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).712,2015, Boston, MA, USA. IEEE, 2015: 3791-3799. doi: 10.1109/cvpr.2015.7299003http://dx.doi.org/10.1109/cvpr.2015.7299003
WANG L G, WANG Y Q, DONG X Y, et al. Unsupervised degradation representation learning for blind super-resolution[EB/OL]. 2021: arXiv: 2104.00416. https://arxiv.org/abs/2104.00416https://arxiv.org/abs/2104.00416. doi: 10.1109/cvpr46437.2021.01044http://dx.doi.org/10.1109/cvpr46437.2021.01044
ZHANG K, ZUO W M, CHEN Y J, et al. Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising[C]. IEEE Transactions on Image Processing. 1,2017, IEEE, 2017: 3142-3155. doi: 10.1109/tip.2017.2662206http://dx.doi.org/10.1109/tip.2017.2662206
FU J, WANG H, XIE Q, et al. KXNet: a model-driven deep neural network for blind super-resolution[EB/OL]. 2022: arXiv: 2209.10305. https://arxiv.org/abs/2209.10305https://arxiv.org/abs/2209.10305. doi: 10.1007/978-3-031-19800-7_14http://dx.doi.org/10.1007/978-3-031-19800-7_14
MAO X J, SHEN C H, YANG Y B. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections[C].Proceedings of the 30th International Conference on Neural Information Processing Systems. 510,2016, Barcelona, Spain. New York: ACM, 2016: 2810-2818.
WANG Y F, WANG L J, WANG H Y, et al. End-to-end image super-resolution via deep and shallow convolutional networks[C].IEEE Access. 13,2019, IEEE, 2019: 31959-31970. doi: 10.1109/access.2019.2903582http://dx.doi.org/10.1109/access.2019.2903582
YANG X, MEI H Y, ZHANG J Q, et al. DRFN: deep recurrent fusion network for single-image super-resolution with large factors[C].IEEE Transactions on Multimedia. 6,2018, IEEE, 2018: 328-337. doi: 10.1109/tmm.2018.2863602http://dx.doi.org/10.1109/tmm.2018.2863602
LIU H, FU Z L, HAN J G, et al. Single image super-resolution using multi-scale deep encoder-decoder with phase congruency edge map guidance[J]. Information Sciences, 2019, 473: 44-58. doi: 10.1016/j.ins.2018.09.018http://dx.doi.org/10.1016/j.ins.2018.09.018
TIAN C W, XU Y, ZUO W M, et al. Coarse-to-fine CNN for image super-resolution[J]. IEEE Transactions on Multimedia, 2021, 23: 1489-1502. doi: 10.1109/tmm.2020.2999182http://dx.doi.org/10.1109/tmm.2020.2999182
TIAN C W, XU Y, ZUO W M, et al. Asymmetric CNN for image superresolution[C].IEEE Transactions on Systems, Man, and Cybernetics: Systems.3,2021, IEEE, 2021: 3718-3730. doi: 10.1109/tsmc.2021.3069265http://dx.doi.org/10.1109/tsmc.2021.3069265
0
Views
21
下载量
0
CSCD
Publicity Resources
Related Articles
Related Author
Related Institution