Infrared and visible image fusion based on multi-scale dense attention connection network

Yong CHEN; Jiaojiao ZHANG; Zhen WANG

doi:10.37188/OPE.20223018.2253

您当前的位置：

首页 >

文章列表页 >

Infrared and visible image fusion based on multi-scale dense attention connection network

Information Sciences | 更新时间：2022-09-26

- Infrared and visible image fusion based on multi-scale dense attention connection network
- Optics and Precision Engineering Vol. 30, Issue 18, Pages: 2253-2266(2022)
- 作者机构：
  
  兰州交通大学电子与信息工程学院，甘肃兰州 730070
- 作者简介：
- 基金信息：
- DOI：10.37188/OPE.20223018.2253
  CLC： TP391
- Received：17 February 2022，
  
  Revised：20 March 2022，
  
  Published：25 September 2022
- 稿件说明：
移动端阅览
陈永,张娇娇,王镇.多尺度密集连接注意力的红外与可见光图像融合[J].光学精密工程,2022,30(18):2253-2266.

CHEN Yong,ZHANG Jiaojiao,WANG Zhen.Infrared and visible image fusion based on multi-scale dense attention connection network[J].Optics and Precision Engineering,2022,30(18):2253-2266.
陈永,张娇娇,王镇.多尺度密集连接注意力的红外与可见光图像融合[J].光学精密工程,2022,30(18):2253-2266. DOI： 10.37188/OPE.20223018.2253.

CHEN Yong,ZHANG Jiaojiao,WANG Zhen.Infrared and visible image fusion based on multi-scale dense attention connection network[J].Optics and Precision Engineering,2022,30(18):2253-2266. DOI： 10.37188/OPE.20223018.2253.

摘要

针对现有红外与可见光图像融合时，融合结果存在细节信息丢失、特征提取不足等问题，提出了一种多尺度密集连接注意力的红外与可见光图像融合深度学习网络模型。首先，设计多尺度卷积提取红外与可见光图像中不同尺度信息，增大感受野特征提取范围，克服了单一尺度特征提取不足的问题。然后，通过密集连接网络增强特征提取，并在编码子网络末端采用提出的可变形卷积注意力机制，密切联系全局上下文信息，增强对红外与可见光图像中重要特征信息的聚焦能力。最后，由全卷积层构成解码网络，重构生成融合图像。本文选取了六种图像融合客观评价指标，红外与可见光图像公开数据集融合实验结果表明：与其他8种方法相比，本文算法对比实验指标均有所提高，其中结构相似性（SSIM）、空间频率（SF）指标分别平均提高了0.26倍、0.45倍。所提方法的融合结果保留了更清晰的边缘及目标信息，具有更好的对比度和清晰度，在客观评价方面均优于对比方法。

Abstract

To solve the loss of detail information and insufficient feature extraction in the fusion results of infrared and visible light images， a deep learning network model for infrared and visible light image fusion with multi-scale densely connected attention is proposed. First， multi-scale convolution is designed to extract information of different scales in infrared and visible light images to increase the feature extraction range in the receptive field and overcome the problem of insufficient feature extraction at a single scale. Then， feature extraction is enhanced through a densely connected network， and an attention mechanism is introduced at the end of the encoding sub-network to closely connect the global context information and enhance the ability to focus on important feature information in infrared and visible light images. Finally， the fully convolutional layers that compose the decoding network are used to reconstruct the fused image. This study selects six objective evaluation indicators of image fusion， and the fusion experiments conducted on public infrared and visible light image datasets show that the proposed algorithm exhibits improved results compared with eight other methods. The structural similarity （SSIM）， spatial frequency （SF） indicators increase by an average of 0.26 and 0.45 times， respectively. The fusion results of the proposed method retain clearer edge and target information with better contrast and clarity， and are superior to the compared methods in both subjective and objective evaluations.

关键词

Keywords

references

刘先红，陈志斌，秦梦泽 . 结合引导滤波和卷积稀疏表示的红外与可见光图像融合［J］. 光学精密工程， 2018 ， 26 （ 5 ）： 1242 - 1253 . doi: 10.3788/OPE.20182605.1242 http://dx.doi.org/10.3788/OPE.20182605.1242

LIU X H ， CHEN Z B ， QIN M Z . Infrared and visible image fusion using guided filter and convolutional sparse representation ［J］. Optics and Precision Engineering ， 2018 ， 26 （ 5 ）： 1242 - 1253 . （in Chinese） . doi: 10.3788/OPE.20182605.1242 http://dx.doi.org/10.3788/OPE.20182605.1242

LI H ， WANG Q ， WANG H ， et al . Infrared small target detection using tensor based least mean square ［J］. Computers & Electrical Engineering ， 2021 ， 91 ： 106994 . doi: 10.1016/j.compeleceng.2021.106994 http://dx.doi.org/10.1016/j.compeleceng.2021.106994

MA J Y ， MA Y ， LI C . Infrared and visible image fusion methods and applications： a survey ［J］. Information Fusion ， 2019 ， 45 ： 153 - 178 . doi: 10.1016/j.inffus.2018.02.004 http://dx.doi.org/10.1016/j.inffus.2018.02.004

DINH P H . Combining Gabor energy with equilibrium optimizer algorithm for multi-modality medical image fusion ［J］. Biomedical Signal Processing and Control ， 2021 ， 68 ： 102696 . doi: 10.1016/j.bspc.2021.102696 http://dx.doi.org/10.1016/j.bspc.2021.102696

LIU S ， ZHAO H ， DU Q ， et al . Novel cross-resolution feature-level fusion for joint classification of multispectral and panchromatic remote sensing images ［J］. IEEE Transactions on Geoscience and Remote Sensing ， 2021 ： 1 - 13 . doi: 10.1109/tgrs.2021.3127710 http://dx.doi.org/10.1109/tgrs.2021.3127710

LIU W ， WANG Z F . A novel multi-focus image fusion method using multiscale shearing non-local guided averaging filter ［J］. Signal Processing ， 2020 ， 166 ： 107252 . doi: 10.1016/j.sigpro.2019.107252 http://dx.doi.org/10.1016/j.sigpro.2019.107252

ZHANG Q ， LIU Y ， BLUM R S ， et al . Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images： a review ［J］. Information Fusion ， 2018 ， 40 ： 57 - 75 . doi: 10.1016/j.inffus.2017.05.006 http://dx.doi.org/10.1016/j.inffus.2017.05.006

于国栋，王春阳，兰孝野，等 . 基于离线标定的快速图像拼接算法［J］. 液晶与显示， 2021 ， 36 （ 10 ）： 1430 - 1436 . doi: 10.37188/CJLCD.2020-0330 http://dx.doi.org/10.37188/CJLCD.2020-0330

YU G D ， WANG C Y ， LAN X Y ， et al . Fast image mosaic algorithm based on offline calibration ［J］. Chinese Journal of Liquid Crystals and Displays ， 2021 ， 36 （ 10 ）： 1430 - 1436 . （in Chinese） . doi: 10.37188/CJLCD.2020-0330 http://dx.doi.org/10.37188/CJLCD.2020-0330

LIU S H ， TREMBLAIS B ， CARRE P ， et al . Image reconstruction from multiscale singular points based on the dual-tree complex wavelet transform ［J］. Security and Communication Networks ， 2021 ， 2021 ： 6752486 . doi: 10.1155/2021/6752486 http://dx.doi.org/10.1155/2021/6752486

LI X S ， ZHOU F Q ， TAN H S ， et al . Multi-focus image fusion based on nonsubsampled contourlet transform and residual removal ［J］. Signal Processing ， 2021 ， 184 ： 108062 . doi: 10.1016/j.sigpro.2021.108062 http://dx.doi.org/10.1016/j.sigpro.2021.108062

刘博，韩广良，罗惠元 . 基于多尺度细节的孪生卷积神经网络图像融合算法［J］. 液晶与显示， 2021 ， 36 （ 9 ）： 1283 - 1293 . doi: 10.37188/CJLCD.2020-0339 http://dx.doi.org/10.37188/CJLCD.2020-0339

LIU B ， HAN G L ， LUO H Y . Image fusion algorithm based on multi-scale detail Siamese convolutional neural network ［J］. Chinese Journal of Liquid Crystals and Displays ， 2021 ， 36 （ 9 ）： 1283 - 1293 . （in Chinese） . doi: 10.37188/CJLCD.2020-0339 http://dx.doi.org/10.37188/CJLCD.2020-0339

ZHENG Y J ， LIU S C ， DU Q ， et al . A novel multitemporal deep fusion network （MDFN） for short-term multitemporal HR images classification ［J］. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing ， 2021 ， 14 ： 10691 - 10704 . doi: 10.1109/jstars.2021.3119942 http://dx.doi.org/10.1109/jstars.2021.3119942

杨艳春，高晓宇，党建武，等 . 基于WEMD和生成对抗网络重建的红外与可见光图像融合［J］. 光学精密工程， 2022 ， 30 （ 3 ）： 320 - 330 . doi: 10.37188/OPE.20223003.0320 http://dx.doi.org/10.37188/OPE.20223003.0320

YANG Y C ， GAO X Y ， DANG J W ， et al . Infrared and visible image fusion based on WEMD and generative adversarial network reconstruction ［J］. Optics and Precision Engineering ， 2022 ， 30 （ 3 ）： 320 - 330 . （in Chinese） . doi: 10.37188/OPE.20223003.0320 http://dx.doi.org/10.37188/OPE.20223003.0320

YAO T T ， HU J C ， ZHANG B ， et al . Scale and appearance variation enhanced Siamese network for thermal infrared target tracking ［J］. Infrared Physics & Technology ， 2021 ， 117 ： 103825 . doi: 10.1016/j.infrared.2021.103825 http://dx.doi.org/10.1016/j.infrared.2021.103825

LIU Y ， CHEN X ， PENG H ， et al . Multi-focus image fusion with a deep convolutional neural network ［J］. Information Fusion ， 2017 ， 36 ： 191 - 207 . doi: 10.1016/j.inffus.2016.12.001 http://dx.doi.org/10.1016/j.inffus.2016.12.001

MA J Y ， YU W ， LIANG P W ， et al . FusionGAN： a generative adversarial network for infrared and visible image fusion ［J］. Information Fusion ， 2019 ， 48 ： 11 - 26 . doi: 10.1016/j.inffus.2018.09.004 http://dx.doi.org/10.1016/j.inffus.2018.09.004

PRABHAKAR K R ， SRIKAR V S ， BABU R V . Deepfuse： A deep unsupervised approach for exposure fusion with extreme exposure image pairs ［C］. 2017 IEEE International Conference on Computer Vision . Venice， Italy . IEEE ， 2017 ： 4724 - 4732 . doi: 10.1109/iccv.2017.505 http://dx.doi.org/10.1109/iccv.2017.505

LI H ， WU X J ， DURRANI T S . Infrared and visible image fusion with ResNet and zero-phase component analysis ［J］. Infrared Physics & Technology ， 2019 ， 102 ： 103039 . doi: 10.1016/j.infrared.2019.103039 http://dx.doi.org/10.1016/j.infrared.2019.103039

AN W B ， WANG H M . Infrared and visible image fusion with supervised convolutional neural network ［J］. Optik ， 2020 ， 219 ： 165120 . doi: 10.1016/j.ijleo.2020.165120 http://dx.doi.org/10.1016/j.ijleo.2020.165120

ZHU W Q ， TANG X Y ， ZHANG R ， et al . Infrared and visible image fusion based on edge-preserving and attention generative adversarial network ［J］. Journal of Infrared and Millimeter Waves ， 2021 ， 40 （ 5 ）： 696 - 708 . doi: 10.3837/tiis.2021.05.010 http://dx.doi.org/10.3837/tiis.2021.05.010

GUO X ， MENG L Y ， MEI L ， et al . Multi-focus image fusion with Siamese self-attention network ［J］. IET Image Processing ， 2020 ， 14 （ 7 ）： 1339 - 1346 . doi: 10.1049/iet-ipr.2019.0883 http://dx.doi.org/10.1049/iet-ipr.2019.0883

张红颖，安征 . 基于改进双流时空网络的人体行为识别［J］. 光学精密工程， 2021 ， 29 （ 2 ）： 420 - 429 . doi: 10.37188/OPE.20212902.0420 http://dx.doi.org/10.37188/OPE.20212902.0420

ZHANG H Y ， AN Z . Human action recognition based on improved two-stream spatiotemporal network ［J］. Optics and Precision Engineering ， 2021 ， 29 （ 2 ）： 420 - 429 . （in Chinese） . doi: 10.37188/OPE.20212902.0420 http://dx.doi.org/10.37188/OPE.20212902.0420

WEI B Z ， FENG X C ， WANG K ， et al . The multi-focus-image-fusion method based on convolutional neural network and sparse representation ［J］. Entropy （Basel， Switzerland）， 2021 ， 23 （ 7 ）： 827 . doi: 10.3390/e23070827 http://dx.doi.org/10.3390/e23070827

LI H ， WU X J ， KITTLER J . MDLatLRR： a novel decomposition method for infrared and visible image fusion ［J］. IEEE Transactions on Image Processing ， 2020 ： 29 ： 4733 - 4746 . doi: 10.1109/tip.2020.2975984 http://dx.doi.org/10.1109/tip.2020.2975984

GAO C ， SONG C C ， ZHANG Y C ， et al . Improving the performance of infrared and visible image fusion based on latent low-rank representation nested with rolling guided image filtering ［J］. IEEE Access ， 2021 ， 9 ： 91462 - 91475 . doi: 10.1109/access.2021.3090436 http://dx.doi.org/10.1109/access.2021.3090436

Views

970

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Application of densely connected network in SAR and multispectral image fusion

Conditional diffusion and multi-channel high-low frequency parallel fusion of infrared and visible light images

Visible-polarized image fusion for nighttime dispersal of mines

Infrared image and visible image fusion algorithm based on secondary image decomposition

PET/CT Cross-modal medical image fusion of lung tumors based on DCIF-GAN

Related Author

Shu-tao WANG

Kai CUI

De-ming KONG

Shi-yu LIU

Xing WU

DI Jing

WANG Heran

LIANG Chan

Related Institution

Key Laboratory of Measurement and Measurement Technology and Instruments， College of Electrical Engineering， Yanshan University

School of Information Science and Engineering， Lanzhou University

Beijing Institute of Space Mechanics & Electricity

School of Mechanical Engineering and Automation， Shanghai University

College of Electronic and Optical Engineering & College of Flexible Electronics （Future Technology），Nanjing University of Posts and Telecommunications

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China Postal code：130033
Tel：0431-86176855 Email：gxjmgc@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰