Infrared image and visible image fusion algorithm based on secondary image decomposition

MA Xin; YU Chunyu; TONG Yixin; ZHANG Jun

doi:10.37188/OPE.20243210.1567

您当前的位置：

首页 >

文章列表页 >

Infrared image and visible image fusion algorithm based on secondary image decomposition

Information Sciences | 更新时间：2024-06-11

- Infrared image and visible image fusion algorithm based on secondary image decomposition
- Optics and Precision Engineering Vol. 32, Issue 10, Pages: 1567-1581(2024)
- 作者机构：
  
  南京邮电大学电子与光学工程学院、柔性电子（未来技术）学院，江苏南京 210023
- 作者简介：
- 基金信息：
- DOI：10.37188/OPE.20243210.1567
  CLC： TP394.1;TH691.9
- Published：25 May 2024，
  
  Received：26 September 2023，
  
  Revised：14 November 2023，
- 稿件说明：
移动端阅览
马鑫,喻春雨,童亦新等.基于二次图像分解的红外图像与可见光图像融合[J].光学精密工程,2024,32(10):1567-1581.

MA Xin,YU Chunyu,TONG Yixin,et al.Infrared image and visible image fusion algorithm based on secondary image decomposition[J].Optics and Precision Engineering,2024,32(10):1567-1581.
马鑫,喻春雨,童亦新等.基于二次图像分解的红外图像与可见光图像融合[J].光学精密工程,2024,32(10):1567-1581. DOI： 10.37188/OPE.20243210.1567.

MA Xin,YU Chunyu,TONG Yixin,et al.Infrared image and visible image fusion algorithm based on secondary image decomposition[J].Optics and Precision Engineering,2024,32(10):1567-1581. DOI： 10.37188/OPE.20243210.1567.

摘要

针对红外图像与可见光图像融合中细节丢失严重，红外图像的特征信息未能突出显示以及源图像的语义信息被忽视的问题，提出一种基于二次图像分解的红外图像与可见光图像融合网络（Secondary Image Decomposition For Infrared And Visible Image Fusion， SIDFuse）。利用编码器对源图像进行二次分解以提取不同尺度的特征信息，然后利用双元素注意力为不同尺度的特征信息分配权重、引入全局语义支路，再采用像素相加法作为融合策略，最后通过解码器重建融合图像。实验选择FLIR数据集用于训练，采用TNO和RoadScene两个数据集进行测试，并选取八种图像融合客观评价参数进行实验对比分析。由TNO数据集的图像融合实验表明，在信息熵、标准差、空间频率、视觉保真度、平均梯度、差异相关系数、多层级结构相似性、梯度融合性能评价指标上，SIDFuse比基于卷积网络中经典融合算法DenseFuse分别平均提高12.2%，9.0%，90.2%，13.9%，85.1%，16.8%，6.7%，30.7%，比最新的融合网络LRRNet分别平均提高2.5%，5.6%，31.5%，5.4%，25.2%，17.9%，7.5%，20.7%。可见本文所提算法融合的图像对比度较高，可以同时更有效保留可见光图像的细节纹理和红外图像的特征信息，在同类方法中占有明显优势。

Abstract

In view of the serious detail loss， the feature information of infrared image is not highlighted and the semantic information of source image is ignored in the fusion of infrared image and visible image， a fusion network of infrared image and visible image based on secondary image decomposition was proposed. The encoder was used to decompose the source image twice to extract the feature information of different scales， then the two-element attention was used to assign weights to the feature information of different scales， the global semantic branch is introduced， the pixel addition method was used as the fusion strategy， and the fusion image was reconstructed by the decoder. In the experiment， FLIR data set was selected for training， TNO and RoadScene data sets were used for testing， and eight objective evaluation parameters of image fusion were selected for comparative analysis. The image fusion experiment of TNO data set shows that in terms of information entropy， standard deviation， spatial frequency， visual fidelity， average gradient and difference correlation coefficient， SIDFuse is 12.2%， 9.0%， 90.2%， 13.9%， 85.1% ， 16.8%，6.7%，30.7% higher than DenseFuse， the classical fusion algorithm based on convolutional networks， respectively. Compared with the latest fusion network LRRNet， the average increase is 2.5%， 5.6%， 31.5%， 5.4%， 25.2% ， 17.9%，7.5%，20.7 respectively. It can be seen that the image fusion algorithm proposed in this paper has a high contrast， and can retain the detail texture of visible image and the feature information of infrared image more effectively at the same time， which has obvious advantages in similar methods.

关键词

图像融合图像二次分解全局语义支路双元素注意力图像对比度

Keywords

image fusionimage secondary decompositionglobal semantic branchtwo-element attentionimage contrast

references

MA J Y， MA Y， LI C. Infrared and visible image fusion methods and applications： a survey［J］. Information Fusion， 2019， 45： 153-178. doi: 10.1016/j.inffus.2018.02.004http://dx.doi.org/10.1016/j.inffus.2018.02.004

LI S T， YANG B， HU J W. Performance comparison of different multi-resolution transforms for image fusion［J］. Information Fusion， 2011， 12（2）： 74-84. doi: 10.1016/j.inffus.2010.03.002http://dx.doi.org/10.1016/j.inffus.2010.03.002

ZONG J J， QIU T. Medical image fusion based on sparse representation of classified image patches［J］. Biomed Signal Process Control， 2017， 34： 195-205. doi: 10.1016/j.bspc.2017.02.005http://dx.doi.org/10.1016/j.bspc.2017.02.005

PATIL U， MUDENGUDI U. Image fusion using hierarchical PCA［C］. 2011 International Conference on Image Information Processing. Shimla， India. IEEE， 2011： 1-6. doi: 10.1109/iciip.2011.6108966http://dx.doi.org/10.1109/iciip.2011.6108966

ZHANG X Y， MA Y， FAN F， et al. Infrared and visible image fusion via saliency analysis and local edge-preserving multi-scale decomposition［J］. Journal of the Optical Society of America A， Optics， Image Science， and Vision， 2017， 34（8）： 1400-1410. doi: 10.1364/josaa.34.001400http://dx.doi.org/10.1364/josaa.34.001400

LI H， WU X J. DenseFuse： a fusion approach to infrared and visible images［J］. IEEE Transactions on Image Processing， 2018： 2614-2623. doi: 10.1109/tip.2018.2887342http://dx.doi.org/10.1109/tip.2018.2887342

LI H， WU X J， DURRANI T. NestFuse： an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models［J］. IEEE Transactions on Instrumentation and Measurement， 2020， 69（12）： 9645-9656. doi: 10.1109/tim.2020.3005230http://dx.doi.org/10.1109/tim.2020.3005230

LI H， WU X J， KITTLER J. RFN-Nest： an end-to-end residual fusion network for infrared and visible images［J］. Information Fusion， 2021， 73（C）： 72-86. doi: 10.1016/j.inffus.2021.02.023http://dx.doi.org/10.1016/j.inffus.2021.02.023

ZHANG Y， LIU Y， SUN P， et al. IFCNN： a general image fusion framework based on convolutional neural network［J］. Information Fusion， 2020， 54： 99-118. doi: 10.1016/j.inffus.2019.07.011http://dx.doi.org/10.1016/j.inffus.2019.07.011

ZHANG H， MA J Y. SDNet： a versatile squeeze-and-decomposition network for real-time image fusion［J］. International Journal of Computer Vision， 2021， 129（10）： 2761-2785. doi: 10.1007/s11263-021-01501-8http://dx.doi.org/10.1007/s11263-021-01501-8

LIU J Y， WU Y H， HUANG Z B， et al. SMoA： searching a modality-oriented architecture for infrared and visible image fusion［J］. IEEE Signal Processing Letters， 2021， 28： 1818-1822. doi: 10.1109/lsp.2021.3109818http://dx.doi.org/10.1109/lsp.2021.3109818

LI H， XU T Y， WU X J， et al. LRRNet： a novel representation learning guided fusion network for infrared and visible images［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2023， 45（9）： 11040-11052. doi: 10.1109/tpami.2023.3268209http://dx.doi.org/10.1109/tpami.2023.3268209

MA J Y， TANG L F， XU M L， et al. STDFusionNet： an infrared and visible image fusion network based on salient target detection［J］. IEEE Transactions on Instrumentation and Measurement， 2021， 70： 5009513. doi: 10.1109/tim.2021.3075747http://dx.doi.org/10.1109/tim.2021.3075747

REN C， HE X H， WANG C C， et al. Adaptive consistency prior based deep network for image denoising［C］. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Nashville， TN， USA. IEEE， 2021： 8592-8602. doi: 10.1109/cvpr46437.2021.00849http://dx.doi.org/10.1109/cvpr46437.2021.00849

KINGMA DP， BA J. Adam： A method for stochastic optimization［C］. International Conference on Learning Representations （ICLR）， 2015： 1-15.

TOET A， HOGERVORST MA. Progress in color night vision［J］. Optical Engineering， 2012， 51（1）： 010901. doi: 10.1117/1.oe.51.1.010901http://dx.doi.org/10.1117/1.oe.51.1.010901

XU H， MA J Y， JIANG J J， et al. U2Fusion： a unified unsupervised image fusion network［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022， 44（1）： 502-518. doi: 10.1109/tpami.2020.3012548http://dx.doi.org/10.1109/tpami.2020.3012548

MA J Y， CHEN C， LI C， et al. Infrared and visible image fusion via gradient transfer and total variation minimization［J］. Information Fusion， 2016， 31（C）： 100-109. doi: 10.1016/j.inffus.2016.02.001http://dx.doi.org/10.1016/j.inffus.2016.02.001

FU Y， WU X J. A dual-branch network for infrared and visible image fusion［C］. 2020 25th International Conference on Pattern Recognition （ICPR）. Milan， Italy. IEEE， 2021： 10675-10680. doi: 10.1109/icpr48806.2021.9412293http://dx.doi.org/10.1109/icpr48806.2021.9412293

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Visible-polarized image fusion for nighttime dispersal of mines

PET/CT Cross-modal medical image fusion of lung tumors based on DCIF-GAN

Infrared and visible image fusion based on target enhancement and butterfly optimization

Underwater polarization image fusion based on unsupervised learning and attention mechanisms

Improved Retinex low-light image enhancement for stage scenes

Related Author

SHEN Kangyu

CUI Bolun

LYU Qifeng

WANG Chi

ZHOU Tao

CHENG Qianru

ZHANG Xiangxiang

LI Qi

Related Institution

Beijing Institute of Space Mechanics & Electricity

School of Mechanical Engineering and Automation， Shanghai University

Key Laboratory of Image and Graphics Intelligent Processing of State Ethnic Affairs Commission， North Minzu University

School of Medical Information and Engineering， Ningxia Medical University

School of Computer Science and Engineering， North Minzu University

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China Postal code：130033
Tel：0431-86176855 Email：gxjmgc@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰