Infrared and visible image fusion based on target enhancement and butterfly optimization

HAO Shuai; LI Tong; MA Xu; HE Tian; SUN Xizi; LI Jiahao

doi:10.37188/OPE.20233123.3490

您当前的位置：

首页 >

文章列表页 >

Infrared and visible image fusion based on target enhancement and butterfly optimization

Information Sciences | 更新时间：2023-12-13

- Infrared and visible image fusion based on target enhancement and butterfly optimization
- Optics and Precision Engineering Vol. 31, Issue 23, Pages: 3490-3503(2023)
- 作者机构：
  
  1.西安科技大学电气与控制工程学院，陕西西安 710054
- 作者简介：
  
  E-mail： haoxust@163.com
- 基金信息：
- DOI：10.37188/OPE.20233123.3490
  CLC：
扫描看全文
郝帅,李彤,马旭等.基于目标增强与蝴蝶优化的红外与可见光图像融合[J].光学精密工程,2023,31(23):3490-3503.

HAO Shuai,LI Tong,MA Xu,et al.Infrared and visible image fusion based on target enhancement and butterfly optimization[J].Optics and Precision Engineering,2023,31(23):3490-3503.
郝帅,李彤,马旭等.基于目标增强与蝴蝶优化的红外与可见光图像融合[J].光学精密工程,2023,31(23):3490-3503. DOI： 10.37188/OPE.20233123.3490.

HAO Shuai,LI Tong,MA Xu,et al.Infrared and visible image fusion based on target enhancement and butterfly optimization[J].Optics and Precision Engineering,2023,31(23):3490-3503. DOI： 10.37188/OPE.20233123.3490.

摘要

针对传统红外与可见光图像融合结果存在目标模糊、纹理细节丢失以及伪影等问题，提出一种基于目标增强与蝴蝶优化的红外与可见光图像融合算法。针对红外图像因成像机理导致的融合目标边缘模糊问题，构造基于目标边缘强化的红外图像增强模块；为解决因可见光图像质量较低而造成的融合图像细节缺失问题，搭建基于带色彩恢复因子的多尺度Retinex可见光增强模块；然后，利用四阶偏微分方程-主成分分析法对红外和可见光增强图像分别进行边缘平滑处理，以解决融合结果存在的伪影问题；最后，为使最终融合结果突出目标的同时保留更多的纹理细节，设计了基于蝴蝶优化的图像重建模块，以实现重建图像权重的自适应分配。为验证所提出算法的优势，将所提出的算法在TNO，INO，M3FD以及RoadScene数据集上与6种经典算法进行比较。实验结果表明，相比于对比算法，本文算法的信息熵、空间频率、均方差、联合熵、视觉信息保真度和自然场景6个客观评价指标上分别平均提高了9.24%，38.88%，51.11%，4.65%，35.44%，19.36%，融合结果目标边缘清晰、对比度强、无伪影且纹理细节丰富。

Abstract

A fusion method for infrared and visible images was developed based on target enhancement and butterfly optimization to address the problems of target blurring， textural detail loss， and artifacts in traditional infrared and visible image fusion results. First， to deal with the blurring of the fusion target edge caused by the imaging mechanism of the infrared image， an infrared image enhancement module based on target edge enhancement was constructed. Second， a visible light enhancement module based on multiscale Retinex with color restoration was developed to solve the problem of missing details in fused images caused by the low quality of visible images. Third， fourth-order partial differential equations and the principal component analysis method were used to smooth out the edges of infrared and visible enhanced images to solve the problem of artifacts in the fusion results. Finally， an image reconstruction module based on butterfly optimization was designed for the adaptive allocation of reconstructed image weights. This allows the target to be highlighted in the final fusion result while retaining more textural details. To verify the advantages of the proposed algorithm， it was compared with six classic algorithms on the TNO， INO， M3FD， and RoadScene datasets. The experimental results show that the fusion results obtained by the proposed algorithm have clear edges， strong contrast， no artifacts， and rich textural details. Compared with the other algorithms， the objective evaluation indicators EN， SF， SD， JE， VIF， and NSS are improved by an average of 9.24%， 38.88%， 51.11%， 4.65%， 35.44% and 19.36%， respectively.

关键词

图像融合红外图像可见光图像多尺度蝴蝶优化

Keywords

image fusioninfrared imagevisible imagemulti-scalebutterfly optimization

references

胡建平，郝梦云，杜影，等. 结构和纹理感知的Retinex融合红外与可见光图像［J］. 光学精密工程， 2022， 30（24）：3225-3238. doi: 10.37188/ope.20223024.3225http://dx.doi.org/10.37188/ope.20223024.3225

HU J P， HAO M Y， DU Y， et al. Fusion of infrared and visible images via structure and texture-aware retinex［J］. Opt. Precision Eng.， 2022， 30（24）：3225-3238. （in Chinese）. doi: 10.37188/ope.20223024.3225http://dx.doi.org/10.37188/ope.20223024.3225

ZHOU T， LI Q， LU H L， et al. GAN review： models and medical image fusion applications［J］. Information Fusion， 2023， 91： 134-148. doi: 10.1016/j.inffus.2022.10.017http://dx.doi.org/10.1016/j.inffus.2022.10.017

王满利，王晓龙，张长森. 基于动态范围压缩增强和NSST的红外与可见光图像融合算法［J］. 光子学报， 2022， 51（9）： 0910002. doi: 10.3788/gzxb20225109.0910002http://dx.doi.org/10.3788/gzxb20225109.0910002

WANG M L， WANG X L， ZHANG CH S. Infrared and visible image fusion algorithm based on dynamic range compression enhancement and NSST［J］. Acta Photonica Sinica， 2022， 51（9）： 0910002.（in Chinese）. doi: 10.3788/gzxb20225109.0910002http://dx.doi.org/10.3788/gzxb20225109.0910002

LI H， WU X J， TARIQ S D. Infrared and visible image fusion with ResNet and zero-phase component analysis［J］. Infrared Physics & Technology， 2019， 102： 103039. doi: 10.1016/j.infrared.2019.103039http://dx.doi.org/10.1016/j.infrared.2019.103039

朱雯青，汤心溢，张瑞，等. 基于边缘保持和注意力生成对抗网络的红外与可见光图像融合［J］. 红外与毫米波学报， 2021， 40（5）：696-708. doi: 10.11972/j.issn.1001-9014.2021.05.017http://dx.doi.org/10.11972/j.issn.1001-9014.2021.05.017

ZHU W Q， TANG X Y， ZHANG R， et al. Infrared and visible image fusion based on edge-preserving and attention generative adversarial network［J］. Journal of Infrared and Millimeter Waves， 2021， 40（5）：696-708. （in Chinese）. doi: 10.11972/j.issn.1001-9014.2021.05.017http://dx.doi.org/10.11972/j.issn.1001-9014.2021.05.017

陈永，张娇娇，王镇. 多尺度密集连接注意力的红外与可见光图像融合［J］. 光学精密工程， 2022， 30（18）：2253-2266. doi: 10.37188/ope.20223018.2253http://dx.doi.org/10.37188/ope.20223018.2253

CHEN Y， ZHANG J J， WANG ZH. Infrared and visible image fusion based on multi-scale dense attention connection network［J］. Opt. Precision Eng.， 2022， 30（18）：2253-2266. （in Chinese）. doi: 10.37188/ope.20223018.2253http://dx.doi.org/10.37188/ope.20223018.2253

HAO S， HE T， AN B， et al. VDFEFuse： a novel fusion approach to infrared and visible images［J］. Infrared Physics & Technology， 2022， 121： 104048. doi: 10.1016/j.infrared.2022.104048http://dx.doi.org/10.1016/j.infrared.2022.104048

JIANG J W， LIU L， WANG L P， et al. Fusion of visible and infrared images based on multiple differential gradients［J］. Journal of Modern Optics， 2020， 67（4）： 329-339. doi: 10.1080/09500340.2020.1718789http://dx.doi.org/10.1080/09500340.2020.1718789

BAVIRISETTI D P， DHULI R. Fusion of infrared and visible sensor images based on anisotropic diffusion and karhunen-loeve transform［J］. IEEE Sensors Journal， 2016， 16（1）： 203-209. doi: 10.1109/jsen.2015.2478655http://dx.doi.org/10.1109/jsen.2015.2478655

ZHOU T， CHENG Q R， LU H L， et al. Deep learning methods for medical image fusion： a review［J］. Computers in Biology and Medicine， 2023， 160： 106959. doi: 10.1016/j.compbiomed.2023.106959http://dx.doi.org/10.1016/j.compbiomed.2023.106959

LI G F， LIN Y J， QU X D. An infrared and visible image fusion method based on multi-scale transformation and norm optimization［J］. Information Fusion， 2021， 71： 109-129. doi: 10.1016/j.inffus.2021.02.008http://dx.doi.org/10.1016/j.inffus.2021.02.008

MA J Y， MA Y， LI C. Infrared and visible image fusion methods and applications： a survey［J］. Information Fusion， 2019， 45： 153-178. doi: 10.1016/j.inffus.2018.02.004http://dx.doi.org/10.1016/j.inffus.2018.02.004

LIU Y， CHEN X， WANG Z F， et al. Deep learning for pixel-level image fusion： recent advances and future prospects［J］. Information Fusion， 2018， 42： 158-173. doi: 10.1016/j.inffus.2017.10.007http://dx.doi.org/10.1016/j.inffus.2017.10.007

LIU Y， CHEN X， WANG Z F， et al. FusionGAN： a generative adversarial network for infrared and visible image fusion［J］. Information Fusion， 2019， 48： 11-26. doi: 10.1016/j.inffus.2018.09.004http://dx.doi.org/10.1016/j.inffus.2018.09.004

杨艳春，高晓宇，党建武，等. 基于WEMD和生成对抗网络重建的红外与可见光图像融合［J］. 光学精密工程， 2022， 30（3）：320-330. doi: 10.37188/OPE.20223003.0320http://dx.doi.org/10.37188/OPE.20223003.0320

YANG Y CH， GAO X Y， DANG J W， et al. Infrared and visible image fusion based on WEMD and generative adversarial network reconstruction［J］. Opt. Precision Eng.， 2022， 30（3）：320-330. （in Chinese）. doi: 10.37188/OPE.20223003.0320http://dx.doi.org/10.37188/OPE.20223003.0320

曹宇彤，宦克为，薛超，等. 基于卷积神经网络结合NSCT的红外与可见光图像融合［J］. 红外技术， 2023， 45（4）： 378-385.

CAO Y T， HUAN K W， XUE CH， et al. Infrared and visible image fusion based on CNN with NSCT［J］. Infrared Technology， 2023， 45（4）： 378-385.（in Chinese）

LI Q Q， HAN G L， LIU P X， et al. A multilevel hybrid transmission network for infrared and visible image fusion［J］. IEEE Transactions on Instrumentation and Measurement， 2022， 71： 1-14. doi: 10.1109/tim.2022.3186048http://dx.doi.org/10.1109/tim.2022.3186048

HAO S， HE T， MA X， et al. NOSMFuse： an infrared and visible image fusion approach based on norm optimization and slime mold architecture［J］.Applied Intelligence， 2023， 53（5）： 5388-5401.

LIU Y H， YAN H M， GAO S B， et al. Criteria to evaluate the fidelity of image enhancement by MSRCR［J］. IET Image Processing， 2018， 12（6）： 880-887. doi: 10.1049/iet-ipr.2017.0171http://dx.doi.org/10.1049/iet-ipr.2017.0171

崔晓荣，沈涛，黄建鲁，等. FABEMD和改进局部能量窗口的红外中波和长波图像融合［J］. 光谱学与光谱分析， 2021， 41（7）：2043-2049. doi: 10.3964/j.issn.1000-0593(2021)07-2043-07http://dx.doi.org/10.3964/j.issn.1000-0593(2021)07-2043-07

CUI X R， SHEN T， HUANG J L， et al. Inf rared mid-wave and long-wave image fusion based on FABEMD and improved local energy window［J］. Spectroscopy and Spectral Analysis， 2021， 41（7）：2043-2049. （in Chinese）. doi: 10.3964/j.issn.1000-0593(2021)07-2043-07http://dx.doi.org/10.3964/j.issn.1000-0593(2021)07-2043-07

SHAMS I， MEKHILEF S， TEY K S. Maximum power point tracking using modified butterfly optimization algorithm for partial shading， uniform shading， and fast varying load conditions［J］. IEEE Transactions on Power Electronics， 2021， 36（5）： 5569-5581. doi: 10.1109/tpel.2020.3029607http://dx.doi.org/10.1109/tpel.2020.3029607

ALEXANDER T. The TNO multiband image data collection［J］. Data in Brief， 2017， 15： 249-251. doi: 10.1016/j.dib.2017.09.038http://dx.doi.org/10.1016/j.dib.2017.09.038

LIU J Y， FAN X， HUANG Z B， et al. Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection［C］. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. June 18-24， 2022. New Orleans， LA， USA. IEEE， 2022： 5792-5801.

XU H， MA J Y， JIANG J J， et al. U2Fusion： a unified unsupervised image fusion network［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022， 44（1）： 502-518. doi: 10.1109/tpami.2020.3012548http://dx.doi.org/10.1109/tpami.2020.3012548

ZHOU Z Q， WANG B， LI S， et al. Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with Gaussian and bilateral filters［J］. Information Fusion， 2016， 30： 15-26. doi: 10.1016/j.inffus.2015.11.003http://dx.doi.org/10.1016/j.inffus.2015.11.003

LIU Y， CHEN X， CHENG J， et al. A medical image fusion method based on convolutional neural networks［C］. 2017 20th International Conference on Information Fusion （Fusion）， Xi'an， China，2017：10-13. doi: 10.23919/icif.2017.8009769http://dx.doi.org/10.23919/icif.2017.8009769

ZHANG Y， LIU Y， SUN P， et al. IFCNN： a general image fusion framework based on convolutional neural network［J］. Information Fusion， 2020， 54： 99-118. doi: 10.1016/j.inffus.2019.07.011http://dx.doi.org/10.1016/j.inffus.2019.07.011

LI H， WU X J， KITTLER J. RFN-Nest： an end-to-end residual fusion network for infrared and visible images［J］. Information Fusion， 2021， 73： 72-86. doi: 10.1016/j.inffus.2021.02.023http://dx.doi.org/10.1016/j.inffus.2021.02.023

LI H， WU X J. DenseFuse： a fusion approach to infrared and visible images［J］. IEEE Transactions on Image Processing， 2019， 28（5）： 2614-2623. doi: 10.1109/tip.2018.2887342http://dx.doi.org/10.1109/tip.2018.2887342

LI H， WU X J， DURRANI T. NestFuse： an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models［J］. IEEE Transactions on Instrumentation and Measurement， 2020， 69（12）： 9645-9656. doi: 10.1109/tim.2020.3005230http://dx.doi.org/10.1109/tim.2020.3005230

杨艳春，李娇，王阳萍. 图像融合质量评价方法研究综述［J］. 计算机科学与探索， 2018， 12（7）： 1021-1035. doi: 10.3778/j.issn.1673-9418.1710001http://dx.doi.org/10.3778/j.issn.1673-9418.1710001

YANG Y CH， LI J， WANG Y P. Review of image fusion quality evaluation methods［J］. Journal of Frontiers of Computer Science & Technology， 2018， 12（7）： 1021-1035.（in Chinese）. doi: 10.3778/j.issn.1673-9418.1710001http://dx.doi.org/10.3778/j.issn.1673-9418.1710001

DAS S， SUGANTHAN P N. Differential evolution： a survey of the state-of-the-art［J］. IEEE Transactions on Evolutionary Computation， 2011， 15（1）： 4-31. doi: 10.1109/tevc.2010.2059031http://dx.doi.org/10.1109/tevc.2010.2059031

SENTHILNATH J， OMKAR S N， MANI V. Clustering using firefly algorithm： performance study［J］. Swarm and Evolutionary Computation， 2011， 1（3）： 164-171. doi: 10.1016/j.swevo.2011.06.003http://dx.doi.org/10.1016/j.swevo.2011.06.003

MORRIS G M， GOODSELL D S， HALLIDAY R S， et al. Automated docking using a Lamarckian genetic algorithm and an empirical binding free energy function［J］. Journal of Computational Chemistry， 1998， 19（14）： 1639-1662. doi: 10.1002/(sici)1096-987x(19981115)19:14<1639::aid-jcc10>3.0.co;2-bhttp://dx.doi.org/10.1002/(sici)1096-987x(19981115)19:14<1639::aid-jcc10>3.0.co;2-b

PARK J B， LEE K S， SHIN J R， et al. A particle swarm optimization for economic dispatch with nonsmooth cost functions［J］. IEEE Transactions on Power Systems，2005，20（1）：34-42. doi: 10.1109/tpwrs.2004.831275http://dx.doi.org/10.1109/tpwrs.2004.831275

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Fusion of infrared and visible images combined with NSDTCT and sparse representation

Fusion of infrared and visible images based on target segmentation and compressed sensing

Fusion of infrared and visual images based on non-sampled Contourlet transform and region classification

Fusion of infrared and visible images based on compressive sensing

Related Author

No data

Related Institution

School of Mathematics, Hefei University of Technology

School of Computer Science & Engineering, Changchun University of Technology

College of Communication Engineering, Jilin University

Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences

University of Chinese Academy of Sciences

Address：No.3888 Dong Nanhu Road, Changchun, Jilin, China Postal code：130033
Tel：0431-86176855 Email：gxjmgc@ciomp.ac.cn
Technical support is provided by Beijing Founder electronics co., LTD 吉ICP备11002662号-17 京公网安备11010802024621
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰