1.重庆大学 光电技术与系统教育部重点实验室,重庆 400044
2.重庆市妇幼保健院 超声科,重庆 401147
3.重庆大学附属肿瘤医院 影像科,重庆 400030
E-mail:hhuang@cqu.edu.cn
扫 描 看 全 文
LI Yuan, SHI Xu, YANG Zhengchun, et al. Spatial-spectral Transformer for classification of medical hyperspectral images. [J]. Optics and Precision Engineering 31(18):2752-2764(2023)
LI Yuan, SHI Xu, YANG Zhengchun, et al. Spatial-spectral Transformer for classification of medical hyperspectral images. [J]. Optics and Precision Engineering 31(18):2752-2764(2023) DOI: 10.37188/OPE.20233118.2752.
高光谱成像技术的飞速发展给非侵入式医学成像带来新的契机,但高光谱医学图像具有高维度、高冗余以及“图谱合一”的特点,亟需针对上述特点设计智能诊断算法。近年来,Transformer已经在高光谱医学图像处理领域得到广泛应用。然而,不同仪器设备、不同采集操作所获得的高光谱医学图像差异较大,这给现有Transformer诊断模型的实际应用带来了巨大挑战。针对上述问题,本文提出了一种空-谱自注意力Transformer (S,3,AT),自适应挖掘像素与像素间、波段与波段间的内蕴联系,并在分类阶段融合多个视野下的预测结果。首先,在Transformer编码器中,设计一种空-谱自注意力机制,获取不同视野下高光谱图像上的关键空间信息和重要波段,并将不同视野下所获得的空-谱自注意力进行融合。其次,在模型分类阶段,将不同视野下的预测结果根据可学习权重进行加权融合,对图像进行综合预测。在 In-vivo Human Brain 和 BloodCell HSI 两个数据集上,本文算法总体分类精度分别达到82.25%和91.74%。实验结果表明,所提出的算法有效改善高光谱医学图像分类性能。
The development of hyperspectral imaging (HSI) technology offers new avenues for non-invasive medical imaging. However, medical hyperspectral images are characterized by high dimensionality, high redundancy, and the property of “graph-spectral uniformity,” necessitating the design of high-precision diagnostic algorithms. In recent years, transformer modes have been widely applied in medical hyperspectral image processing. However, medical hyperspectral images obtained using various instruments and acquisition methods have significant differences; this considerably hinders the practical applications of existing transformer-based diagnostic models. To address the aforementioned issues, a spatial–spectral self-attention transformer (S3AT) algorithm is proposed to adaptively mine the intrinsic relations between pixels and bands. First, in the transformer encoder, a spatial–spectral self-attention mechanism, which is designed to obtain key spatial information and important bands on hyperspectral images from different viewpoints, is employed. Thus, the spectral–spectral self-attention obtained from different views is fused. Second, in the classification stage, the predictions from different views are fused according to the learned weights. The experimental result on in-vivo human brain and blood cell HSI datasets indicate that the overall classification accuracies reach 82.25% and 91.74%, respectively. This demonstrates that the proposed S3AT algorithm yields enhanced classification performance on medical hyperspectral images.
高光谱医学图像Transformer空-谱自注意力预测融合
medical hyperspectral imagestransformerspatial-spectral self-attentionpredictions fusion
李伟, 吕蒙, 陈天虹, 等. 高光谱图像在生物医学中的应用[J]. 中国图象图形学报, 2021, 26(8): 1764-1785.
LI W, LYU M, CHEN T H, et al. Application of a hyperspectral image in medical field: a review[J]. Journal of Image and Graphics, 2021, 26(8): 1764-1785.(in Chinese)
CHEN Y R, WANG Y N, ZHANG H, et al. 医学高光谱显微成像与智能分析关键技术研究及应用[J]. AI-View, 2022(3): 22-32.(in Chinese)
陈煜嵘, 王耀南, 张辉, 等. 医学高光谱显微成像与智能分析关键技术研究及应用[J]. 人工智能, 2022(3): 22-32.
UL A, REHMAN. A review of the medical hyperspectral imaging systems and unmixing algorithms’ in biological tissues[J]. Photodiagnosis and Photodynamic Therapy, 2021, 33: 102165. doi: 10.1016/j.pdpdt.2020.102165http://dx.doi.org/10.1016/j.pdpdt.2020.102165
闫敬文, 陈宏达, 刘蕾. 高光谱图像分类的研究进展[J]. 光学 精密工程, 2019, 27(3): 680-693. doi: 10.3788/ope.20192703.0680http://dx.doi.org/10.3788/ope.20192703.0680
YAN J W, CHEN H, LIU L. Overview of hyperspectral image classification [J]. Opt. Precision Eng., 2019, 27(3): 680-693. (in Chinese). doi: 10.3788/ope.20192703.0680http://dx.doi.org/10.3788/ope.20192703.0680
宋蓓蓓, 马穗娜, 何帆, 等. Res2-Unet深度学习网络的RGB-高光谱图像重建[J]. 光学 精密工程, 2022, 30(13): 1606-1619. doi: 10.37188/OPE.2021.0433http://dx.doi.org/10.37188/OPE.2021.0433
SONG B B, MA S N, HE F, et al. Hyperspectral reconstruction from RGB images based on Res2-Unet deep learning network[J]. Opt. Precision Eng., 2022, 30(13): 1606-1619.(in Chinese). doi: 10.37188/OPE.2021.0433http://dx.doi.org/10.37188/OPE.2021.0433
时旭, 李远, 黄鸿. 面向高光谱显微图像血细胞分类的空-谱可分离卷积神经网络[J]. 光学 精密工程, 2022, 30(8):960-969. doi: 10.37188/ope.20223008.0960http://dx.doi.org/10.37188/ope.20223008.0960
SHI X, LI Y, HUANG H. Spatial-spectral separable convolutional neural network for cell classification of hyperspectral microscopic images[J]. Opt. Precision Eng., 2022, 30(8):960-969.(in Chinese). doi: 10.37188/ope.20223008.0960http://dx.doi.org/10.37188/ope.20223008.0960
郑少佳, 邱崧, 李庆利, 等. 傅里叶变换通道注意力网络的胆管癌高光谱图像分割[J]. 中国图象图形学报, 2021, 26(8):1836-1846. doi: 10.11834/jig.210207http://dx.doi.org/10.11834/jig.210207
ZHENG S J, QIU S, LI Q L, et al. Fourier transform channel attention network for cholangiocarcinoma hyperspectral image segmentation[J]. Journal of Image and Graphics, 2021, 26(8):1836-1846.(in Chinese). doi: 10.11834/jig.210207http://dx.doi.org/10.11834/jig.210207
FABELO H, ORTEGA S, SZOLNA A, et al. In-vivo hyperspectral human brain image database for brain cancer detection[J]. IEEE Access, 2019, 7: 39098-39116. doi: 10.1109/access.2019.2904788http://dx.doi.org/10.1109/access.2019.2904788
BØTKER J, WU J X, RANTANEN J. Data Handling in Science and Technology[M]. Elsevier, 2020. doi: 10.1016/b978-0-444-63977-6.00022-5http://dx.doi.org/10.1016/b978-0-444-63977-6.00022-5
YIFAN, DUAN, . Leukocyte classification based on spatial and spectral features of microscopic hyperspectral images[J]. Optics & Laser Technology, 2019, 112: 530-538. doi: 10.1016/j.optlastec.2018.11.057http://dx.doi.org/10.1016/j.optlastec.2018.11.057
RUIZ L, MARTÍN A, URBANOS G, et al. Multiclass brain tumor classification using hyperspectral imaging and supervised machine learning[C]. 2020 XXXV Conference on Design of Circuits and Integrated Systems (DCIS).1820,2020, Segovia, Spain. IEEE, 2020: 1-6. doi: 10.1109/dcis51330.2020.9268650http://dx.doi.org/10.1109/dcis51330.2020.9268650
BALTUSSEN E J M, KOK E N D, DE KONING S G B, et al. Hyperspectral imaging for tissue classification, a way toward smart laparoscopic colorectal surgery[J]. Journal of Biomedical Optics, 2019, 24(1): 016002. doi: 10.1117/1.jbo.24.1.016002http://dx.doi.org/10.1117/1.jbo.24.1.016002
HUANG Q, LI W, ZHANG B C, et al. Blood cell classification based on hyperspectral imaging with modulated Gabor and CNN[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 24(1): 160-170. doi: 10.1109/jbhi.2019.2905623http://dx.doi.org/10.1109/jbhi.2019.2905623
WEI X L, LI W, ZHANG M M, et al. Medical hyperspectral image classification based on end-to-end fusion deep neural network[J]. IEEE Transactions on Instrumentation and Measurement, 2019, 68(11): 4481-4492. doi: 10.1109/tim.2018.2887069http://dx.doi.org/10.1109/tim.2018.2887069
ZHANG Q, WANG Y, QIU S, et al. 3D-PulCNN: pulmonary cancer classification from hyperspectral images using convolution combination unit based CNN[J]. Journal of Biophotonics, 2021, 14(12): e202100142. doi: 10.1002/jbio.202100142http://dx.doi.org/10.1002/jbio.202100142
HU B L, DU J, ZHANG Z F, et al. Tumor tissue classification based on micro-hyperspectral technology and deep learning[J]. Biomedical Optics Express, 2019, 10(12): 6370. doi: 10.1364/boe.10.006370http://dx.doi.org/10.1364/boe.10.006370
DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale[EB/OL]. 2020: arXiv: 2010.11929. https://arxiv.org/abs/2010.11929.pdfhttps://arxiv.org/abs/2010.11929.pdf
ZHOU Z H, QIU S, WANG Y, et al. Swin-spectral transformer for cholangiocarcinoma hyperspectral image segmentation[C]. 2021 14th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI).23-25, 2021, Shanghai, China. IEEE, 2021: 1-6. doi: 10.1109/cisp-bmei53629.2021.9624405http://dx.doi.org/10.1109/cisp-bmei53629.2021.9624405
LI N Y, XUE J Q, JIA S. Spectral context-aware transformer for cholangiocarcinoma hyperspectral image segmentation[C]. Proceedings of the 2022 5th International Conference on Image and Graphics Processing. January 7 - 9, 2022, Beijing, China. New York: ACM, 2022: 209-213. doi: 10.1145/3512388.3512419http://dx.doi.org/10.1145/3512388.3512419
LI Y, SHI X, YANG L P, et al. MC-GAT: multi-layer collaborative generative adversarial transformer for cholangiocarcinoma classification from hyperspectral pathological images[J]. Biomedical Optics Express, 2022, 13(11): 5794-5812. doi: 10.1364/boe.472106http://dx.doi.org/10.1364/boe.472106
PENG Y S, ZHANG Y W, TU B, et al. Spatial-spectral transformer with cross-attention for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-15. doi: 10.1109/tgrs.2022.3203476http://dx.doi.org/10.1109/tgrs.2022.3203476
OUYANG E, LI B, HU W J, et al. When multigranularity meets spatial–spectral attention: a hybrid transformer for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-18. doi: 10.1109/tgrs.2023.3242978http://dx.doi.org/10.1109/tgrs.2023.3242978
LIU B, YU A Z, GAO K L, et al. DSS-TRM: deep spatial-spectral transformer for hyperspectral image classification[J]. European Journal of Remote Sensing, 2022, 55(1): 103-114. doi: 10.1080/22797254.2021.2023910http://dx.doi.org/10.1080/22797254.2021.2023910
ROY S K, KRISHNA G, DUBEY S R, et al. HybridSN: exploring 3-D-2-D CNN feature hierarchy for hyperspectral image classification[J]. IEEE Geoscience and Remote Sensing Letters, 2020, 17(2): 277-281. doi: 10.1109/lgrs.2019.2918719http://dx.doi.org/10.1109/lgrs.2019.2918719
ZHONG Z L, LI J, LUO Z M, et al. Spectral-spatial residual network for hyperspectral image classification: a 3-D deep learning framework[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(2): 847-858. doi: 10.1109/tgrs.2017.2755542http://dx.doi.org/10.1109/tgrs.2017.2755542
LI R, ZHENG S Y, DUAN C X, et al. Classification of hyperspectral image based on double-branch dual-attention mechanism network[J]. Remote Sensing, 2020, 12(3): 582. doi: 10.3390/rs12030582http://dx.doi.org/10.3390/rs12030582
HONG D F, HAN Z, YAO J, et al. SpectralFormer: rethinking hyperspectral image classification with transformers[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-15. doi: 10.1109/tgrs.2021.3130716http://dx.doi.org/10.1109/tgrs.2021.3130716
SUN L, ZHAO G R, ZHENG Y H, et al. Spectral–spatial feature tokenization transformer for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-14. doi: 10.1109/tgrs.2022.3144158http://dx.doi.org/10.1109/tgrs.2022.3144158
ZHANG J J, MENG Z, ZHAO F, et al. Convolution transformer mixer for hyperspectral image classification[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 1-5. doi: 10.1109/lgrs.2022.3208935http://dx.doi.org/10.1109/lgrs.2022.3208935
0
Views
21
下载量
0
CSCD
Publicity Resources
Related Articles
Related Author
Related Institution