浏览全部资源
扫码关注微信
1.战略支援部队信息工程大学 地理空间信息学院,河南 郑州 450001
2.智慧中原地理信息技术河南省协同创新中心,河南 郑州 450001
Received:08 August 2022,
Revised:10 September 2022,
Published:10 March 2023
移动端阅览
戴莫凡,徐青,邢帅等.采用独立分支网络的多源遥感数据自适应融合地物分类[J].光学精密工程,2023,31(05):644-655.
DAI Mofan,XU Qing,XING Shuai,et al.Semantic segmentation of multi-source remote sensing data self-adaptive fusion with independent branch network[J].Optics and Precision Engineering,2023,31(05):644-655.
戴莫凡,徐青,邢帅等.采用独立分支网络的多源遥感数据自适应融合地物分类[J].光学精密工程,2023,31(05):644-655. DOI: 10.37188/OPE.20233105.0644.
DAI Mofan,XU Qing,XING Shuai,et al.Semantic segmentation of multi-source remote sensing data self-adaptive fusion with independent branch network[J].Optics and Precision Engineering,2023,31(05):644-655. DOI: 10.37188/OPE.20233105.0644.
针对现有基于深度学习的地物分类方法大多面向遥感影像,而对点云数据的空间信息利用不足,特别是对点云和影像这种异源特征融合不够充分的问题,提出了一种采用独立分支网络结构的多源遥感数据自适应融合地物分类方法。首先,对配准好的LiDAR点云和遥感影像分别采用三维网络和二维网络提取各模态的空间几何特征和语义特征;其次,在点云空间对影像特征进行交叉模态采样和特征对齐得到基于点的多源特征;最后,采用一种基于注意力机制的非线性自适应特征融合方法实现二、三维语义特征的融合。实验结果表明,本文方法通过网络训练能够实现自适应数据特征的多源遥感数据融合分类,针对ISPRS多源遥感数据集的植被、建筑物和地面三类地物平均分类精度达到85.87 %,相较三维点云语义分割的分类精度提高了10.12%。本文提出的独立分支融合网络能够实现二、三维数据的交互学习与深度融合,为遥感多源数据地物分类提供了一种新的思路。
Existing deep learning-based terrain classification methods are mainly for remote sensing imagery; however, the spatial information of point clouds is underutilized. Specifically, the fusion of heterologous features is insufficient for point clouds and imagery. To utilize multi-source features fully, we propose a self-adaptive fusion classification method of multi-source remote sensing data based on independent branch network in this study. First, three-dimensional (3D) and two-dimensional (2D) networks are used to extract the semantic features of registered LiDAR point clouds and remote sensing imagery. From the 3D space, the features of imagery are then sampled and aligned with those of point clouds. Finally, a nonlinear self-adaptive feature fusion module is proposed to realize the fusion of multi-source semantic features. The experimental results indicate that the proposed method achieves an average classification accuracy of 85.87% on the vegetation, building, and ground of the ISPRS multi-source remote sensing dataset. Through network training, multi-source remote sensing data can be more data feature-adaptive fused and classified; further, the accuracy is significantly improved by 10.12% compared with the 3D classification result. The proposed independent branch fusion network can realize interactive learning and deep fusion of 2D and 3D data, and it provide a new idea for terrain classification based on remote sensing multimodal data fusion.
毕凯 , 桂德竹 . 浅谈地理国情监测与基础测绘 [J]. 遥感信息 , 2014 , 29 ( 4 ): 10 - 15 . doi: 10.3969/j.issn.1000-3177.2014.04.003 http://dx.doi.org/10.3969/j.issn.1000-3177.2014.04.003
BI K , GUI D Z . Brief probe into national geographic conditions monitoring and fundamental surveying and mapping [J]. Remote Sensing Information , 2014 , 29 ( 4 ): 10 - 15 . (in Chinese) . doi: 10.3969/j.issn.1000-3177.2014.04.003 http://dx.doi.org/10.3969/j.issn.1000-3177.2014.04.003
李德仁 , 姚远 , 邵振峰 . 智慧城市中的大数据 [J]. 武汉大学学报·信息科学版 , 2014 , 39 ( 6 ): 631 - 640 .
LI D R , YAO Y , SHAO Z F . Big data in smart city [J]. Geomatics and Information Science of Wuhan University , 2014 , 39 ( 6 ): 631 - 640 . (in Chinese)
李华玉 , 陈永富 , 陈巧 , 等 . 基于遥感技术的森林树种识别研究进展 [J]. 西北林学院学报 , 2021 , 36 ( 6 ): 220 - 229 . doi: 10.3969/j.issn.1001-7461.2021.06.31 http://dx.doi.org/10.3969/j.issn.1001-7461.2021.06.31
LI H Y , CHEN Y F , CHEN Q , et al . Research progress of forest tree species identification based on remote sensing technology [J]. Journal of Northwest Forestry University , 2021 , 36 ( 6 ): 220 - 229 . (in Chinese) . doi: 10.3969/j.issn.1001-7461.2021.06.31 http://dx.doi.org/10.3969/j.issn.1001-7461.2021.06.31
HUANG J F , ZHANG X C , XIN Q C , et al . Automatic building extraction from high-resolution aerial images and LiDAR data using gated residual refinement network [J]. ISPRS Journal of Photogrammetry and Remote Sensing , 2019 , 151 : 91 - 105 . doi: 10.1016/j.isprsjprs.2019.02.019 http://dx.doi.org/10.1016/j.isprsjprs.2019.02.019
BERTASIUS G , TORRESANI L , YU S X , et al . Convolutional random walk networks for semantic image segmentation [C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu, HI, USA . IEEE , 2017 : 858 - 866 . doi: 10.1109/cvpr.2017.650 http://dx.doi.org/10.1109/cvpr.2017.650
GARCIA-GARCIA A , ORTS-ESCOLANO S , OPREA S , et al . A review on deep learning techniques applied to semantic segmentation [J]. arXiv preprint arXiv: 1704.06857 , 2017 . doi: 10.1016/j.asoc.2018.05.018 http://dx.doi.org/10.1016/j.asoc.2018.05.018
PANBOONYUEN T , JITKAJORNWANICH K , LAWAWIROJWONG S , et al . Road segmentation of remotely-sensed images using deep convolutional neural networks with landscape metrics and conditional random fields [J]. Remote Sensing , 2017 , 9 ( 7 ): 680 . doi: 10.3390/rs9070680 http://dx.doi.org/10.3390/rs9070680
ELSHEHABY A R , TAHA L G E D . A new expert system module for building detection in urban areas using spectral information and LIDAR data [J]. Applied Geomatics , 2009 , 1 ( 4 ): 97 - 110 . doi: 10.1007/s12518-009-0013-1 http://dx.doi.org/10.1007/s12518-009-0013-1
PEDERGNANA M , MARPU P R , DALLA MURA M , et al . Classification of remote sensing optical and LiDAR data using extended attribute profiles [J]. IEEE Journal of Selected Topics in Signal Processing , 2012 , 6 ( 7 ): 856 - 865 . doi: 10.1109/jstsp.2012.2208177 http://dx.doi.org/10.1109/jstsp.2012.2208177
MALLET C , BRETAR F . Full-waveform topographic lidar: state-of-the-art [J]. ISPRS Journal of Photogrammetry and Remote Sensing , 2009 , 64 ( 1 ): 1 - 16 . doi: 10.1016/j.isprsjprs.2008.09.007 http://dx.doi.org/10.1016/j.isprsjprs.2008.09.007
RASTI B , GHAMISI P , GLOAGUEN R . Hyperspectral and LiDAR fusion using extinction profiles and total variation component analysis [J]. IEEE Transactions on Geoscience and Remote Sensing , 2017 , 55 ( 7 ): 3997 - 4007 . doi: 10.1109/tgrs.2017.2686450 http://dx.doi.org/10.1109/tgrs.2017.2686450
GHAMISI P , RASTI B , YOKOYA N , et al . Multisource and multitemporal data fusion in remote sensing: a comprehensive review of the state of the art [J]. IEEE Geoscience and Remote Sensing Magazine , 2019 , 7 ( 1 ): 6 - 39 . doi: 10.1109/mgrs.2018.2890023 http://dx.doi.org/10.1109/mgrs.2018.2890023
CHEN Y S , LI C Y , GHAMISI P , et al . Deep fusion of remote sensing data for accurate classification [J]. IEEE Geoscience and Remote Sensing Letters , 2017 , 14 ( 8 ): 1253 - 1257 . doi: 10.1109/lgrs.2017.2704625 http://dx.doi.org/10.1109/lgrs.2017.2704625
DEBES C , MERENTITIS A , HEREMANS R , et al . Hyperspectral and LiDAR data fusion: outcome of the 2013 GRSS data fusion contest [J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2014 , 7 ( 6 ): 2405 - 2418 . doi: 10.1109/jstars.2014.2305441 http://dx.doi.org/10.1109/jstars.2014.2305441
PIRAMANAYAGAM S , SABER E , SCHWARTZKOPF W , et al . Supervised classification of multisensor remotely sensed images using a deep learning framework [J]. Remote Sensing , 2018 , 10 ( 9 ): 1429 . doi: 10.3390/rs10091429 http://dx.doi.org/10.3390/rs10091429
LI H , GHAMISI P , SOERGEL U , et al . Hyperspectral and LiDAR fusion using deep three-stream convolutional neural networks [J]. Remote Sensing , 2018 , 10 ( 10 ): 1649 . doi: 10.3390/rs10101649 http://dx.doi.org/10.3390/rs10101649
ZHANG M M , LI W , DU Q , et al . Feature extraction for classification of hyperspectral and LiDAR data using patch-to-patch CNN [J]. IEEE Transactions on Cybernetics , 2020 , 50 ( 1 ): 100 - 111 . doi: 10.1109/tcyb.2018.2864670 http://dx.doi.org/10.1109/tcyb.2018.2864670
XU X D , LI W , RAN Q , et al . Multisource remote sensing data classification based on convolutional neural network [J]. IEEE Transactions on Geoscience and Remote Sensing , 2018 , 56 ( 2 ): 937 - 949 . doi: 10.1109/tgrs.2017.2756851 http://dx.doi.org/10.1109/tgrs.2017.2756851
HONG D F , GAO L R , YOKOYA N , et al . More diverse means better: multimodal deep learning meets remote-sensing imagery classification [J]. IEEE Transactions on Geoscience and Remote Sensing , 2021 , 59 ( 5 ): 4340 - 4354 . doi: 10.1109/tgrs.2020.3016820 http://dx.doi.org/10.1109/tgrs.2020.3016820
赵传 , 张保明 , 余东行 , 等 . 利用迁移学习的机载激光雷达点云分类 [J]. 光学 精密工程 , 2019 , 27 ( 7 ): 1601 - 1612 . doi: 10.3788/OPE.20192707.1601 http://dx.doi.org/10.3788/OPE.20192707.1601
ZHAO C , ZHANG B M , YU D H , et al . Airborne LiDAR point cloud classification using transfer learning [J]. Opt. Precision Eng. , 2019 , 27 ( 7 ): 1601 - 1612 . (in Chinese) . doi: 10.3788/OPE.20192707.1601 http://dx.doi.org/10.3788/OPE.20192707.1601
EITEL A , SPRINGENBERG J T , SPINELLO L , et al . Multimodal deep learning for robust RGB-D object recognition [C]. 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Hamburg , Germany . IEEE , 2015 : 681 - 687 . doi: 10.1109/iros.2015.7353446 http://dx.doi.org/10.1109/iros.2015.7353446
HAZIRBAS C , MA L N , DOMOKOS C , et al . FuseNet: incorporating depth into semantic segmentation via fusion-based CNN architecture [C]. Asian Conference on Computer Vision. Springer , Cham , 2017 : 213 - 228 . doi: 10.1007/978-3-319-54181-5_14 http://dx.doi.org/10.1007/978-3-319-54181-5_14
LIU Z Y , ZHANG W , ZHAO P . A cross-modal adaptive gated fusion generative adversarial network for RGB-D salient object detection [J]. Neurocomputing , 2020 , 387 : 210 - 220 . doi: 10.1016/j.neucom.2020.01.045 http://dx.doi.org/10.1016/j.neucom.2020.01.045
ZHANG W K , HUANG H , SCHMITZ M , et al . Effective fusion of multi-modal remote sensing data in a fully convolutional network for semantic labeling [J]. Remote Sensing , 2017 , 10 ( 2 ): 52 . doi: 10.3390/rs10010052 http://dx.doi.org/10.3390/rs10010052
HOSSEINPOUR H , SAMADZADEGAN F , JAVAN F D . CMGFNet: a deep cross-modal gated fusion network for building extraction from very high-resolution remote sensing images [J]. ISPRS Journal of Photogrammetry and Remote Sensing , 2022 , 184 : 96 - 115 . doi: 10.1016/j.isprsjprs.2021.12.007 http://dx.doi.org/10.1016/j.isprsjprs.2021.12.007
ALSHEHHI R , MARPU P R , WOON W L , et al . Simultaneous extraction of roads and buildings in remote sensing imagery with convolutional neural networks [J]. ISPRS Journal of Photogrammetry and Remote Sensing , 2017 , 130 : 139 - 149 . doi: 10.1016/j.isprsjprs.2017.05.002 http://dx.doi.org/10.1016/j.isprsjprs.2017.05.002
HU X Y , YUAN Y . Deep-learning-based classification for DTM extraction from ALS point cloud [J]. Remote Sensing , 2016 , 8 ( 9 ): 730 . doi: 10.3390/rs8090730 http://dx.doi.org/10.3390/rs8090730
SU H , MAJI S , KALOGERAKIS E , et al . Multi-view convolutional neural networks for 3D shape recognition [C]. 2015 IEEE International Conference on Computer Vision . Santiago, Chile . IEEE , 2015 : 945 - 953 . doi: 10.1109/iccv.2015.114 http://dx.doi.org/10.1109/iccv.2015.114
WU Z R , SONG S R , KHOSLA A , et al . 3D ShapeNets: a deep representation for volumetric shapes [C]. 2015 IEEE Conference on Computer Vision and Pattern Recognition . Boston, MA, USA . IEEE , 2015 : 1912 - 1920 . doi: 10.1109/cvpr.2015.7298801 http://dx.doi.org/10.1109/cvpr.2015.7298801
QI C R , SU H , MO K , et al . PointNet: deep learning on point sets for 3D classification and segmentation [C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu, HI, USA . IEEE , 2017 : 77 - 85 . doi: 10.1109/cvpr.2017.16 http://dx.doi.org/10.1109/cvpr.2017.16
QI C R , YI L , SU H , et al . PointNet++: deep hierarchical feature learning on point sets in a metric space [J]. Advances in Neural Information Processing Systems , 2017 , 30 .
ZHOU Y , SUN P , ZHANG Y , et al . End-to-end multi-view fusion for 3d object detection in lidar point clouds [C]. Conference on Robot Learning.PLMR , 2020 : 923 - 932 .
EIMADAWI K , RASHED H , EISALLAB AEL , et al . RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving [C]. 2019 IEEE Intelligent Transportation Systems Conference. Auckland , New Zealand . IEEE , 2019 : 7 - 12 . doi: 10.1109/itsc.2019.8917447 http://dx.doi.org/10.1109/itsc.2019.8917447
ZHANG F H , FANG J , WAH B , et al . Deep FusionNet for point cloud semantic segmentation [C]. European Conference on Computer Vision. Spring , Cham , 2020 : 644 - 663 . doi: 10.1007/978-3-030-58586-0_38 http://dx.doi.org/10.1007/978-3-030-58586-0_38
LIU Z , TANG H , LIN Y , et al .. Point-Voxel CNN for Efficient 3D Deep Learning [C]. Advances in Neural Information Processing Systems . 2019 , 32 . doi: 10.7551/mitpress/11474.003.0014 http://dx.doi.org/10.7551/mitpress/11474.003.0014
王朝莹 , 邢帅 , 戴莫凡 . 遥感影像与LiDAR点云多尺度深度特征融合的地物分类方法 [J]. 测绘科学技术学报 , 2021 , 38 ( 6 ): 604 - 610, 617 . doi: 10.3969/j.issn.1673-6338.2021.06.009 http://dx.doi.org/10.3969/j.issn.1673-6338.2021.06.009
WANG Z Y , XING S , DAI M F . A method of ground object classification based on multi-scale deep feature fusion of remote sensing image and LiDAR point cloud [J]. Journal of Geomatics Science and Technology , 2021 , 38 ( 6 ): 604 - 610, 617 . (in Chinese) . doi: 10.3969/j.issn.1673-6338.2021.06.009 http://dx.doi.org/10.3969/j.issn.1673-6338.2021.06.009
WIDYANINGRUM E , BAI Q , FAJARI M K , et al . Airborne laser scanning point cloud classification using the DGCNN deep learning method [J]. Remote Sensing , 2021 , 13 ( 5 ): 859 . doi: 10.3390/rs13050859 http://dx.doi.org/10.3390/rs13050859
POLIYAPRAM V , WANG W M , NAKAMURA R . A point-wise LiDAR and image multimodal fusion network (PMNet) for aerial point cloud 3D semantic segmentation [J]. Remote Sensing , 2019 , 11 ( 24 ): 2961 . doi: 10.3390/rs11242961 http://dx.doi.org/10.3390/rs11242961
王庆超 , 付光远 , 汪洪桥 , 等 . 多核融合多尺度特征的高光谱影像地物分类 [J]. 光学 精密工程 , 2018 , 26 ( 4 ): 980 - 988 . doi: 10.3788/ope.20182604.0980 http://dx.doi.org/10.3788/ope.20182604.0980
WANG Q C , FU G Y , WANG H Q , et al . Fusion of multi-scale feature using multiple kernel learning for hyperspectral image land cover classification [J]. Opt. Precision Eng. , 2018 , 26 ( 4 ): 980 - 988 . (in Chinese) . doi: 10.3788/ope.20182604.0980 http://dx.doi.org/10.3788/ope.20182604.0980
戴莫凡 , 邢帅 , 徐青 , 等 . 多特征融合与几何卷积的机载LiDAR点云地物分类 [J]. 中国图象图形学报 , 2022 , 27 ( 2 ): 574 - 585 . doi: 10.11834/jig.210555 http://dx.doi.org/10.11834/jig.210555
DAI M F , XING S , XU Q , et al . Semantic segmentation of airborne LiDAR point cloud based on multi-feature fusion and geometric convolution [J]. Journal of Image and Graphics , 2022 , 27 ( 2 ): 574 - 585 . (in Chinese) . doi: 10.11834/jig.210555 http://dx.doi.org/10.11834/jig.210555
DAI Y M , GIESEKE F , OEHMCKE S , et al . Attentional feature fusion [C]. 2021 IEEE Winter Conference on Applications of Computer Vision. Waikoloa , HI, USA . IEEE , 2021 : 3559 - 3568 . doi: 10.1109/wacv48630.2021.00360 http://dx.doi.org/10.1109/wacv48630.2021.00360
ROTTENSTEINER F , SOHN G , JUNG J , et al . The isprs benchmark on urban object classification and 3d building reconstruction [J]. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences I-3 ( 2012 ),Nr.1, 2012, 1 ( 1 ): 293 - 298 .
JIANG M , WU Y , ZHAO T , et al . Pointsift: A sift-like network module for 3d point cloud semantic segmentation [J]. arXiv preprint arXiv: 1807.00652 , 2018 .
WANG Y , SUN Y B , LIU Z W , et al . Dynamic graph CNN for learning on point clouds [J]. ACM Transactions on Graphics , 2019 , 38 ( 5 ): 1 - 12 . doi: 10.1145/3326362 http://dx.doi.org/10.1145/3326362
HU Q Y , YANG B , XIE L H , et al . RandLA-net: efficient semantic segmentation of large-scale point clouds [C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle , WA , USA . IEEE , 2020 : 11105 - 11114 . doi: 10.1109/cvpr42600.2020.01112 http://dx.doi.org/10.1109/cvpr42600.2020.01112
0
Views
948
下载量
0
CSCD
Publicity Resources
Related Articles
Related Author
Related Institution