浏览全部资源
扫码关注微信
1.中国科学院 光电信息处理重点实验室,辽宁 沈阳 110016
2.中国科学院 沈阳自动化研究所,辽宁 沈阳 110016
3.中国科学院 机器人与智能制造创新研究院,辽宁 沈阳 110169
4.中国科学院大学,北京 100049
[ "梁奥(1998-),男,湖北襄阳人,硕士研究生,2021年于福州大学获得机器人工程专业学士学位,辅修人工智能专业,现主要从事基于激光雷达的目标检测及点云处理的研究。E-mail: liangao@sia.cn" ]
[ "花海洋(1978-),男,辽宁抚顺人,副研究员,研究生导师,2001年于东北大学测控技术与仪器专业获工学学士学位; 2006年于中国科学院沈阳自动化研究所模式识别与智能系统专业工学硕士学位。主要从事光电系统性能评估理论与方法、光电仿真、目标光学特性分析与建模等领域的研究。E-mail: c3ill@sia.cn" ]
收稿日期:2022-08-29,
修回日期:2022-09-26,
纸质出版日期:2023-04-25
移动端阅览
梁奥,张浩,花海洋.面向点云识别网络的显著图生成[J].光学精密工程,2023,31(08):1188-1201.
LIANG Ao,ZHANG Hao,HUA Haiyang.Saliency maps for point cloud recognition models[J].Optics and Precision Engineering,2023,31(08):1188-1201.
梁奥,张浩,花海洋.面向点云识别网络的显著图生成[J].光学精密工程,2023,31(08):1188-1201. DOI: 10.37188/OPE.20233108.1188.
LIANG Ao,ZHANG Hao,HUA Haiyang.Saliency maps for point cloud recognition models[J].Optics and Precision Engineering,2023,31(08):1188-1201. DOI: 10.37188/OPE.20233108.1188.
点云结构上的特殊性质,导致解释其深度模型学习特征的结果存在困难。提出了一种获得点云目标识别模型显著图的方法,首先在点云空间中随机释放若干自由因子并输入到模型中,然后根据设计的贡献度评价指标,基于梯度下降使骨干网络输出的池化特征尽可能偏离目标点云识别过程中输出的池化特征并更新因子位置。迭代后的因子无法参与识别过程,其对模型的预测“零贡献”,将目标点云中的点移动到这些因子的位置后对识别结果的影响与丢弃该点完全相同。点的移动过程可微,最后可根据梯度信息获得显著图。本文的方法在ModelNet40数据集上生成PointNet模型的显著图,相较于用点云中心生成显著图的方法,理论依据更强且适用的数据集更多。移动点至“零贡献”因子位置后对模型的影响较移动点至点云中心与丢弃点更相似。按本文的方法丢弃点使模型精度下降得更快,在仅丢弃100个点的情况下,模型的OA(overall accuracy)由90.4%下降至81.1%。同时经DGCNN和PointMLP评估,该显著性结果具有良好的通用性。该方法生成的显著性分数精度更高,且由模型驱动不含任何假设,适用于绝大多数点云识别模型和数据集,其显著性分析结果对目标识别网络的搭建与数据增强具有指导意义。
The specific properties of point cloud structures lead to difficulties in the interpretation of features learned from deep neural networks (DNNs). In this study, a method is proposed to obtain saliency maps for point cloud target recognition models. First, a number of free factors are randomly released in the point cloud space and input into the model. Then, based on the designed contribution evaluation index, the pooled features output by the backbone are deflected as much as possible using the target point cloud recognition process based on gradient descent, and the factor positions are updated. The iterated factors do not participate in the recognition process, thus contributing "zero" to the prediction of the model such that moving the points in the target point cloud to the positions of these factors has exactly the same effect on the recognition result as dropping the points. The process of moving the points is differentiable, and the saliency maps can be obtained from the gradient information. The saliency maps for PointNet were generated on ModelNet40 using the proposed method. This method has a strong theoretical basis for generating saliency maps and is applicable to more number of datasets compared with the method using point cloud centers. The effect of shifting points to the no-contribution factor position is more similar to dropping points than shifting points to the center of the point cloud. In this study, dropping points by the saliency scores rapidly reduced the overall accuracy of the model from 90.4% to 81.1% with only 100 points dropped. Meanwhile, the saliency scores displayed good generality as evaluated using deep graph convolutional neural network (DGCNN) and PointMLP. The proposed method is driven by the model without prior assumptions and generates significance scores with higher precision; it is applicable to most point cloud recognition models. The results of saliency analysis are instructive for the construction of target recognition networks and data augmentation.
RAJ T , HASHIM F H , HUDDIN A B , et al . A survey on LiDAR scanning mechanisms [J]. Electronics , 2020 , 9 ( 5 ): 741 . doi: 10.3390/electronics9050741 http://dx.doi.org/10.3390/electronics9050741
MEHENDALE N , NEOGE S . Review on LiDAR technology [J]. SSRN Electronic Journal , 2020 .
FEI B , YANG W , CHEN W , et al . Comprehensive Review of Deep Learning-Based 3 D Point Cloud Completion Processing and Analysis [EB/OL]. 2022 : arXiv : 2203 . 03311 . https://arxiv.org/abs/2203.03311 https://arxiv.org/abs/2203.03311 . doi: 10.1109/tits.2022.3195555 http://dx.doi.org/10.1109/tits.2022.3195555
ZAMANAKOS G , TSOCHATZIDIS L , AMANATIADIS A , et.al . A comprehensive survey of LIDAR-based 3D object detection methods with deep learning for autonomous driving [J]. Computers & Graphics , 2021 , 99 : 153 - 181 . doi: 10.1016/j.cag.2021.07.003 http://dx.doi.org/10.1016/j.cag.2021.07.003
QIAN R , LAI X , LI X . 3D object detection for autonomous driving: a survey [J]. Pattern Recognition , 2022 , 130 : 108796 . doi: 10.1016/j.patcog.2022.108796 http://dx.doi.org/10.1016/j.patcog.2022.108796
RORIZ R , CABRAL J , GOMES T . Automotive LiDAR technology: a survey [J]. IEEE Transactions on Intelligent Transportation Systems , 2022 , 23 ( 7 ): 6282 - 6297 . doi: 10.1109/tits.2021.3086804 http://dx.doi.org/10.1109/tits.2021.3086804
GUPTA A , WATSON S , YIN H J . 3D point cloud feature explanations using gradient-based methods [C]. 2020 International Joint Conference on Neural Networks (IJCNN). 1924,2020 , Glasgow, UK. IEEE , 2020 : 1 - 8 . doi: 10.1109/ijcnn48605.2020.9206688 http://dx.doi.org/10.1109/ijcnn48605.2020.9206688
CHARLES R Q , HAO S , MO K C , et al . Pointnet: deep learning on point sets for 3D classification and segmentation [C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2126,2017 , Honolulu, HI, USA. IEEE , 2017 : 77 - 85 . doi: 10.1109/cvpr.2017.16 http://dx.doi.org/10.1109/cvpr.2017.16
QI C R , YI L , SU H , et al . Pointnet++: deep hierarchical feature learning on point sets in a metric space [C]. Proceedings of the 31st International Conference on Neural Information Processing Systems. December 4 - 9 , 2017, Long Beach, California, USA. New York : ACM , 2017: 5105 - 5114 .
ZHAO H S , JIANG L , FU C W , et al . Pointweb: enhancing local neighborhood features for point cloud processing [C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 1520,2019 , Long Beach, CA, USA. IEEE , 2020 : 5560 - 5568 . doi: 10.1109/cvpr.2019.00571 http://dx.doi.org/10.1109/cvpr.2019.00571
MA X , QIN C , YOU H , et al . Rethinking Network Design and Local Geometry in Point Cloud : a Simple Residual MLP Framework [EB/OL]. 2022 : arXiv : 2202 . 07123 . https://arxiv.org/abs/2202.07123 https://arxiv.org/abs/2202.07123
QIAN G , LI Y , PENG H , et al . Pointnext : Revisiting Pointnet++ with Improved Training and Scaling Strategies [EB/OL]. 2022 : arXiv : 2206 . 04670 . https://arxiv.org/abs/2206.04670 https://arxiv.org/abs/2206.04670
YANG J , ZHANG Q , FANG R , et al . Adversarial Attack And Defense On Point Sets [EB/OL]. 2019 : arXiv : 1902 . 10899 . https://arxiv.org/abs/1902.10899 https://arxiv.org/abs/1902.10899
TAN H , KOTTHAUS H . Surrogate model-based explainability methods for point cloud nns [C]. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022 : 2239 - 2248 . doi: 10.1109/wacv51458.2022.00298 http://dx.doi.org/10.1109/wacv51458.2022.00298
TAGHANAKI S A , LUO J , ZHANG R , et al . Robustpointset : A Dataset for Benchmarking Robustness of Point Cloud Classifiers [EB/OL]. 2020 : arXiv : 2011 . 11572 . https://arxiv.org/abs/2011.11572 https://arxiv.org/abs/2011.11572
ZHANG Y , TIŇO P , LEONARDIS A , et al . A survey on neural network interpretability [J]. IEEE Transactions on Emerging Topics in Computational Intelligence , 2021 , 5 ( 5 ): 726 - 742 . doi: 10.1109/tetci.2021.3100641 http://dx.doi.org/10.1109/tetci.2021.3100641
SCHINAGL D , KRISPEL G , POSSEGGER H , et al . Occam’S laser: occlusion-based attribution maps for 3D object detectors on lidar data [C]. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 1824,2022 , New Orleans, LA, USA. IEEE , 2022 : 1131 - 1140 . doi: 10.1109/cvpr52688.2022.00121 http://dx.doi.org/10.1109/cvpr52688.2022.00121
ZHENG T H , CHEN C Y , YUAN J S , et al . Pointcloud saliency maps [C]. 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 272,2019 , Seoul, Korea (South). IEEE , 2020 : 1598 - 1606 . doi: 10.1109/iccv.2019.00168 http://dx.doi.org/10.1109/iccv.2019.00168
LV D K , YING X X , CUI Y J , et al . Research on the technology of LiDAR data processing [C]. 2017 First International Conference on Electronics Instrumentation & Information Systems (EIIS). 35,2017 , Harbin, China. IEEE , 2018 : 1 - 5 . doi: 10.1109/eiis.2017.8298694 http://dx.doi.org/10.1109/eiis.2017.8298694
BELLO S A , YU S S , WANG C , et al . Review: deep learning on 3D point clouds [J]. Remote Sensing , 2020 , 12 ( 11 ): 1729 . doi: 10.3390/rs12111729 http://dx.doi.org/10.3390/rs12111729
BAI S , BAI X , ZHOU Z C , et al . GIFT: A real-time and scalable 3D shape search engine [C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2730,2016 , Las Vegas, NV, USA. IEEE , 2016 : 5023 - 5032 . doi: 10.1109/cvpr.2016.543 http://dx.doi.org/10.1109/cvpr.2016.543
SU H , MAJI S , KALOGERAKIS E , et al . Multi-view convolutional neural networks for 3D shape recognition [C]. 2015 IEEE International Conference on Computer Vision (ICCV). 713,2015 , Santiago, Chile. IEEE , 2016 : 945 - 953 . doi: 10.1109/iccv.2015.114 http://dx.doi.org/10.1109/iccv.2015.114
QI C R , SU H , NIEßNER M , et al . Volumetric and multi-view CNNs for object classification on 3D data [C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2730,2016 , Las Vegas, NV, USA. IEEE , 2016 : 5648 - 5656 . doi: 10.1109/cvpr.2016.609 http://dx.doi.org/10.1109/cvpr.2016.609
MATURANA D , SCHERER S . 3D convolutional neural networks for landing zone detection From Lidar [C]. 2015 IEEE International Conference on Robotics and Automation (ICRA). 2630,2015 , Seattle, WA, USA. IEEE , 2015 : 3471 - 3478 . doi: 10.1109/icra.2015.7139679 http://dx.doi.org/10.1109/icra.2015.7139679
MATURANA D , SCHERER S . Voxnet: a 3D convolutional neural network for real-time object recognition [C]. 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 282,2015 , Hamburg, Germany. IEEE , 2015 : 922 - 928 . doi: 10.1109/iros.2015.7353481 http://dx.doi.org/10.1109/iros.2015.7353481
WU Z R , SONG S R , KHOSLA A , et al . 3D ShapeNets: a deep representation for volumetric shapes [C]. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 712,2015 , Boston, MA, USA. IEEE , 2015 : 1912 - 1920 . doi: 10.1109/cvpr.2015.7298801 http://dx.doi.org/10.1109/cvpr.2015.7298801
WANG C , CHENG M , SOHEL F , et al . NormalNet: a voxel-based CNN for 3D object classification and retrieval [J]. Neurocomputing , 2019 , 323 : 139 - 147 . doi: 10.1016/j.neucom.2018.09.075 http://dx.doi.org/10.1016/j.neucom.2018.09.075
LI J X , CHEN B M , LEE G H . SO-Net: Self-organizing network for point cloud analysis [C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . 1823,2018 , Salt Lake City, UT, USA . IEEE , 2018 : 9397 - 9406 . doi: 10.1109/cvpr.2018.00979 http://dx.doi.org/10.1109/cvpr.2018.00979
ZHANG W X , XIAO C X . PCAN: 3D attention map learning using contextual information for point cloud based retrieval [C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 1520,2019 , Long Beach, CA, USA. IEEE , 2020 : 12428 - 12437 . doi: 10.1109/cvpr.2019.01272 http://dx.doi.org/10.1109/cvpr.2019.01272
KLOKOV R , LEMPITSKY V . Escape from cells: deep kd-networks for the recognition of 3D point cloud models [C]. 2017 IEEE International Conference on Computer Vision (ICCV). 2229,2017 , Venice, Italy. IEEE , 2017 : 863 - 872 . doi: 10.1109/iccv.2017.99 http://dx.doi.org/10.1109/iccv.2017.99
WANG C , SAMARI B , SIDDIQI K . Local Spectral Graph Convolution for Point Set Feature Learning [EB/OL]. 2018 : arXiv : 1803 . 05827 . https://arxiv.org/abs/1803.05827 https://arxiv.org/abs/1803.05827 . doi: 10.1007/978-3-030-01225-0_4 http://dx.doi.org/10.1007/978-3-030-01225-0_4
ZHAO H S , JIANG L , JIA J Y , et al . Point transformer [C]. 2021 IEEE/CVF International Conference on Computer Vision (ICCV). 1017,2021 , Montreal, QC, Canada. IEEE , 2022 : 16239 - 16248 . doi: 10.1109/iccv48922.2021.01595 http://dx.doi.org/10.1109/iccv48922.2021.01595
ENGEL N , BELAGIANNIS V , DIETMAYER K . Point transformer [J]. IEEE Access , 2021 , 9 : 134826 - 134840 . doi: 10.1109/access.2021.3116304 http://dx.doi.org/10.1109/access.2021.3116304
GUO M H , CAI J X , LIU Z N , et al . PCT: Point cloud transformer [J]. Computational Visual Media , 2021 , 7 ( 2 ): 187 - 199 . doi: 10.1007/s41095-021-0229-5 http://dx.doi.org/10.1007/s41095-021-0229-5
PANG Y , WANG W , TAY F E H , et al . Masked Autoencoders for Point Cloud Self-Supervised Learning [EB/OL]. 2022 : arXiv : 2203 . 06604 . https://arxiv.org/abs/2203.06604 https://arxiv.org/abs/2203.06604 . doi: 10.1007/978-3-031-20086-1_35 http://dx.doi.org/10.1007/978-3-031-20086-1_35
LIPTON Z . The mythos of model interpretability: in machine learning, the concept of interpretability is both important and slippery [J]. Queue , 2018 , 16 ( 3 ): 31 - 57 . doi: 10.1145/3236386.3241340 http://dx.doi.org/10.1145/3236386.3241340
RUDIN C . Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead [J]. Nature Machine Intelligence , 2019 , 1 ( 5 ): 206 - 215 . doi: 10.1038/s42256-019-0048-x http://dx.doi.org/10.1038/s42256-019-0048-x
KIM J , HUA B S , NGUYEN D T , et al . Minimal Adversarial Examples for Deep Learning on 3D point clouds [C]. 2021 IEEE/CVF International Conference on Computer Vision (ICCV). 1017,2021 , Montreal, QC, Canada. IEEE , 2022 : 7777 - 7786 . doi: 10.1109/iccv48922.2021.00770 http://dx.doi.org/10.1109/iccv48922.2021.00770
DOSOVITSKIY A , BROX T . Inverting visual representations wdith convolutional networks [C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2730, 2016 , Las Vegas, NV, USA. IEEE , 2016 : 4829 - 4837 . doi: 10.1109/cvpr.2016.522 http://dx.doi.org/10.1109/cvpr.2016.522
MAHENDRAN A , VEDALDI A . Understanding deep image representations by inverting them [C]. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 712,2015 , Boston, MA, USA. IEEE , 2015 : 5188 - 5196 . doi: 10.1109/cvpr.2015.7299155 http://dx.doi.org/10.1109/cvpr.2015.7299155
ERHAN D , BENGIO Y , COURVILLE A , et al . Visualizing higher-layer features of a deep network [J]. University of Montreal , 2009 , 1341 ( 3 ): 1 .
NGUYEN A , DOSOVITSKIY A , YOSINSKI J , et al . Synthesizing the preferred inputs for neurons in neural networks via deep generator networks [C]. Proceedings of the 30th International Conference on Neural Information Processing Systems . 510,2016 , Barcelona, Spain. New York : ACM , 2016 : 3395 - 3403 .
YOSINSKI J , CLUNE J , NGUYEN A , et al . Understanding Neural Networks through Deep Visualization [EB/OL]. 2015 : arXiv : 1506 . 06579 . https://arxiv.org/abs/1506.06579 https://arxiv.org/abs/1506.06579
ADLER P , FALK C , FRIEDLER S A , et al . Auditing black-box models for indirect influence [J]. Knowledge and Information Systems , 2018 , 54 ( 1 ): 95 - 122 . doi: 10.1007/s10115-017-1116-3 http://dx.doi.org/10.1007/s10115-017-1116-3
KÁDÁR Á , CHRUPAŁA G , ALISHAHI A . Representation of linguistic form and function in recurrent neural networks [J]. Computational Linguistics , 2017 , 43 ( 4 ): 761 - 780 . doi: 10.1162/coli_a_00300 http://dx.doi.org/10.1162/coli_a_00300
LI J , MONROE W , JURAFSKY D . Understanding Neural Networks Through Representation Erasure [EB/OL]. 2016 : arXiv : 1612 . 08220 . https://arxiv.org/abs/1612.08220 https://arxiv.org/abs/1612.08220
SIMONYAN K , VEDALDI A , ZISSERMAN A . Deep Inside Convolutional Networks : Visualising Image Classification Models and Saliency Maps [EB/OL]. 2013 : arXiv : 1312 . 6034 . http://arxiv.org/abs/1312.6034 http://arxiv.org/abs/1312.6034
SMILKOV D , THORAT N , KIM B , et al . Smoothgrad : Removing Noise by Adding Noise [EB/OL]. 2017 : arXiv : 1706 . 03825 . https://arxiv.org/abs/1706.03825 https://arxiv.org/abs/1706.03825
SUNDARARAJAN M , TALY A , YAN Q Q . Axiomatic attribution for deep networks [C]. Proceedings of the 34th International Conference on Machine Learning-Volume 70. August 6 - 11 , 2017, Sydney, NSW, Australia. New York : ACM , 2017: 3319 - 3328 .
ZHOU B L , KHOSLA A , LAPEDRIZA A , et al . Learning deep features for discriminative localization [C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2730,2016 , Las Vegas, NV, USA. IEEE , 2016 : 2921 - 2929 . doi: 10.1109/cvpr.2016.319 http://dx.doi.org/10.1109/cvpr.2016.319
KRISHNAN R , SIVAKUMAR G , BHATTACHARYA P . Extracting decision trees from trained neural networks [J]. Pattern Recognition , 1999 , 32 ( 12 ): 1999 - 2009 . doi: 10.1016/s0031-3203(98)00181-2 http://dx.doi.org/10.1016/s0031-3203(98)00181-2
YANG C L , RANGARAJAN A , RANKA S . Global model interpretation via recursive partitioning [C]. 2018 IEEE 20th International Conference on High Performance Computing and Communications; IEEE 16th International Conference on Smart City; IEEE 4th International Conference on Data Science and Systems (HPCC/SmartCity/DSS). 2830,2018 , Exeter, UK. IEEE , 2019 : 1563 - 1570 . doi: 10.1109/hpcc/smartcity/dss.2018.00256 http://dx.doi.org/10.1109/hpcc/smartcity/dss.2018.00256
SETIONO R , LIU H . Understanding neural networks via rule extraction [C]. IJCAI , 1995 , 1 : 480 - 485 .
RIBEIRO M T , SINGH S , GUESTRIN C . “Why Should I Trust You?”: Explaining the predictions of any classifier [C]. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1317,2016 , San Francisco, California, USA. New York : ACM , 2016 : 1135 - 1144 . doi: 10.1145/2939672.2939778 http://dx.doi.org/10.1145/2939672.2939778
LIU D , YU R , SU H . Extending adversarial attacks and defenses to deep 3D point cloud classifiers [C]. 2019 IEEE International Conference on Image Processing (ICIP). 2225,2019 , Taipei, China. IEEE , 2019 : 2279 - 2283 . doi: 10.1109/icip.2019.8803770 http://dx.doi.org/10.1109/icip.2019.8803770
WANG Y , SUN Y B , LIU Z W , et al . Dynamic graph CNN for learning on point clouds [J]. ACM Transactions on Graphics , 2019 , 38 ( 5 ): 1 - 12 . doi: 10.1145/3326362 http://dx.doi.org/10.1145/3326362
LOSHCHILOV I , HUTTER F . SGDR : Stochastic Gradient Descent with Warm Restarts [EB/OL]. 2016 : arXiv : 1608 . 03983 . https://arxiv.org/abs/1608.03983 https://arxiv.org/abs/1608.03983
0
浏览量
845
下载量
0
CSCD
关联资源
相关文章
相关作者
相关机构