Rede de reconstrução 3D de fusão adaptativa de pesos de múltiplas características em múltiplas camadas para objetos com informações de alta frequência

  • role: First author第一作者
  • Affiliation:

    School of Instrument Science and Opto-electronics Engineering, Hefei University of Technology, Hefei 230000, China

  • Email:wangbiao@hfut.edu.cn
  • Introduction:王 标(1980-),男,博士,副教授,硕士生导师,2009年于合肥工业大学获得博士学位,2019年至2020年在新加坡南洋理工大学进行访问学者研究,主要研究包括精度检测与在线测量、微元件材料特性测试、机器视觉以及图像处理。E-mail: wangbiao@hfut.edu.cn
WANG Biao,  
  • Affiliation:

    School of Instrument Science and Opto-electronics Engineering, Hefei University of Technology, Hefei 230000, China

LI Ying,  
  • Affiliation:

    School of Instrument Science and Opto-electronics Engineering, Hefei University of Technology, Hefei 230000, China

RONG Baichuan,  
  • Affiliation:

    School of Instrument Science and Opto-electronics Engineering, Hefei University of Technology, Hefei 230000, China

LIU Jing,  
  • Affiliation:

    School of Instrument Science and Opto-electronics Engineering, Hefei University of Technology, Hefei 230000, China

ZHANG Jin,  
  • role: Corresponding author通信作者
  • Affiliation:

    School of Instrument Science and Opto-electronics Engineering, Hefei University of Technology, Hefei 230000, China

  • Email:yhwang@hfut.edu.cn
  • Introduction:王永红(1972-),男,博士,教授,博士生导师,2004年获得博士学位,2007-2008年在美国Oakland University做博士后,主要研究包括光学精密测试技术、激光散斑干涉测量、机器视觉与图像处理、材料变形应变机械性能测量和复合材料无损检测。E-mail: yhwang@hfut.edu.cn
WANG Yonghong*

Resumo

Para resolver o problema da perda de informações de alta frequência na textura da superfície durante o processo de reconstrução dos vetores normais da superfície do objeto no aprendizado profundo de estereoscopia fotométrica, o que causa grandes desvios na precisão da reconstrução, foi proposto uma rede de reconstrução 3D de fusão adaptativa de pesos de múltiplas características em múltiplas camadas (MMF-Net). A rede utiliza principalmente o PS-FCN como modelo base, introduzindo uma estrutura simétrica de codificador-decodificador para aumentar a capacidade de aprendizado e expressão de características da rede, melhorando a integração das características entre diferentes níveis; projetou camadas convolucionais multivariadas com ajuste adaptativo de pesos entre camadas independentes, adicionando pesos treináveis adicionais, considerando tanto informações de forma quanto de textura, e capturando melhor as variações detalhadas da textura da superfície, tornando o desempenho da rede mais estável e preciso em cenários com informações de alta frequência densas; auxiliou na adição de uma estrutura de conexões de salto, conectando características das camadas intermediárias para níveis posteriores, preservando as informações de alta frequência do objeto e reforçando as informações de baixa frequência para realizar a aplicação de fusão de características de alta e baixa frequência. Foram realizados testes utilizando o conjunto de dados padrão DiLiGenT, e os resultados experimentais mostraram que o MMF-Net pode alcançar um MAE médio de 6,94°, melhorando em 6% em comparação com o PS-FCN (Norm) de 7,39°, com erro médio de reconstrução de 11,03° para dois objetos contendo informações de alta frequência, melhorando 12% em comparação com o método anterior FUPS-Net de 12,52%. O MMF-Net realizou a aquisição eficaz de informações de baixa e alta frequência da superfície dos objetos em estereoscopia fotométrica, fornecendo uma referência para reconstrução 3D de alta precisão baseada nos vetores normais da superfície do objeto.

Palavras-chave

aprendizado profundo;estereoscopia fotométrica;convolução multivariada;fusão de características;peso adaptativo

References

  1. 1.
    HORN B K P. Shape From Shading: a Method for Obtaining the Shape of a Smooth Opaque Object From One View[D]. Cambridge: Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, 1970.
  2. 2.
    HORN B K P. Robot Vision[M]. Wang L, Jiang X L,Transl. Beijing: China Youth Publishing Group, 2014.
  3. 3.
    WOODHAM R J. Photometric method for determining surface orientation from multiple images[J]. Optical Engineering, 1980, 19(1): 139. doi:
  4. 4.
    CHO D, MATSUSHITA Y, TAI Y W, et al. Semi-calibrated photometric stereo[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(1): 232-245. doi:
  5. 5.
    KAYA B, KUMAR S, OLIVEIRA C, et al. Uncalibrated neural inverse rendering for photometric stereo of general surfaces[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 20-25, 2021. Nashville, TN, USA. IEEE, 2021: 3803-3813.
  6. 6.
    HAYAKAWA H. Photometric stereo under a light source with arbitrary motion[J]. Journal of the Optical Society of America A, 1994, 11(11): 3079-3089. doi:
  7. 7.
    PAPADHIMITRI T, FAVARO P. A closed-form, consistent and robust solution to uncalibrated photometric stereo via local diffuse reflectance maxima[J]. International Journal of Computer Vision, 2014, 107(2): 139-154. doi:
  8. 8.
    SENGUPTA S, ZHOU H, FORKEL W, et al. Solving uncalibrated photometric stereo using fewer images by jointly optimizing low-rank matrix completion and integrability[J]. Journal of Mathematical Imaging and Vision, 2018, 60(4): 563-575. doi:
  9. 9.
    BOYD S. Distributed optimization and statistical learning via the alternating direction method of multipliers[J]. Foundations and Trends® in Machine Learning, 2010, 3(1): 1-122. doi:
  10. 10.
    CHO D, MATSUSHITA Y, TAI Y W, et al. Photometricstereo Under Non-uniform Light Intensities and Exposures[M]. Leibe B, Matas J, Sebe N,et al. Computer vision-ECCV 2016. Lecture notes in computer science. Cham: Springer, 2016, 9906: 170-186. doi:
  11. 11.
    QUÉAU Y, WU T, LAUZE F, et al. A non-convex variational approach to photometric stereo under inaccurate lighting[C]. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).21-26,2017, Honolulu, HI, USA. IEEE, 2017: 350-359. doi:
  12. 12.
    ZHANG Y F, YANG Y H. Illuminant direction determination for multiple light sources[C]. Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000.15-15,2000, Hilton Head Island, SC. IEEE, 2000: 269-276.
  13. 13.
    POWELL M W, SARKAR S, GOLDGOF D. Calibration of light sources[C]. Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000.15-15,2000, Hilton Head, SC, USA. IEEE, 2000: 263-269.
  14. 14.
    HARA K, NISHINO K, LKEUCHI K. Light source position and reflectance estimation from a single view without the distant illumination assumption[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(4): 493-505. doi:
  15. 15.
    QUÉAU Y, DURIX B, WU T, et al. LED-based photometric stereo: modeling, calibration and numerical solution[J]. Journal of Mathematical Imaging and Vision, 2018, 60(3): 313-340. doi:
  16. 16.
    XIE L M, SONG Z, JIAO G H, et al. A practical means for calibrating an LED-based photometric stereo system[J]. Optics and Lasers in Engineering, 2015, 64: 42-50. doi:
  17. 17.
    WU L, GANESH A, SHI B X, et al. Robust Photometric Stereo Via Low-rank Matrix Completion and Recovery[M]. Computer Vision-ACCV 2010. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011: 703-717. doi:
  18. 18.
    WU T P, TANG C K. Photometric stereo via expectation maximization[C]. IEEE Transactions on Pattern Analysis and Machine Intelligence. 23,2009, IEEE, 2010: 546-560. doi:
  19. 19.
    IKEHATA S, WIPF D, MATSUSHITA Y, et al. Robust photometric stereo using sparse regression[C]. 2012 IEEE Conference on Computer Vision and Pattern Recognition. 16-21,2012, Providence, RI, USA. IEEE, 2012: 318-325. doi:
  20. 20.
    RUITERS R, KLEIN R. Heightfield and spatially varying BRDF reconstruction for materials with interreflections[J]. Computer Graphics Forum, 2009, 28(2): 513-522. doi:
  21. 21.
    HERTZMANN A, SEITZ S M. Example-based photometric stereo: shape reconstruction with general, varying BRDFs[C]. IEEE Transactions on Pattern Analysis and Machine Intelligence. 31,2005, IEEE, 2005: 1254-1264. doi:
  22. 22.
    SANTO H, SAMEJIMA M, SUGANO Y, et al. Deep photometric stereo network[C]. 2017 IEEE International Conference on Computer Vision Workshops (ICCVW).22-29,2017, Venice, Italy. IEEE, 2017: 501-509. doi:
  23. 23.
    NIELSEN J B, JENSEN H W, RAMAMOORTHI R. On optimal, minimal BRDF sampling for reflectance acquisition[J]. ACM Transactions on Graphics, 2015, 34(6): 1-11. doi:
  24. 24.
    XU Z X, NIELSEN J B, YU J Y, et al. Minimal BRDF sampling for two-shot near-field reflectance acquisition[J]. ACM Transactions on Graphics, 2016, 35(6): 1-12. doi:
  25. 25.
    ZHENG Q, SHI B X, PAN G. Summary study of data-driven photometric stereo methods[J]. Virtual Reality & Intelligent Hardware, 2020, 2(3): 213-221. doi:
  26. 26.
    CHEN G Y, HAN K, WONG K Y K. PS-FCN: a Flexible Learning Framework for Photometric Stereo[M]. Computer vision-ECCV 2018. Lecture notes in computer science. Cham:Springer, 2018, 11213: 3-19. doi:
  27. 27.
    SHELHAMER E, LONG J, DARRELL T. Fully convolutional networks for semantic segmentation[C]. IEEE Transactions on Pattern Analysis and Machine Intelligence. 24,2016, IEEE, 2017: 640-651. doi:
  28. 28.
    IKEHATA S. CNN-PS: CNN-based photometric stereo for general non-convex surfaces[C]. Computer Vision -ECCV 2018. Cham: Springer, 2018: 3-19. doi:
  29. 29.
    WANG R S, ZHEN C, DONG N. SDPS-M2CAN: improving predictive performance of photometric stereo networks on broad spectrum reflective surfaces with minor indentations or bumps[C]. 2024 China Automation Congress (CAC).1-3,2024, Qingdao, China. IEEE, 2024: 3347-3352. doi:
  30. 30.
    FANG M Q, SONG Y. Photometric stereo method based on deep learning and self-attention mechanism[J]. Software Engineering, 2024, 27(8): 46-50. (in Chinese)
  31. 31.
    YAO Z K. Photometric Stereo Method Based on the Unstructured Neural Network[D]. Tianjin: Tianjin University, 2021. (in Chinese)
  32. 32.
    LIU Y R, JU Y K, JIAN M W, et al. A deep-shallow and global-local multi-feature fusion network for photometric stereo[J]. Image and Vision Computing, 2022, 118: 104368. doi:
  33. 33.
    JU Y K, SHI B X, WEN B H, et al. Revisiting one-stage deep uncalibrated photometric stereo via Fourier embedding[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025, 47(8): 6185-6199. doi:
  34. 34.
    ZHANG L, SHI Y, LU W Y, et al. 3D reconstruction technique based on SURF-OKG feature matching[J]. Opt. Precision Eng., 2024, 32(6): 915-929. (in Chinese). doi:
  35. 35.
    DENG G X, LIU Z H, KE L Y, et al. Adaptive discontinuity segmentation in phase measurement[J]. Opt. Precision Eng., 2024, 32(8): 1252-1260. (in Chinese). doi:
  36. 36.
    LI M, ZHOU Z L, WU Z, et al. Multi-view photometric stereo: a robust solution and benchmark dataset for spatially varying isotropic materials[J]. IEEE Transactions on Image Processing, 2020. doi:
  37. 37.
    JOHNSON M K, ADELSON E H. Shape estimation in natural illumination[C]. CVPR 2011.20-25,2011, Colorado Springs, CO, USA. IEEE, 2011: 2553-2560. doi:
  38. 38.
    WILES O, ZISSERMAN A. SilNet: single- and multi-view reconstruction by learning from silhouettes[C]. Proceedings ofthe British Machine Vision Conference 2017. London, UK. British Machine Vision Association, 2017. doi:
  39. 39.
    SHI B X, WU Z, MO Z P, et al. A benchmark dataset and evaluation for non-lambertian and uncalibrated photometric stereo[C]. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).27-30,2016, Las Vegas, NV, USA. IEEE, 2016: 3707-3716. doi:
  40. 40.
    SHI B X, MO Z P, WU Z, et al. A benchmark dataset and evaluation for non-lambertian and uncalibrated photometric stereo[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(2): 271-284. doi:
  41. 41.
    ALLDRIN N, ZICKLER T, KRIEGMAN D. Photometric stereo with non-parametric and spatially-varying reflectance[C]. 2008 IEEE Conference on Computer Vision and Pattern Recognition. 23-28,2008, Anchorage, AK, USA. IEEE, 2008: 1-8. doi:
  42. 42.
    IKEHATA S, AIZAWA K. Photometric stereo using constrained bivariate regression for general isotropic surfaces[C]. 2014 IEEE Conference on Computer Vision and Pattern Recognition. 23-28,2014, Columbus, OH, USA. IEEE, 2014: 2187-2194. doi:
  43. 43.
    SHI B X, TAN P, MATSUSHITA Y, et al. Bi-polynomial modeling of low-frequency reflectances[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(6): 1078-1091. doi:
  44. 44.
    ZHENG Q, JIA Y M, SHI B X, et al. SPLINE-net: sparse photometric stereo through lighting interpolation and normal estimation networks[C]. 2019 IEEE/CVF International Conference on Computer Vision (ICCV).27-2,2019, Seoul, Korea. IEEE, 2019: 8548-8557. doi:
  45. 45.
    ENOMOTO K, WAECHTER M, KUTULAKOS K N, et al. Photometric stereo via discrete hypothesis-and-test search[C]. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 13-19, 2020. Seattle, WA, USA. IEEE, 2020: 2308-2316.
  46. 46.
    CHENG K H M, KUMAR A. Revisiting outlier rejection approach for non-lambertian photometric stereo[J]. IEEE Transactions on Image Processing, 2018. doi:
  47. 47.
    TANIAI T, MAEHARA T. Neural inverse rendering for general reflectance photometric stereo[C]. International Conference on Machine Learning., 2024.
  48. 48.
    LI J X, ROBLES-KELLY A, YOU S D, et al. Learning to minify photometric stereo[C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 15-20, 2019. Long Beach, CA, USA. IEEE, 2019: 7560-7568.
  49. 49.
    CHEN G Y, HAN K, SHI B X, et al. Self-calibrating deep photometric stereo networks[C]. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 15-20, 2019. Long Beach, CA, USA. IEEE, 2019: 8731-8739.
  50. 50.
    JU Y K, LAM K M, CHEN Y, et al. Pay attention to Devils: a photometric stereo network for better details[C]. Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence. July 11-17, 2020. Yokohama, Japan. International Joint Conferences on Artificial Intelligence Organization, 2020: 694-700.
  51. 51.
    JU Y K, DONG J Y, CHEN S. Recovering surface normal and arbitrary images: a dual regression network for photometric stereo[J]. IEEE Transactions on Image Processing, 2021, 30: 3676-3690. doi:
  52. 52.
    YAO Z, LI K, FU Y, et al.Gps-net: Graph-based photometric stereo network[C]. Proceedings of the Advances in Neural Information Processing Systems, 33, 2020,pp. 10306-10316.
  53. 53.
    JU Y K, PENG Y X, JIAN M W, et al. Learning conditional photometric stereo with high-resolution features[J]. Computational Visual Media, 2022, 8(1): 105-118. doi:
  54. 54.
    CHEN G Y, HAN K, SHI B X, et al. Deep photometric stereo for non-lambertian surfaces[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 129-142. doi:
  55. 55.
    JU Y K, JIAN M W, RAO Y, et al. MASR-PSN: a low-resolution photometric stereo images-relevant deep learning model for high-resolution surface normal reconstruction[J]. Journal of Image and Graphics, 2023, 28(7): 2120-2134. doi:

Leia o texto completo

The above content is generated by Large Model Translation. The translated content is for reference only. We do not assume any commercial or legal responsibilty for any consequences arising from the use of our website