Image restoration in diffractive optical systems using deep learning and deconvolution
Nikonorov A.V., Petrov M.V., Bibikov S.A., Kutikova V.V., Morozov A.A., Kazanskiy N.L.

 

Image Processing Systems Institute оf the RAS – Branch of the FSRC “Crystallography and Photonics” RAS, Samara, Russia,
Samara National Research University, Samara, Russia

Full text of article: Russian language.

 PDF

Abstract:
In recent years, several pioneering works were dedicated to imaging systems based on simple diffractive structures like Fresnel lenses or phase zone plates. Such systems are much lighter and cheaper than classical refractive optical systems. However, the quality of images obtained by diffractive optics suffers from stronger distortions of various types. In this paper, we show that a combination of the high-precision lens design with post-capture computational reconstruction allows one to attain a much higher image quality. The proposed reconstruction procedure uses a sequence of color correction, deconvolution, and a feedforward deep learning neural network. An improvement both in lens manufacturing and in image processing may contribute to the emergence of ultra-lightweight imaging systems varying from cameras for nano- and picosatellites to surveillance systems.

Keywords:
harmonic lens, remote sensing, deconvolution, deep learning, PSF estimation, color correction.

Citation:
Nikonorov AV, Petrov MV, Bibikov SA, Kutikova VV, Morozov AA, Kazanskiy NL. Image restoration in diffractive optical systems using deep learning and deconvolution. Computer Optics 2017; 41(6): 875-887. DOI: 10.18287/2412-6179-2017-41-6-875-887.

References:

  1. Nikonorov A, Skidanov R, Fursov V, Petrov M, Bibikov S, Yuzifovich Y. Fresnel Lens Imaging with Post-Capture Image Processing. CVPRW 2015: 33-41. DOI: 10.1109/CVPRW.2015.7301373.
  2. Hasinoff SW, Kutulakos KN. Light-efficient photography. ECCV2008: 45-59. DOI: 10.1007/978-3-540-88693-8_4.
  3. Wang P, Mohammad N, Menon R. Chromatic-aberration-corrected diffractive lenses for ultra-broadband focusing. Sci Rep 2016; 6: 21545. DOI: 10.1038/srep21545.
  4. Kazanskii NL, Khonina SN, Skidanov RV, Morozov AA, Kharitonov SI, Volotovsky SG. Formation of images using multilevel diffractive lens [In Russian]. Computer Optics 2014; 38(3): 425-434.
  5. Peng Y, Fu Q, Amata H, Su S, Heide F, Heidrich W. Computational imaging using lightweight diffractive-refractive optics. Opt Express 2015; 23(24): 31393-31407. DOI: 10.1364/OE.23.031393.
  6. Peng Y, Fu Q, Heide F, Heidrich W. The diffractive achromat full spectrum computational imaging with diffractive optics. SIGGRAPH ASIA 2016; 4. DOI: 10.1145/2992138.2992145.
  7. Nikonorov A, Petrov M, Bibikov S, Yuzifovich Y, Yakimov P, Kazanskiy N, Skidanov R, Fursov V. Comparative evaluation of deblurring techniques for Fresnel lens computational imaging. ICPR 2016. DOI: 10.1109/ICPR.2016.7899729.
  8. Heide F, Fu Q, Peng Y, Heidrich W. Encoded diffractive optics for full-spectrum computational imaging. Sci Rep 2016;6: 33543. DOI: 10.1038/srep33543.
  9. Heide F, Steinberger M, Tsai Y-T, Rouf M, Pajak D, Reddy D, Gallo O, Liu J, Heidrich W, Egiazarian K, Kautz J, Pulli K. FlexISP: A flexible camera image processing framework. ACM Trans Graph 2014; 33(6): 231. DOI: 10.1145/2661229.2661260.
  10. Chambolle A, Pock T. A first-order primal-dual algorithm for convex problems with applications to imaging. J Math Imaging Vision 2011;40: 120-145. DOI: 10.1007/s10851-010-0251-1.
  11. Genevet P, Capasso F, Aieta F, Khorasaninejad M, Devlin R. Recent advances in planar optics: from plasmonic to dielectric metasurfaces. Optica 2017;4(1):139-152. DOI: 10.1364/OPTICA.4.000139.
  12. Schuler CJ, Hirsch M, Harmeling S, Schölkopf B. Learning to deblur. IEEE Trans Pattern Anal Mach Intel2016; 38(7): 1439-1451. DOI: 10.1109/TPAMI.2015.2481418.
  13. Dong C, Loy CC, He K, Tang X. Image superresolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intel 2015; 38(2): 295-307. DOI: 10.1109/TPAMI.2015.2439281.
  14. Kim J, Lee JK, Lee KM. Accurate image super-resolution using very deep convolutional networks. CVPR 2016: 1646-1654. DOI: 10.1109/CVPR.2016.182.
  15. Lai W-S, Huang J-B, Ahuja N, Yang M-H. Deep laplacian pyramid networks for fast and accurate super-resolution. Source: <https://arxiv.org/pdf/1704.03915.pdf>.
  16. Yuan Y, Zheng X, Lu X. Hyperspectral Image Superresolution by Transfer Learning. IEEE J Sel Top Appl Earth Obs Remote Sens 2017; 10(5): 1963-1974. DOI: 10.1109/JSTARS.2017.2655112.
  17. Sweeney DW, Sommargren GE. Harmonic diffractive lenses. Appl Opt 1995; 34(14): 2469-2475. DOI: 10.1364/AO.34.002469.
  18. Soifer VA, ed. Computer design of diffractive optics. Cambridge: Woodhead Publishing; 2012. ISBN: 978-1-845696351, DOI: 10.1533/9780857093745.
  19. Poleshchuk AG, Korolkov VP, Nasyrov RK. Diffractive optical elements: fabrication and application. Proc SPIE 2014; 9283:928302. DOI: 10.1117/12.2073301.
  20. Nikonorov A, Bibikov S, Myasnikov V, Yuzifovich Y, Fursov V. Correcting color and hyperspectral images with identification of distortion model. Pattern Recognit Lett 2016; 83(2): 178-187. DOI: 10.1016/j.patrec.2016.06.027.
  21. Kharitonov SI, Volotovsky SG, Khonina SN. Geometric-optical calculation of the focal spot of a harmonic diffractive lens [In Russian]. Computer Optics 2016; 40(3): 331-337. DOI: 10.18287/2412-6179-2016-40-3-331-337.
  22. Chakrabarti A, Zickler T. Fast deconvolution with color constraints on gradients. Harvard Computer Science Group Technical Report TR-06-12 2012.
  23. Krishnan D, Fergus R. Fast image deconvolution using Hyper-Laplacian priors. NIPS 2009; 1033-1041.
  24. He K, Sun J, Tang X. Guided image filtering. IEEE Trans Pattern Anal Mach Intell 2013; 35(6): 1397-1409. DOI: 10.1109/TPAMI.2012.213.
  25. Van De Weijer J, Gevers T, Gijsenij A. Edge-based color constancy. IEEE Trans Image Process 2007; 16(9): 2207-2214. DOI: 10.1109/TIP.2007.901808.
  26. Deqing S, Roth S, Black MJ. Secrets of optical flow estimation and their principles. CVPR 2010; 2432-2439. DOI: 10.1109/CVPR.2010.5539939.
  27. He K, Zhang X, Ren S, Sun J. Delving deep into rectifiers: Surpassing human-level performance on image net classification. ICCV 2015; 1026-1034. DOI: 10.1109/ICCV.2015.123.
  28. Kingma DP, Ba JL. Adam: A method for stochastic optimization. Source: <https://arxiv.org/pdf/1412.6980.pdf>.
  29. Han S, Pool J, Tran J, Dally W. Learning both weights and connections for efficient neural network. NIPS 2015; 1135-1143.
© 2009, IPSI RAS
151, Molodogvardeiskaya str., Samara, 443001, Russia; E-mail: journal@computeroptics.ru ; Tel: +7 (846) 242-41-24 (Executive secretary), +7 (846)332-56-22 (Issuing editor), Fax: +7 (846) 332-56-20