(46-1) 16 * << * >> * Russian * English * Content * All Issues

Algorithms for multi-frame image super-resolution under applicative noise based on deep neural networks
S.V. Savvin 1, A.A. Sirota 1

1 Voronezh State University, 394018, Voronezh, Russia, Universitetskaya pl. 1

 PDF, 1298 kB

DOI: 10.18287/2412-6179-CO-904

Pages: 130-138.

Full text of article: Russian language.

Abstract:
The article describes algorithms for multi-frame image super-resolution, which recover high-resolution images from a sequence of low-resolution images of the same scene under applicative noise. Applicative noise generates local regions of outlying observations in each image and reduces the image resolution. So far, little attention has been paid to this problem. At the same time, the use of deep neural networks is considered to be a promising method of image processing, including multi-frame image super-resolution. The article considers the existing solutions to the problem and suggests a new approach based on using several pre-trained convolutional neural networks and directed acyclic graph neural networks trained by the authors. The developed approach and the algorithms based on this approach involve iterative processing of the input sequence of low-resolution images using different neural networks at different processing stages. The stages include registration of low-resolution images, their segmentation performed in order to determine regions damaged by applicative noise, and transformation performed in order to increase the resolution. The approach combines the strengths of the existing solutions while lacking their drawbacks resulting from the use of approximate mathematical data models required for the synthesis of the image processing algorithms within the statistical theory of solutions. The experimental studies demonstrated that the suggested algorithm is fully functional and allows more accurate recovery of high-resolution images than the existing analogues.

Keywords:
digital image processing, multi-frame superresolution, convolutional neural networks, deep learning, applicative noise.

Citation:
Savvin SV, Sirota AA. Algorithms for multi-frame image super-resolution under applicative noise based on deep neural networks. Computer Optics 2022; 46(1): 130-138. DOI: 10.18287/2412-6179-CO-904.

References:
  1. Villegas OV, Nandayapa M, Solo I, eds. Advanced topics on computer vision. Springer; 2018.
  2. Huang D, Liu H. A short survey of image super resolution algorithms. Journal of Computer Science Technology Updates 2015; 2(2): 19-29. DOI: 10.15379/2410-2938.2015.02.02.03.
  3. Sajjadi M, Vemulapalli R, Brown M. Frame-recurrent video super-resolution. IEEE Conf on Computer Vision and Pattern Recognition (CVPR) 2018: 6626-6634. DOI: 10.1109/CVPR.2018.00693.
  4. Kawulok M, Benecki P, Piechaczek S, Hrynczenko K, Kostrzewa D, Nalepa J. Deep learning for multiple-image super-resolution. IEEE Geosci Remote Sens Lett 2019; 17(6): 1062-1066. DOI: 10.1109/LGRS.2019.2940483.
  5. Li Z, Li S, Wang J, Wang H. A novel multi-frame color images super-resolution framework based on deep convolutional neural network. Proc 5th Int Conf on Measurement, Instrumentation and Automation (ICMIA) 2016: 634-639. DOI: 10.2991/icmia-16.2016.115.
  6. Kappeler A, Yoo S, Dai Q, Katsaggelos AK. Video super-resolution with convolutional neural networks. IEEE Trans Comput Imaging 2016; 2(2): 109-122. DOI: 10.1109/TCI.2016.2532323.
  7. Molini AB, Valsesia D, Fracastoro G, Magli E. DeepSUM: Deep neural network for super-resolution of unregistered multitemporal images. IEEE Trans Geosci Remote Sens 2020; 58(5): 3644-3656. DOI: 10.1109/TGRS.2019.2959248.
  8. Kim J, Lee JK, Lee KM. Deeply-recursive convolutional network for image super-resolution. IEEE Conf on Computer Vision and Pattern Recognition (CVPR) 2016: 1637-1645. DOI: 10.1109/CVPR.2016.181.
  9. Deudon M, Kalaitzis A, Goytom I, Arefin R, Lin Z, Sankaran K, Michalski V, Kahou S, Cornebise J, Bengio Y. HighRes-net: Recursive fusion for multi-frame super-resolution of satellite imagery. arXiv Preprint arXiv:2002.06460 2020. Source: <https://arxiv.org/abs/2002.06460>.
  10. Kanemura A, Maeda S, Ishii S. Image superresolution under spatially structured noise. 2007 IEEE Int Symposium on Signal Processing and Information Technology 2007: 275-280. DOI: 10.1109/ISSPIT.2007.4458156.
  11. Fukuda W, Kanemura A, Maeda S, Ishii S. Superresolution from occluded scenes. In Book: Leung CS, Lee M, Chan JH, eds. ICONIP '09: Proceedings of the 16th International Conference on Neural Information Processing 2009, II: 19-27. DOI: 10.1007/978-3-642-10684-2_3.
  12. Negi CS, Mandal K, Sahay RR, Kankanhalli MS. Super-resolution de-fencing: Simultaneous fence removal and high-resolution image recovery using videos. Proc IEEE Int Conf on Multimedia and Expo Workshops 2014: 1-6. DOI: 10.1109/ICMEW.2014.6890641.
  13. Savvin SV, Sirorta AA, Ivankov AYu. Methods and algorithms of super-resolution for image sequence under applicative noise [In Russian]. Izvestiya RAN. Control theory and systems 2021; 3: 136-148. DOI: 10.31857/S0002338821030070.
  14. Savvin SV, Sirota AA. Performance of super-resolution algorithms under applicative noise. J Phys Conf Ser 2019; 1479: 11-13. DOI: 10.1088/1742-6596/1479/1/012080.
  15. Savvin SV, Sirota AA. An algorithm for multi-fame image super-resolution under applicative noise based on a convolutional neural network. 2nd Int Conf on Control Systems, Mathematical Modeling, Automation and Energy Efficiency (SUMMA) 2020: 422-424. DOI: 10.1109/SUMMA50634.2020.9280698.
  16. Hui TW, Tang X, Loy CC. A lightweight optical flow cnn – revisiting data fidelity and regularization. IEEE Trans Pattern Anal Mach Intell 2020; 43(8): 2555-2569. DOI: 10.1109/TPAMI.2020.2976928.
  17. Balas VE, Roy SS, Sharma D, Samui P, eds. Handbook of deep learning applications. Springer; 2019.
  18. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Commun ACM 2017; 60(6): 84-90. DOI: 10.1145/3065386.
  19. Kim J, Lee JK, Lee KM. Accurate image super-resolution using very deep convolutional networks. IEEE Conf on Computer Vision and Pattern Recognition (CVPR) 2016: 1646-1654. DOI: 10.1109/CVPR.2016.182.
  20. Xiao J, Hays J, Ehinger K, Oliva A, Torral A. SUN database: Large-scale scene recognition from Abbey to Zoo. IEEE Conf on Computer Vision and Pattern Recognition 2010: 3485-3492. DOI: 10.1109/CVPR.2010.5539970.
  21. Algazinov EK, Sirota AA. Analysis and computer modelling of informational processes and systems [In Russian]. Moscow: "Dialog-MIFI: Publisher; 2009.
  22. Wang Z, Bovik AC, Sheikh AR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 2004; 13(4): 600-612. DOI: 10.1109/TIP.2003.819861.

© 2009, IPSI RAS
151, Molodogvardeiskaya str., Samara, 443001, Russia; E-mail: journal@computeroptics.ru ; Tel: +7 (846) 242-41-24 (Executive secretary), +7 (846) 332-56-22 (Issuing editor), Fax: +7 (846) 332-56-20