(48-6) 16 * << * >> * Russian * English * Content * All Issues

A unified neural network-based single super-resolution method for heterogeneous digital earth remote sensing images
V.F.  Konovalov 1, V.V. Myasnikov 1, V.V. Sergeev 1

1 Samara National Research University,
443086, Samara, Russia, Moskovskoye Shosse 34

 PDF, 994 kB

DOI: 10.18287/2412-6179-CO-1610

Pages: 944-955.

Full text of article: Russian language.

Abstract:
This paper is devoted to finding a unified solution for the practical problem of increasing the resolution of heterogeneous digital images of remote sensing of the Earth: panchromatic images, color images, multispectral and hyperspectral images, as well as images obtained by synthetic aperture radar (SAR). To obtain such a solution, an aggregated dataset was collected from the existing datasets, including both pairs of high- and low-resolution images and individual images acting as high-resolution images. For the collected database and a typical distortion model, an experimental comparison of various modern neural network models of resolution enhancement (including pre-tuned and retrained options) was carried out, which are listed in reviews as state-of-the-art: convolutional, generative adversarial and transform (based on the attention mechanism). Taking into account possible limitations of the models on the number of layers of space images, two options for image preprocessing were considered. The final result of the work is a single neural network model for increasing the resolution of digital space images, supplemented by adapters for their various types, which, when trained, surpasses or is not inferior to modern specialized state-of-the-art solutions.

Keywords:
digital remote sensing images, image enhancement, multispectral and hyperspectral images, radar images.

Citation:
Konovalov VF, Myasnikov VV, Sergeev VV. A unified neural network-based single super-resolution method for heterogeneous digital earth remote sensing images. Computer Optics 2024; 48(6): 944-955. DOI: 10.18287/2412-6179-CO-1610.

Acknowledgements:
This work was supported by the Ministry of Science and Higher Education, Russia (Agreement No.075-15-2024-558).

References:
  1. Wang Y, Bashir SMA, Khan M, Ullah Q, Wang R, Song Y, Guo Z, Niu Y. Remote sensing image super-resolution and object detection: Benchmark and state of the art. Expert Syst Appl 2022; 197: 116793. DOI: 10.1016/j.eswa.2022.116793.
  2. Meng X, Shen H, Li H, Zhang L, Fu R. Review of the pansharpening methods for remote sensing images based on the idea of meta-analysis: Practical discussion and challenges. Inf Fusion 2018; 46: 102-113. DOI: 10.1016/j.inffus.2018.05.006.
  3. Schowengerdt RA. Remote sensing, models, and methods for image processing. San Diego: Academic Press; 1997. ISBN: 0-12-628981-6.
  4. Lia J, Hong D, Gao L, Yao J, Zheng K, Zhang B, Chanussot J. Deep learning in multimodal remote sensing data fusion: A comprehensive review. Int J Appl Earth Obs Geoinf 2022; 112: 102926. DOI: 10.1016/j.jag.2022.102926.
  5. Malhotra R, Singh P. Recent advances in deep learning models: a systematic literature review. Multimed Tools Appl 2023; 82(29): 44977-45060. DOI: 10.1007/s11042-023-15295-z.
  6. Wang X, Yi J, Guo J, Song Y, Lyu J, Xu J, Yan W, Zhao J, Cai Q, Min H. A review of image super-resolution approaches based on deep learning and applications in remote sensing. Remote Sens 2022; 14(21): 5423. DOI: 10.3390/rs14215423.
  7. Reshad Md, Burks R, Kwan C, Li J. Deep learning for remote sensing image super-resolution. 2019 IEEE 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conf (UEMCON) 2019: 286-292. DOI: 10.1109/UEMCON47517.2019.8993047.
  8. Zhao X, Wang L, Zhang Y. A review of convolutional neural networks in computer vision. Artif Intell Rev 2024; 57(4): 99. DOI: 10.1007/s10462-024-10721-6.
  9. Wu J, Cong R, Fang L. Unpaired remote sensing image super-resolution with content-preserving weak supervision neural network. Sci China Inf Sci 2023; 66: 119105. DOI: 10.1007/s11432-021-3575-1.
  10. Wang P, Zhang H, Zhou F, Jiang Z. Unsupervised remote sensing image super-resolution using cycle CNN. IGARSS 2019 – 2019 IEEE Int Geoscience and Remote Sensing Symposium 2019: 3117-3120. DOI: 10.1109/IGARSS.2019.8898648.
  11. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville AC, Bengio Y. Generative adversarial nets. Neural Information Processing Systems 2014: 1-9.
  12. Vaswani A, et al. Attention is all you need. NIPS'17: Proc 31st Int Conf on Neural Information Processing Systems 2017: 6000-6010.
  13. Kolesnikov A, Dosovitskiy A, Weissenborn D, Heigold G, Uszkoreit J, Beyer L, Minderer M, Dehghani M, Houlsby N, Gelly S, Unterthiner T, Zhai X. An image is worth 16x16 words: Transformers for image recognition at scale. Int Conf on Learning Representations 2021: 1-21.
  14. Wang P, Bayram B, Sertel E. A comprehensive review on deep learning based remote sensing image super-resolution methods. Earth-Sci Rev 2022; 232: 104110. DOI: 10.1016/j.earscirev.2022.104110.
  15. Tuna C, Unal G, Sertel E. Single-frame super resolution of remote-sensing images by convolutional neural networks. Int J Remote Sens 2018; 39(8): 2463-2479. DOI: 10.1080/01431161.2018.1425561.
  16. Zhou S, Zhang J, Zuo W, Loy CC. Cross-scale internal graph neural network for image super-resolution. 34th Conf on Neural Information Processing Systems (NeurIPS 2020) 2020; 1-13.
  17. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP. Image quality assessment: from error visibility to structural similarity. IEEE Trans on Image Process 2004; 13(4): 600-612. DOI: 10.1109/TIP.2003.819861.
  18. Zhang R, Isola P, Efros AA, Shechtman E, Wang O. The unreasonable effectiveness of deep features as a perceptual metric. 2018 IEEE/CVF Conf on Computer Vision and Pattern Recognition 2018: 586-595. DOI: 10.1109/CVPR.2018.00068.
  19. Dong C, Loy CC, He K, Tang X. Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Machine Intell 2016; 38(2): 295-307. DOI: 10.1109/TPAMI.2015.2439281.
  20. Kim J, Lee JK, Lee KM. Accurate image super-resolution using very deep convolutional networks. Proc 2016 IEEE Conf on Computer Vision and Pattern Recognition 2016: 1646-1654. DOI: 10.1109/CVPR.2016.182.
  21. Ji Y, Zhang H, Gao F, Sun H, Wei H, Wang N, Yang B. LGCNet: A local-to-global context-aware feature augmentation network for salient object detection. Inf Sci 2017; 584: 399-416. DOI: 10.1016/j.ins.2021.10.055.
  22. Kui J, Zhongyuan W, Pen Y, Junjun J. A progressively enhanced network for video satellite imagery superresolution. IEEE Signal Process Lett 2018; 25(11): 1630-1634. DOI: 10.1109/LSP.2018.2870536.
  23. Rika S, Bayu SA, Donny D, Ketut W. Generative adversarial network with residual dense generator for remote sensing image super resolution. ICRAMET 2020: 34-39. DOI: 10.1109/ICRAMET51080.2020.9298648.
  24. Jiang K, Wang Z, Yi P, Jiang J, Xiao J, Yao Y. Deep distillation recursive network for remote sensing imagery super-resolution. Remote Sens 2018; 10(11): 1700. DOI: 10.3390/rs10111700.
  25. Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, Shi W. Photo-realistic single image super-resolution using a generative adversarial network. Proc IEEE Conf on Computer Vision and Pattern Recognition 2017: 4681-4690. DOI: 10.1109/CVPR.2017.19.
  26. Wang X, Xie L, Dong C, Shan Y. Real-ESRGAN: Training real-world blind super-resolution with pure synthetic data. 2021 IEEE/CVF Int Conf on Computer Vision Workshops (ICCVW) 2021: 1905-1914. DOI: 10.1109/ICCVW54120.2021.00217.
  27. Kui J, Wang Z, Yi P, Wang G, Lu T, Jiang J. Edge-enhanced GAN for remote sensing image superresolution. IEEE Trans Geosci Remote Sens 2019; 57(8): 5799-5812. DOI: 10.1109/TGRS.2019.2902431.
  28. Zhang, Yulun, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, Yun Fu. Image super-resolution using very deep residual channel attention networks. In Book: Ferrari V, Hebert M, Sminchisescu C, Weiss Y, eds. Computer Vision – ECCV 2018. 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part VII. Cham, Switzerland: Springer Nature Switzerland AG; 2018: 286-301. DOI: 10.1007/978-3-030-01234-2_18.
  29. Haut JM, Fernandez-Beltran R, Paoletti ME, Plaza J, Plaza A. Remote sensing image superresolution using deep residual channel attention. IEEE Trans Geosci Remote Sens 2019; 57(11): 9277-9289. DOI: 10.1109/TGRS.2019.2924818.
  30. Niu B, Wen W, Ren W, Zhang X, Yang L, Wang S, Zhang K, Cao X, Shen H. Single image super-resolution via a holistic attention network. In Book: Vedaldi A, Bischof H, Brox T, Frahm J-M, eds. Computer Vision – ECCV 2020. 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII. Cham, Switzerland: Springer Nature Switzerland AG; 2020: 191-207. DOI: 10.1007/978-3-030-58610-2_12.
  31. Dai T, Cai J, Zhang Y, Xia S-T, Zhang L. Second-order attention network for single image super-resolution. 2019 IEEE/CVF Conf on Computer Vision and Pattern Recognition (CVPR) 2019: 11057-11066. DOI: 10.1109/CVPR.2019.01132.
  32. Zhang D, Shao J, Li X, Shen HT. Remote sensing image super-resolution via mixed high-order attention network. IEEE Trans Geosci Remote Sens 2021; 59(6): 5183-5196. DOI: 10.1109/TGRS.2020.3009918.
  33. Wang P, Bayram B, Sertel E. Super-resolution of remotely sensed data using channel attention based deep learning approach. Int J Remote Sens 2021; 42(16): 6048-6065. DOI: 10.1080/01431161.2021.1934598.
  34. Liang J, Cao J, Sun G, Zhang K, Van Gool L, Timofte R. SwinIR: Image restoration using swin transformer. 2021 IEEE/CVF Int Conf on Computer Vision Workshops (ICCVW) 2021: 1833-1844. DOI: 10.1109/ICCVW54120.2021.00210.
  35. Rossi L, Bernuzzi V, Fontanini T, Bertozzi M, Prati A. Swin2-MoSE: A new single image super-resolution model for remote sensing. arXiv Preview. 2024. Source: <https://arxiv.org/abs/2404.18924>. DOI: 10.48550/arXiv.2404.18924.
  36. Michel J, Vinasco-Salinas J, Inglada J, Hagolle O. SEN2VENµS, a dataset for the training of Sentinel-2 super-resolution algorithms. Data 2022; 7(7): 96. DOI: 10.3390/data7070096.
  37. Chen H, He X, Qing L, Wu Y, Ren C, Sheriff RE, Zhu C. Real-world single image super-resolution: A brief review. Inf Fusion 2022; 79: 124-145. DOI: 10.1016/j.inffus.2021.09.005.
  38. Kehtarnavaz N, Kim N. Digital signal processing system-level design using LabVIEW. Newnes; 2005: 43-53. ISBN: 978-0750679145.
  39. Al-Mekhlafi H, Liu S. Single image super-resolution: a comprehensive review and recent insight. Front Comput Sci 2024; 18: 181702. 2024. DOI: 10.1007/s11704-023-2588-9.
  40. New trend in image restoration and enhancement. 2024. Source: <https://cvlai.net/ntire/2024/>.
  41. Bavrina AYu, Myasnikov VV, Sergeev AV. Method of parametric estimation of optoelectronic tract of remote sensed optical image formation. Computer Optics 2011; 35(4): 500-507.
  42. Bavrina AYu, Myasnikov VV, Sergeyev VV, Tresheva EV, Chupshev NV. Modelling of videoinformational tract of optoelectronic remote sensing systems of Earth: solutions, problems and tasks. Computer Optics 2012; 36(4): 572-585.

© 2009, IPSI RAS
151, Molodogvardeiskaya str., Samara, 443001, Russia; E-mail: journal@computeroptics.ru ; Tel: +7 (846) 242-41-24 (Executive secretary), +7 (846) 332-56-22 (Issuing editor), Fax: +7 (846) 332-56-20