(44-5) 11 * << * >> * Russian * English * Content * All Issues

Remote sensing data retouching based on image inpainting algorithms in the forgery generation problem
A.V. Kuznetsov 1,2, M.V. Gashnikov 1,2

1 Samara National Research University, 443086, Samara, Russia, Moskovskoye Shosse 34,
2 IPSI RAS – Branch of the FSRC "Crystallography and Photonics" RAS,
443001, Samara, Russia, Molodogvardeyskaya 151

 PDF, 11 MB

DOI: 10.18287/2412-6179-CO-721

Pages: 763-771.

Full text of article: Russian language.

Abstract:
We investigate image retouching algorithms for generating forgery Earth remote sensing data. We provide an overview of existing neural network solutions in the field of generation and inpainting of remote sensing images. To retouch Earth remote sensing data, we use image-inpainting algorithms based on convolutional neural networks and generative-adversarial neural networks. We pay special attention to a generative neural network with a separate contour prediction block that includes two series-connected generative-adversarial subnets. The first subnet inpaints contours of the image within the retouched area. The second subnet uses the inpainted contours to generate the resulting retouch area. As a basis for comparison, we use exemplar-based algorithms of image inpainting. We carry out computational experiments to study the effectiveness of these algorithms when retouching natural data of remote sensing of various types. We perform a comparative analysis of the quality of the algorithms considered, depending on the type, shape and size of the retouched objects and areas. We give qualitative and quantitative characteristics of the efficiency of the studied image inpainting algorithms when retouching Earth remote sensing data. We experimentally prove the advantage of generative-competitive neural networks in the construction of forgery remote sensing data.

Keywords:
forgery generation, retouching, image inpainting, neural networks, remote sensing data.

Citation:
Kuznetsov AV, Gashnikov MV. Remote sensing data retouching based on the image inpainting algorithms in the forgery generation problem. Computer Optics 2020; 44(5): 763-771. DOI: 10.18287/2412-6179-CO-721.

Acknowledgements:
The work was funded by the Russian Foundation for Basic Research under RFBR grants ## 20-37-70053, 19-07-00138, 18-01-00667 and the RF Ministry of Science and Higher Education within the state project of FSRC “Crystallography and Photonics” RAS.

References:
  1. Elharrouss O, Almaadeed N, Al-Maadeed S, Akbari Y. Image inpainting: A review. Neural Processing Letters 2020; 51: 2007-2028.
  2. Lu Q, Zhang G. Review of Image Inpainting. 2018 8th Int Conf on Manufacturing Science and Engineering (ICMSE) 2018: 655-658.
  3. Li Q, Chen G, Zhang X, Saruta K, Terata Y. Image Inpainting based on sparse representation with histogram dictionary. J Comput 2018; 13(10): 1145-1155.
  4. Amrani N, Serra-Sagristà J, Peter P, Weickert J. Diffusion-based inpainting for coding remote-sensing data. IEEE Geosci Remote Sens Lett 2017; 14(8): 1203-1207.
  5. Barnes C, Shechtman E, Finkelstein A, Goldman DB. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Trans Graph 2009; 28(3): 24.
  6. Goodfellow I, Bengio Y, Courville A. Deep learning. Cambridge, MA: MIT Press; 2016. ISBN: 978-0-262-33737-3.
  7. Zhang C, Du F, Zhang Y. A brief review of image restoration techniques based on generative adversarial models. In Book: Park JJ, Yang LT, Jeong Y-S, Hao F, eds. Advanced multimedia and ubiquitous engineering. Singapore: Springer Nature Singapore Pte Ltd; 2020: 169-175.
  8. Goodfellow I, et al. Generative adversarial nets. Proc 27th Int Conf on Neural Information Processing Systems 2014; 2: 2672-2680.
  9. Ren CX, Ziemann A, Durieux A, Theiler J. Cycle-consistent adversarial networks for realistic pervasive change generation in remote sensing imagery. arXiv preprint. Source: <https://arxiv.org/abs/1911.12546>.
  10. Lou S, Fan Q, Chen F, Wang C, Li J. Preliminary investigation on single remote sensing image inpainting through a modified gan. IEEE 10th IAPR Workshop on Pattern Recognition in Remote Sensing (PRRS) 2018; 1-6.
  11. Dong J, Yin R, Sun X, Li Q, Yang Y, Qin X. Inpainting of remote sensing SST images with deep convolutional generative adversarial network. IEEE Geosci Remote Sens Lett 2018; 16(2): 173-177.
  12. Singh P, Komodakis N. Cloud-GAN: Cloud removal for Sentinel-2 imagery using a cyclic consistent generative adversarial networks. IGARSS IEEE International Geoscience and Remote Sensing Symposium 2018; 1772-1775.
  13. Kokoshkin AV, Korotkov VA, Korotkov KV, Novichikhin EP. Retouching and restoration of missing image fragments by means of the iterative calculation of their spectra. Computer Optics 2019; 43(6): 1030-1040. DOI: 10.18287/2412-6179-2019-43-6-1030-1040.
  14. Lin D, Xu G, Wang Y, Sun X, Fu K. Dense-Add Net: An novel convolutional neural network for remote sensing image inpainting. IGARSS IEEE International Geoscience and Remote Sensing Symposium 2018; 4985-4988.
  15. Zhang Q, Yuan Q, Zeng C, Li X, Wei Y. Missing data reconstruction in remote sensing image with a unified spatial–temporal–spectral deep convolutional neural network. IEEE Trans Geosci Remote Sens 2018; 56(8): 4274-4288.
  16. Ashishrai A. Generation remote sensing images using generative adversarial networks (GAN). 2019. Source: <https://github.com/aashishrai3799/Remote-Sensing-Image-Generation>.
  17. Zhao C. Inpainting to hide structures in satellite images. 2018. Source: <https://github.com/ChenchaoZhao/NeuralCamouflage>.
  18. Zhao C. Fingerprints of the invisible hand. 2018. Source: <https://github.com/ChenchaoZhao/FingerprintsOfTheInvisibleHand>.
  19. Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention 2015; 234-241.
  20. Nazeri K, Ng E, Joseph T, Qureshi FZ, Ebrahimi M. EdgeConnect: Generative image inpainting with adversarial edge learning. arXiv preprint 2019. Source: <https://arxiv.org/abs/1901.00212>.
  21. Collobert R, Kavukcuoglu K, Farabet C. Torch7: A matlab-like environment for machine learning. BigLearn NIPS Workshop 2011.
  22. Isola P, Zhu JY, Zhou T, Efros AA. Image-to-image translation with conditional adversarial networks. Proc IEEE Conf on Comput Vis Pattern Recogn 2017; 1125-1134.
  23. Rong W, Li Z, Zhang W, Sun L. An improved CANNY edge detection algorithm. IEEE Int Conf on Mechatronics and Automation 2014; 577-582.
  24. Roscosmos. Informational resources. Source: <https://www.roscosmos.ru> . Google Earth. Source: <https://www.google.com/earth>.

© 2009, IPSI RAS
151, Molodogvardeiskaya str., Samara, 443001, Russia; E-mail: ko@smr.ru ; Tel: +7 (846) 242-41-24 (Executive secretary), +7 (846) 332-56-22 (Issuing editor), Fax: +7 (846) 332-56-20