(50-5) 11 * << * >> * Russian * English * Content * All Issues

A method for cloud soft segmentation using multimodal remote sensing data
V.F. Konovalov1, V.V. Myasnikov1, V.V. Sergeev1

1 Samara National Research University, Moskovskoye Shosse 34, Samara, 443086, Russia

  Full text (PDF)

DOI: 10.18287/COJ2005

Article ID: 2005

Language: Russian

Abstract:
This paper presents a method for estimating cloud transparency fields using the TerraMind foundation neural network model. The TerraMind model utilizes multispectral and synthetic aperture radar images as multimodal data for the survey area. The TerraMind model was trained on the SEN12MSCR dataset, containing pairs of Sentinel-1 and Sentinel-2 satellite images, both with and without cloud cover. To further train the fundamental model to solve the desired problem, we used augmentation, synthesizing cloud cover and the corresponding true transparency fields using the Satellite Cloud Generation Toolbox. Numerical results from the experimental study demonstrate the high quality of the proposed estimation method, and the qualitative results of estimating transparency fields on real-world images confirm the method's effectiveness on real data.

Keywords:
Sentinel 2, Sentinel 1, synthetic aperture radar, multispectral imagery, neural networks, cloud detection, image dehazing, synthetic dataset, TerraMind.

Acknowledgements:
The study was financially supported by the Ministry of Education and Science of the Russian Federation under contract No. 075-15-2024-558.

Citation:
Konovalov VF, Myasnikov VV, Sergeev VV. A method for cloud soft segmentation using multimodal remote sensing data. Computer Optics 2026; 50(5): 2005. doi: 10.18287/COJ2005.

References:

  1. King MD, Platnick S, Menzel WP, Ackerman SA, Hubanks PA. Spatial and temporal distribution of clouds observed by MODIS onboard the Terra and Aqua satellites. IEEE Transactions on Geoscience and Remote Sensing 2013; 51(7): 3826-3852. DOI: 10.1109/TGRS.2012.2227333.
  2. Konovalov VF, Myasnikov VV, Sergeev VV. Reconstruction of optical remote sensing images using synthetic aperture radar data and diffusion neural networks. Optoelectronics, Instrumentation and Data Processing 2025; 61(4): 449-457.
  3. Meraner A, Ebel P, Zhu XX, Schmitt M. Cloud removal in Sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion. ISPRS Journal of Photogrammetry and Remote Sensing 2020; 166: 333-346. DOI: 10.1016/j.isprsjprs.2020.05.013.
  4. Valovcin FR. Snow/Cloud Discrimination. Hanscom AFB, MA: Air Force Geophysics Laboratory, Air Force Systems Command, United States Air Force; 1976. 17 p. Air Force Surveys in Geophysics No. 349. Report AFGL-TR-76-0174.
  5. Kyle HL, Curran RJ, Barnes WL, Escoe D. A cloud physics radiometer. In: Proceedings of the Third Conference on Atmospheric Radiation; June 28-30, 1978; Davis, CA, USA. Boston, MA: American Meteorological Society; 1978: 107.
  6. Hall DK, Riggs GA, Salomonson VV. Development of methods for mapping global snow cover using Moderate Resolution Imaging Spectroradiometer data. Remote Sensing of Environment 1995; 54(2): 127-140.
  7. Wang Z, Zhao L, Meng J, Han Y, Li X, Jiang R, Chen J, Li H. Deep learning-based cloud detection for optical remote sensing images: A survey. Remote Sensing 2024; 16(23): 4583. DOI: 10.3390/rs16234583.
  8. Li Z, Shen H, Weng Q, Zhang Y, Dou P, Zhang L. Cloud and cloud shadow detection for optical satellite imagery: Features, algorithms, validation, and prospects. ISPRS Journal of Photogrammetry and Remote Sensing 2022; 188: 89-108. DOI: 10.1016/j.isprsjprs.2022.04.018.
  9. Huang L, Jiang B, Lv S, Liu Y, Fu Y. Deep learning-based semantic segmentation of remote sensing images: A survey. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 2024; 17: 8370-8396. DOI: 10.1109/JSTARS.2024.3388508.
  10. Mahajan S, Fataniya B. Cloud detection methodologies: Variants and development—a review. Complex & Intelligent Systems 2020; 6: 251-261. DOI: 10.1007/s40747-019-00128-0.
  11. Zhu X, Helmer EH. An automatic method for screening clouds and cloud shadows in optical satellite image time series in cloudy regions. Remote Sensing of Environment 2018; 214: 135-153. DOI: 10.1016/j.rse.2018.05.024.
  12. Frantz D, Haß E, Uhl A, Stoffels J, Hill J. Improvement of the Fmask algorithm for Sentinel-2 images: Separating clouds from bright surfaces based on parallax effects. Remote Sensing of Environment 2018; 215: 471-481. DOI: 10.1016/j.rse.2018.04.046.
  13. Qiu S, Zhu Z, He B. Fmask 4.0: Improved cloud and cloud shadow detection in Landsats 4-8 and Sentinel-2 imagery. Remote Sensing of Environment 2019; 231: 111205. DOI: 10.1016/j.rse.2019.05.024.
  14. Ning J, Xie L, Yin J, Liu Y. Cloud removal advances: A comprehensive review and analysis for optical remote sensing images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 2025; 18: 18621-18641. DOI: 10.1109/JSTARS.2025.3580718.
  15. Joglekar M, Deshpande A. Remote sensing image dehazing techniques: Systematic review, performance evaluation and future directions. Archives of Computational Methods in Engineering 2026. DOI: 10.1007/s11831-026-10500-2.
  16. Aybar C, Bautista L, Montero D, Contreras J, Ayala D, Prudencio F, Loja J, Ysuhuaylas L, Herrera F, Gonzales K, et al. CloudSEN12+: The largest dataset of expert-labeled pixels for cloud and cloud shadow detection in Sentinel-2. Data in Brief 2024; 56: 110852. DOI: 10.1016/j.dib.2024.110852.
  17. Main-Knorn M, Pflug B, Louis J, Debaecker V, Müller-Wilm U, Gascon F. Sen2Cor for Sentinel-2. In: Proceedings of the Living Planet Symposium 2016; May 9-13, 2016; Prague, Czech Republic. ESA Special Publication SP-740; 2017: 1-8.
  18. Hu J, Wei Y, Chen Z, Zhi X, Zhang W. CM-YOLO: Typical object detection method in remote sensing cloud and mist scene images. Remote Sensing 2025; 17(1): 125. DOI: 10.3390/rs17010125.
  19. Zhang J, Li Y, Yang X, Jiang R, Zhang L. RSAM-Seg: A SAM-based model with prior knowledge integration for remote sensing image semantic segmentation. Remote Sensing 2025; 17(4): 590. DOI: 10.3390/rs17040590.
  20. Yu Z, Idris MYI, Wang P. When cloud removal meets diffusion model in remote sensing. arXiv preprint arXiv:2504.14785; 2025. DOI: 10.48550/arXiv.2504.14785.
  21. Liu Y, Li W, Guan J, Zhou S, Zhang Y. Effective cloud removal for remote sensing images by an improved mean-reverting denoising model with elucidated design space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2025: 17851-17861.
  22. Grohnfeldt C, Schmitt M, Zhu XX. A conditional generative adversarial network to fuse SAR and multispectral optical data for cloud removal from Sentinel-2 images. In: Proceedings of the IEEE International Geoscience and Remote Sensing Symposium. IEEE; 2018: 1726-1729. DOI: 10.1109/IGARSS.2018.8519215.
  23. Chen S, Zhang W, Li Z, Wang Y, Zhang B. Cloud removal with SAR-optical data fusion and graph-based feature aggregation network. Remote Sensing 2022; 14(14): 3374. DOI: 10.3390/rs14143374.
  24. Tu K, Yang C, Qing Y, Qi K, Chen N, Gong J. Cloud removal with optical and SAR imagery via multimodal similarity attention. ISPRS Journal of Photogrammetry and Remote Sensing 2025; 226: 116-126. DOI: 10.1016/j.isprsjprs.2025.05.004.
  25. Zhu ZH, Lu W, Chen SB, Ding C, Tang J, Luo B. Real-world remote sensing image dehazing: Benchmark and baseline. IEEE Transactions on Geoscience and Remote Sensing 2025; 63: 1-16. DOI: 10.1109/TGRS.2025.3584234.
  26. Zhao Z, Yan J, Li C, Wang X, Jiang P, Tang J. DehazeMamba: SAR-guided optical remote sensing image dehazing with adaptive state space model. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 2026; 19: 778-794. DOI: 10.1109/JSTARS.2025.3634360.
  27. Makarau A, Richter R, Müller R, Reinartz P. Haze detection and removal in remotely sensed multispectral imagery. IEEE Transactions on Geoscience and Remote Sensing 2014; 52(9): 5895-5905. DOI: 10.1109/TGRS.2013.2293662.
  28. Huang B, Li Z, Yang C, Sun F, Song Y. Single satellite optical imagery dehazing using SAR image prior based on conditional generative adversarial networks. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision. IEEE; 2020: 1795-1802. DOI: 10.1109/WACV45572.2020.9093471.
  29. Shen H, Zhong T, Jia Y, Wu C. Remote sensing image dehazing using generative adversarial network with texture and color space enhancement. Scientific Reports 2024; 14: 12382. DOI: 10.1038/s41598-024-63259-6.
  30. Zhou H, Wu X, Chen H, Chen X, He X. RSDehamba: Lightweight vision Mamba for remote sensing satellite image dehazing. arXiv preprint arXiv:2405.10030; 2024. DOI: 10.48550/arXiv.2405.10030.
  31. Ma X, Wang Q, Tong X. Nighttime light remote sensing image haze removal based on a deep learning model. Remote Sensing of Environment 2025; 318: 114575. DOI: 10.1016/j.rse.2024.114575.
  32. Zhu Z, Luo Y, Wei H, Li Y, Qi G, Mazur N, Li Y, Li P. Atmospheric light estimation based remote sensing image dehazing. Remote Sensing 2021; 13(13): 2432. DOI: 10.3390/rs13132432.
  33. Jakubik J, Yang F, Blumenstiel B, et al. TerraMind: Large-scale generative multimodality for Earth observation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision; 2025.
  34. Ebel P, Xu Y, Schmitt M, Zhu XX. SEN12MS-CR-TS: A remote sensing data set for multimodal multitemporal cloud removal. IEEE Transactions on Geoscience and Remote Sensing 2022; 60: 5222414. DOI: 10.1109/TGRS.2022.3146246.
  35. Song Y, He Z, Qian H, Du X. Vision transformers for single image dehazing. arXiv preprint arXiv:2204.03883; 2022.
  36. Schmitt M, Hughes LH, Qiu C, Zhu XX. SEN12MS: A curated dataset of georeferenced multi-spectral Sentinel-1/2 imagery for deep learning and data fusion. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences 2019; IV-2/W7: 153-160. DOI: 10.5194/isprs-annals-IV-2-W7-153-2019.
  37. Jiang B, Chen G, Wang J, Ma H, Wang L, Wang Y, Chen X. Deep dehazing network for remote sensing image with non-uniform haze. Remote Sensing 2021; 13(21): 4443. DOI: 10.3390/rs13214443.
  38. Czerkawski M, Atkinson R, Michie C, Tachtatzis C. SatelliteCloudGenerator: Controllable cloud and shadow synthesis for multispectral optical satellite images. Remote Sensing 2023; 15(17): 4138. DOI: 10.3390/rs15174138.
  39. Kong F, Huang B, Bradbury K, Malof J. The Synthinel-1 dataset: A collection of high-resolution synthetic overhead imagery for building segmentation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. IEEE; 2020: 1814-1823. DOI: 10.1109/WACV45572.2020.9093500.
  40. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N. An image is worth 16×16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929; 2021. DOI: 10.48550/arXiv.2010.11929.
  41. Mentzer F, Minnen D, Agustsson E, Tschannen M. Finite scalar quantization: VQ-VAE made simple. arXiv preprint arXiv:2309.15505; 2023. DOI: 10.48550/arXiv.2309.15505.

151, Molodogvardeiskaya str., Samara, 443001, Russia; E-mail: journal@computeroptics.ru; Tel: +7 (846) 242-41-24 (Executive secretary), +7 (846) 332-56-22 (Issuing editor), Fax: +7 (846) 332-56-20