(48-6) 11 * << * >> * Russian * English * Content * All Issues

Semantic segmentation of hyperspectral images using convolutional neural networks and the attention mechanism
D.N. Gribanov 1, A.V. Mukhin 1, I.A. Kilbas 1, R.A. Paringer 1

1 Samara National Research University,
443086, Samara, Russia, Moskovskoye Shosse 34

 PDF, 4250 kB

DOI: 10.18287/2412-6179-CO-1371

Pages: 894-902.

Full text of article: Russian language.

Abstract:
This paper investigates an effect of the attention mechanism on the accuracy of hyperspectral image segmentation by convolutional neural networks in agriculture. The study compares two modifications of neural network architectures: with and without the attention mechanism. The attention mechanism is implemented as two modules: position-based (PAM) and channel-based (CAM). The positional module (PAM) considers the global context using information about the spatial domain of the whole image. The channel module (CAM) in turn takes into account the information of all spectral components. L2Net and U-Net architectures are used for a comparative study. Modified versions with the addition of the attention mechanism are developed: L2AT-Net and ULAT-Net. The experimental results show that adding the attention mechanism to the U-Net and L2Net architectures increases the mean value of the F1 metric from 0.80 to 0.83 and from 0.74 to 0.78, respectively. The results show that the application of the attention mechanism can improve the quality of semantic segmentation of hyperspectral images.

Keywords:
semantic segmentation, attention mechanism, hyperspectral data, neural network, machine learning.

Citation:
Gribanov DN, Mukhin AV, Kilbas IA, Paringer RA. Semantic segmentation of hyperspectral images using convolutional neural networks and the attention mechanism. Computer Optics 2024; 48(6): 894-902. DOI: 10.18287/2412-6179-CO-1371.

Acknowledgements:
The work was partly funded by the Russian Federation Ministry of Science and Higher Education under the state project FSSS-2021-0016, “Photonics for a smart home and smart city” (theoretical part and technology development) and under the state research project FSSS-2024-0014) (software implementation).

References:
  1. Fabelo H, Ortega S, Ravi D, et al. Spatio-spectral classification of hyperspectral images for brain cancer detection during surgical operations. PLOS ONE 2018; 13(3): e0193721. DOI: 10.1371/journal.pone.0193721.
  2. Paringer RA, Mukhin AV, Kupriyanov AV. Formation of an informative index for recognizing specified objects in hyperspectral data. Computer Optics 2021; 45(6): 873-878. DOI: 10.18287/2412-6179-CO-930.
  3. Amini S, Homayouni S, Safari A, Darvishsefat AA. Object-based classification of hyperspectral data using Random Forest algorithm. Geo-Spat Inf Sci 2018; 21: 127-138. DOI: 10.1080/10095020.2017.1399674.
  4. Paoletti ME, Haut JM, Plaza J, Plaza A. A new deep convolutional neural network for fast hyperspectral image classification. ISPRS J Photogramm Remote Sens 2018; 145: 120-147. DOI: 10.1016/j.isprsjprs.2017.11.021.
  5. Grewal R, Singh Kasana S, Kasana G. Machine learning and deep learning techniques for spectral spatial classification of hyperspectral images: A comprehensive survey. Electronics 2023; 12: 488. DOI: 10.3390/electronics12030488.
  6. Leiva-Valenzuela GA, Lu R, Aguilera JM. Prediction of firmness and soluble solids content of blueberries using hyperspectral reflectance imaging. J Food Eng 2013; 115: 91-98. DOI: 10.1016/j.jfoodeng.2012.10.001.
  7. Li Z, Wang H, Li Q. tongue tumor detection in medical hyperspectral images. Sensors 2011; 12: 162-174. DOI: 10.3390/s120100162.
  8. Liu LY, Wang JH, Huang WJ, Zhao CJ, Zhang B, Tong QX. Improving winter wheat yield prediction by novel spectral index. Trans CSAE 2004; 20: 172-175.
  9. Kutser T, Paavel B, Verpoorter C, Kauer T, Vahtmäe E. Remote sensing of water quality in optically complex lakes. Int Arch Photogramm Remote Sens Spatial Inf Sci 2012; XXXIX-B8: 165-169. DOI: 10.5194/isprsarchives-xxxix-b8-165-2012.
  10. Ishihara M, Inoue Y, Ono K, Shimizu M, Matsuura S. The impact of sunlight conditions on the consistency of vegetation indices in croplands–effective usage of vegetation indices from continuous ground-based spectral measurements. Remote Sens 2015; 7: 14079-14098. DOI: 10.3390/rs71014079.
  11. Graña M. Hyperspectral remote sensing scenes – Grupo de Inteligencia Computacional (GIC). Wwwehueus 2011. Source:  <http://www.ehu.eus/ccwintco/index.php?title=Hyperspectral_Remote_Sensing_Scenes>.
  12. Fei-Fei L, Deng J, Li K. ImageNet: Constructing a large-scale image database. J Vision 2010; 9: 1037. DOI: 10.1167/9.8.1037.
  13. Li J, Bioucas-Dias JM, Plaza A. Semisupervised hyperspectral image segmentation using multinomial logistic regression with active learning. IEEE Trans Geosci Remote Sens 2010; 48(11): 4085-4098. DOI: 10.1109/tgrs.2010.2060550.
  14. Prasad S, Bruce LM. Limitations of principal components analysis for hyperspectral target recognition. IEEE Geosci Remote Sens Lett 2008; 5: 625-629. DOI: 10.1109/lgrs.2008.2001282.
  15. Villa A, Benediktsson JA, Chanussot J, Jutten C. Hyperspectral image classification with independent component discriminant analysis. IEEE Trans Geosci Remote Sens 2011; 49: 4865-4876. DOI: 10.1109/tgrs.2011.2153861.
  16. Bandos TV, Bruzzone L, Camps-Valls G. Classification of hyperspectral images with regularized linear discriminant analysis. IEEE Tran Geosci Remote Sens 2009; 47: 862-873. DOI: 10.1109/tgrs.2008.2005729.
  17. Ran L, Zhang Y, Wei W, Zhang Q. A hyperspectral image classification framework with spatial pixel pair features. Sensors 2017; 17: 2421. DOI: 10.3390/s17102421.
  18. Trajanovski S, Shan C, Weijtmans PJC, de Koning SGB, Ruers TJM. Tongue tumor detection in hyperspectral images using deep learning semantic segmentation. IEEE Trans Biomed Eng 2021; 68(4): 1330-1340. DOI: 10.1109/tbme.2020.3026683.
  19. Mukhin A, Danil G, Paringer R. semantic segmentation of hyperspectral imaging using convolutional neural networks. Optical Memory and Neural Networks 2022; 31: 38-47. DOI: 10.3103/s1060992x22050071.
  20. Kirillov A, Mintun E, Ravi N, et al. Segment anything. arXiv Preprint. 2023. Source: <https://arxiv.org/abs/2304.02643>. DOI: 10.48550/arxiv.2304.02643.
  21. Chen B, Liu YQ, Zhang Z, Lu G, Kong AWK. TransAttUnet: Multi-level attention-guided U-Net with transformer for medical image segmentation. IEEE Trans Emerg Top Comput Intell 2023; 8(1): 55-68. DOI: 10.1109/tetci.2023.3309626.
  22. Vanian V, Zamanakos G, Pratikakis I. Improving performance of deep learning models for 3D point cloud semantic segmentation via attention mechanisms. Computers & Graphics 2022; 106: 277-287. DOI: 10.1016/j.cag.2022.06.010.
  23. Han Z, Hong D, Gao L, Yao J, Zhang B, Chanussot J. Multimodal hyperspectral unmixing: Insights from attention networks. IEEE Trans Geosci Remote Sens 2022; 60: 5524913. DOI: 10.1109/tgrs.2022.3155794.
  24. Shi C, Liao D, Zhang T, Wang L. Hyperspectral image classification based on 3D coordination attention mechanism network. Remote Sens 2022; 14: 608. DOI: 10.3390/rs14030608.
  25. Mohla S, Pande S, Banerjee B, Chaudhuri S. FusAtNet: Dual attention based spectrospatial multimodal fusion network for hyperspectral and LiDAR classification. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 2020: 416-425. DOI: 10.1109/cvprw50498.2020.00054.
  26. Zheng X, Chen W, Lu X. Spectral super-resolution of multispectral images using spatial–spectral residual attention network. IEEE Trans Geosci Remote Sens 2022; 60: 5404114. DOI: 10.1109/tgrs.2021.3104476.
  27. Zhang H, Yao J, Li N, Gao L, Huang M. Multimodal attention-aware convolutional neural networks for classification of hyperspectral and LiDAR data. IEEE J Sel Top Appl Earth Obs Remote Sens 2023; 16: 3635-3644. DOI: 10.1109/jstars.2022.3187730.
  28. Fu J, Liu J, Tian H, et al. Dual attention network for scene segmentation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2019: 3141-3149. DOI: 10.1109/cvpr.2019.00326.
  29. Sinha A, Dolz J. Multi-scale self-guided attention for medical image segmentation. IEEE J Biomed Health Inform 2021; 25: 121-130. DOI: 10.1109/jbhi.2020.2986926.
  30. Huang G, Liu Z, van der Maaten L, Weinberger KQ. Densely connected convolutional networks. arXiv Preprint. 2016. Source: <https://arxiv.org/abs/1608.06993>. DOI: 10.48550/arxiv.1608.06993.
  31. Ronneberger O, Fischer P, Brox T. U-Net: Convolutional networks for biomedical image segmentation. arXiv Preprint. 2015. Source: <https://arxiv.org/abs/1505.04597>. DOI: 10.48550/arxiv.1505.04597.
  32. HSI-Dataset-API: API for accessing HSI datasets. 2021. Source: <https://pypi.org/project/HSI-Dataset-API>.
  33. Wold S, Esbensen K, Geladi P. Principal component analysis. Chemom Intell Lab Syst 1987; 2: 37-52. DOI: 10.1016/0169-7439(87)80084-9.
  34. Paszke A, Gross S, Massa F, et al. PyTorch: An imperative style, high-performance deep learning library. Proc 33rd Int Conf on Neural Information Processing Systems 2019: 8026-8037.
  35. Lin T-Y, Goyal P, Girshick R, He K, Dollar P. Focal loss for dense object detection. IEEE Trans Pattern Anal Mach Intell 2018; 42(2): 318-327. DOI: 10.1109/tpami.2018.2858826.

© 2009, IPSI RAS
151, Molodogvardeiskaya str., Samara, 443001, Russia; E-mail: journal@computeroptics.ru ; Tel: +7 (846) 242-41-24 (Executive secretary), +7 (846) 332-56-22 (Issuing editor), Fax: +7 (846) 332-56-20