(50-5) 17 * << * >> * Russian * English * Content * All Issues

Semantic gap analysis in image classification by functional purpose
A.N. Zhdanova1,*, A.V. Kupriyanov1,2

1 Samara National Research University, Moskovskoye Shosse 34, Samara, 443086, Russia;
2 Image Processing Systems Institute, NRC “Kurchatov Institute”,
Molodogvardeiskaya Str. 151, Samara, 443001, Russia;
* Corresponding Author

  Full text (PDF)

DOI: 10.18287/COJ2079

Article ID: 2079

Language: Russian

Abstract:
The paper addresses the problem of image classification by functional purpose, which differs from traditional object classification in that the category is determined by the functional role of the image and depends on the composition of scene objects, their spatial arrangement, and contextual information. The study investigates the semantic gap between the object feature space formed by convolutional neural networks pretrained on object-centered datasets and the space of functional categories. A taxonomy of functional image categories is proposed, together with formal classification criteria based on the relative importance of objects and their spatial relationships.

Experimental evaluation was conducted on a dataset of 1,722 real social media profile images annotated by experts (Cohen’s kappa coefficient κ = 0.89). Using a ResNet50 architecture with a two-stage transfer learning strategy, classification performance with an accuracy of 0.85 and an F1 score of 0.83 was achieved. For comparison, an object-based approach using YOLOv8 object detection combined with expert-defined rules was also evaluated, yielding an Accuracy of 0.72. Analysis of classification errors showed that 67% of misclassifications were associated with borderline cases in which the object composition alone was insufficient to determine the functional category unambiguously.

Feature space analysis revealed reduced separability of categories distinguished by relationships between objects compared with categories distinguished by object composition. The obtained results indicate the limitations of directly transferring object-level features to functional image classification tasks and highlight the importance of incorporating object relationships into image analysis models.

Keywords:
image classification; functional categories, semantic gap, transfer learning, convolutional neural networks, feature space.

Acknowledgements:
The research was carried out within the state assignment theme FSSS-2026-0003.

Citation:
Zhdanova AN, Kupriyanov AV. Semantic gap analysis in image classification by functional purpose. Computer Optics 2026; 50(5): 2079. doi: 10.18287/COJ2079.

References:

  1. Smeulders AWM, Worring M, Santini S, Gupta A, Jain R. Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence 2000; 22(12): 1349-1380. DOI: 10.1109/34.895972.
  2. Datta R, Joshi D, Li J, Wang JZ. Image retrieval: Ideas, influences, and trends of the new age. ACM Computing Surveys 2008; 40(2): 5. DOI: 10.1145/1348246.1348248.
  3. Zhou B, Lapedriza A, Khosla A, Oliva A, Torralba A. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 2018; 40(6): 1452-1464. DOI: 10.1109/TPAMI.2017.2723009.
  4. Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L. ImageNet: A large-scale hierarchical image database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE; 2009: 248-255. DOI: 10.1109/CVPR.2009.5206848.
  5. Krizhevsky A. Learning multiple layers of features from tiny images. Technical Report. Toronto, Canada: University of Toronto; 2009.
  6. Lin TY, Maire M, Belongie S, et al. Microsoft COCO: Common objects in context. In: Computer Vision – ECCV 2014. Lecture Notes in Computer Science. Vol 8693. Springer; 2014: 740-755. DOI: 10.1007/978-3-319-10602-1_48.
  7. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE; 2016: 770-778. DOI: 10.1109/CVPR.2016.90.
  8. Andriyanov NA, Dementiev VE, Tashlinsky AG. Object detection in images: From Bayes and Neyman-Pearson criteria to neural network-based EfficientDet detectors [In Russian]. Computer Optics 2022; 46(1): 139-159. DOI: 10.18287/2412-6179-CO-922.
  9. Sokolova AD, Savchenko AV, Nikolenko SI. Rare data search in the problem of face recognition in images [In Russian]. Computer Optics 2022; 46(5): 801-807. DOI: 10.18287/2412-6179-CO-1061.
  10. Egorov AD, Reznik MS. Interpretation of animal actions from their image in near real time [In Russian]. Computer Optics 2023; 47(2): 278-286. DOI: 10.18287/2412-6179-CO-1138.
  11. Yosinski J, Clune J, Bengio Y, Lipson H. How transferable are features in deep neural networks? In: Advances in Neural Information Processing Systems. Vol 27. MIT Press; 2014: 3320-3328.
  12. Kornblith S, Shlens J, Le QV. Do better ImageNet models transfer better? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE; 2019: 2661-2671. DOI: 10.1109/CVPR.2019.00277.
  13. Johnson J, Krishna R, Stark M, Li LJ, Shamma DA, Bernstein MS, Fei-Fei L. Image retrieval using scene graphs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE; 2015: 3668-3678. DOI: 10.1109/CVPR.2015.7298990.
  14. Lu C, Krishna R, Bernstein M, Fei-Fei L. Visual relationship detection with language priors. In: Computer Vision – ECCV 2016. Lecture Notes in Computer Science. Vol 9905. Springer; 2016: 852-869. DOI: 10.1007/978-3-319-46448-0_51.
  15. Krishna R, Zhu Y, Groth O, et al. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 2017; 123(1): 32-73. DOI: 10.1007/s11263-016-0981-7.
  16. Jia M, Wu Z, Reiter A, Cardie C, Belongie S, Lim SN. Intentonomy: A dataset and study towards human intent understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE; 2021: 12986-12996.
  17. Liu L, Preotiuc-Pietro D, Samani ZR, Moghaddam ME, Ungar L. Analyzing personality through social media profile picture choice. In: Proceedings of the Tenth International AAAI Conference on Web and Social Media. AAAI Press; 2016: 211-220.
  18. Lang P, inventor; Imaging Therapeutics Inc, assignee. Methods and devices for quantitative analysis of X-ray images. U.S. Patent 7,050,534. August 20, 2002.

151, Molodogvardeiskaya str., Samara, 443001, Russia; E-mail: journal@computeroptics.ru; Tel: +7 (846) 242-41-24 (Executive secretary), +7 (846) 332-56-22 (Issuing editor), Fax: +7 (846) 332-56-20