On the quantitative performance evaluation of image analysis algorithms
P.P. Koltsov, A.S. Osipov, A.S. Kutsaev, A.A. Kravchenko, N.V. Kotovich, A.V. Zakharov

 

Scientific-Research Institute for System Analysis, Russian Academy of Sciences

Full text of article: Russian language.

 PDF

Abstract:
The paper contains a brief review of main approaches to the comparative performance evaluation of image analysis algorithms. Some empirical methods used for the comparative evaluation of edge detectors and image segmentation algorithms are considered and quantitative criteria employed in these methods are studied. Problems associated with the use of these criteria are described. Finally, using the edge detector evaluation as an example, we propose an empirical method, called EDEM, which is implemented using our proprietary software system PICASSO.

Keywords:
comparative study, image analysis, edge detectors, image segmentation, performance measures, ground truth image, fuzzy sets.

Citation:
Koltsov PP, Osipov AS, Kutsaev AS, Kravchenko AA, Kotovich NV, Zakharov AV. On the quantitative performance evaluation of image analysis algorithms. Computer Optics 2015; 39(4): 542-56. DOI: 10.18287/0134-2452-2015-39-4-542-556.

References:

  1. Kirsch RA, Kahn L, Ray C, Urban GH. Experiments in processing pictorial information with a digital computer.Proceedings of the Easterm Joint Computer conference 1957; 221-9.
  2. Zhang YJ. Evaluation and comparison of different segmentation algorithms. Pattern Recognition Letters 1997; 18(10): 963-74.
  3. Heath MD, Sarkar S, Sanocki T, Bowyer K. Robust visual method for assessing the relative performance of edge detection algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence 1997; 19(12): 1338-59.
  4. Wirth MA. Performance evaluation of image processing algorithms in CADe. Technology in Cancer Research and Treatment 2005; 4(2): 159-72.
  5. Zhang H, Fritts JE, Goldman SA. Image segmentation evaluation: A survey of unsupervised methods. Computer Vision and Image Understanding 2008; 110(2): 260-80.
  6. Shin MC, Goldgof D, Bowyer K. Comparison of edge detector performance through use in an object recognition task.Computer Vision and Image Understanding2001; 84(1): 160-78.
  7. Cardoso JS, Corte-Real L. Toward a Generic Evaluation of Image Segmentation. IEEE Transactions on Image Processing 2005; 14(11): 1773-82.
  8. Thomas GA, Grau O. 3d image sequence acquisition for tv and film production. Proceedings of 1st International Symposium on 3D Data Processing, Visualisation and Transmission 2002; 320-6.
  9. Zhang YJ. A survey on evaluation methods for image segmentation. Pattern Recognition1996; 29(8): 1335-46.
  10.  Zhang YJ. Image segmentation evaluation in this century. Encyclopedia of Information Science and Technology. 2nd edition. IGI Global; 2009: 1812-17.
  11. Haralick RM, Shapiro LG. Image segmentation techniques Computer Vision, Graphics, and Image Processing 1985; 29(1): 100-32.
  12. Canny J. A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 1986; 8(6): 679-98.
  13. Gribkov IV, Koltsov PP, Kotovich NV, Kravchenko AA, Kutsaev AS, Osipov AS, Zakharov AV. On some issues of the quantitative performance evaluation of edge detectors [In Russian]. Programmnye Produkty I Sistemy 2011; 2: 13-9.
  14. Baddeley AJ. Errors in binary images and Lp version of the Hausdorff Metric.Nieuw Archief voor Wiskunde1992; 10: 157-83.
  15. Yasnoff WA, Mui JK, Bacus JW. Error measures for scene segmentation.PatternRecognition 1977; 9(4): 217-31.
  16. Van Droogenbroeck M, Barnich O. Design of Statistical Measures for the Assessment of Image Segmentation Schemes. Proceedings of 11th International Conference on Computer Analysis of Images and Patterns (CAIP2005), Lecture Notes in Computer Science 2005; 3691: 280-7.
  17. Strasters KS, Gerbrands JJ. Three-dimensional image segmentation using a split, merge and group approach. Pattern Recognition Letters 1991; 12(5): 307-25.
  18. Zhang YJ, Gerbrands JJ. Objective and quantitative segmentation evaluation and comparison. Signal Processing 1994; 39(1-2): 43-54.
  19. Koltsov PP, Kotovich NV, Kravchenko AA, Kutsaev AS, Osipov AS, Zakharov AV. Criteria for evaluating image segmentation [In Russian]. Trudy NIISI RAN 2012; 2(2): 87-99.
  20. Gribkov IV, Koltsov PP, Kotovich NV, Kravchenko AA, Kutsaev AS, Osipov AS, Zakharov AV. Testing of image segmentation methods in PICASSO system [In Russian]. Moscow: NIISI RAN; 2007.
  21. Gribkov IV, Koltsov PP, Kotovich NV, Kravchenko AA, Kutsaev AS, Nikolaev VK, Zakharov AV. PICASSO – A System for Evaluating Edge Detection Algorithms. Pattern Recognition and Image Analysis 2003; 13(4): 617-22.
  22. Gribkov IV, Koltsov PP, Kotovich NV, Kravchenko AA, Kutsaev AS, Osipov AS, Zakharov AV. Edge Detection under Affine Transformations: Comparative Study by PICASSO 2 System. WSEAS Transactions on Signal Processing 2006; 2(9): 1215-21.
  23. Osipov A. A fuzzy approach to performance evaluation of edge detectors. Lecture Notes in Signal Science, Internet and Education. WSEAS Press; 2007: 94-9.
  24. Koltsov PP, Kotovich NV, Kravchenko AA, Kutsaev AS, Kuznetsov AB, Osipov AS, Sukhenko EP, Zakharov AV. On one approach to blood cell image segmentation.The 11th International Conference "Pattern Recognition and Image Analysis" (PRIA-11-2013), Conference Proceedings 2013; 2: 615-18.
  25. Belyakov VK, Koltsov PP, Kotovich NV, Kravchenko AA, Kutsaev AS, Kuznetsov AB, Osipov AS, Sukhenko EP, Zakharov AV. On one method of blood cell classification and its software implementation [In Russian] Programmnye Produkty I Sistemy 2014; 2: 46-56.
  26. Koltsov PP, Kotovich NV, Kravchenko AA, Kutsaev AS, Osipov AS, Zakharov AV. Direct assessment of software quality. Criteria and materials for testing [In Russian]. Programmnye Produkty, Sistemy I Algoritmy 2014; 3: P. 1-8.
© 2009, IPSI RAS
151, Molodogvardeiskaya str., Samara, 443001, Russia; E-mail:journal@computeroptics.ru; Tel: +7 (846) 242-41-24 (Executive secretary), +7 (846) 332-56-22 (Issuing editor), Fax: +7 (846) 332-56-20