(48-2) 13 * << * >> * Russian * English * Content * All Issues

Non-convex optimization with using positive-negative moment estimation and its application for skin cancer recognition with a neural network
P.A. Lyakhov 1,2, U.A. Lyakhova 1,2, R.I. Abdulkadirov 2

1 North-Caucasus Federal University,
355009, Russia, Stavropol, Pushkin str. 1;
2 North-Caucasus Center for Mathematical Research,
355009, Russia, Stavropol, Pushkin str. 1

 PDF, 961 kB

DOI: 10.18287/2412-6179-CO-1308

Pages: 260-271.

Full text of article: Russian language.

Abstract:
The main problem of using standard optimization methods is the need to change all parameters in same-size steps, regardless of the behavior of the gradient. A more efficient way to optimize a neural network is to set adaptive step sizes for each parameter. Standard methods are based on the square roots of exponential estimates of the moments of the squares of past gradients and do not use the local variation in gradients. The paper presents methods of adaptive non-convex and belief-based optimization with a positive-negative estimate of the moments with the corresponding theoretical guarantees of convergence. These approaches allow the loss function to more accurately converge in the neighborhood of the global minimum in a smaller number of iterations. The utilization of transformed positive-negative moment estimates and an additional parameter that controls the step size allows one to avoid local extremes for achieving higher performance, compared to similar methods. The introduction of the developed algorithms into the learning process of various architectures of multimodal neural network systems for analyzing heterogeneous data has made it possible to increase the accuracy of recognizing pigmented skin lesions by 2.33 – 5.69 percentage points, compared to the original optimization methods. Multimodal neural network systems for analyzing heterogeneous dermatological data, using the proposed optimization algorithms, can be applied as a tool for auxiliary medical diagnostics, which will reduce the consumption of financial and labor resources involved in the medical industry, as well as increase the chance of early detection of pigmentary oncopathologies.

Keywords:
optimization, natural gradient descent, artificial intelligence, multimodal neural networks, heterogeneous data, skin cancer, melanoma.

Citation:
Lyakhov PA, Lyakhova UA, Abdulkadirov RI. Non-convex optimization with using positive-negative moment estimation and its application for skin cancer recognition with a neural network. Computer Optics 2024; 48(2): 260-271. DOI: 10.18287/2412-6179-CO-1308.

Acknowledgements:
The authors thank the North-Caucasus Federal University for the award of funding in the contest of competitive projects of scientific groups and individual scientists of North-Caucasus Federal University. The research in section 2 was supported by the North-Caucasus Center for Mathematical Research under agreement with the Ministry of Science and Higher Education of the Russian Federation (Agreement No. 075-02-2023-938). The research in section 3 was supported by the Russian Science Foundation (Project No. 23-71-10013).

References:
  1. Kaul V, Enslin S, Gross SA. History of artificial intelligence in medicine. Gastrointest Endosc Mosby 2020; 92(4): 807-812. DOI: 10.1016/J.GIE.2020.06.040.
  2. Hamet P, Tremblay J. Artificial intelligence in medicine. Metabolism 2017; 69: 36-40. DOI: 10.1016/J.METABOL.2017.01.011.
  3. Brinker TJ, Hekler A, Enk AH, et al. Deep learning outperformed 136 of 157 dermatologists in a head-to-head dermoscopic melanoma image classification task. Eur J Cancer 2019; 113: 47-54. DOI: 10.1016/J.EJCA.2019.04.001.
  4. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter, SM, Blau HM, Thrun S. Dermatologist-level classification of skin cancer with deep neural networks. Nature 2017; 542: 115-118. DOI: 10.1038/nature21056.
  5. Haggenmüller S, Maron RC, Hekler A, et al. Skin cancer classification via convolutional neural networks: Systematic review of studies involving human experts. Eur J Cancer 2021; 156: 202-216. DOI: 10.1016/J.EJCA.2021.06.049.
  6. Wiens J, Saria S, Sendak M, et al. Author correction: Do no harm: A roadmap for responsible machine learning for health care. Nature Medicine 2019; 25(9): 1337-1340. DOI: 10.1038/S41591-019-0548-6.
  7. Hwang J, Bose N, Fan S. AUV adaptive sampling methods: A review. Appl Sci 2019; 9: 3145-2019. DOI: 10.3390/APP9153145.
  8. Hospedales T, Antoniou A, Micaelli P, Storkey A. Meta-learning in neural networks: A survey. IEEE Trans Pattern Anal Mach Intell 2022; 44: 5149-5169. DOI: 10.1109/TPAMI.2021.3079209.
  9. Gao S, Pei Z, Zhang Y, Li T. Bearing fault diagnosis based on adaptive convolutional neural network with Nesterov momentum. IEEE Sens J 2021; 21: 9268-9276.
  10. Kingma DP, Ba JL. Adam: A method for stochastic optimization. 3rd Int Conf on Learning Representations (ICLR 2015) 2015: 1-13. DOI: 10.48550/arxiv.1412.6980.
  11. Wang S, Yang Y, Sun J, Xu Z. Variational HyperAdam: A meta-learning approach to network training. IEEE Trans Pattern Anal Mach Intell 2022; 44: 4469-4484. DOI: 10.1109/TPAMI.2021.3061581.
  12. Abdulkadirov RI, Lyakhov PA, Nagornov NN. Accelerating extreme search of multidimensional functions based on natural gradient descent with Dirichlet distributions. Mathematics 2022; 10: 3556-2022. DOI: 10.3390/MATH10193556.
  13. Dubey SR, Chakraborty S, Roy SK, Mukherjee S, Singh SK, Chaudhuri BB. DiffGrad: An optimization method for convolutional neural networks. IEEE Trans Neural Netw Learn Syst 2020; 31: 4500-4511. DOI: 10.1109/TNNLS.2019.2955777.
  14. Zaheer M, Reddi SJ, Sachan D, Kale S, Research G, Kumar S. Adaptive methods for nonconvex optimization. Proc 32nd Int Conf on Neural Information Processing Systems (NIPS'18) 2018: 9815-9825.
  15. Zhuang J, Tang T, Ding Y, Tatikonda SC, Dvornek N, Papademetris X, Duncan JS. AdaBelief optimizer: Adapting stepsizes by the belief in observed gradients. Adv Neural Inf Process Syst 2020; 33: 18795-18806.
  16. Xie Z, Yuan L, Zhu Z, Sugiyama M. Positive-negative momentum: Manipulating stochastic gradient noise to improve generalization. Thirty-eighth Int Conf on Machine Learning (ICML 2021) 2021: 11448-11458.
  17. Kurtansky NR, Dusza SW, Halpern AC, Hartman RI, Geller AC, Marghoob AA, Rotemberg VM, Marchetti MA. An epidemiologic analysis of melanoma overdiagnosis in the United States, 1975-2017. J Invest Dermatol 2022; 142: 1804-1811. DOI: 10.1016/J.JID.2021.12.003.
  18. Turkay C, Lundervold A, Lundervold AJ, Hauser H. Hypothesis generation by interactive visual exploration of heterogeneous medical data. In Book: Holzinger A, Pasi G, eds. Human-computer interaction and knowledge discovery in complex, unstructured, big data. Third international workshop (HCI-KDD 2013). Berlin, Heidelberg: Springer-Verlag; 2013: 1-12. DOI: 10.1007/978-3-642-39146-0_1.
  19. Wang S, Yin Y, Wang D, Wang Y, Jin Y. Interpretability-based multimodal convolutional neural networks for skin lesion diagnosis. IEEE Trans Cybern 2022; 52(12): 12623-12637.
  20. Goh G, Carter S, Petrov M, Schubert L, Radford A, Olah, C. Multimodal neurons in artificial neural networks. Distill 2021; 6: 30. DOI: 10.23915/DISTILL.00030.
  21. Liu K, Li Y, Xu N, Natarajan P. Learn to combine modalities in multimodal deep learning. arXiv Preprint. 2023. Source: <https://arxiv.org/abs/1805.11730>.
  22. Lyakhov PA, Lyakhova UA, Nagornov NN. System for the recognizing of pigmented skin lesions with fusion and analysis of heterogeneous data based on a multimodal neural network. Cancers 2022; 14: 1819-2022. DOI: 10.3390/CANCERS14071819.
  23. Banerjee. S. Estimatation of body weight at different ages using linear and some non linear regression equations in a duck breed reared in hot and humid climate of Eastern India. Am-Eurasian J Sci Res 2011; 6(4): 201-204.
  24. Tukey JW. The practical relationship between the common transformations of percentages or fractions and of amounts. In Book: Mallows CL, ed. The collected works of John W. Tukey. Volume VI: More mathematical. Pacific Grove, CA: Wadsworth & Brooks-Cole; 1990: 211-219.

© 2009, IPSI RAS
151, Molodogvardeiskaya str., Samara, 443001, Russia; E-mail: journal@computeroptics.ru ; Tel: +7 (846) 242-41-24 (Executive secretary), +7 (846) 332-56-22 (Issuing editor), Fax: +7 (846) 332-56-20