(47-3) 18 * << * >> * Russian * English * Content * All Issues

Development and research of a neural network alternate incremental learning algorithm
A.A. Orlov 1, E.S. Abramova 1

1 Murom Institute (branch) of Vladimir State University named after Alexander and Nickolay Stoletovs,
602264, Murom, Russia, Orlovskaya Street 23

 PDF, 1392 kB

DOI: 10.18287/2412-6179-CO-1203

Pages: 491-498.

Full text of article: Russian language.

Abstract:
In this paper, the relevance of developing methods and algorithms for neural network incremental learning is shown. Families of incremental learning techniques are presented. A possibility of using the extreme learning machine for incremental learning is assessed. Experiments show that the extreme learning machine is suitable for incremental learning, but as the number of training examples increases, the neural network becomes unsuitable for further learning. To solve this problem, we propose a neural network incremental learning algorithm that alternately uses the extreme learning machine to correct the only output layer network weights (operation mode) and the backpropagation method (deep learning) to correct all network weights (sleep mode). During the operation mode, the neural network is assumed to produce results or learn from new tasks, optimizing its weights in the sleep mode. The proposed algorithm features the ability for real-time adaption to changing external conditions in the operation mode. The effectiveness of the proposed algorithm is shown by an example of solving the approximation problem. Approximation results after each step of the algorithm are presented. A comparison of the mean square error values when using the extreme learning machine for incremental learning and the developed algorithm of neural network alternate incremental learning is made.

Keywords:
incremental learning methods, artificial neural networks, extreme learning machine, functioning and sleeping states.

Citation:
Orlov AA, Abramova ES. Development and research of a neural network alternate incremental learning algorithm. Computer Optics 2023; 47(3): 491-498. DOI: 10.18287/2412-6179-CO-1203.

References:
  1. Ganguly S, Chatterjee A, Bhoumik D, Majumdar R. An empirical study of incremental learning in neural network with noisy training set. In Book: Das NR, Sarkar S, eds. Computers and devices for communication. Springer Nature Singapore Pte Ltd; 2021: 72-77. DOI: 10.1007/978-981-15-8366-7_11.
  2. Wang JH, Wang HY, Chen YL, Liu CM. A constructive algorithm for unsupervised learning with incremental neural network. J Appl Res Technol 2015; 13: 188-196. DOI: 10.1016/j.jart.2015.06.017.
  3. Abramova ES, Orlov AA, Makarov KV. Possibilities of using neural network incremental learning. Bull South Ural State Univ Ser Comput Technol Autom Control Radioelectron 2021; 21: 19-27. DOI: 10.14529/ctcr210402.
  4. Sarwar SS, Ankit A, Roy K. Incremental Learning in Deep Convolutional Neural Networks Using Partial Network Sharing. IEEE Access 2020; 8: 4615-4628. DOI: 10.1109/ACCESS.2019.2963056.
  5. Mallya A, Davis D, Lazebnik S. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Book: Ferrari V, Hebert M, Sminchisescu C, Weiss Y, eds. Computer Vision – ECCV 2018. Springer Verlag; 2018: 72-88. DOI: 10.1007/978-3-030-01225-0_5.
  6. Mallya A, Lazebnik S. PackNet: Adding multiple tasks to a single network by iterative pruning. 2018 IEEE/CVF Conf Computer Vision and Pattern Recognition 2018: 7765-7773. DOI: 10.1109/CVPR.2018.00810.
  7. Dai X, Yin H, Jha NK. Incremental learning using a grow-and-prune paradigm with efficient neural networks. IEEE Trans Emerg Top Comput 2020. DOI: 10.1109/TETC.2020.3037052.
  8. Zenke F, Poole B, Ganguli S. Continual learning through synaptic intelligence. Proc 34th Int Conf on Machine Learning 2017: 3987-3995.
  9. Xiang Y, Miao Y, Chen J, Xuan Q. Efficient incremental learning using dynamic correction vector. IEEE Access 2020; 8: 23090-23099. DOI: 10.1109/ACCESS.2019.2963461.
  10. Li P, Chen Z, Yang LT, Gao J, Zhang Q, Deen MJ. An incremental deep convolutional computation model for feature learning on industrial big data. IEEE Trans Industr Inform 2019; 15: 1341-1349. DOI: 10.1109/TII.2018.2871084.
  11. Paik I, Oh S, Kwak T, Kim I. Overcoming catastrophic forgetting by neuron-level plasticity control. Proc AAAI Conf on Artificial Intelligence 2020; 34(04): 5339-5346. DOI: 10.1609/aaai.v34i04.5981.
  12. Masse NY, Grant GD, Freedman DJ. Alleviating catastrophic forgetting using contextdependent gating and synaptic stabilization. PNAS 2018; 115(44): E104657-E104675. DOI: 10.1073/pnas.1803839115.
  13. Gaurav A, Abdelzad V, Vernekar S, Czarnecki K, Lee J, Sedwards S. Simple continual learning strategies for safer classifers. CEUR Workshop Proc 2020; 2560: 96-104.
  14. Rebuffi SA, Kolesnikov A, Sperl G, Lampert CH. iCaRL: Incremental classifier and representation learning. IEEE Conf on Computer Vision and Pattern Recognition (CVPR) 2017: 5533-5542. DOI: 10.1109/CVPR.2017.587.
  15. Wu Y, Chen Y, Wang L, et al. Large scale incremental learning. IEEE/CVF Conf on Computer Vision and Pattern Recognition (CVPR) 2019: 374-382. DOI: 10.1109/CVPR.2019.00046.
  16. Stojanov S, Mishra S, Thai NA, et al. Incremental object learning from contiguous views. IEEE/CVF Conf on Computer Vision and Pattern Recognition (CVPR) 2019: 8769-8778. DOI: 10.1109/CVPR.2019.00898.
  17. Xiong W, Wang Y, Cheng L. Fisher discriminant analysis random forest for online class incremental learning. IEEE Intl Conf on Parallel & Distributed Processing with Applications, Ubiquitous Computing & Communications, Big Data & Cloud Computing, Social Computing & Networking, Sustainable Computing & Communications (ISPA/IUCC/BDCloud/SocialCom/SustainCom) 2019: 597-604. DOI: 10.1109/BDCloud.2018.00093.
  18. Tasar O, Tarabalka Y, Alliez P. Incremental learning for semantic segmentation of large-scale remote sensing data. IEEE J Sel Top Appl Earth Obs Remote Sens 2019; 12: 3524-3537. DOI: 10.1109/JSTARS.2019.2925416.
  19. Nakano S, Hattori M. Characteristics of contrastive Hebbian learning with pseudorehearsal for multilayer neural networks on reduction of catastrophic forgetting. Int J Comput Intell Stud 2018; 7: 289. DOI: 10.1504/ijcistudies.2018.10017450.
  20. Huang G-B, Zhu Q-Y, Siew C-K. Extreme learning machine: Theory and applications. Neurocomputing 2006; 70: 489-501. DOI: 10.1016/j.neucom.2005.12.126.
  21. Poggio T, Girosi F. Networks for approximation and learning. Proc IEEE 1990; 78: 1481-1497. DOI: 10.1109/5.58326.
  22. Schmidt WF, Kraaijveld MA, Duin RPW. Feed forward neural networks with random weights. 11th IAPR Int Conf on Pattern Recognition. Vol II. Conference B: Pattern Recognition Methodology and Systems 1992: 1-4. DOI: 10.1109/ICPR.1992.201708.
  23. Pao YH, Park GH, Sobajic DJ. Learning and generalization characteristics of the random vector functional-link net. Neurocomputing 1994; 6: 163-180. DOI: 10.1016/0925-2312(94)90053-1.
  24. Huang G-B, Zhou H, Ding X, Zhang R. Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man, Cybern Part B Cybern 2012; 42: 513-529. DOI: 10.1109/TSMCB.2011.2168604.
  25. Zainuddin Z, Ong P. Function approximation using artificial neural networks. WSEAS Trans Math 2008; 7: 333-338.
  26. Gopalakrishnan V, Ramaswamy C. Patient opinion mining to analyze drugs satisfaction using supervised learning. J Appl Res Technol 2017; 15: 311-319. DOI: 10.1016/j.jart.2017.02.005.
  27. Hopfield JJ, Feinstein DI, Palmer RG. “Unlearning” has a stabilizing effect in collective memories. Nature 1983; 304: 158-159. DOI: 10.1038/304158a0.
  28. Fachechi A, Agliari E, Barra A. Dreaming neural networks: Forgetting spurious memories and reinforcing pure ones. Neural Networks 2019; 112: 24-40. DOI: 10.1016/j.neunet.2019.01.006.
  29. Yamauchi K, Hayami J. Sleep learning – An incremental learning system inspired by sleep behavior-. IEEE Int Conf on Fuzzy Systems 2006: 1186-1193. DOI: 10.1109/FUZZY.2006.1681860.
  30. Tadros T, Krishnan G, Ramyaa R, Bazhenov M. Biologically inspired sleep algorithm for reducing catastrophic forgetting in neural networks (student abstract). Proc AAAI Conf on Artificial Intelligence 2020; 34(10): 13933-13934. DOI: 10.1609/AAAI.V34I10.7239.
  31. Abramova ES, Orlov AA, Makarov KV. Applying regularization to the neural network extreme learning machine. Optoelectronic devices and devices in image recognition and image processing systems. Recognition – 2021 [In Russian]. Sat materials of the XVI Intern sci-tech conf. Kursk: Publisher of Southwest state University; 2021.

© 2009, IPSI RAS
151, Molodogvardeiskaya str., Samara, 443001, Russia; E-mail: journal@computeroptics.ru ; Tel: +7 (846) 242-41-24 (Executive secretary), +7 (846) 332-56-22 (Issuing editor), Fax: +7 (846) 332-56-20