(48-4) 12 * << * >> * Русский * English * Содержание * Все выпуски

Uncertainty-based quantization method for stable training of binary neural networks
A.V. Trusov 1,2,3, D.N. Putintsev 2,3, E.E. Limonova 2,3

Moscow Institute of Physics and Technology (National Research University),
141701, Russia, Dolgoprudnii, Institutskiy per. 9;
Federal Research Center “Computer Science and Control” of Russian Academy of Sciences,
19333, Russia, Moscow, Vavilova str. 44, corp. 2;
LLC “Smart Engines Service”,
117312, Russia, Moscow, prospect 60-letiya Oktyabrya 9

  PDF, 1382 kB

DOI: 10.18287/2412-6179-CO-1427

Страницы: 573-581.

Язык статьи: English.

Аннотация:
Binary neural networks (BNNs) have gained attention due to their computational efficiency. However, training BNNs has proven to be challenging. Existing algorithms either fail to produce stable and high-quality results or are overly complex for practical use. In this paper, we introduce a novel quantizer called UBQ (Uncertainty-based quantizer) for BNNs, which combines the advantages of existing methods, resulting in stable training and high-quality BNNs even with a low number of trainable parameters. We also propose a training method involving gradual network freezing and batch normalization replacement, facilitating a smooth transition from training mode to execution mode for BNNs.
     To evaluate UBQ, we conducted experiments on the MNIST and CIFAR-10 datasets and compared our method to existing algorithms. The results demonstrate that UBQ outperforms previous methods for smaller networks and achieves comparable results for larger networks.

Ключевые слова:
binary networks, neural networks training, quantization, gradient estimation, approximation.

Citation:
Trusov AV, Putintsev DN, Limonova EE. Uncertainty-based quantization method for stable training of binary neural networks. Computer Optics 2024; 48(4): 573-581. DOI: 10.18287/2412-6179-CO-1427.

References:

  1. Hoffmann J, Borgeaud S, Mensch A, et al. An empirical analysis of compute-optimal large language model training. Adv Neural Inf Process Syst 2022; 35: 30016-30030.
  2. Rombach R, Blattmann A, Lorenz D, Esser P, Ommer B. High-resolution image synthesis with latent diffusion models. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition 2022: 10684-10695.
  3. Ilyuhin SA, Sheshkus AV, Arlazarov VL. Recognition of images of korean characters using embedded networks. Proc SPIE 2020; 11433: 1143311. DOI: 10.1117/12.2559453.
  4. Bulatov K, Arlazarov VV, Chernov T, Slavin O, Nikolaev D. Smart IDReader: Document recognition in video stream, 2017 14th IAPR Int Conf on Document Analysis and Recognition (ICDAR) 2017: 39-44. DOI: 10.1109/ICDAR.2017.347.
  5. Sheshkus A, Chirvonaya A, Arlazarov VL. Tiny CNN for feature point description for document analysis: approach and dataset. Computer Optics 2022; 46(3): 429-435. DOI: 10.18287/2412-6179-CO-1016.
  6. Chernyshova YS, Chirvonaya AN, Sheshkus AV. Localization of characters horizontal bounds in text line images with fully convolutional network. Proc SPIE 2020; 11433: 114333F. DOI: 10.1117/12.2559449.
  7. Liang T, Bao H, Pan W, Pan F. ALODAD: An anchor-free lightweight object detector for autonomous driving, IEEE Access 2022; 10: 40701-40714. DOI: 10.1109/ACCESS.2022.3166923.
  8. Sivapalan G, Nundy KK, Dev S, Cardiff B, John D. Annet: a lightweight neural network for ecg anomaly detection in iot edge sensors. IEEE Trans Biomed Circuits Syst 2022; 16(1): 24-35. DOI: 10.1109/TBCAS.2021.3137646.
  9. He Z, Zhang X, Cao Y, Liu Z, Zhang B, Wang X. LiteNet: Lightweight neural network for detecting arrhythmias at resource-constrained mobile devices. Sensors 2018; 18(4): 1229. DOI: 10.3390/s18041229.
  10. Gholami A. Kim S, Dong Z, Yao Z, Mahoney MW, Keutzer K. A survey of quantization methods for efficient neural network inference. In Book: Thiruvathukal GK, Lu Y-H, Kim J, Chen Y, Chen B, eds. Low-power computer vision. New York: Chapman and Hall/CRC; 2022: 291-326.
  11. Rastegari M, Ordonez V, Redmon J, Farhadi A. XNOR-Net: Imagenet classification using binary convolutional neural networks, In Book: Leibe B, Matas J, Sebe N, Welling M, eds. European conference on computer vision. Cham: Springer International Publishing AG; 2016: 525-542. DOI: 10.1007/978-3-319-46493-0_32.
  12. Moss DJ, Nurvitadhi E, Sim J, Mishra A, Marr D, Subhaschandra S, Leong PH. High performance binary neural networks on the Xeon+FPGA™ platform. 2017 27th Int Conf on Field Programmable Logic and Applications (FPL) 2017: 1-4. DOI: 10.23919/FPL.2017.8056823.
  13. He S, Meng H, Zhou Z, Liu Y, Huang K, Chen G. An efficient GPU-accelerated inference engine for binary neural network on mobile phones. J Syst Archit 2021; 117: 102156.
  14. Zhang J, Pan Y, Yao T, Zhao H, Mei T. daBNN: A super fast inference framework for binary neural networks on arm devices. Proc 27th ACM Int Conf on Multimedia 2019: 2272-2275.
  15. Frickenstein A, Vemparala M-R, Mayr J, Nagaraja N-S, Unger C, Tombari F, Stechele W. Binary DAD-Net: Binarized driveable area detection network for autonomous driving. 2020 IEEE Int Conf on Robotics and Automation (ICRA) 2020: 2295-2301.
  16. Xiang X, Qian Y, Yu K. Binary deep neural networks for speech recognition. INTERSPEECH 2017: 533-537.
  17. Alemdar H, Leroy V, Prost-Boucle A, Pétrot F. Ternary neural networks for resource-efficient ai applications. 2017 Int Joint Conf on Neural Networks (IJCNN) 2017: 2547-2554.
  18. Courbariaux M, Bengio Y, David J-P. Binaryconnect: Training deep neural networks with binary weights during propagations. Proc 28th Int Conf on Neural Information Processing Systems (NIPS'15) 2015; 2: 3123-3131.
  19. Liu B, Li F, Wang X, Zhang B, Yan J. Ternary weight networksю ICASSP 2023-2023 IEEE Int Conf on Acoustics, Speech and Signal Processing (ICASSP) 2023: 1-5. DOI: 10.1109/ICASSP49357.2023.10094626.
  20. Liu Z, Wu B, Luo W, Yang X, Liu W, Cheng K-T. Bi-Real Net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. Proc European Conf on Computer Vision (ECCV) 2018: 722-737.
  21. Chen H, Wang Y, Xu C, Shi B, Xu C, Tian Q, Xu C. AdderNet: Do we really need multiplications in deep learning? Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition 2020: 1468-1477.
  22. Limonova EE. Fast and gate-efficient approximated activations for bipolar morphological neural networks. Informatsionnye Tekhnologii i Vychslitel'nye Sistemy 2022; 2: 3-10. DOI: 10.14357/20718632220201.
  23. Bengio Y, Léonard N, Courville A. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv Preprint. 2013. Source: <https://arxiv.org/abs/1308.3432>.
  24. Bethge J, Yang H, Bornstein M, Meinel C. Back to simplicity: How to train accurate bnns from scratch? arXiv Preprint. 2019. Source: <https://arxiv.org/abs/1906.08637>.
  25. Bulat A, Tzimiropoulos G. XNOR-Net++: Improved binary neural networks. arXiv Preprint. 2019. Source: <https://arxiv.org/abs/1909.13863>.
  26. Xu Z, Lin M, Liu J, Chen J, Shao L, Gao Y, Tian Y, Ji R. Recu: Reviving the dead weights in binary neural networks. Proc IEEE/CVF Int Conf on Computer Vision 2021: 5198-5208.
  27. Gong R, Liu X, Jiang S, Li T, Hu P, Lin J, Yu F, Yan J. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. Proc IEEE/CVF Int Conf on Computer Vision 2019: 4852-4861.
  28. Lahoud F, Achanta R, Márquez-Neila P, Süsstrunk S. Self-binarizing networks. arXiv Preprint. 2019. Source: <https://arxiv.org/abs/1902.00730>.
  29. Yang J, Shen X, Xing J, Tian X, Li H, Deng B, Huang J, Hua X-s. Quantization networks. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition 2019: 7308-7316.
  30. Meng X, Bachmann R, Khan ME. Training binary neural networks using the bayesian learning rule. Int Conf on Machine Learning (PMLR) 2020: 6852-6861.
  31. Jacob B, Kligys S, Chen B, Zhu M, Tang M, Howard A, Adam H, Kalenichenko D. Quantization and training of neural networks for efficient integer-arithmetic-only inference. Proc IEEE Conf on Computer Vision and Pattern Recognition 2018: 2704-2713.
  32. Paszke A, Gross S, Massa F, et al. PyTorch: An imperative style, high-performance deep learning library. Proc 33rd Int Conf on Neural Information Processing Systems (NIPS'19) 2019: 8026-8037.
  33. Keras: Simple. Flexible. Powerful. 2023. Source: <https://keras.io>.
  34. Xue P, Lu Y, Chang J, Wei X, Wei Z. Fast and accurate binary neural networks based on depth-width reshaping. Proc AAAI Conf on Artificial Intelligence 2023; 37: 10684-10692.
  35. Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proc 32nd Int Conf on on Machine Learning (ICML'15) 2015; 37: 448-456.
  36. LeCun Y. The mnist database of handwritten digits. 1998. Source: <http://yann.lecun.com/exdb/mnist/>.
  37. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv Preprint. 2014. Source: <https://arxiv.org/abs/1409.1556>.
  38. Krizhevsky A, Hinton G, et al. Learning multiple layers of features from tiny images. 2009. Source:       <https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf>.
  39. Hubara I, Courbariaux M, Soudry D, El-Yaniv R, Bengio Y. Binarized neural networks. In Book: Lee D, Sugiyama M, Luxburg U, Guyon I, Garnett R, eds. Advances in Neural Information Processing Systems 29 (NIPS 2016). 2016. Source:       <https://proceedings.neurips.cc/paper_files/paper/2016/file/d8330f857a17c53d217014ee776bfd50-Paper.pdf>.
  40. Trusov AV, Limonova EE, Nikolaev DP, Arlazarov VV. Fast matrix multiplication for binary and ternary CNNs on ARM CPU. 2022 26th Int Conf on Pattern Recognition (ICPR) 2022: 3176-3182. DOI: 10.1109/ICPR56361.2022.9956533.
  41. Zhang D, Yang J, Ye D, Hua G. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Book: Ferrari V, Hebert M, Sminchisescu C, Weiss Y, eds. Computer Vision – ECCV 2018. Cham: Springer Nature Switzerland AG; 2018: 365-382.
  42. Darabi S, Belbahri M, Courbariaux M, Nia VP. Regularized binary network training. arXiv Preprint. 2018. Source: <https://arxiv.org/abs/1812.11800>.
  43. Sher AV, Trusov AV, Limonova EE, Nikolaev DP, Arlazarov VV. Neuron-by-neuron quantization for efficient low-bit qnn training. Mathematics 2023; 11(9): 2112. DOI: 10.3390/math11092112.
  44. Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR. Improving neural networks by preventing co-adaptation of feature detectors. arXiv Preprint. 2012. Source: <https://arxiv.org/abs/1207.0580>.
  45. Kingma DP, Ba J. Adam: A method for stochastic optimization, arXiv Preprint. 2014. Source: <https://arxiv.org/abs/1412.6980>.

© 2009, IPSI RAS
Россия, 443001, Самара, ул. Молодогвардейская, 151; электронная почта: journal@computeroptics.ru; тел: +7 (846) 242-41-24 (ответственный секретарь), +7 (846) 332-56-22 (технический редактор), факс: +7 (846) 332-56-20