(48-4) 12 * << * >> * Russian * English * Content * All Issues

Uncertainty-based quantization method for stable training of binary neural networks
A.V. Trusov 1,2,3, D.N. Putintsev 2,3, E.E. Limonova 2,3

1 Moscow Institute of Physics and Technology (National Research University),
141701, Russia, Dolgoprudnii, Institutskiy per. 9;
2 Federal Research Center “Computer Science and Control” of Russian Academy of Sciences,
19333, Russia, Moscow, Vavilova str. 44, corp. 2;
3 LLC “Smart Engines Service”,
117312, Russia, Moscow, prospect 60-letiya Oktyabrya 9

 PDF, 1382 kB

DOI: 10.18287/2412-6179-CO-1427

Pages: 573-581.

Full text of article: English language.

Abstract:
Binary neural networks (BNNs) have gained attention due to their computational efficiency. However, training BNNs has proven to be challenging. Existing algorithms either fail to produce stable and high-quality results or are overly complex for practical use. In this paper, we introduce a novel quantizer called UBQ (Uncertainty-based quantizer) for BNNs, which combines the advantages of existing methods, resulting in stable training and high-quality BNNs even with a low number of trainable parameters. We also propose a training method involving gradual network freezing and batch normalization replacement, facilitating a smooth transition from training mode to execution mode for BNNs.
     To evaluate UBQ, we conducted experiments on the MNIST and CIFAR-10 datasets and compared our method to existing algorithms. The results demonstrate that UBQ outperforms previous methods for smaller networks and achieves comparable results for larger networks.

Keywords:
binary networks, neural networks training, quantization, gradient estimation, approximation.

Citation:
Trusov AV, Putintsev DN, Limonova EE. Uncertainty-based quantization method for stable training of binary neural networks. Computer Optics 2024; 48(4): 573-581. DOI: 10.18287/2412-6179-CO-1427.

References:
  1. Hoffmann J, Borgeaud S, Mensch A, et al. An empirical analysis of compute-optimal large language model training. Adv Neural Inf Process Syst 2022; 35: 30016-30030.
  2. Rombach R, Blattmann A, Lorenz D, Esser P, Ommer B. High-resolution image synthesis with latent diffusion models. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition 2022: 10684-10695.
  3. Ilyuhin SA, Sheshkus AV, Arlazarov VL. Recognition of images of korean characters using embedded networks. Proc SPIE 2020; 11433: 1143311. DOI: 10.1117/12.2559453.
  4. Bulatov K, Arlazarov VV, Chernov T, Slavin O, Nikolaev D. Smart IDReader: Document recognition in video stream, 2017 14th IAPR Int Conf on Document Analysis and Recognition (ICDAR) 2017: 39-44. DOI: 10.1109/ICDAR.2017.347.
  5. Sheshkus A, Chirvonaya A, Arlazarov VL. Tiny CNN for feature point description for document analysis: approach and dataset. Computer Optics 2022; 46(3): 429-435. DOI: 10.18287/2412-6179-CO-1016.
  6. Chernyshova YS, Chirvonaya AN, Sheshkus AV. Localization of characters horizontal bounds in text line images with fully convolutional network. Proc SPIE 2020; 11433: 114333F. DOI: 10.1117/12.2559449.
  7. Liang T, Bao H, Pan W, Pan F. ALODAD: An anchor-free lightweight object detector for autonomous driving, IEEE Access 2022; 10: 40701-40714. DOI: 10.1109/ACCESS.2022.3166923.
  8. Sivapalan G, Nundy KK, Dev S, Cardiff B, John D. Annet: a lightweight neural network for ecg anomaly detection in iot edge sensors. IEEE Trans Biomed Circuits Syst 2022; 16(1): 24-35. DOI: 10.1109/TBCAS.2021.3137646.
  9. He Z, Zhang X, Cao Y, Liu Z, Zhang B, Wang X. LiteNet: Lightweight neural network for detecting arrhythmias at resource-constrained mobile devices. Sensors 2018; 18(4): 1229. DOI: 10.3390/s18041229.
  10. Gholami A. Kim S, Dong Z, Yao Z, Mahoney MW, Keutzer K. A survey of quantization methods for efficient neural network inference. In Book: Thiruvathukal GK, Lu Y-H, Kim J, Chen Y, Chen B, eds. Low-power computer vision. New York: Chapman and Hall/CRC; 2022: 291-326.
  11. Rastegari M, Ordonez V, Redmon J, Farhadi A. XNOR-Net: Imagenet classification using binary convolutional neural networks, In Book: Leibe B, Matas J, Sebe N, Welling M, eds. European conference on computer vision. Cham: Springer International Publishing AG; 2016: 525-542. DOI: 10.1007/978-3-319-46493-0_32.
  12. Moss DJ, Nurvitadhi E, Sim J, Mishra A, Marr D, Subhaschandra S, Leong PH. High performance binary neural networks on the Xeon+FPGA™ platform. 2017 27th Int Conf on Field Programmable Logic and Applications (FPL) 2017: 1-4. DOI: 10.23919/FPL.2017.8056823.
  13. He S, Meng H, Zhou Z, Liu Y, Huang K, Chen G. An efficient GPU-accelerated inference engine for binary neural network on mobile phones. J Syst Archit 2021; 117: 102156.
  14. Zhang J, Pan Y, Yao T, Zhao H, Mei T. daBNN: A super fast inference framework for binary neural networks on arm devices. Proc 27th ACM Int Conf on Multimedia 2019: 2272-2275.
  15. Frickenstein A, Vemparala M-R, Mayr J, Nagaraja N-S, Unger C, Tombari F, Stechele W. Binary DAD-Net: Binarized driveable area detection network for autonomous driving. 2020 IEEE Int Conf on Robotics and Automation (ICRA) 2020: 2295-2301.
  16. Xiang X, Qian Y, Yu K. Binary deep neural networks for speech recognition. INTERSPEECH 2017: 533-537.
  17. Alemdar H, Leroy V, Prost-Boucle A, Pétrot F. Ternary neural networks for resource-efficient ai applications. 2017 Int Joint Conf on Neural Networks (IJCNN) 2017: 2547-2554.
  18. Courbariaux M, Bengio Y, David J-P. Binaryconnect: Training deep neural networks with binary weights during propagations. Proc 28th Int Conf on Neural Information Processing Systems (NIPS'15) 2015; 2: 3123-3131.
  19. Liu B, Li F, Wang X, Zhang B, Yan J. Ternary weight networksю ICASSP 2023-2023 IEEE Int Conf on Acoustics, Speech and Signal Processing (ICASSP) 2023: 1-5. DOI: 10.1109/ICASSP49357.2023.10094626.
  20. Liu Z, Wu B, Luo W, Yang X, Liu W, Cheng K-T. Bi-Real Net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. Proc European Conf on Computer Vision (ECCV) 2018: 722-737.
  21. Chen H, Wang Y, Xu C, Shi B, Xu C, Tian Q, Xu C. AdderNet: Do we really need multiplications in deep learning? Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition 2020: 1468-1477.
  22. Limonova EE. Fast and gate-efficient approximated activations for bipolar morphological neural networks. Informatsionnye Tekhnologii i Vychslitel'nye Sistemy 2022; 2: 3-10. DOI: 10.14357/20718632220201.
  23. Bengio Y, Léonard N, Courville A. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv Preprint. 2013. Source: <https://arxiv.org/abs/1308.3432>.
  24. Bethge J, Yang H, Bornstein M, Meinel C. Back to simplicity: How to train accurate bnns from scratch? arXiv Preprint. 2019. Source: <https://arxiv.org/abs/1906.08637>.
  25. Bulat A, Tzimiropoulos G. XNOR-Net++: Improved binary neural networks. arXiv Preprint. 2019. Source: <https://arxiv.org/abs/1909.13863>.
  26. Xu Z, Lin M, Liu J, Chen J, Shao L, Gao Y, Tian Y, Ji R. Recu: Reviving the dead weights in binary neural networks. Proc IEEE/CVF Int Conf on Computer Vision 2021: 5198-5208.
  27. Gong R, Liu X, Jiang S, Li T, Hu P, Lin J, Yu F, Yan J. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. Proc IEEE/CVF Int Conf on Computer Vision 2019: 4852-4861.
  28. Lahoud F, Achanta R, Márquez-Neila P, Süsstrunk S. Self-binarizing networks. arXiv Preprint. 2019. Source: <https://arxiv.org/abs/1902.00730>.
  29. Yang J, Shen X, Xing J, Tian X, Li H, Deng B, Huang J, Hua X-s. Quantization networks. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition 2019: 7308-7316.
  30. Meng X, Bachmann R, Khan ME. Training binary neural networks using the bayesian learning rule. Int Conf on Machine Learning (PMLR) 2020: 6852-6861.
  31. Jacob B, Kligys S, Chen B, Zhu M, Tang M, Howard A, Adam H, Kalenichenko D. Quantization and training of neural networks for efficient integer-arithmetic-only inference. Proc IEEE Conf on Computer Vision and Pattern Recognition 2018: 2704-2713.
  32. Paszke A, Gross S, Massa F, et al. PyTorch: An imperative style, high-performance deep learning library. Proc 33rd Int Conf on Neural Information Processing Systems (NIPS'19) 2019: 8026-8037.
  33. Keras: Simple. Flexible. Powerful. 2023. Source: <https://keras.io>.
  34. Xue P, Lu Y, Chang J, Wei X, Wei Z. Fast and accurate binary neural networks based on depth-width reshaping. Proc AAAI Conf on Artificial Intelligence 2023; 37: 10684-10692.
  35. Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. Proc 32nd Int Conf on on Machine Learning (ICML'15) 2015; 37: 448-456.
  36. LeCun Y. The mnist database of handwritten digits. 1998. Source: <http://yann.lecun.com/exdb/mnist/>.
  37. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv Preprint. 2014. Source: <https://arxiv.org/abs/1409.1556>.
  38. Krizhevsky A, Hinton G, et al. Learning multiple layers of features from tiny images. 2009. Source:       <https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf>.
  39. Hubara I, Courbariaux M, Soudry D, El-Yaniv R, Bengio Y. Binarized neural networks. In Book: Lee D, Sugiyama M, Luxburg U, Guyon I, Garnett R, eds. Advances in Neural Information Processing Systems 29 (NIPS 2016). 2016. Source:       <https://proceedings.neurips.cc/paper_files/paper/2016/file/d8330f857a17c53d217014ee776bfd50-Paper.pdf>.
  40. Trusov AV, Limonova EE, Nikolaev DP, Arlazarov VV. Fast matrix multiplication for binary and ternary CNNs on ARM CPU. 2022 26th Int Conf on Pattern Recognition (ICPR) 2022: 3176-3182. DOI: 10.1109/ICPR56361.2022.9956533.
  41. Zhang D, Yang J, Ye D, Hua G. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Book: Ferrari V, Hebert M, Sminchisescu C, Weiss Y, eds. Computer Vision – ECCV 2018. Cham: Springer Nature Switzerland AG; 2018: 365-382.
  42. Darabi S, Belbahri M, Courbariaux M, Nia VP. Regularized binary network training. arXiv Preprint. 2018. Source: <https://arxiv.org/abs/1812.11800>.
  43. Sher AV, Trusov AV, Limonova EE, Nikolaev DP, Arlazarov VV. Neuron-by-neuron quantization for efficient low-bit qnn training. Mathematics 2023; 11(9): 2112. DOI: 10.3390/math11092112.
  44. Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR. Improving neural networks by preventing co-adaptation of feature detectors. arXiv Preprint. 2012. Source: <https://arxiv.org/abs/1207.0580>.
  45. Kingma DP, Ba J. Adam: A method for stochastic optimization, arXiv Preprint. 2014. Source: <https://arxiv.org/abs/1412.6980>.

© 2009, IPSI RAS
151, Molodogvardeiskaya str., Samara, 443001, Russia; E-mail: journal@computeroptics.ru ; Tel: +7 (846) 242-41-24 (Executive secretary), +7 (846) 332-56-22 (Issuing editor), Fax: +7 (846) 332-56-20