On Computer Memory Saving Methods in Performing Data Classification Using Fully Connected Neural Networks
DOI:
https://doi.org/10.24160/1993-6982-2021-3-103-109Keywords:
linear polynomials, integers, fully connected neural networkAbstract
In solving the classification problem, a fully connected trainable neural network (with adjusting the parameters represented by double-precision real numbers) is used as a mathematical model. After the training is completed, the neural network parameters are rounded and represented as fixed-point numbers (integers). The aim of the study is to reduce the required amount of the computing system memory for storing the obtained integer parameters.
To reduce the amount of memory, the following methods for storing integer parameters are developed, which are based on representing the linear polynomials included in a fully connected neural network using compositions of simpler functions:
- a method based on representing the considered polynomial as a sum of simpler polynomials;
- a method based on separately storing the information about additions and multiplications.
In the experiment with the MNIST data set, it took 1.41 MB to store real parameters of a fully connected neural network, 0.7 MB to store integer parameters without using the proposed methods, 0.47 MB in the RAM and 0.3 MB in compressed form on the disk when using the first method, and 0.25 MB on the disk when using the second method.
In the experiment with the USPS data set, it took 0.25 MB to store real parameters of a fully connected neural network, 0.1 MB to store integer parameters without using the proposed methods, 0.05 MB in the RAM and approximately the same amount in compressed form on the disk when using the first method, and 0.03 MB on the disk when using the second method.
The study results can be applied in using fully connected neural networks to solve various recognition problems under the conditions of limited hardware capacities.
References
1. Anguita D., Ghio A., Pischiutta S., Ridella S. A Support Vector Machine with Integer Parameters // Neurocomputing. 2008. V. 72(1—3). Pp. 480—489.
2. Tschiatschek S., Paul K., Pernkopf F. Integer Bayesian Network Classifiers // Machine Learning and Knowledge Discovery in Databases. Lecture Notes in Computer Sci. 2014. V. 8726. Pp. 209—224.
3. Han S., Mao H., Dally W.J. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding // Proc. Intern. Conf. Learning Representations. 2016. Pp. 1—14.
4. Liu B., Wang M., Foroosh H., Tappen M., Penksy M. Sparse Convolutional Neural Networks // IEEE Conf. Computer Vision and Pattern Recognition. 2015. Pp. 806—814.
5. Соловьев Р.А., Кустов А.Г., Тельпухов Д.В., Рухлов В.С. Прототипирование высокоскоростной нейронной сети в ПЛИС для классификации изображений видеопотока // Cloud of Sci. 2018. № 4. С. 680—703.
6. Алексиадис Н.Ф. Алгоритмическая неразрешимость задачи о нахождении базиса конечной полной системы полиномов с целыми коэффициентами // Интеллектуальные системы. Теория и приложения. 2016. Т. 20. Вып. 3. С. 19—23.
7. Мамонтов А.И., Мещанинов Д.Г. Проблема полноты в функциональной системе линейных полиномов с целыми коэффициентами // Дискретная математика. 2010. Т. 22. № 4. С. 64—82.
8. Мамонтов А.И., Мещанинов Д.Г. Алгоритм распознавания полноты в функциональной системе L(Z) // Дискретная математика. 2014. Т. 26. № 1. С. 85—95.
9. Мамонтов А.И. О повышении эффективности вычислений при классификации изображений // Вестник МЭИ. 2019. № 5. С. 129—134.
10. Мамонтов А.И. О способах экономии памяти при решении задач классификации линейным методом опорных векторов // Вестник МЭИ. 2020. № 4. С. 129—135.
11. Gorban A.N., Wunsch II D.C. The General Approximation Theorem // Proc. Intern. Joint Conf. Neural Networks. 1998. Pp. 1271—1274.
12. Горбань А.Н. Обобщенная аппроксимационная теорема и вычислительные возможности нейронных сетей // Сибирский журнал вычислительной математики. 1998. Т. 1. № 1. С. 11—24.
13. Половников В.С. О нелинейных характеристиках нейронных схем в произвольных базисах // Интеллектуальные системы. 2013. Т. 17. № 1—4. С. 87—90.
14. Anguita D., Pischiutta S., Ridella S., Sterpi D. Feed-forward SVM without Multipliers // IEEE Trans. Neural Networks. 2006. V. 17. Pp. 1328—1331.
15. Anguita D., Boni A., Ridella S. A Digital Architecture for Support Vector Machines: Theory, Algorithm and FPGA Implementation // IEEE Trans. Neural Networks. 2003. V. 14. Pp. 993—1009.
16. Anguita D., Bozza G. The effects of quantization on support vector machines with Gaussian kernel // IEEE Intern. Joint Conf. Neural Networks. 2005. V. 2. Pp. 681—684.
17. Lesser B., Mücke M., Gansterer W.N. Effects of Reduced Precision on Floating-point SVM Classification Accuracy // Procedia Computer Sci. 2011. V. 4. Pp. 508—517.
18. Tschiatschek S., Pernkopf F. On Bayesian Network Classifiers with Reduced Precision Parameters // IEEE Trans. Pattern Analysis and Machine Intelligence. 2015. V. 37(4). Pp. 774—785.
19. Yi Y., Hangping Z., Bin Z. A New Learning Algorithm for Neural Networks with Integer Weights and Quantized Non-linear Activation Functions // Artificial Intelligence in Theory and Practice. 2008. V. 276. Pp. 427—431.
20. Shuang Wu, Guoqi Li, Feng Chen, Luping Shi. Training and Inference with Integers in Deep Neural Networks // Proc. Intern. Conf. Learning Representations. 2018. Pp. 1—14.
---
Для цитирования: Мамонтов А.И. О способах экономии компьютерной памяти при классификации данных с помощью полносвязных нейронных сетей // Вестник МЭИ. 2021.
№ 3. С. 103—109. DOI: 10.24160/1993-6982-2021-3-103-109.
---
Работа выполнена при поддержке: РФФИ (проект № 19-01-00294)
#
1. Anguita D., Ghio A., Pischiutta S., Ridella S. A Support Vector Machine with Integer Parameters. Neurocomputing. 2008;72(1—3):480—489.
2. Tschiatschek S., Paul K., Pernkopf F. Integer Bayesian Network Classifiers. Machine Learning and Knowledge Discovery in Databases. Lecture Notes in Computer Sci. 2014;8726:209—224.
3. Han S., Mao H., Dally W.J. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. Proc. Intern. Conf. Learning Representations. 2016:1—14.
4. Liu B., Wang M., Foroosh H., Tappen M., Penksy M. Sparse Convolutional Neural Networks. IEEE Conf. Computer Vision and Pattern Recognition. 2015:806—814.
5. Solov'ev R.A., Kustov A.G., Tel'pukhov D.V., Rukhlov V.S. Prototipirovanie Vysokoskorostnoy Neyronnoy Seti v PLIS dlya Klassifikatsii Izobrazheniy Videopotoka. Cloud of Sci. 2018;4:680—703. (in Russian).
6. Aleksiadis N.F. Algoritmicheskaya Nerazreshimost' Zadachi o Nakhozhdenii Bazisa Konechnoy Polnoy Sistemy Polinomov s Tselymi Koeffitsientami. Intellektual'nye Sistemy. Teoriya i Prilozheniya. 2016;20;3:19—23. (in Russian).
7. Mamontov A.I., Meshchaninov D.G. Problema Polnoty v Funktsional'noy Sisteme Lineynykh Polinomov s Tselymi Koeffitsientami. Diskretnaya Matematika. 2010;22;4:64—82. (in Russian).
8. Mamontov A.I., Meshchaninov D.G. Algoritm Raspoznavaniya Polnoty v Funktsional'noy Sisteme L(Z). Diskretnaya Matematika. 2014;26;1:85—95. (in Russian).
9. Mamontov A.I. O Povyshenii Effektivnosti Vychisleniy pri Klassifikatsii Izobrazheniy. Vestnik MEI. 2019;5:129—134. (in Russian).
10. Mamontov A.I. O Sposobakh Ekonomii Pamyati pri Reshenii Zadach Klassifikatsii Lineynym Metodom Opornykh Vektorov. Vestnik MEI. 2020;4:129—135. (in Russian).
11. Gorban A.N., Wunsch II D.C. The General Approximation Theorem. Proc. Intern. Joint Conf. Neural Networks. 1998:1271—1274.
12. Gorban' A.N. Obobshchennaya Approksimatsionnaya Teorema i Vychislitel'nye Vozmozhnosti Neyronnykh Setey. Sibirskiy Zhurnal Vychislitel'noy Matematiki. 1998;1;1:11—24. (in Russian).
13. Polovnikov V.S. O Nelineynykh Kharakteristikakh Neyronnykh Skhem v Proizvol'nykh Bazisakh. Intellektual'nye Sistemy. 2013;17;1—4:87—90. (in Russian).
14. Anguita D., Pischiutta S., Ridella S., Sterpi D. Feed-forward SVM without Multipliers. IEEE Trans. Neural Networks. 2006;17:1328—1331.
15. Anguita D., Boni A., Ridella S. A Digital Architecture for Support Vector Machines: Theory, Algorithm and FPGA Implementation. IEEE Trans. Neural Networks. 2003;14:993—1009.
16. Anguita D., Bozza G. The effects of quantization on support vector machines with Gaussian kernel. IEEE Intern. Joint Conf. Neural Networks. 2005;2:681—684.
17. Lesser B., Mücke M., Gansterer W.N. Effects of Reduced Precision on Floating-point SVM Classification Accuracy. Procedia Computer Sci. 2011;4:508—517.
18. Tschiatschek S., Pernkopf F. On Bayesian Network Classifiers with Reduced Precision Parameters. IEEE Trans. Pattern Analysis and Machine Intelligence. 2015;37(4):774—785.
19. Yi Y., Hangping Z., Bin Z. A New Learning Algorithm for Neural Networks with Integer Weights and Quantized Non-linear Activation Functions. Artificial Intelligence in Theory and Practice. 2008;276:427—431.
20. Shuang Wu, Guoqi Li, Feng Chen, Luping Shi. Training and Inference with Integers in Deep Neural Networks. Proc. Intern. Conf. Learning Representations. 2018:1—14.
---
For citation: Mamontov A.I. On Computer Memory Saving Methods in Performing Data Classification Using Fully Connected Neural Networks. Bulletin of MPEI. 2021;3:103—109. (in Russian). DOI: 10.24160/1993-6982-2021-3-103-109.
---
The work is executed at support: RFBR (Project No. 19-01-00294)
2. Tschiatschek S., Paul K., Pernkopf F. Integer Bayesian Network Classifiers // Machine Learning and Knowledge Discovery in Databases. Lecture Notes in Computer Sci. 2014. V. 8726. Pp. 209—224.
3. Han S., Mao H., Dally W.J. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding // Proc. Intern. Conf. Learning Representations. 2016. Pp. 1—14.
4. Liu B., Wang M., Foroosh H., Tappen M., Penksy M. Sparse Convolutional Neural Networks // IEEE Conf. Computer Vision and Pattern Recognition. 2015. Pp. 806—814.
5. Соловьев Р.А., Кустов А.Г., Тельпухов Д.В., Рухлов В.С. Прототипирование высокоскоростной нейронной сети в ПЛИС для классификации изображений видеопотока // Cloud of Sci. 2018. № 4. С. 680—703.
6. Алексиадис Н.Ф. Алгоритмическая неразрешимость задачи о нахождении базиса конечной полной системы полиномов с целыми коэффициентами // Интеллектуальные системы. Теория и приложения. 2016. Т. 20. Вып. 3. С. 19—23.
7. Мамонтов А.И., Мещанинов Д.Г. Проблема полноты в функциональной системе линейных полиномов с целыми коэффициентами // Дискретная математика. 2010. Т. 22. № 4. С. 64—82.
8. Мамонтов А.И., Мещанинов Д.Г. Алгоритм распознавания полноты в функциональной системе L(Z) // Дискретная математика. 2014. Т. 26. № 1. С. 85—95.
9. Мамонтов А.И. О повышении эффективности вычислений при классификации изображений // Вестник МЭИ. 2019. № 5. С. 129—134.
10. Мамонтов А.И. О способах экономии памяти при решении задач классификации линейным методом опорных векторов // Вестник МЭИ. 2020. № 4. С. 129—135.
11. Gorban A.N., Wunsch II D.C. The General Approximation Theorem // Proc. Intern. Joint Conf. Neural Networks. 1998. Pp. 1271—1274.
12. Горбань А.Н. Обобщенная аппроксимационная теорема и вычислительные возможности нейронных сетей // Сибирский журнал вычислительной математики. 1998. Т. 1. № 1. С. 11—24.
13. Половников В.С. О нелинейных характеристиках нейронных схем в произвольных базисах // Интеллектуальные системы. 2013. Т. 17. № 1—4. С. 87—90.
14. Anguita D., Pischiutta S., Ridella S., Sterpi D. Feed-forward SVM without Multipliers // IEEE Trans. Neural Networks. 2006. V. 17. Pp. 1328—1331.
15. Anguita D., Boni A., Ridella S. A Digital Architecture for Support Vector Machines: Theory, Algorithm and FPGA Implementation // IEEE Trans. Neural Networks. 2003. V. 14. Pp. 993—1009.
16. Anguita D., Bozza G. The effects of quantization on support vector machines with Gaussian kernel // IEEE Intern. Joint Conf. Neural Networks. 2005. V. 2. Pp. 681—684.
17. Lesser B., Mücke M., Gansterer W.N. Effects of Reduced Precision on Floating-point SVM Classification Accuracy // Procedia Computer Sci. 2011. V. 4. Pp. 508—517.
18. Tschiatschek S., Pernkopf F. On Bayesian Network Classifiers with Reduced Precision Parameters // IEEE Trans. Pattern Analysis and Machine Intelligence. 2015. V. 37(4). Pp. 774—785.
19. Yi Y., Hangping Z., Bin Z. A New Learning Algorithm for Neural Networks with Integer Weights and Quantized Non-linear Activation Functions // Artificial Intelligence in Theory and Practice. 2008. V. 276. Pp. 427—431.
20. Shuang Wu, Guoqi Li, Feng Chen, Luping Shi. Training and Inference with Integers in Deep Neural Networks // Proc. Intern. Conf. Learning Representations. 2018. Pp. 1—14.
---
Для цитирования: Мамонтов А.И. О способах экономии компьютерной памяти при классификации данных с помощью полносвязных нейронных сетей // Вестник МЭИ. 2021.
№ 3. С. 103—109. DOI: 10.24160/1993-6982-2021-3-103-109.
---
Работа выполнена при поддержке: РФФИ (проект № 19-01-00294)
#
1. Anguita D., Ghio A., Pischiutta S., Ridella S. A Support Vector Machine with Integer Parameters. Neurocomputing. 2008;72(1—3):480—489.
2. Tschiatschek S., Paul K., Pernkopf F. Integer Bayesian Network Classifiers. Machine Learning and Knowledge Discovery in Databases. Lecture Notes in Computer Sci. 2014;8726:209—224.
3. Han S., Mao H., Dally W.J. Deep Compression: Compressing Deep Neural Network with Pruning, Trained Quantization and Huffman Coding. Proc. Intern. Conf. Learning Representations. 2016:1—14.
4. Liu B., Wang M., Foroosh H., Tappen M., Penksy M. Sparse Convolutional Neural Networks. IEEE Conf. Computer Vision and Pattern Recognition. 2015:806—814.
5. Solov'ev R.A., Kustov A.G., Tel'pukhov D.V., Rukhlov V.S. Prototipirovanie Vysokoskorostnoy Neyronnoy Seti v PLIS dlya Klassifikatsii Izobrazheniy Videopotoka. Cloud of Sci. 2018;4:680—703. (in Russian).
6. Aleksiadis N.F. Algoritmicheskaya Nerazreshimost' Zadachi o Nakhozhdenii Bazisa Konechnoy Polnoy Sistemy Polinomov s Tselymi Koeffitsientami. Intellektual'nye Sistemy. Teoriya i Prilozheniya. 2016;20;3:19—23. (in Russian).
7. Mamontov A.I., Meshchaninov D.G. Problema Polnoty v Funktsional'noy Sisteme Lineynykh Polinomov s Tselymi Koeffitsientami. Diskretnaya Matematika. 2010;22;4:64—82. (in Russian).
8. Mamontov A.I., Meshchaninov D.G. Algoritm Raspoznavaniya Polnoty v Funktsional'noy Sisteme L(Z). Diskretnaya Matematika. 2014;26;1:85—95. (in Russian).
9. Mamontov A.I. O Povyshenii Effektivnosti Vychisleniy pri Klassifikatsii Izobrazheniy. Vestnik MEI. 2019;5:129—134. (in Russian).
10. Mamontov A.I. O Sposobakh Ekonomii Pamyati pri Reshenii Zadach Klassifikatsii Lineynym Metodom Opornykh Vektorov. Vestnik MEI. 2020;4:129—135. (in Russian).
11. Gorban A.N., Wunsch II D.C. The General Approximation Theorem. Proc. Intern. Joint Conf. Neural Networks. 1998:1271—1274.
12. Gorban' A.N. Obobshchennaya Approksimatsionnaya Teorema i Vychislitel'nye Vozmozhnosti Neyronnykh Setey. Sibirskiy Zhurnal Vychislitel'noy Matematiki. 1998;1;1:11—24. (in Russian).
13. Polovnikov V.S. O Nelineynykh Kharakteristikakh Neyronnykh Skhem v Proizvol'nykh Bazisakh. Intellektual'nye Sistemy. 2013;17;1—4:87—90. (in Russian).
14. Anguita D., Pischiutta S., Ridella S., Sterpi D. Feed-forward SVM without Multipliers. IEEE Trans. Neural Networks. 2006;17:1328—1331.
15. Anguita D., Boni A., Ridella S. A Digital Architecture for Support Vector Machines: Theory, Algorithm and FPGA Implementation. IEEE Trans. Neural Networks. 2003;14:993—1009.
16. Anguita D., Bozza G. The effects of quantization on support vector machines with Gaussian kernel. IEEE Intern. Joint Conf. Neural Networks. 2005;2:681—684.
17. Lesser B., Mücke M., Gansterer W.N. Effects of Reduced Precision on Floating-point SVM Classification Accuracy. Procedia Computer Sci. 2011;4:508—517.
18. Tschiatschek S., Pernkopf F. On Bayesian Network Classifiers with Reduced Precision Parameters. IEEE Trans. Pattern Analysis and Machine Intelligence. 2015;37(4):774—785.
19. Yi Y., Hangping Z., Bin Z. A New Learning Algorithm for Neural Networks with Integer Weights and Quantized Non-linear Activation Functions. Artificial Intelligence in Theory and Practice. 2008;276:427—431.
20. Shuang Wu, Guoqi Li, Feng Chen, Luping Shi. Training and Inference with Integers in Deep Neural Networks. Proc. Intern. Conf. Learning Representations. 2018:1—14.
---
For citation: Mamontov A.I. On Computer Memory Saving Methods in Performing Data Classification Using Fully Connected Neural Networks. Bulletin of MPEI. 2021;3:103—109. (in Russian). DOI: 10.24160/1993-6982-2021-3-103-109.
---
The work is executed at support: RFBR (Project No. 19-01-00294)
Downloads
Published
2020-11-12
Issue
Section
Theoretical Foundations of Computer Science (05.13.17)
