Применение нейронных сетей для аппроксимации Q-функции в методах обучения с подкреплением
Ключевые слова:
интеллектуальная система, обучение с подкреплением, нейронные сети
Аннотация
Рассмотрена возможность интеграции методов обучения с подкреплением и искусственных нейронных сетей. Представлена реализация методов аппроксимации Q-функции в методах машинного обучения с подкреплением. Методы выполнены на основе классической нейронной сети и самоорганизующейся нечеткой нейронной сети. Описан процесс автоматической настройки самоорганизующейся нейронной сети. Даны результаты тестирования методов аппроксимации на модельных задачах стабилизации перевернутого маятника и горного автомобиля. Работа выполнена в рамках исследований методов обучения с подкреплением, применяемых в системах искусственного интеллекта различного назначения, в том числе в интеллектуальных системах поддержки принятия решений реального времени. Приведены основные преимущества и недостатки подходов аппроксимации с помощью нейронных сетей Q-функции в методах машинного обучения с подкреплением.
Литература
1. Andrychowicz M., Baker B. Learning Dexterous In-hand Manipulation [Электрон. ресурс] http://phys2018.csail.mit.edu/papers/4.pdf (дата обращения 04.08.2025).
2. Silver D. e. a. Mastering the Game of go with Deep Neural Networks and Tree Search // Nature. 2016. V. 529. Pp. 484—489.
3. Gatti C.J., Embrechts M.J. Reinforcement Learning with Neural Networks: Tricks of the Trade // Advances in Intelligent Signal Processing and Data Mining. Studies in Computational Intelligence. Berlin: Springer, 2013. V. 410.
4. Ramkumar J., Ravindran D. Machine Learning and Robotics in Urban Traffic Flow Optimization with Graph Neural Networks and Reinforcement Learning // Machine Learning and Robotics in Urban Planning and Management. N.-Y.: IGI Global Scientific Publ., 2025. Pp. 83—104.
5. Prudencio R.F., Maximo M., Colombini E.L. A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems // IEEE Trans. Neural Networks and Learning Systems. 2023. V. 99. P. 1.
6. Саттон Р.С., Барто Э.Г. Обучение с подкреплением. М.: Бином. Лаборатория знаний, 2017.
7. François-Lavet V. е. а. An Introduction to Deep Reinforcement Learning // Foundations and Trends in Machine Learning. 2018. V. 11(3—4). Pp. 219—354.
8. Khakimov P., Savin S., Klimchik A. Trajectory Optimization for Underactuated Systems Using Reinforcement Learning: Cart-pole Problem // Proc. III School on Dynamics of Complex Networks and Their Application in Intellectual Robotics. Innopolis, 2019. Pp. 71—73.
9. Еремеев А.П., Полюшкин И.А., Сергеев М.Д. Сравнительный анализ гибких алгоритмов поиска решения для интеллектуальных систем реального времени // Интегрированные модели и мягкие вычисления в искусственном интеллекте: Сб. научных трудов X Междунар. науч.-техн. конф. Т. 2. Смоленск: Универсум, 2021. С. 115—122.
10. Еремеев А.П., Герасимова А.Е., Кожухов А.А. Сравнительный анализ методов машинного обучения с подкреплением применительно к системам реального времени // Интеллектуальные информационные технологии: Труды Междунар. науч.-техн. конгресса. Т. 1. Таганрог: Изд-во Ступина С.А., 2019.
11. Eremeev A.P. e. a. Prototype of Intelligent real-Time Decision Support System Based on Anytime Algorithms and NO-SQL Database // Proc. III Intern. Youth Conf. Radio Electronics, Electrical and Power Eng. 2021. Pр. 1—6.
12. Eremeev A.P., Poliushkin I.A., Sergeev M.D. Instrumental Software Environment for Teaching Students the Technology of Designing Intelligent Decision Support Systems // Proc. VI Intern. Conf. Information Technol. Engineering Education (Inforino). Moscow, 2022. Pp. 1—4.
13. Casillas J. e. a. Interpretability Issues in Fuzzy Modeling. N.-Y.: Springer, 2013.
14. Muslimi B. e. a. An Efficient Technique for Extracting Fuzzy Rules from Neural Networks // Intern. J. Electrical and Computer Eng. 2008. V. 2(4). Pp. 1231—1237.
15. Vashishtha Kritika, Anas Saad, Fengfeng Xi, Intelligent Adaptive Lightingalgorithm: Integrating Reinforcement Learning and Fuzzy Logic for Personal-ized Interior Lighting // Engineering Appl. of Artificial Intelligence. 2024. V. 133(3). Pp. 108512.
16. Kasabov N.K., Song Q. DENFIS: Dynamic Evolving Neural-fuzzy Inference System and Its Application for Time-series Prediction // IEEE Trans. Fuzzy Systems. 2002. V. 2(10). Pp. 144—154.
17. Angelov P., Gu X. Empirical Fuzzy Sets // Intern. J. Intelligent Systems. 2017. V. 33. Pp. 135—155.
18. Kim J., Kasabov N. HyFIS: Adaptive Neuro-fuzzy Inference Systems and Their Application to Nonlinear Dynamical Systems // Neural Networks. 1999. V. 12. Pp. 1301—1319.
19. Kosko B. Fuzzy Systems as Universal Approximators // IEEE Trans. Comput. 1994. V. 43. Pp. 1329—1333.
20. Lindskog P. Fuzzy Identification from a Grey Box Modeling Point of View. Berlin: Springer, 1997. Pp. 3—50.
---
Для цитирования: Еремеев А.П., Сергеев М.Д. Применение нейронных сетей для аппроксимации Q-функции в методах обучения с подкреплением // Вестник МЭИ. 2025. № 5. С. 160—166. DOI: 10.24160/1993-6982-2025-5-160-166
---
Работа выполнена при поддержке Российского научного фонда (проект № 24-11-00285) https://rscf.ru/project/24-11-00285/
---
Конфликт интересов: авторы заявляют об отсутствии конфликта интересов
#
1. Andrychowicz M., Baker B. Learning Dexterous In-hand Manipulation [Elektron. Resurs] http://phys2018.csail.mit.edu/papers/4.pdf (Data Obrashcheniya 04.08.2025).
2. Silver D. e. a. Mastering the Game of go with Deep Neural Networks and Tree Search. Nature. 2016;529:484—489.
3. Gatti C.J., Embrechts M.J. Reinforcement Learning with Neural Networks: Tricks of the Trade. Advances in Intelligent Signal Processing and Data Mining. Studies in Computational Intelligence. Berlin: Springer, 2013;410.
4. Ramkumar J., Ravindran D. Machine Learning and Robotics in Urban Traffic Flow Optimization with Graph Neural Networks and Reinforcement Learning. Machine Learning and Robotics in Urban Planning and Management. N.-Y.: IGI Global Scientific Publ., 2025:83—104.
5. Prudencio R.F., Maximo M., Colombini E.L. A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems. IEEE Trans. Neural Networks and Learning Systems. 2023;99”1.
6. Satton R.S., Barto E.G. Obuchenie s Podkrepleniem. M.: Binom. Laboratoriya Znaniy, 2017. (in Russian).
7. François-Lavet V. е. а. An Introduction to Deep Reinforcement Learning. Foundations and Trends in Machine Learning. 2018;11(3—4):219—354.
8. Khakimov P., Savin S., Klimchik A. Trajectory Optimization for Underactuated Systems Using Reinforcement Learning: Cart-pole Problem. Proc. III School on Dynamics of Complex Networks and Their Application in Intellectual Robotics. Innopolis, 2019:71—73.
9. Eremeev A.P., Polyushkin I.A., Sergeev M.D. Sravnitel'nyy Analiz Gibkikh Algoritmov Poiska Resheniya dlya Intellektual'nykh Sistem Real'nogo Vremeni. Integrirovannye Modeli i Myagkie Vychisleniya v Iskusstvennom Intellekte: Sb. Nauchnykh Trudov X Mezhdunar. Nauch.-tekhn. Konf. T. 2. Smolensk: Universum, 2021:115—122. (in Russian).
10. Eremeev A.P., Gerasimova A.E., Kozhukhov A.A. Sravnitel'nyy Analiz Metodov Mashinnogo Obucheniya s Podkrepleniem Primenitel'no k Sistemam Real'nogo Vremeni. Intellektual'nye Informatsionnye Tekhnologii: Trudy Mezhdunar. Nauch.-tekhn. Kongressa. T. 1. Taganrog: Izd-vo Stupina S.A., 2019. (in Russian).
11. Eremeev A.P. e. a. Prototype of Intelligent real-Time Decision Support System Based on Anytime Algorithms and NO-SQL Database. Proc. III Intern. Youth Conf. Radio Electronics, Electrical and Power Eng. 2021:1—6.
12. Eremeev A.P., Poliushkin I.A., Sergeev M.D. Instrumental Software Environment for Teaching Students the Technology of Designing Intelligent Decision Support Systems. Proc. VI Intern. Conf. Information Technol. Engineering Education (Inforino). Moscow, 2022:1—4.
13. Casillas J. e. a. Interpretability Issues in Fuzzy Modeling. N.-Y.: Springer, 2013.
14. Muslimi B. e. a. An Efficient Technique for Extracting Fuzzy Rules from Neural Networks. Intern. J. Electrical and Computer Eng. 2008;2(4):1231—1237.
15. Vashishtha Kritika, Anas Saad, Fengfeng Xi, Intelligent Adaptive Lightingalgorithm: Integrating Reinforcement Learning and Fuzzy Logic for Personal-ized Interior Lighting. Engineering Appl. of Artificial Intelligence. 2024;133(3):108512.
16. Kasabov N.K., Song Q. DENFIS: Dynamic Evolving Neural-fuzzy Inference System and Its Application for Time-series Prediction. IEEE Trans. Fuzzy Systems. 2002;2(10):144—154.
17. Angelov P., Gu X. Empirical Fuzzy Sets. Intern. J. Intelligent Systems. 2017;33:135—155.
18. Kim J., Kasabov N. HyFIS: Adaptive Neuro-fuzzy Inference Systems and Their Application to Nonlinear Dynamical Systems. Neural Networks. 1999;12:1301—1319.
19. Kosko B. Fuzzy Systems as Universal Approximators. IEEE Trans. Comput. 1994;43:1329—1333.
20. Lindskog P. Fuzzy Identification from a Grey Box Modeling Point of View. Berlin: Springer, 1997:3—50
---
For citation: Eremeev A.P., Sergeev M.D. The Use of Neural Networks to Approximate the Q-function in Reinforcement Learning Methods. Bulletin of MPEI. 2025;5:160—166. (in Russian). DOI: 10.24160/1993-6982-2025-5-160-166
---
The Work was Carried Out Russian Science Foundation (Project No. 24-11-00285) https://rscf.ru/project/24-11-00285/
---
Conflict of interests: the authors declare no conflict of interest
2. Silver D. e. a. Mastering the Game of go with Deep Neural Networks and Tree Search // Nature. 2016. V. 529. Pp. 484—489.
3. Gatti C.J., Embrechts M.J. Reinforcement Learning with Neural Networks: Tricks of the Trade // Advances in Intelligent Signal Processing and Data Mining. Studies in Computational Intelligence. Berlin: Springer, 2013. V. 410.
4. Ramkumar J., Ravindran D. Machine Learning and Robotics in Urban Traffic Flow Optimization with Graph Neural Networks and Reinforcement Learning // Machine Learning and Robotics in Urban Planning and Management. N.-Y.: IGI Global Scientific Publ., 2025. Pp. 83—104.
5. Prudencio R.F., Maximo M., Colombini E.L. A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems // IEEE Trans. Neural Networks and Learning Systems. 2023. V. 99. P. 1.
6. Саттон Р.С., Барто Э.Г. Обучение с подкреплением. М.: Бином. Лаборатория знаний, 2017.
7. François-Lavet V. е. а. An Introduction to Deep Reinforcement Learning // Foundations and Trends in Machine Learning. 2018. V. 11(3—4). Pp. 219—354.
8. Khakimov P., Savin S., Klimchik A. Trajectory Optimization for Underactuated Systems Using Reinforcement Learning: Cart-pole Problem // Proc. III School on Dynamics of Complex Networks and Their Application in Intellectual Robotics. Innopolis, 2019. Pp. 71—73.
9. Еремеев А.П., Полюшкин И.А., Сергеев М.Д. Сравнительный анализ гибких алгоритмов поиска решения для интеллектуальных систем реального времени // Интегрированные модели и мягкие вычисления в искусственном интеллекте: Сб. научных трудов X Междунар. науч.-техн. конф. Т. 2. Смоленск: Универсум, 2021. С. 115—122.
10. Еремеев А.П., Герасимова А.Е., Кожухов А.А. Сравнительный анализ методов машинного обучения с подкреплением применительно к системам реального времени // Интеллектуальные информационные технологии: Труды Междунар. науч.-техн. конгресса. Т. 1. Таганрог: Изд-во Ступина С.А., 2019.
11. Eremeev A.P. e. a. Prototype of Intelligent real-Time Decision Support System Based on Anytime Algorithms and NO-SQL Database // Proc. III Intern. Youth Conf. Radio Electronics, Electrical and Power Eng. 2021. Pр. 1—6.
12. Eremeev A.P., Poliushkin I.A., Sergeev M.D. Instrumental Software Environment for Teaching Students the Technology of Designing Intelligent Decision Support Systems // Proc. VI Intern. Conf. Information Technol. Engineering Education (Inforino). Moscow, 2022. Pp. 1—4.
13. Casillas J. e. a. Interpretability Issues in Fuzzy Modeling. N.-Y.: Springer, 2013.
14. Muslimi B. e. a. An Efficient Technique for Extracting Fuzzy Rules from Neural Networks // Intern. J. Electrical and Computer Eng. 2008. V. 2(4). Pp. 1231—1237.
15. Vashishtha Kritika, Anas Saad, Fengfeng Xi, Intelligent Adaptive Lightingalgorithm: Integrating Reinforcement Learning and Fuzzy Logic for Personal-ized Interior Lighting // Engineering Appl. of Artificial Intelligence. 2024. V. 133(3). Pp. 108512.
16. Kasabov N.K., Song Q. DENFIS: Dynamic Evolving Neural-fuzzy Inference System and Its Application for Time-series Prediction // IEEE Trans. Fuzzy Systems. 2002. V. 2(10). Pp. 144—154.
17. Angelov P., Gu X. Empirical Fuzzy Sets // Intern. J. Intelligent Systems. 2017. V. 33. Pp. 135—155.
18. Kim J., Kasabov N. HyFIS: Adaptive Neuro-fuzzy Inference Systems and Their Application to Nonlinear Dynamical Systems // Neural Networks. 1999. V. 12. Pp. 1301—1319.
19. Kosko B. Fuzzy Systems as Universal Approximators // IEEE Trans. Comput. 1994. V. 43. Pp. 1329—1333.
20. Lindskog P. Fuzzy Identification from a Grey Box Modeling Point of View. Berlin: Springer, 1997. Pp. 3—50.
---
Для цитирования: Еремеев А.П., Сергеев М.Д. Применение нейронных сетей для аппроксимации Q-функции в методах обучения с подкреплением // Вестник МЭИ. 2025. № 5. С. 160—166. DOI: 10.24160/1993-6982-2025-5-160-166
---
Работа выполнена при поддержке Российского научного фонда (проект № 24-11-00285) https://rscf.ru/project/24-11-00285/
---
Конфликт интересов: авторы заявляют об отсутствии конфликта интересов
#
1. Andrychowicz M., Baker B. Learning Dexterous In-hand Manipulation [Elektron. Resurs] http://phys2018.csail.mit.edu/papers/4.pdf (Data Obrashcheniya 04.08.2025).
2. Silver D. e. a. Mastering the Game of go with Deep Neural Networks and Tree Search. Nature. 2016;529:484—489.
3. Gatti C.J., Embrechts M.J. Reinforcement Learning with Neural Networks: Tricks of the Trade. Advances in Intelligent Signal Processing and Data Mining. Studies in Computational Intelligence. Berlin: Springer, 2013;410.
4. Ramkumar J., Ravindran D. Machine Learning and Robotics in Urban Traffic Flow Optimization with Graph Neural Networks and Reinforcement Learning. Machine Learning and Robotics in Urban Planning and Management. N.-Y.: IGI Global Scientific Publ., 2025:83—104.
5. Prudencio R.F., Maximo M., Colombini E.L. A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems. IEEE Trans. Neural Networks and Learning Systems. 2023;99”1.
6. Satton R.S., Barto E.G. Obuchenie s Podkrepleniem. M.: Binom. Laboratoriya Znaniy, 2017. (in Russian).
7. François-Lavet V. е. а. An Introduction to Deep Reinforcement Learning. Foundations and Trends in Machine Learning. 2018;11(3—4):219—354.
8. Khakimov P., Savin S., Klimchik A. Trajectory Optimization for Underactuated Systems Using Reinforcement Learning: Cart-pole Problem. Proc. III School on Dynamics of Complex Networks and Their Application in Intellectual Robotics. Innopolis, 2019:71—73.
9. Eremeev A.P., Polyushkin I.A., Sergeev M.D. Sravnitel'nyy Analiz Gibkikh Algoritmov Poiska Resheniya dlya Intellektual'nykh Sistem Real'nogo Vremeni. Integrirovannye Modeli i Myagkie Vychisleniya v Iskusstvennom Intellekte: Sb. Nauchnykh Trudov X Mezhdunar. Nauch.-tekhn. Konf. T. 2. Smolensk: Universum, 2021:115—122. (in Russian).
10. Eremeev A.P., Gerasimova A.E., Kozhukhov A.A. Sravnitel'nyy Analiz Metodov Mashinnogo Obucheniya s Podkrepleniem Primenitel'no k Sistemam Real'nogo Vremeni. Intellektual'nye Informatsionnye Tekhnologii: Trudy Mezhdunar. Nauch.-tekhn. Kongressa. T. 1. Taganrog: Izd-vo Stupina S.A., 2019. (in Russian).
11. Eremeev A.P. e. a. Prototype of Intelligent real-Time Decision Support System Based on Anytime Algorithms and NO-SQL Database. Proc. III Intern. Youth Conf. Radio Electronics, Electrical and Power Eng. 2021:1—6.
12. Eremeev A.P., Poliushkin I.A., Sergeev M.D. Instrumental Software Environment for Teaching Students the Technology of Designing Intelligent Decision Support Systems. Proc. VI Intern. Conf. Information Technol. Engineering Education (Inforino). Moscow, 2022:1—4.
13. Casillas J. e. a. Interpretability Issues in Fuzzy Modeling. N.-Y.: Springer, 2013.
14. Muslimi B. e. a. An Efficient Technique for Extracting Fuzzy Rules from Neural Networks. Intern. J. Electrical and Computer Eng. 2008;2(4):1231—1237.
15. Vashishtha Kritika, Anas Saad, Fengfeng Xi, Intelligent Adaptive Lightingalgorithm: Integrating Reinforcement Learning and Fuzzy Logic for Personal-ized Interior Lighting. Engineering Appl. of Artificial Intelligence. 2024;133(3):108512.
16. Kasabov N.K., Song Q. DENFIS: Dynamic Evolving Neural-fuzzy Inference System and Its Application for Time-series Prediction. IEEE Trans. Fuzzy Systems. 2002;2(10):144—154.
17. Angelov P., Gu X. Empirical Fuzzy Sets. Intern. J. Intelligent Systems. 2017;33:135—155.
18. Kim J., Kasabov N. HyFIS: Adaptive Neuro-fuzzy Inference Systems and Their Application to Nonlinear Dynamical Systems. Neural Networks. 1999;12:1301—1319.
19. Kosko B. Fuzzy Systems as Universal Approximators. IEEE Trans. Comput. 1994;43:1329—1333.
20. Lindskog P. Fuzzy Identification from a Grey Box Modeling Point of View. Berlin: Springer, 1997:3—50
---
For citation: Eremeev A.P., Sergeev M.D. The Use of Neural Networks to Approximate the Q-function in Reinforcement Learning Methods. Bulletin of MPEI. 2025;5:160—166. (in Russian). DOI: 10.24160/1993-6982-2025-5-160-166
---
The Work was Carried Out Russian Science Foundation (Project No. 24-11-00285) https://rscf.ru/project/24-11-00285/
---
Conflict of interests: the authors declare no conflict of interest
Опубликован
2025-09-15
Выпуск
Раздел
Информатика и информационные процессы (технические науки) (2.3.8)
