Multimodal Information Integration Method for Road Traffic Accident Detection and Prediction by the Neural Network Method
DOI:
https://doi.org/10.24160/1993-6982-2025-3-153-160Keywords:
traffic accident detection, prediction of frames and location, scene context, adversarial learning, computer vision, machine learning, artificial intelligence, artificial neural networksAbstract
A method for detecting road traffic accidents (RTA) from dashcam videos is proposed. The method utilizes neural networks and does not require labeled data for training, as it employs unsupervised learning. An RTA is regarded as an abnormal situation—out of distribution—since accidents occur quite rarely. A difference between the predicted frame and the actually observed frame is used as the criterion for detecting abnormal situations. The frames are compared on the basis of three criteria: pixel-wise comparison of a frame, overlap of highlighted areas of RTA participants, and similarity of the contextual representation of the entire traffic scene. The novelty of the work lies in the combination of different criteria for comparing the scene. A method for analyzing the scene context using a graph neural networks and generative adversarial models has been proposed for the first time.
The proposed method consists of four basic blocks: an image processing block, an optical flow processing block, an RTA participants detection (bounding box) block, and a scene context analysis block. In the first three blocks, pre-trained encoders are used to extract features (embeddings) that are utilized to predict the subsequent frame. The predicted frame, optical flow, and detection blocks are fed into a graph neural network, in which all modalities are combined. Using the generative adversarial model GAN, the output of the graph neural network for the predicted frame is modified to make it minimally different from the contextual features of the real frame. If such difference is still significant, it points to a high probability of a road traffic accident.
References
1. World Health Organization. Global Status Report on Road Safety [Электрон. ресурс] https://www.who.int/publications/i/item/9789240086517 (дата обращения 20.01.2025).
2. Kholmatov U.S. e. a. Causing Factors of Road Transport Incidents in Traffic // Intern. J. Education, Social Sci. & Humanities. 2024. V. 12(5). Pp. 1524—1534.
3. Ghahremannezhad H., Shi H., Liu C. A Real Time Accident Detection Framework for Traffic Video Analysis // Proc. XVI Intern. Conf. Machine Learning and Data Mining. 2020. Pp. 77—92.
4. Sun Y., Mallick T., Balaprakash P., Macfarlane J. A Datacentric Weak Supervised Learning for Highway Traffic Incident Detection // Accident Analysis and Prevention. 2022. V. 176. P. 106779.
5. Sherimon V. e. a. An Overview of Different Deep Learning Techniques Used in Road Accident Detection // Intern. J. Advanced Computer Sci. & Appl. 2023. V. 14(11). P. 0141144.
6. Shunsuke Kamijo e. a. Traffic Monitoring and Accident Detection at Intersections // IEEE Trans. Intelligent Transportation Syst. 2000. V. 1. Pp. 108—118.
7. Xu Y. Traffic Incident Detection Based on HMM // Proc. IEEE Third Intern. Conf. Information Sci. and Technol. Yangzhou, 2013. Pp. 942—945.
8. Kimin Yun e. a. Motion Interaction Field for Accident Detection in Traffic Surveillance Video // Proc. XXII Intern. Conf. Pattern Recognition. 2014. Pp. 3062—3067.
9. Dogru N, Subasi A. Traffic Accident Detection Using Random Forest Classifier // Proc. XV Learning and Technol. Conf. 2018. Pp. 40—45.
10. Haresh S. e. a. Towards Anomaly Detection in Dashcam Videos // Proc. IEEE IV Intelligent Vehicles Symp. 2020. Pp. 1407—1414.
11. Li S., Fang J., Xu H., Xue J. Video Frame Prediction by Deep Multi-branch Mask Network // IEEE Trans. Circuits Syst. Video Technol. 2021. V. 31(4). Pp. 1283—1295.
12. Hasan M. e. a. Learning Temporal Regularity in Video Sequences // Proc. IEEE Conf. Computer Vision and Pattern Recognition. 2016. Pp. 733—742.
13. Liu W. e a. Future Frame Prediction for Anomaly Detection — a New Baseline // Proc. IEEE Conf. Computer Vision and Pattern Recognition. 2018. Pp. 6536—6545.
14. Yu Yao e. a. DoTA: Unsupervised Detection of Traffic Anomaly in Driving Videos // IEEE Trans. Pattern Anal. Mach. Intell. 2023. V. 45(1). Pp. 444—459.
15. Васюк М.А., Карпета Т.В. Алгоритм обнаружения дорожно-транспортных происшествий на перекрестках в режиме реального времени с помощью нейронной сети YOLOv4 // Южно-Уральская молодежная школа по математическому моделированию: Сб. трудов IV Всеросс. науч.-техн. конф. Челябинск: Издат. центр ЮУрГУ, 2021. С. 51—58.
16. Ротарь В.Г., Беляев С.И. Обнаружение автомобильных аварий в реальном времени с применением нейросети архитектуры YOLOv3 // Молодежь и современные информационные технологии: Сб. трудов XVIII Междунар. науч.-практ. конф. студентов, аспирантов и молодых учёных. Томск: Томский политехн. ун-т, 2021. С. 49—50.
17. Hajri F., Fradi H. Vision Transformers for Road Accident Detection from Dashboard Cameras // Proc. XVIII Intern. Conf. Advances Video and Signal Based Surveillance. 2022. Pp. 1—8.
18. Khan S. e. a. Transformers in Vision: a Survey // ACM Comput. Surv. 2022. V. 54(10). Pp. 1—41.
19. Yu Yao e. a. Unsupervised Traffic Accident Detection in First-person Videos // Proc. IEEE Intern. Conf. Intelligent Robots and Systems. Macau, 2019. Pp. 273—280.
20. Girshick R. Fast R-CNN // Proc. IEEE Intern. Conf. Computer Vision. 2015. Pp. 1440—1448.
21. Wojke N., Bewley A., Paulus D. Simple Online and Realtime Tracking with a Deep Association Metric // Proc. IEEE Intern. Conf. Image Proc. Beijing, 2017. Pp. 3645—3649.
22. He K. e. a. Mask R-CNN // IEEE Intern. Conf. Computer Vision. 2017. Pp. 2980—2988.
23. Kipf T.N., Welling M. Semi-supervised Classification with Graph Convolutional Networks // Proc. Intern. Conf. Learning Representations. 2017. Pp. 1—14.
24. Goodfellow I. e. a. Generative Adversarial Networks // Communications of the ACM. 2020. V. 63(11). Pp. 139—144.
25. Schafer R.W. What is a Savitzky-Golay filter? // IEEE Signal Process. Mag. 2011. V. 28(4). Pp. 111—117.
26. Fang J. e. a. DADA-2000: Can Driving Accident be Predicted by Driver Attention? Analyzed by a Benchmark // Proc. IEEE Intelligent Transportation Syst. Conf. Aucklend, 2019. Pp. 4303—4309.
27. Yao Y. e. a. Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systems // Proc. Intern. Conf. Robotics and Automation. Montreal, 2019. Pp. 9711—9717.
---
цитирования: Цяо Цзяхуань, Кружилов И.С., Шамаева О.Ю. Метод интеграции мультимодальной информации для обнаружения и прогнозирования дорожно-транспортных происшествий на основе нейросетевого подхода // Вестник МЭИ. 2025. № 3. С. 153—160. DOI: 10.24160/1993-6982-2025-3-153-160
---
Конфликт интересов: авторы заявляют об отсутствии конфликта интересов
#
1. World Health Organization. Global Status Report on Road Safety [Elektron. Resurs] https://www.who.int/publications/i/item/9789240086517 (Data Obrashcheniya 20.01.2025).
2. Kholmatov U.S. e. a. Causing Factors of Road Transport Incidents in Traffic. Intern. J. Education, Social Sci. & Humanities. 2024;12(5):1524—1534.
3. Ghahremannezhad H., Shi H., Liu C. A Real Time Accident Detection Framework for Traffic Video Analysis. Proc. XVI Intern. Conf. Machine Learning and Data Mining. 2020:77—92.
4. Sun Y., Mallick T., Balaprakash P., Macfarlane J. A Datacentric Weak Supervised Learning for Highway Traffic Incident Detection. Accident Analysis and Prevention. 2022;176:106779.
5. Sherimon V. e. a. An Overview of Different Deep Learning Techniques Used in Road Accident Detection. Intern. J. Advanced Computer Sci. & Appl. 2023;14(11):0141144.
6. Shunsuke Kamijo e. a. Traffic Monitoring and Accident Detection at Intersections. IEEE Trans. Intelligent Transportation Syst. 2000;1:108—118.
7. Xu Y. Traffic Incident Detection Based on HMM. Proc. IEEE Third Intern. Conf. Information Sci. and Technol. Yangzhou, 2013:942—945.
8. Kimin Yun e. a. Motion Interaction Field for Accident Detection in Traffic Surveillance Video. Proc. XXII Intern. Conf. Pattern Recognition. 2014:3062—3067.
9. Dogru N, Subasi A. Traffic Accident Detection Using Random Forest Classifier. Proc. XV Learning and Technol. Conf. 2018:40—45.
10. Haresh S. e. a. Towards Anomaly Detection in Dashcam Videos. Proc. IEEE IV Intelligent Vehicles Symp. 2020:1407—1414.
11. Li S., Fang J., Xu H., Xue J. Video Frame Prediction by Deep Multi-branch Mask Network. IEEE Trans. Circuits Syst. Video Technol. 2021;31(4):1283—1295.
12. Hasan M. e. a. Learning Temporal Regularity in Video Sequences. Proc. IEEE Conf. Computer Vision and Pattern Recognition. 2016:733—742.
13. Liu W. e a. Future Frame Prediction for Anomaly Detection — a New Baseline. Proc. IEEE Conf. Computer Vision and Pattern Recognition. 2018:6536—6545.
14. Yu Yao e. a. DoTA: Unsupervised Detection of Traffic Anomaly in Driving Videos. IEEE Trans. Pattern Anal. Mach. Intell. 2023;45(1):444—459.
15. Vasyuk M.A., Karpeta T.V. Algoritm Obnaruzheniya Dorozhno-transportnykh Proisshestviy na Perekrestkakh v Rezhime Real'nogo Vremeni s Pomoshch'yu Neyronnoy Seti Yolov4. Yuzhno-Ural'skaya Molodezhnaya Shkola po Matematicheskomu Modelirovaniyu: Sb. Trudov IV Vseross. Nauch.-tekhn. Konf. Chelyabinsk: Izdat. Tsentr YUUrGU, 2021:51—58. (in Russian).
16. Rotar' V.G., Belyaev S.I. Obnaruzhenie Avtomobil'nykh Avariy v Real'nom Vremeni s Primeneniem Neyroseti Arkhitektury YOLOv3. Molodezh' i Sovremennye Informatsionnye Tekhnologii: Sb. Trudov XVIII Mezhdunar. Nauch.-prakt. Konf. Studentov, Aspirantov i Molodykh Uchenykh. Tomsk: Tomskiy Politekhn. Un-t, 2021:49—50. (in Russian).
17. Hajri F., Fradi H. Vision Transformers for Road Accident Detection from Dashboard Cameras. Proc. XVIII Intern. Conf. Advances Video and Signal Based Surveillance. 2022:1—8.
18. Khan S. e. a. Transformers in Vision: a Survey. ACM Comput. Surv. 2022;54(10):1—41.
19. Yu Yao e. a. Unsupervised Traffic Accident Detection in First-person Videos. Proc. IEEE Intern. Conf. Intelligent Robots and Systems. Macau, 2019:273—280.
20. Girshick R. Fast R-CNN. Proc. IEEE Intern. Conf. Computer Vision. 2015:1440—1448.
21. Wojke N., Bewley A., Paulus D. Simple Online and Realtime Tracking with a Deep Association Metric. Proc. IEEE Intern. Conf. Image Proc. Beijing, 2017:3645—3649.
22. He K. e. a. Mask R-CNN. IEEE Intern. Conf. Computer Vision. 2017:2980—2988.
23. Kipf T.N., Welling M. Semi-supervised Classification with Graph Convolutional Networks. Proc. Intern. Conf. Learning Representations. 2017:1—14.
24. Goodfellow I. e. a. Generative Adversarial Networks. Communications of the ACM. 2020;63(11):139—144.
25. Schafer R.W. What is a Savitzky-Golay filter?. IEEE Signal Process. Mag. 2011;28(4):111—117.
26. Fang J. e. a. DADA-2000: Can Driving Accident be Predicted by Driver Attention? Analyzed by a Benchmark. Proc. IEEE Intelligent Transportation Syst. Conf. Aucklend, 2019:4303—4309.
27. Yao Y. e. a. Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systems. Proc. Intern. Conf. Robotics and Automation. Montreal, 2019:9711—9717
---
For citation: Qiao Jiahuan, Kruzhilov I.S., Shamaeva O.Yu. Multimodal Information Integration Method for Road Traffic Accident Detection and Prediction by the Neural Network Method. Bulletin of MPEI. 2025;3:153—160. (in Russian). DOI: 10.24160/1993-6982-2025-3-153-160
---
Conflict of interests: the authors declare no conflict of interest
2. Kholmatov U.S. e. a. Causing Factors of Road Transport Incidents in Traffic // Intern. J. Education, Social Sci. & Humanities. 2024. V. 12(5). Pp. 1524—1534.
3. Ghahremannezhad H., Shi H., Liu C. A Real Time Accident Detection Framework for Traffic Video Analysis // Proc. XVI Intern. Conf. Machine Learning and Data Mining. 2020. Pp. 77—92.
4. Sun Y., Mallick T., Balaprakash P., Macfarlane J. A Datacentric Weak Supervised Learning for Highway Traffic Incident Detection // Accident Analysis and Prevention. 2022. V. 176. P. 106779.
5. Sherimon V. e. a. An Overview of Different Deep Learning Techniques Used in Road Accident Detection // Intern. J. Advanced Computer Sci. & Appl. 2023. V. 14(11). P. 0141144.
6. Shunsuke Kamijo e. a. Traffic Monitoring and Accident Detection at Intersections // IEEE Trans. Intelligent Transportation Syst. 2000. V. 1. Pp. 108—118.
7. Xu Y. Traffic Incident Detection Based on HMM // Proc. IEEE Third Intern. Conf. Information Sci. and Technol. Yangzhou, 2013. Pp. 942—945.
8. Kimin Yun e. a. Motion Interaction Field for Accident Detection in Traffic Surveillance Video // Proc. XXII Intern. Conf. Pattern Recognition. 2014. Pp. 3062—3067.
9. Dogru N, Subasi A. Traffic Accident Detection Using Random Forest Classifier // Proc. XV Learning and Technol. Conf. 2018. Pp. 40—45.
10. Haresh S. e. a. Towards Anomaly Detection in Dashcam Videos // Proc. IEEE IV Intelligent Vehicles Symp. 2020. Pp. 1407—1414.
11. Li S., Fang J., Xu H., Xue J. Video Frame Prediction by Deep Multi-branch Mask Network // IEEE Trans. Circuits Syst. Video Technol. 2021. V. 31(4). Pp. 1283—1295.
12. Hasan M. e. a. Learning Temporal Regularity in Video Sequences // Proc. IEEE Conf. Computer Vision and Pattern Recognition. 2016. Pp. 733—742.
13. Liu W. e a. Future Frame Prediction for Anomaly Detection — a New Baseline // Proc. IEEE Conf. Computer Vision and Pattern Recognition. 2018. Pp. 6536—6545.
14. Yu Yao e. a. DoTA: Unsupervised Detection of Traffic Anomaly in Driving Videos // IEEE Trans. Pattern Anal. Mach. Intell. 2023. V. 45(1). Pp. 444—459.
15. Васюк М.А., Карпета Т.В. Алгоритм обнаружения дорожно-транспортных происшествий на перекрестках в режиме реального времени с помощью нейронной сети YOLOv4 // Южно-Уральская молодежная школа по математическому моделированию: Сб. трудов IV Всеросс. науч.-техн. конф. Челябинск: Издат. центр ЮУрГУ, 2021. С. 51—58.
16. Ротарь В.Г., Беляев С.И. Обнаружение автомобильных аварий в реальном времени с применением нейросети архитектуры YOLOv3 // Молодежь и современные информационные технологии: Сб. трудов XVIII Междунар. науч.-практ. конф. студентов, аспирантов и молодых учёных. Томск: Томский политехн. ун-т, 2021. С. 49—50.
17. Hajri F., Fradi H. Vision Transformers for Road Accident Detection from Dashboard Cameras // Proc. XVIII Intern. Conf. Advances Video and Signal Based Surveillance. 2022. Pp. 1—8.
18. Khan S. e. a. Transformers in Vision: a Survey // ACM Comput. Surv. 2022. V. 54(10). Pp. 1—41.
19. Yu Yao e. a. Unsupervised Traffic Accident Detection in First-person Videos // Proc. IEEE Intern. Conf. Intelligent Robots and Systems. Macau, 2019. Pp. 273—280.
20. Girshick R. Fast R-CNN // Proc. IEEE Intern. Conf. Computer Vision. 2015. Pp. 1440—1448.
21. Wojke N., Bewley A., Paulus D. Simple Online and Realtime Tracking with a Deep Association Metric // Proc. IEEE Intern. Conf. Image Proc. Beijing, 2017. Pp. 3645—3649.
22. He K. e. a. Mask R-CNN // IEEE Intern. Conf. Computer Vision. 2017. Pp. 2980—2988.
23. Kipf T.N., Welling M. Semi-supervised Classification with Graph Convolutional Networks // Proc. Intern. Conf. Learning Representations. 2017. Pp. 1—14.
24. Goodfellow I. e. a. Generative Adversarial Networks // Communications of the ACM. 2020. V. 63(11). Pp. 139—144.
25. Schafer R.W. What is a Savitzky-Golay filter? // IEEE Signal Process. Mag. 2011. V. 28(4). Pp. 111—117.
26. Fang J. e. a. DADA-2000: Can Driving Accident be Predicted by Driver Attention? Analyzed by a Benchmark // Proc. IEEE Intelligent Transportation Syst. Conf. Aucklend, 2019. Pp. 4303—4309.
27. Yao Y. e. a. Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systems // Proc. Intern. Conf. Robotics and Automation. Montreal, 2019. Pp. 9711—9717.
---
цитирования: Цяо Цзяхуань, Кружилов И.С., Шамаева О.Ю. Метод интеграции мультимодальной информации для обнаружения и прогнозирования дорожно-транспортных происшествий на основе нейросетевого подхода // Вестник МЭИ. 2025. № 3. С. 153—160. DOI: 10.24160/1993-6982-2025-3-153-160
---
Конфликт интересов: авторы заявляют об отсутствии конфликта интересов
#
1. World Health Organization. Global Status Report on Road Safety [Elektron. Resurs] https://www.who.int/publications/i/item/9789240086517 (Data Obrashcheniya 20.01.2025).
2. Kholmatov U.S. e. a. Causing Factors of Road Transport Incidents in Traffic. Intern. J. Education, Social Sci. & Humanities. 2024;12(5):1524—1534.
3. Ghahremannezhad H., Shi H., Liu C. A Real Time Accident Detection Framework for Traffic Video Analysis. Proc. XVI Intern. Conf. Machine Learning and Data Mining. 2020:77—92.
4. Sun Y., Mallick T., Balaprakash P., Macfarlane J. A Datacentric Weak Supervised Learning for Highway Traffic Incident Detection. Accident Analysis and Prevention. 2022;176:106779.
5. Sherimon V. e. a. An Overview of Different Deep Learning Techniques Used in Road Accident Detection. Intern. J. Advanced Computer Sci. & Appl. 2023;14(11):0141144.
6. Shunsuke Kamijo e. a. Traffic Monitoring and Accident Detection at Intersections. IEEE Trans. Intelligent Transportation Syst. 2000;1:108—118.
7. Xu Y. Traffic Incident Detection Based on HMM. Proc. IEEE Third Intern. Conf. Information Sci. and Technol. Yangzhou, 2013:942—945.
8. Kimin Yun e. a. Motion Interaction Field for Accident Detection in Traffic Surveillance Video. Proc. XXII Intern. Conf. Pattern Recognition. 2014:3062—3067.
9. Dogru N, Subasi A. Traffic Accident Detection Using Random Forest Classifier. Proc. XV Learning and Technol. Conf. 2018:40—45.
10. Haresh S. e. a. Towards Anomaly Detection in Dashcam Videos. Proc. IEEE IV Intelligent Vehicles Symp. 2020:1407—1414.
11. Li S., Fang J., Xu H., Xue J. Video Frame Prediction by Deep Multi-branch Mask Network. IEEE Trans. Circuits Syst. Video Technol. 2021;31(4):1283—1295.
12. Hasan M. e. a. Learning Temporal Regularity in Video Sequences. Proc. IEEE Conf. Computer Vision and Pattern Recognition. 2016:733—742.
13. Liu W. e a. Future Frame Prediction for Anomaly Detection — a New Baseline. Proc. IEEE Conf. Computer Vision and Pattern Recognition. 2018:6536—6545.
14. Yu Yao e. a. DoTA: Unsupervised Detection of Traffic Anomaly in Driving Videos. IEEE Trans. Pattern Anal. Mach. Intell. 2023;45(1):444—459.
15. Vasyuk M.A., Karpeta T.V. Algoritm Obnaruzheniya Dorozhno-transportnykh Proisshestviy na Perekrestkakh v Rezhime Real'nogo Vremeni s Pomoshch'yu Neyronnoy Seti Yolov4. Yuzhno-Ural'skaya Molodezhnaya Shkola po Matematicheskomu Modelirovaniyu: Sb. Trudov IV Vseross. Nauch.-tekhn. Konf. Chelyabinsk: Izdat. Tsentr YUUrGU, 2021:51—58. (in Russian).
16. Rotar' V.G., Belyaev S.I. Obnaruzhenie Avtomobil'nykh Avariy v Real'nom Vremeni s Primeneniem Neyroseti Arkhitektury YOLOv3. Molodezh' i Sovremennye Informatsionnye Tekhnologii: Sb. Trudov XVIII Mezhdunar. Nauch.-prakt. Konf. Studentov, Aspirantov i Molodykh Uchenykh. Tomsk: Tomskiy Politekhn. Un-t, 2021:49—50. (in Russian).
17. Hajri F., Fradi H. Vision Transformers for Road Accident Detection from Dashboard Cameras. Proc. XVIII Intern. Conf. Advances Video and Signal Based Surveillance. 2022:1—8.
18. Khan S. e. a. Transformers in Vision: a Survey. ACM Comput. Surv. 2022;54(10):1—41.
19. Yu Yao e. a. Unsupervised Traffic Accident Detection in First-person Videos. Proc. IEEE Intern. Conf. Intelligent Robots and Systems. Macau, 2019:273—280.
20. Girshick R. Fast R-CNN. Proc. IEEE Intern. Conf. Computer Vision. 2015:1440—1448.
21. Wojke N., Bewley A., Paulus D. Simple Online and Realtime Tracking with a Deep Association Metric. Proc. IEEE Intern. Conf. Image Proc. Beijing, 2017:3645—3649.
22. He K. e. a. Mask R-CNN. IEEE Intern. Conf. Computer Vision. 2017:2980—2988.
23. Kipf T.N., Welling M. Semi-supervised Classification with Graph Convolutional Networks. Proc. Intern. Conf. Learning Representations. 2017:1—14.
24. Goodfellow I. e. a. Generative Adversarial Networks. Communications of the ACM. 2020;63(11):139—144.
25. Schafer R.W. What is a Savitzky-Golay filter?. IEEE Signal Process. Mag. 2011;28(4):111—117.
26. Fang J. e. a. DADA-2000: Can Driving Accident be Predicted by Driver Attention? Analyzed by a Benchmark. Proc. IEEE Intelligent Transportation Syst. Conf. Aucklend, 2019:4303—4309.
27. Yao Y. e. a. Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systems. Proc. Intern. Conf. Robotics and Automation. Montreal, 2019:9711—9717
---
For citation: Qiao Jiahuan, Kruzhilov I.S., Shamaeva O.Yu. Multimodal Information Integration Method for Road Traffic Accident Detection and Prediction by the Neural Network Method. Bulletin of MPEI. 2025;3:153—160. (in Russian). DOI: 10.24160/1993-6982-2025-3-153-160
---
Conflict of interests: the authors declare no conflict of interest
Downloads
Published
2025-04-22
Issue
Section
Informatics and information processes (technical sciences) (2.3.8.)
