Simplification of deep reinforcement learning in traffic control using the Bonsai Platform

• Autorzy: Skuba Michał , Janota Aleš

Identyfikatory

DOI

10.24136/tren.2020.014

Warianty tytułu

PL

Uproszczenie uczenia się przez głębokie wzmocnienie w zarządzaniu ruchem z wykorzystaniem platformy Bonsai

• Języki publikacji: EN

Abstrakty

EN

The paper deals with the problem of traffic light control of road intersection. The authors use a model of a real road junction created in the AnyLogic modelling tool. For two scenarios, there are three simulation experiments performed – fixed time control, fixed time control after AnyLogic-based optimizations, and dynamic control obtained through the cooperation of the AnyLogic tool and the Bonsai platform, utilizing benefits of deep reinforcement learning. At present, there are trends to simplify machine learning processes as much as possible to make them accessible to practitioners with no artificial intelligence background and without the need to become data scientists. Project Bonsai represents an easy-to-use connector, that allows to use AnyLogic models connected to the Bonsai platform - a novel approach to machine learning without the need to set any hyper-parameters. Due to unavailability of real operational data, the model uses simulation data only, with presence and movement of vehicles only (no pedestrians). The optimization problem consists in minimizing the average time that agents (vehicles) must spend in the model, passing the modelled intersection. Another observed parameter is the maximum time of individual vehicles spent in the model. The authors share their practical, mainly methodological, experiences with the simulation process and indicate economic cost needed for training as well.

PL

Artykuł dotyczy problemu sterowania sygnalizacją świetlną na skrzyżowaniach dróg. Autorzy wykorzystują model rzeczywistego węzła drogowego utworzony w narzędziu do modelowania AnyLogic. Dla dwóch scenariuszy wykonywane są trzy eksperymenty symulacyjne - sterowanie światłami sygnalizacyjnymi o stałym czasie działania, sterowanie światłami sygnalizacyjnymi o stałym czasie działania po optymalizacji w oparciu o AnyLogic, i sterowanie dynamiczne dzięki współpracy między AnyLogic i platformą Bonsai, wykorzystując korzyści płynące z uczenia się przez głębokie wzmocnienie. Obecnie istnieją tendencje do maksymalnego upraszczania procesów uczenia maszynowego, aby były dostępne dla praktyków bez doświadczenia w zakresie sztucznej inteligencji i bez konieczności zostania naukowcami danych. Project Bonsai to łatwe w obsłudze złącze, które pozwala na korzystanie z modeli AnyLogic podłączonych do platformy Bonsai - nowatorskie podejście do uczenia maszynowego bez konieczności ustawiania hiperparametrów. Ze względu na niedostępność rzeczywistych danych eksploatacyjnych model wykorzystuje tylko dane symulacyjne, tylko z obecnością i ruchem pojazdów (bez pieszych). Problem optymalizacji polega na zminimalizowaniu średniego czasu, jaki agenci (pojazdy) muszą spędzać w modelu, mijając modelowane skrzyżowanie. Kolejnym obserwowanym parametrem jest maksymalny czas przebywania poszczególnych pojazdów w modelu. Autorzy dzielą się praktycznymi, głównie metodologicznymi, doświadczeniami związanymi z procesem symulacji oraz wskazują koszty ekonomiczne potrzebne do uczenia.

Słowa kluczowe

EN

control; deep reinforcement learning; model; simulation; traffic

PL

sterowanie; uczenie głębokie; uczenie przez wzmacnianie; symulacja; ruch drogowy

• Wydawca: Uniwersytet Technologiczno-Humanistyczny im. Kazimierza Pułaskiego w Radomiu
• Czasopismo: Journal of civil engineering and transport
• Rocznik: 2020
• Tom: Vol. 2, No. 4
• Strony: 191--202
• Opis fizyczny: Bibliogr. 32 poz., rys., tab.

Twórcy

autor

Skuba Michał

• University of Žilina, Faculty of Electrical Engineering and Information Technology, Univerzitná 1, 010 26 Žilina, Slovakia

• https://orcid.org/0000-0002-5453-9648

autor

Janota Aleš

• University of Žilina, Faculty of Electrical Engineering and Information Technology, Univerzitná 1, 010 26 Žilina, Slovakia,

• https://orcid.org/0000-0003-2132-1295

Bibliografia

• [1] Zhonghe H, Chi Z, Li W. (2015) “Consensus feedback control for urban road traffic networks”, 54th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE), pp. 1413-1418. https://doi.org/10.1109/SICE.2015.7285401
• [2] Zhao J, Ma W. (2021) “An alternative design for the intersections with limited traffic lanes and queuing space”. IEEE Transactions on intelligent transportation systems, Vol. 22, No. 3, pp. 1473-1483. https://doi.org/10.1109/TITS.2020.2971353
• [3] Wu W, Mingjun W. (2003) “Research on traffic signal control based on intelligence techniques”, Proceedings of the 2003 IEEE International Conference on Intelligent Transportation Systems, vol. 1, pp. 892-896. https://doi.org/10.1109/ITSC.2003.1252078
• [4] Vogel A, Oremović I, Šimić R, Ivanjko E. (2018) “Improving Traffic Light Control by Means of Fuzzy Logic”, 2018 International Symposium ELMAR, pp. 51-56. https://doi.org/10.23919/ELMAR.2018.8534692
• [5] Vogel A, Oremović I, Šimić R, Ivanjko E (2019) “Fuzzy Traffic Light Control Based on Phase Urgency", 2019 International Symposium ELMAR, pp. 9-14. https://doi.org/10.1109/ELMAR.2019.8918675
• [6] Chai L, Shen G, Ye W. (2006) “The Traffic Flow Model for Single Intersection and its Traffic Light Intelligent Control Strategy”, 2006 6th World Congress on Intelligent Control and Automation, pp. 8558-8562. https://doi.org/10.1109/WCICA.2006.1713650
• [7] Yadav A, Nuthong C. (2020) “Traffic signal timings optimization based on genetic algorithm and gradient descent”, 2020 5th International Conference on Computer and Communication Systems (ICCCS), pp. 670-674. https://doi.org/10.1109/ICCCS49078.2020.9118450
• [8] Slowik A, Kwasnicka H. (2020) “Evolutionary algorithms and their applications to engineering problems”. Neural Computing and Applications, 32, pp. 12363-12379. https://doi.org/10.1007/s00521-020-04832-8
• [9] Jiang L, Li Y, Liu Y, Chen C. (2017) “Traffic signal light control model based on evolutionary programming algorithm optimization BP neural network”, 2017 7th IEEE International Conference on Electronics Information and Emergency Communication (ICEIEC), pp. 564-567. https://doi.org/10.1109/ICEIEC.2017.8076629
• [10] Mohammadian M. (2006) “Multi-Agents Systems for Intelligent Control of Traffic Signals”, 2006 International Conference on Computational Inteligence for Modelling Control and Automation and International Conference on Intelligent Agents Web Technologies and International Commerce (CIMCA'06), pp. 270-270. https://doi.org/10.1109/CIMCA.2006.152
• [11] Haydari A, Yilmaz Y. (2020) “Deep reinforcement learning for intelligent transportation systems: A survey”. ArXiv, abs/2005.00935, pp. 1-22. https://arxiv.org/pdf/2005.00935.pdf
• [12] Wei H, Zheng G, Gaah V, Li Z. (2021) “Recent advances in reinforcement learning for traffic signal control: A survey of models and evaluation”. ACM SIGKDD Explorations Newsletter. https://doi.org/10.1145/3447556.3447565
• [13] Arulkumaran K, Deisenroth M.P, Brundage M, Bharath A.A. (2017) “A brief survey of deep reinforcement learning". IEEE Signal Processing Magazine, Special issue on deep learning for image understanding (arXiv extended version), arXiv: 1708.05866. https://doi.org/10.1109/MSP.2017.2743240
• [14] Nachum O, Norouzi M, Xu K, Schuurmans D. (2017) “Bridging the gap between value and policy based reinforcement learning”. 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, pp. 2772–2782. https://dl.acm.org/doi/pdf/10.5555/3294996.3295037
• [15] Farazi NP, Ahamed T, Barua L, Zou B. (2020) “Deep reinforcement learning and transportation research: A comprehensive review”. ArXiv abs/2020.06187, pp. 1-60. https://arxiv.org/ftp/arxiv/papers/2010/2010.06187.pdf
• [16] Fadlullah ZM et al. (2017) “State-of-the-Art Deep Learning: Evolving Machine Intelligence Toward Tomorrow’s Intelligent Network Traffic Control Systems”, IEEE Communications Surveys & Tutorials. Vol. 19, No. 4, pp. 2432-2455, Fourthquarter 2017. https://doi.org/10.1109/COMST.2017.2707140
• [17] Paul A, Mitra S. (2020) “Deep reinforcement learning based traffic signal optimization for multiple intersections in ITS”, 2020 IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS), pp. 1-6. https://doi.org/10.1109/ANTS50601.2020.9342819
• [18] Liang X, Du X, Wang G, Han Z. (2019) “A Deep Reinforcement Learning Network for Traffic Light Cycle Control”. IEEE Transactions on Vehicular Technology. Vol. 68, No. 2, pp. 1243-1253. https://doi.org/10.1109/TVT.2018.2890726
• [19] Wei H, Zheng G, Yao H, Li Z. (2018) “IntelliLight: A reinforcement learning approach for intelligent traffic light control”, Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2496-2505. https://doi.org/10.1145/3219819.3220096
• [20] Coşkun M, Baggag A, Chawla S. (2018) “Deep Reinforcement Learning for Traffic Light Optimization”, 2018 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 564-571. https://doi.org/10.1109/ICDMW.2018.00088
• [21] Rosyadi AR, Wirayuda TAB, Al-Faraby S. (2016) “Intelligent traffic light control using collaborative Q-Learning algorithms”, 2016 4th International Conference on Information and Communication Technology (ICoICT), pp. 1-6. https://doi.org/10.1109/ICoICT.2016.7571925
• [22] Pálos P, Huszák Á. (2020) “Comparison of Q- Learning based Traffic Light Control Methods and Objective Functions”, 2020 International Conference on Software, Telecommunications and Computer Networks (SoftCOM), pp. 1-6. https://doi.org/10.23919/SoftCOM50211.2020.9238290
• [23] Wu T, Kong F, Fan Z. (2019) “Road Model Design Based on Reward Function in Traffic Light Control”, 2019 5th International Conference on Control, Automation and Robotics (ICCAR), pp. 407-412. https://doi.org/10.1109/ICCAR.2019.8813381
• [24] Kao Y, Wu C. (2018) “A Self-Organizing Map-Based Adaptive Traffic Light Control System with Reinforcement Learning”, 2018 52nd Asilomar Conference on Signals, Systems, and Computers, pp. 2060-2064. https://doi.org/10.1109/ACSSC.2018.8645125
• [25] Bhave N, Dhagavkar A, Dhande K, Bana M, Joshi J. (2019) “Smart Signal – Adaptive Traffic Signal Control using Reinforcement Learning and Object Detection”, 2019 Third International conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), pp. 624-628. https://doi.org/10.1109/I-SMAC47947.2019.9032589
• [26] Garg D, Chli M, Vogiatzis G. (2018) “Deep Reinforcement Learning for Autonomous Traffic Light Control”, 3rd IEEE International Conference on Intelligent Transportation Engineering (ICITE), pp. 214-218. https://doi.org/10.1109/ICITE.2018.8492537
• [27] Khamis M.A, Gomaa W. (2012) “Enhanced multiagent multi-objective reinforcement learning for urban traffic light control”, 2012 11th International Conference on Machine Learning and Applications, pp. 586-591. https://doi.org/10.1109/ICMLA.2012.108
• [28] Wu T. et al. (2020) “Multi-Agent Deep Reinforcement Learning for Urban Traffic Light Control in Vehicular Networks”. IEEE Transactions on Vehicular Technology. Vol. 69, No. 8, pp. 8243-8256. https://doi.org/10.1109/TVT.2020.2997896
• [29] Shabestary SMA, Abdulhai B. (2018) “Deep Learning vs. Discrete Reinforcement Learning for Adaptive Traffic Signal Control”, 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 286-293. https://doi.org/10.1109/ITSC.2018.8569549
• [30] Zeng J, Hu J, Zhang Y. (2019) “Training Reinforcement Learning Agent for Traffic Signal Control under Different Traffic Conditions”, 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 4248-4254. https://doi.org/10.1109/ITSC.2019.8917342
• [31] Skuba M. (2021) “Deep reinforcement learning use in road traffic modelling and simulation”. MSc. Thesis, DCIS FEEIT University of Žilina, 52 p.
• [32] Benčat G, Janota A. (2020) “Road traffic modelling based on the hybrid modelling tool AnyLogic”. Journal of civil engineering and transport, Vol. 3, Issue 1, pp. 19-35. https://doi.org/10.24136/tren.2020.006