Human-like decision making for autonomous lane change driving: a hybrid inverse reinforcement learning with game-theoretical vehicle interaction model

• Autorzy: Jiang Yalan , Wu Xuncheng , Zhang Weiwei , Guo Wenfeng , Yu Wangpengfei , Li Jun

Identyfikatory

DOI

10.24425/bpasts.2024.152602

• Języki publikacji: EN

Abstrakty

EN

The development of automated driving vehicles aims to provide safer, comfortable, and more efficient mobility options. However, the decision-making control of autonomous vehicles still faces limitations of human performance mimicry. These limitations become particularly evident in complex and unfamiliar driving scenarios, where weak decision-making abilities and poor adaptation of vehicle behaviour are prominent issues. This paper proposes a game-theoretic decision-making algorithm for human-like driving in the vehicle lane change scenario. Firstly, an inverse reinforcement learning (IRL) model is used to quantitatively analyze the lane change trajectories of the natural driving dataset, establishing the human-like human cost function. Subsequently, joint safety, and comfort to build the comprehensive decision cost function. The combined decision cost function is used to conduct a noncooperative game of vehicle lane changing decisions to solve the optimal decision of host vehicle lane changing. The host vehicle lane-changing decision problem is formulated as a Stackelberg game optimization problem. To verify the feasibility and effectiveness of the algorithm proposed in this study, a lane change test scenario was established. Firstly, we analyze the human-like decision-making model derived from the maximum entropy inverse reinforcement learning algorithm to verify the effectiveness and robustness of the IRL algorithm. Secondly, the human-like game decision-making algorithm in this paper is validated by conducting an interactive lane-changing experiment with obstacle vehicles of different driving styles. The experimental results prove that the human-like driving decision-making model proposed in this study can make lane-changing behaviours in line with human driving patterns in lane-changing scenarios of the expressway.

Słowa kluczowe

EN

expressway lane-changing scenarios; inverse reinforcement learning; interaction model; Stackelberg game; human-like decision making; IRL

PL

model interakcji; gra Stackelberga; odwrotne uczenie przez wzmacnianie; zmiana pasa ruchu; podejmowanie decyzji

• Wydawca: Polska Akademia Nauk, Wydział IV Nauk Technicznych
• Czasopismo: Bulletin of the Polish Academy of Sciences. Technical Sciences
• Rocznik: 2025
• Tom: Vol. 73, nr 1
• Strony: art. no. e152602
• Opis fizyczny: Bibliogr. 25 poz., rys., tab., wykr.

Twórcy

autor

Jiang Yalan

• School of Mechanical and Automotive Engineering, Shanghai University of Engineering Science, Shanghai, 201620, China

• https://orcid.org/0009-0008-5672-5728

autor

Wu Xuncheng

• School of Mechanical and Automotive Engineering, Shanghai University of Engineering Science, Shanghai, 201620, China

autor

Zhang Weiwei

• Shanghai Smart Vehicle Cooperating Innovation Center Co., Ltd., Shanghai, 201805, China

autor

Guo Wenfeng

• School of Vehicle and Mobility, Tsinghua University, Beijing, 100084, China

autor

Yu Wangpengfei

• Shanghai Smart Vehicle Cooperating Innovation Center Co., Ltd., Shanghai, 201805, China

autor

Li Jun

• School of Vehicle and Mobility, Tsinghua University, Beijing, 100084, China

Bibliografia

• [1] W. Wang, L. Wang, C. Zhang, C. Liu, and L. Sun, “Social Inter-actions for Autonomous Driving: A Review and Perspectives,” Found. Trends Robot., vol. 10, no. 3–4, pp. 198–376, 2022, doi: 10.1561/2300000078.
• [2] C. Dong, J.M. Dolan, and B. Litkouhi, “Intention estimation for ramp merging control in autonomous driving,” in 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, USA: IEEE, Jun. 2017, pp. 1584–1589, doi: 10.1109/IVS.2017.7995935.
• [3] E. Galceran, A.G. Cunningham, R.M. Eustice, and E. Olson, “Multipolicy decision-making for autonomous driving via changepoint-based behavior prediction: Theory and experiment,” Auton. Robot., vol. 41, no. 6, pp. 1367–1382, Aug. 2017, doi: 10.1007/s10514-017-9619-z.
• [4] C.L. Baker and J.B. Tenenbaum, “Modeling Human Plan Recognition Using Bayesian Theory of Mind,” in Plan, Activity, and Intent Recognition, Elsevier, 2014, pp. 177–204. doi: 10.1016/B978-0-12-398532-3.00007-5.
• [5] J. Wiest, M. Hoffken, U. Kresel, and K. Dietmayer, “Probabilistic trajectory prediction with Gaussian mixture models,” in 2012 IEEE Intelligent Vehicles Symposium, Madrid, Spain: IEEE, Jun. 2012, pp. 141–146. doi: 10.1109/IVS.2012.6232277.
• [6] L. Hou, L. Xin, S.E. Li, B. Cheng, and W. Wang, “Interactive Trajectory Prediction of Surrounding Road Users for Autonomous Driving Using Structural-LSTM Network,” IEEE Trans. Intell. Transport. Syst., vol. 21, no. 11, pp. 4615–4625, Nov. 2020, doi: 10.1109/TITS.2019.2942089.
• [7] Z. Huang, J. Wang, L. Pi, X. Song, and L. Yang, “LSTM based trajectory prediction model for cyclist utilizing multiple interactions with environment,” Pattern Recognit., vol. 112, p. 107800, Apr. 2021, doi: 10.1016/j.patcog.2020.107800.
• [8] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” Aug. 28, 2017, arXiv: arXiv:1707.06347. [Online]. Available: http://arxiv.org/abs/1707.06347 (Accessed: Jun. 13, 2024).
• [9] F. Ye, X. Cheng, P. Wang, C.-Y. Chan, and J. Zhang, “Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning,” May 20, 2020, arXiv: arXiv:2002.02667. [Online]. Available: http://arxiv.org/abs/2002.02667 (Accessed: Jun. 13, 2024).
• [10] Y. Shi, Y. Liu, Y. Qi, and Q. Han, “A Control Method with Reinforcement Learning for Urban Un-Signalized Intersection in Hybrid Traffic Environment,” Sensors, vol. 22, no. 3, p. 779, Jan. 2022, doi: 10.3390/s22030779.
• [11] R. Trumpp, H. Bayerlein, and D. Gesbert, “Modeling Interactions of Autonomous Vehicles and Pedestrians with Deep Multi-Agent Reinforcement Learning for Collision Avoidance,” in 2022 IEEE Intelligent Vehicles Symposium (IV), Aachen, Germany: IEEE, Jun. 2022, pp. 331–336, doi: 10.1109/IV51971.2022.9827451.
• [12] S. Brechtel, T. Gindele, and R. Dillmann, “Probabilistic MDP-behavior planning for cars,” in 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), Washington, USA: IEEE, Oct. 2011, pp. 1537–1542, doi: 10.1109/ITSC.2011.6082928.
• [13] C. Hubmann, M. Becker, D. Althoff, D. Lenz, and C. Stiller, “Decision making for autonomous driving considering interaction and uncertain prediction of surrounding vehicles,” in 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, USA: IEEE, Jun. 2017, pp. 1671–1678. doi: 10.1109/IVS.2017.7995949.
• [14] H. Yu, H.E. Tseng, and R. Langari, “A human-like game theory-based controller for automatic lane changing,” Transp. Res. Part C-Emerg. Technol., vol. 88, pp. 140–158, Mar. 2018, doi: 10.1016/j.trc.2018.01.016.
• [15] L. Li, W. Zhao, and C. Wang, “POMDP Motion Planning Algorithm Based on Multi-Modal Driving Intention,” IEEE Trans. Intell. Veh., vol. 8, no. 2, pp. 1777–1786, Feb. 2023, doi: 10.1109/TIV.2022.3209926.
• [16] Z. Zhu and H. Zhao, “Learning Autonomous Control Policy for Intersection Navigation With Pedestrian Interaction,” IEEE Trans. Intell. Veh., vol. 8, no. 5, pp. 3270–3284, May 2023, doi: 10.1109/TIV.2023.3256972.
• [17] Z. Huang, J. Wu, and C. Lv, “Driving Behavior Modeling using Naturalistic Human Driving Data with Inverse Reinforcement Learning,” Jul. 19, 2021, arXiv: arXiv:2010.03118. [Online]. Available: http://arxiv.org/abs/2010.03118 (Accessed: Jun. 13, 2024).
• [18] X. Wen, S. Jian, and D. He, “Modeling the Effects of Autonomous Vehicles on Human Driver Car-Following Behaviors Using Inverse Reinforcement Learning,” IEEE Trans. Intell. Transport. Syst., vol. 24, no. 12, pp. 13903–13915, Dec. 2023, doi: 10.1109/TITS.2023.3298150.
• [19] X. Di and R. Shi, “A survey on autonomous vehicle control in the era of mixed-autonomy: From physics-based to AI-guided driving policy learning,” Transp. Res. Part C-Emerg. Technol., vol. 125, p. 103008, Apr. 2021, doi: 10.1016/j.trc.2021.103008.
• [20] J. Yoo and R. Langari, “A Game-Theoretic Model of Human Driving and Application to Discretionary Lane-Changes,” Mar. 22, 2020, arXiv: arXiv: 2003.09783. [Online]. Available: https://arxiv.org/abs/2003.09783
• [21] H. Yu, H.E. Tseng, and R. Langari, “A human-like game theory-based controller for automatic lane changing,” Transp. Res. Part C-Emerg. Technol., vol. 88, pp. 140–158, Mar. 2018, doi: 10.1016/j.trc.2018.01.016.
• [22] A. Talebpour, H.S. Mahmassani, and S.H. Hamdar, “Modeling Lane-Changing Behavior in a Connected Environment: A Game Theory Approach,” Transp. Res. Procedia, vol. 7, pp. 420–440, 2015, doi: 10.1016/j.trpro.2015.06.022.
• [23] Q. Zhang, R. Langari, H.E. Tseng, D. Filev, S. Szwabowski, and S. Coskun, “A Game Theoretic Model Predictive Controller With Aggressiveness Estimation for Mandatory Lane Change,” IEEE Trans. Intell. Veh., vol. 5, no. 1, pp. 75–89, Mar. 2020, doi: 10.1109/TIV.2019.2955367.
• [24] G.S. Sankar and K. Han, “Adaptive Robust Game-Theoretic Decision Making Strategy for Autonomous Vehicles in Highway,” IEEE Trans. Veh. Technol., vol. 69, no. 12, pp. 14484–14493, Dec. 2020, doi: 10.1109/TVT.2020.3041152.
• [25] N. Li, D.W. Oyler, M. Zhang, Y. Yildiz, I. Kolmanovsky, and A.R. Girard, “Game Theoretic Modeling of Driver and Vehicle Interactions for Verification and Validation of Autonomous Vehicle Control Systems,” IEEE Trans. Contr. Syst. Technol., vol. 26, no. 5, pp. 1782–1797, Sep. 2018, doi: 10.1109/TCST.2017.2723574.

Uwagi

Opracowanie rekordu ze środków MNiSW, umowa nr POPUL/SP/0154/2024/02 w ramach programu "Społeczna odpowiedzialność nauki II" - moduł: Popularyzacja nauki (2025).