{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:20:39Z","timestamp":1760145639708,"version":"build-2065373602"},"reference-count":41,"publisher":"MDPI AG","issue":"16","license":[{"start":{"date-parts":[[2024,8,11]],"date-time":"2024-08-11T00:00:00Z","timestamp":1723334400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"publisher","award":["52202495","52202494","52394261"],"award-info":[{"award-number":["52202495","52202494","52394261"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>The centralized coordination of Connected and Automated Vehicles (CAVs) at unsignalized intersections aims to enhance traffic efficiency, driving safety, and passenger comfort. Autonomous Intersection Management (AIM) systems introduce a novel approach for centralized coordination. However, existing rule-based and optimization methods often face the challenges of poor generalization and low computational efficiency when dealing with complex traffic environments and highly dynamic traffic conditions. Additionally, current Reinforcement Learning (RL)-based methods encounter difficulties around policy inference and safety. To address these issues, this study proposes Constraint-Guided Behavior Transformer for Safe Reinforcement Learning (CoBT-SRL), which uses transformers as the policy network to achieve efficient decision-making for vehicle driving behaviors. This method leverages the ability of transformers to capture long-range dependencies and improve data sample efficiency by using historical states, actions, and reward and cost returns to predict future actions. Furthermore, to enhance policy exploration performance, a sequence-level entropy regularizer is introduced to encourage policy exploration while ensuring the safety of policy updates. Simulation results indicate that CoBT-SRL exhibits stable training progress and converges effectively. CoBT-SRL outperforms other RL methods and vehicle intersection coordination schemes (VICS) based on optimal control in terms of traffic efficiency, driving safety, and passenger comfort.<\/jats:p>","DOI":"10.3390\/s24165187","type":"journal-article","created":{"date-parts":[[2024,8,12]],"date-time":"2024-08-12T11:23:46Z","timestamp":1723461826000},"page":"5187","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Constraint-Guided Behavior Transformer for Centralized Coordination of Connected and Automated Vehicles at Intersections"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1597-1961","authenticated-orcid":false,"given":"Rui","family":"Zhao","sequence":"first","affiliation":[{"name":"College of Automotive Engineering, Jilin University, Changchun 130025, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-8309-7396","authenticated-orcid":false,"given":"Yuze","family":"Fan","sequence":"additional","affiliation":[{"name":"College of Automotive Engineering, Jilin University, Changchun 130025, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-7824-7751","authenticated-orcid":false,"given":"Yun","family":"Li","sequence":"additional","affiliation":[{"name":"Graduate School of Information and Science Technology, The University of Tokyo, Tokyo 113-8654, Japan"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0009-0000-0821-1660","authenticated-orcid":false,"given":"Kui","family":"Wang","sequence":"additional","affiliation":[{"name":"School of Mechanical Engineering, Beijing Institute of Technology, Beijing 100081, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4195-5033","authenticated-orcid":false,"given":"Fei","family":"Gao","sequence":"additional","affiliation":[{"name":"College of Automotive Engineering, Jilin University, Changchun 130025, China"},{"name":"National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University, Changchun 130025, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4623-3956","authenticated-orcid":false,"given":"Zhenhai","family":"Gao","sequence":"additional","affiliation":[{"name":"College of Automotive Engineering, Jilin University, Changchun 130025, China"},{"name":"National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University, Changchun 130025, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,8,11]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"5439","DOI":"10.1109\/TVT.2015.2483779","article-title":"Eco-departure of connected vehicles with V2X communication at signalized intersections","volume":"64","author":"Li","year":"2015","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"103923","DOI":"10.1016\/j.artint.2023.103923","article-title":"Risk-Aware Controller for Autonomous Vehicles Using Model-Based Collision Prediction and Reinforcement Learning","volume":"320","author":"Candela","year":"2023","journal-title":"Artif. Intell."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1109\/COMST.2014.2339817","article-title":"A communications-oriented perspective on traffic management systems for smart cities: Challenges and innovative approaches","volume":"17","author":"Djahel","year":"2014","journal-title":"IEEE Commun. Surv. Tutorials"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"182","DOI":"10.1049\/iet-its.2016.0043","article-title":"Autonomous Intersection Management Systems: Criteria, Implementation and Evaluation","volume":"11","author":"Qian","year":"2017","journal-title":"IET Intell. Transp. Syst."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"250","DOI":"10.1080\/15472450.2018.1519706","article-title":"Incorporating insights from signal optimization into reservation-based intersection controls","volume":"23","author":"Lukose","year":"2019","journal-title":"J. Intell. Transp. Syst."},{"key":"ref_6","first-page":"1352","article-title":"Priority-Based Coordination of Robots","volume":"30","author":"Gregoire","year":"2014","journal-title":"IEEE Trans. Robot."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"246","DOI":"10.1016\/j.trc.2019.04.012","article-title":"DCL-AIM: Decentralized Coordination Learning of Autonomous Intersection Management for Connected and Automated Vehicles","volume":"103","author":"Wu","year":"2019","journal-title":"Transp. Res. Part C Emerg. Technol."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1908","DOI":"10.1109\/TITS.2018.2850335","article-title":"Developing an Optimal Intersection Control System for Automated Connected Vehicles","volume":"20","author":"Bichiou","year":"2019","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Lu, Q., and Kim, K.D. (2018, January 27\u201330). A Mixed Integer Programming Approach for Autonomous and Connected Intersection Crossing Traffic Control. Proceedings of the 2018 IEEE 88th Vehicular Technology Conference (VTC-Fall), Chicago, IL, USA.","DOI":"10.1109\/VTCFall.2018.8690681"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"1136","DOI":"10.1109\/TITS.2014.2354380","article-title":"A Vehicle-Intersection Coordination Scheme for Smooth Flows of Traffic without Using Traffic Lights","volume":"16","author":"Kamal","year":"2014","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"18288","DOI":"10.1109\/TITS.2022.3162038","article-title":"Fully Distributed Model Predictive Control of Connected Automated Vehicles in Intersections: Theory and Vehicle Experiments","volume":"23","author":"Katriniok","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"278","DOI":"10.1061\/(ASCE)0733-947X(2003)129:3(278)","article-title":"Reinforcement Learning for True Adapt. Traffic Signal Control","volume":"129","author":"Abdulhai","year":"2003","journal-title":"J. Transp. Eng."},{"key":"ref_13","doi-asserted-by":"crossref","first-page":"433","DOI":"10.1109\/TITS.2019.2942014","article-title":"Development of an Efficient Driving Strategy for Connected and Automated Vehicles at Signalized Intersections: A Reinforcement Learning Approach","volume":"21","author":"Zhou","year":"2020","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"2434","DOI":"10.1109\/TCAD.2022.3226673","article-title":"FairLight: Fairness-Aware Autonomous Traffic Signal Control with Hierarchical Action Space","volume":"42","author":"Ye","year":"2023","journal-title":"IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"12597","DOI":"10.1109\/TVT.2020.3026111","article-title":"Centralized Cooperation for Connected and Automated Vehicles at Intersections by Proximal Policy Optimization","volume":"69","author":"Guan","year":"2020","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"12345","DOI":"10.1109\/TITS.2023.3285440","article-title":"Self-Learned Autonomous Driving at Unsignalized Intersections: A Hierarchical Reinforced Learning Approach for Feasible Decision-Making","volume":"24","author":"Dempster","year":"2023","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"5390","DOI":"10.1109\/TITS.2023.3243940","article-title":"Real-Time Cooperative Vehicle Coordination at Unsignalized Road Intersections","volume":"24","author":"Luo","year":"2023","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_18","first-page":"5998","article-title":"Attention is all you need","volume":"30","author":"Vaswani","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_19","unstructured":"Zambaldi, V., Raposo, D., Santoro, A., Bapst, V., Li, Y., Babuschkin, I., Tuyls, K., Reichert, D., Lillicrap, T., and Lockhart, E. (2019, January 6\u20139). Deep reinforcement learning with relational inductive biases. Proceedings of the International Conference on Learning Representations, New Orleans, LA, USA."},{"key":"ref_20","first-page":"15084","article-title":"Decision transformer: Reinforcement learning via sequence modeling","volume":"34","author":"Chen","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_21","first-page":"1273","article-title":"Offline reinforcement learning as one big sequence modeling problem","volume":"34","author":"Janner","year":"2021","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_22","unstructured":"Furuta, H., Matsuo, Y., and Gu, S.S. (2021). Generalized decision transformer for offline hindsight information matching. arXiv."},{"key":"ref_23","unstructured":"Yamagata, T., Khalil, A., and Santos-Rodriguez, R. (2023, January 23\u201329). Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL. Proceedings of the 40th International Conference on Machine Learning, Honolulu, HI, USA. Proceedings of Machine Learning Research."},{"key":"ref_24","unstructured":"Zheng, Q., Zhang, A., and Grover, A. (2022, January 17\u201323). Online decision transformer. Proceedings of the International Conference on Machine Learning, PMLR, Baltimore, MD, USA."},{"key":"ref_25","unstructured":"Liu, Z., Guo, Z., Yao, Y., Cen, Z., Yu, W., Zhang, T., and Zhao, D. (2023, January 23\u201329). Constrained decision transformer for offline safe reinforcement learning. Proceedings of the International Conference on Machine Learning, PMLR, Honolulu, HI, USA."},{"key":"ref_26","unstructured":"Zhang, A., and Coauthors, A. (2023). SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning. arXiv."},{"key":"ref_27","unstructured":"Gu, S., Yang, L., Du, Y., Chen, G., Walter, F., Wang, J., Yang, Y., and Knoll, A. (2023). A Review of Safe Reinforcement Learning: Methods, Theory and Applications. arXiv."},{"key":"ref_28","first-page":"1","article-title":"Risk-constrained reinforcement learning with percentile risk criteria","volume":"18","author":"Chow","year":"2018","journal-title":"J. Mach. Learn. Res."},{"key":"ref_29","unstructured":"Achiam, J., Held, D., Tamar, A., and Abbeel, P. (2017, January 6\u201311). Constrained Policy Optimization. Proceedings of the 34th International Conference on Machine Learning, Sydney, NSW, Australia."},{"key":"ref_30","unstructured":"Chow, Y., Nachum, O., Faust, A., Duenez-Guzman, E., and Ghavamzadeh, M. (2019). Lyapunov-based safe policy optimization for continuous control. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Rasmussen, C.E. (2003). Gaussian processes in machine learning. Summer School on Machine Learning, Springer.","DOI":"10.1007\/978-3-540-28650-9_4"},{"key":"ref_32","first-page":"8092","article-title":"A lyapunov-based approach to safe reinforcement learning","volume":"31","author":"Chow","year":"2018","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_33","first-page":"084501","article-title":"Hybrid traffic flow model for intelligent vehicles exiting to off-ramp","volume":"67","author":"Hao","year":"2018","journal-title":"Acta Phys. Sin."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"1611","DOI":"10.1109\/TVT.2022.3214515","article-title":"An evolutionary learning framework of lane-changing control for autonomous vehicles at freeway off-ramps","volume":"72","author":"Dong","year":"2022","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"3104","DOI":"10.1109\/TITS.2019.2925319","article-title":"Route control strategies for autonomous vehicles exiting to off-ramps","volume":"21","author":"Dong","year":"2019","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_36","unstructured":"Stooke, A., Achiam, J., and Abbeel, P. (2020, January 13\u201318). Responsive safety in reinforcement learning by pid lagrangian methods. Proceedings of the International Conference on Machine Learning, PMLR, Virtual Event."},{"key":"ref_37","unstructured":"Sootla, A., Cowen-Rivers, A.I., Jafferjee, T., Wang, Z., Mguni, D.H., Wang, J., and Ammar, H. (2022, January 17\u201323). Saut\u00e9 rl: Almost surely safe reinforcement learning using state augmentation. Proceedings of the International Conference on Machine Learning, Baltimore, MD, USA."},{"key":"ref_38","first-page":"34464","article-title":"Enhancing safe exploration using safety state augmentation","volume":"35","author":"Sootla","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_39","unstructured":"Ji, J., Zhou, J., Zhang, B., Dai, J., Pan, X., Sun, R., Huang, W., Geng, Y., Liu, M., and Yang, Y. (2023). Omnisafe: An infrastructure for accelerating safe reinforcement learning research. arXiv."},{"key":"ref_40","unstructured":"Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018, January 10\u201315). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_41","unstructured":"Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. (2017, January 13\u201315). CARLA: An open urban driving simulator. Proceedings of the Conference on Robot Learning, Mountain View, CA, USA."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/16\/5187\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:34:49Z","timestamp":1760110489000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/16\/5187"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,8,11]]},"references-count":41,"journal-issue":{"issue":"16","published-online":{"date-parts":[[2024,8]]}},"alternative-id":["s24165187"],"URL":"https:\/\/doi.org\/10.3390\/s24165187","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2024,8,11]]}}}