{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,16]],"date-time":"2026-06-16T13:01:14Z","timestamp":1781614874933,"version":"3.54.5"},"reference-count":31,"publisher":"MDPI AG","issue":"19","license":[{"start":{"date-parts":[[2024,9,25]],"date-time":"2024-09-25T00:00:00Z","timestamp":1727222400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"publisher","award":["52202495"],"award-info":[{"award-number":["52202495"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"publisher","award":["52202494"],"award-info":[{"award-number":["52202494"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"publisher","award":["52394261"],"award-info":[{"award-number":["52394261"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100001809","name":"National Science Foundation of China","doi-asserted-by":"publisher","award":["202302013"],"award-info":[{"award-number":["202302013"]}],"id":[{"id":"10.13039\/501100001809","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Science and Technology Development Project of Jilin Province","award":["52202495"],"award-info":[{"award-number":["52202495"]}]},{"name":"Science and Technology Development Project of Jilin Province","award":["52202494"],"award-info":[{"award-number":["52202494"]}]},{"name":"Science and Technology Development Project of Jilin Province","award":["52394261"],"award-info":[{"award-number":["52394261"]}]},{"name":"Science and Technology Development Project of Jilin Province","award":["202302013"],"award-info":[{"award-number":["202302013"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Urban traffic congestion poses significant economic and environmental challenges worldwide. To mitigate these issues, Adaptive Traffic Signal Control (ATSC) has emerged as a promising solution. Recent advancements in deep reinforcement learning (DRL) have further enhanced ATSC\u2019s capabilities. This paper introduces a novel DRL-based ATSC approach named the Sequence Decision Transformer (SDT), employing DRL enhanced with attention mechanisms and leveraging the robust capabilities of sequence decision models, akin to those used in advanced natural language processing, adapted here to tackle the complexities of urban traffic management. Firstly, the ATSC problem is modeled as a Markov Decision Process (MDP), with the observation space, action space, and reward function carefully defined. Subsequently, we propose SDT, specifically tailored to solve the MDP problem. The SDT model uses a transformer-based architecture with an encoder and decoder in an actor\u2013critic structure. The encoder processes observations and outputs, both encoded data for the decoder, and value estimates for parameter updates. The decoder, as the policy network, outputs the agent\u2019s actions. Proximal Policy Optimization (PPO) is used to update the policy network based on historical data, enhancing decision-making in ATSC. This approach significantly reduces training times, effectively manages larger observation spaces, captures dynamic changes in traffic conditions more accurately, and enhances traffic throughput. Finally, the SDT model is trained and evaluated in synthetic scenarios by comparing the number of vehicles, average speed, and queue length against three baselines, including PPO, a DQN tailored for ATSC, and FRAP, a state-of-the-art ATSC algorithm. SDT shows improvements of 26.8%, 150%, and 21.7% over traditional ATSC algorithms, and 18%, 30%, and 15.6% over the FRAP. This research underscores the potential of integrating Large Language Models (LLMs) with DRL for traffic management, offering a promising solution to urban congestion.<\/jats:p>","DOI":"10.3390\/s24196202","type":"journal-article","created":{"date-parts":[[2024,9,25]],"date-time":"2024-09-25T17:12:04Z","timestamp":1727284324000},"page":"6202","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["Sequence Decision Transformer for Adaptive Traffic Signal Control"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1597-1961","authenticated-orcid":false,"given":"Rui","family":"Zhao","sequence":"first","affiliation":[{"name":"College of Automotive Engineering, Jilin University, Changchun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0001-0124-6040","authenticated-orcid":false,"given":"Haofeng","family":"Hu","sequence":"additional","affiliation":[{"name":"College of Automotive Engineering, Jilin University, Changchun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0002-7824-7751","authenticated-orcid":false,"given":"Yun","family":"Li","sequence":"additional","affiliation":[{"name":"Graduate School of Information and Science Technology, The University of Tokyo, Tokyo 113-8654, Japan"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0003-8309-7396","authenticated-orcid":false,"given":"Yuze","family":"Fan","sequence":"additional","affiliation":[{"name":"College of Automotive Engineering, Jilin University, Changchun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4195-5033","authenticated-orcid":false,"given":"Fei","family":"Gao","sequence":"additional","affiliation":[{"name":"National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University, Changchun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0153-7040","authenticated-orcid":false,"given":"Zhenhai","family":"Gao","sequence":"additional","affiliation":[{"name":"National Key Laboratory of Automotive Chassis Integration and Bionics, Jilin University, Changchun 130025, China"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2024,9,25]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"825","DOI":"10.1016\/j.future.2019.02.058","article-title":"Smart fog based workflow for traffic control networks","volume":"97","author":"Wu","year":"2019","journal-title":"Future Gener. Comput. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Bao, Z., Ng, S.T., Yu, G., Zhang, X., and Ou, Y. (2023). The effect of the built environment on spatial-temporal pattern of traffic congestion in a satellite city in emerging economies. Dev. Built Environ., 14.","DOI":"10.1016\/j.dibe.2023.100173"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"4919","DOI":"10.1109\/TITS.2020.2984033","article-title":"Fuzzy inference enabled deep reinforcement learning-based traffic light control for intelligent transportation system","volume":"22","author":"Kumar","year":"2020","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"12","DOI":"10.1145\/3447556.3447565","article-title":"Recent advances in reinforcement learning for traffic signal control: A survey of models and evaluation","volume":"22","author":"Wei","year":"2021","journal-title":"ACM SIGKDD Explor. Newsl."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"313","DOI":"10.1016\/j.trc.2019.01.026","article-title":"Urban traffic signal control with connected and automated vehicles: A survey","volume":"101","author":"Guo","year":"2019","journal-title":"Transp. Res. Part C Emerg. Technol."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"9126","DOI":"10.1109\/TITS.2021.3091014","article-title":"Using reinforcement learning to control traffic signals in a real-world scenario: An approach based on linear function approximation","volume":"23","author":"Alegre","year":"2021","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Han, G., Liu, X., Wang, H., Dong, C., and Han, Y. (2024). An Attention Reinforcement Learning\u2013Based Strategy for Large-Scale Adaptive Traffic Signal Control System. J. Transp. Eng. Part A Syst., 150.","DOI":"10.1061\/JTEPBS.TEENG-8261"},{"key":"ref_8","first-page":"190","article-title":"The SCOOT on-line traffic signal optimisation technique","volume":"23","author":"Hunt","year":"1982","journal-title":"Traffic Eng. Control"},{"key":"ref_9","unstructured":"Luk, J. (1984). Two traffic-responsive area traffic control methods: SCAT and SCOOT. Traffic Eng. Control, 25."},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Sun, Q.W., Han, S.Y., Zhou, J., Chen, Y.H., and Yao, K. (2022, January 9\u201312). Deep Reinforcement-Learning-Based Adaptive Traffic Signal Control with Real-Time Queue Lengths. Proceedings of the 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Prague, Czech Republic.","DOI":"10.1109\/SMC53654.2022.9945292"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Kong, A.Y., Lu, B.X., Yang, C.Z., and Zhang, D.M. (2022, January 8\u201312). A deep reinforcement learning framework with memory network to coordinate traffic signal control. Proceedings of the 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), Macau, China.","DOI":"10.1109\/ITSC55140.2022.9921752"},{"key":"ref_12","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Noaeen, M., Naik, A., Goodman, L., Crebo, J., Abrar, T., Abad, Z.S.H., Bazzan, A.L., and Far, B. (2022). Reinforcement learning in urban network traffic signal control: A systematic literature review. Expert Syst. Appl., 199.","DOI":"10.1016\/j.eswa.2022.116830"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Wang, X., Sanner, S., and Abdulhai, B. (2022). A Critical Review of Traffic Signal Control and A Novel Unified View of Reinforcement Learning and Model Predictive Control Approaches for Adaptive Traffic Signal Control. arXiv.","DOI":"10.4337\/9781803929545.00029"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"278","DOI":"10.1061\/(ASCE)0733-947X(2003)129:3(278)","article-title":"Reinforcement learning for true adaptive traffic signal control","volume":"129","author":"Abdulhai","year":"2003","journal-title":"J. Transp. Eng."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Raeis, M., and Leon-Garcia, A. (2021, January 19\u201322). A deep reinforcement learning approach for fair traffic signal control. Proceedings of the 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), Indianapolis, IN, USA.","DOI":"10.1109\/ITSC48978.2021.9564847"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"208016","DOI":"10.1109\/ACCESS.2020.3034141","article-title":"Deep reinforcement learning for traffic signal control: A review","volume":"8","author":"Rasheed","year":"2020","journal-title":"IEEE Access"},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"17899","DOI":"10.1109\/TITS.2022.3159714","article-title":"A gain with no pain: Exploring intelligent traffic signal control for emergency vehicles","volume":"23","author":"Cao","year":"2022","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Ma, Z., Cui, T., Deng, W., Jiang, F., and Zhang, L. (2021). Adaptive optimization of traffic signal timing via deep reinforcement learning. J. Adv. Transp., 2021.","DOI":"10.1155\/2021\/6616702"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Zhang, G., Chang, F., Jin, J., Yang, F., and Huang, H. (2024). Multi-objective deep reinforcement learning approach for adaptive traffic signal control system with concurrent optimization of safety, efficiency, and decarbonization at intersections. Accid. Anal. Prev., 199.","DOI":"10.1016\/j.aap.2023.107451"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"247","DOI":"10.1109\/JAS.2016.7508798","article-title":"Traffic signal timing via deep reinforcement learning","volume":"3","author":"Li","year":"2016","journal-title":"IEEE CAA J. Autom. Sin."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Wei, H., Zheng, G., Yao, H., and Li, Z. (2018, January 19\u201323). Intellilight: A reinforcement learning approach for intelligent traffic light control. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, New York, NY, USA.","DOI":"10.1145\/3219819.3220096"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Nishi, T., Otaki, K., Hayakawa, K., and Yoshimura, T. (2018, January 4\u20137). Traffic signal control based on reinforcement learning with graph convolutional neural nets. Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA.","DOI":"10.1109\/ITSC.2018.8569301"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"4","DOI":"10.1109\/TNNLS.2020.2978386","article-title":"A comprehensive survey on graph neural networks","volume":"32","author":"Wu","year":"2020","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"6774","DOI":"10.1109\/TITS.2021.3062072","article-title":"Traffic signal control with reinforcement learning based on region-aware cooperative strategy","volume":"23","author":"Wang","year":"2021","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Zheng, G., Xiong, Y., Zang, X., Feng, J., Wei, H., Zhang, H., Li, Y., Xu, K., and Li, Z. (2019, January 3\u20137). Learning phase competition for traffic signal control. Proceedings of the 28th ACM International Conference on Information and Knowledge Management, Beijing, China.","DOI":"10.1145\/3357384.3357900"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Zhang, H., Liu, C., Zhang, W., Zheng, G., and Yu, Y. (2020, January 19\u201323). Generalight: Improving environment generalization of traffic signal control via meta reinforcement learning. Proceedings of the 29th ACM International Conference on Information & Knowledge Management, New York, NY, USA.","DOI":"10.1145\/3340531.3411859"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Du, W., Ye, J., Gu, J., Li, J., Wei, H., and Wang, G. (2023, January 7\u201314). Safelight: A reinforcement learning method toward collision-free traffic signal control. Proceedings of the AAAI Conference on Artificial Intelligence, Washington, DC, USA.","DOI":"10.1609\/aaai.v37i12.26729"},{"key":"ref_29","first-page":"16509","article-title":"Multi-agent reinforcement learning is a sequence modeling problem","volume":"35","author":"Wen","year":"2022","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Lopez, P.A., Behrisch, M., Bieker-Walz, L., Erdmann, J., Fl\u00f6tter\u00f6d, Y.P., Hilbrich, R., L\u00fccken, L., Rummel, J., Wagner, P., and Wie\u00dfner, E. (2018, January 4\u20137). Microscopic traffic simulation using sumo. Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA.","DOI":"10.1109\/ITSC.2018.8569938"},{"key":"ref_31","unstructured":"Gao, J., Shen, Y., Liu, J., Ito, M., and Shiratori, N. (2017). Adaptive traffic signal control: Deep reinforcement learning algorithm with experience replay and target network. arXiv."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/19\/6202\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T16:02:27Z","timestamp":1760112147000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/19\/6202"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,25]]},"references-count":31,"journal-issue":{"issue":"19","published-online":{"date-parts":[[2024,10]]}},"alternative-id":["s24196202"],"URL":"https:\/\/doi.org\/10.3390\/s24196202","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,25]]}}}