{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,23]],"date-time":"2026-03-23T22:45:26Z","timestamp":1774305926060,"version":"3.50.1"},"reference-count":47,"publisher":"MDPI AG","issue":"10","license":[{"start":{"date-parts":[[2024,5,15]],"date-time":"2024-05-15T00:00:00Z","timestamp":1715731200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Key Research and Development Program of China","award":["2021YFB2501403"],"award-info":[{"award-number":["2021YFB2501403"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Autonomous driving has the potential to revolutionize transportation, but developing safe and reliable systems remains a significant challenge. Reinforcement learning (RL) has emerged as a promising approach for learning optimal control policies in complex driving environments. However, existing RL-based methods often suffer from low sample efficiency and lack explicit safety constraints, leading to unsafe behaviors. In this paper, we propose a novel framework for safe reinforcement learning in autonomous driving that addresses these limitations. Our approach incorporates a latent dynamic model that learns the underlying dynamics of the environment from bird\u2019s-eye view images, enabling efficient learning and reducing the risk of safety violations by generating synthetic data. Furthermore, we introduce state-wise safety constraints through a barrier function, ensuring safety at each state by encoding constraints directly into the learning process. Experimental results in the CARLA simulator demonstrate that our framework significantly outperforms baseline methods in terms of both driving performance and safety. Our work advances the development of safe and efficient autonomous driving systems by leveraging the power of reinforcement learning with explicit safety considerations.<\/jats:p>","DOI":"10.3390\/s24103139","type":"journal-article","created":{"date-parts":[[2024,5,15]],"date-time":"2024-05-15T06:14:42Z","timestamp":1715753682000},"page":"3139","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":4,"title":["Safe Autonomous Driving with Latent Dynamics and State-Wise Constraints"],"prefix":"10.3390","volume":"24","author":[{"given":"Changquan","family":"Wang","sequence":"first","affiliation":[{"name":"Institute of Microelectronics of the Chinese Academy of Sciences, Beijing 100029, China"},{"name":"University of Chinese Academy of Sciences, Beijing 101408, China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yun","family":"Wang","sequence":"additional","affiliation":[{"name":"Institute of Microelectronics of the Chinese Academy of Sciences, Beijing 100029, China"},{"name":"University of Chinese Academy of Sciences, Beijing 101408, China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2024,5,15]]},"reference":[{"key":"ref_1","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv."},{"key":"ref_2","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv."},{"key":"ref_3","unstructured":"Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018, January 10\u201315). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Wolf, P., Hubschneider, C., Weber, M., Bauer, A., H\u00e4rtl, J., D\u00fcrr, F., and Z\u00f6llner, J.M. (2017, January 11\u201314). Learning how to drive in a real world simulation with deep q-networks. Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA.","DOI":"10.1109\/IVS.2017.7995727"},{"key":"ref_5","unstructured":"Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D. (2015). Continuous control with deep reinforcement learning. arXiv."},{"key":"ref_6","doi-asserted-by":"crossref","unstructured":"Chen, J., Wang, Z., and Tomizuka, M. (2018, January 26\u201330). Deep hierarchical reinforcement learning for autonomous driving with distinct behaviors. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China.","DOI":"10.1109\/IVS.2018.8500368"},{"key":"ref_7","unstructured":"Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. (2019, January 10\u201315). Learning latent dynamics for planning from pixels. Proceedings of the International Conference on Machine Learning, Long Beach, CA, USA."},{"key":"ref_8","unstructured":"Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. (2019). Dream to Control: Learning Behaviors by Latent Imagination. arXiv."},{"key":"ref_9","unstructured":"Hafner, D., Lillicrap, T., Norouzi, M., and Ba, J. (2020). Mastering Atari with Discrete World Models. arXiv."},{"key":"ref_10","unstructured":"Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. (2023). Mastering Diverse Domains through World Models. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Chen, J., Yuan, B., and Tomizuka, M. (2019, January 27\u201330). Model-free deep reinforcement learning for urban autonomous driving. Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand.","DOI":"10.1109\/ITSC.2019.8917306"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"5068","DOI":"10.1109\/TITS.2020.3046646","article-title":"Interpretable end-to-end urban autonomous driving with latent deep reinforcement learning","volume":"23","author":"Chen","year":"2021","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Toromanoff, M., Wirbel, E., and Moutarde, F. (2020, January 13\u201319). End-to-end model-free reinforcement learning for urban driving using implicit affordances. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA.","DOI":"10.1109\/CVPR42600.2020.00718"},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Zhao, W., He, T., Chen, R., Wei, T., and Liu, C. (2023). State-wise safe reinforcement learning: A survey. arXiv.","DOI":"10.24963\/ijcai.2023\/763"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Xiao, W., Belta, C., and Cassandras, C.G. (2019, January 16\u201318). Decentralized merging control in traffic networks: A control barrier function approach. Proceedings of the 10th ACM\/IEEE International Conference on Cyber-Physical Systems, Montreal, QC, Canada.","DOI":"10.1145\/3302509.3311054"},{"key":"ref_16","unstructured":"Xiao, W., Wang, T.H., Chahine, M., Amini, A., Hasani, R., and Rus, D. (2022). Differentiable control barrier functions for vision-based end-to-end autonomous driving. arXiv."},{"key":"ref_17","unstructured":"Zhan, S.S., Wang, Y., Wu, Q., Jiao, R., Huang, C., and Zhu, Q. (2023). State-wise safe reinforcement learning with pixel observations. arXiv."},{"key":"ref_18","unstructured":"Hogewind, Y., Simao, T.D., Kachman, T., and Jansen, N. (2022, January 25\u201329). Safe reinforcement learning from pixels using a stochastic latent representation. Proceedings of the Eleventh International Conference on Learning Representations, Virtual."},{"key":"ref_19","unstructured":"Ray, A., Achiam, J., and Amodei, D. (2019). Benchmarking safe exploration in deep reinforcement learning. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Zhang, Z., Liniger, A., Dai, D., Yu, F., and Van Gool, L. (2021, January 11\u201317). End-to-end urban driving by imitating a reinforcement learning coach. Proceedings of the IEEE\/CVF International Conference on Computer Vision, Montreal, QC, Canada.","DOI":"10.1109\/ICCV48922.2021.01494"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"3861","DOI":"10.1109\/TAC.2016.2638961","article-title":"Control barrier function based quadratic programs for safety critical systems","volume":"62","author":"Ames","year":"2016","journal-title":"IEEE Trans. Autom. Control"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"1749","DOI":"10.1109\/TRO.2022.3232542","article-title":"Safe control with learned certificates: A survey of neural lyapunov, barrier, and contraction methods for robotics and control","volume":"39","author":"Dawson","year":"2023","journal-title":"IEEE Trans. Robot."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Yang, Z., Huang, C., Chen, X., Lin, W., and Liu, Z. (2016, January 9\u201311). A linear programming relaxation based approach for generating barrier certificates of hybrid systems. Proceedings of the FM 2016: Formal Methods: 21st International Symposium, Limassol, Cyprus. Proceedings 21.","DOI":"10.1007\/978-3-319-48989-6_44"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Wang, Y., Zhan, S., Wang, Z., Huang, C., Wang, Z., Yang, Z., and Zhu, Q. (2023, January 9\u201312). Joint differentiable optimization and verification for certified reinforcement learning. Proceedings of the ACM\/IEEE 14th International Conference on Cyber-Physical Systems (with CPS-IoT Week 2023), San Antonio, TX, USA.","DOI":"10.1145\/3576841.3585919"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Ames, A.D., Coogan, S., Egerstedt, M., Notomista, G., Sreenath, K., and Tabuada, P. (2019, January 25\u201328). Control barrier functions: Theory and applications. Proceedings of the 2019 18th European control conference (ECC), Naples, Italy.","DOI":"10.23919\/ECC.2019.8796030"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Choi, J., Castaneda, F., Tomlin, C.J., and Sreenath, K. (2020). Reinforcement learning for safety-critical control under model uncertainty, using control lyapunov functions and control barrier functions. arXiv.","DOI":"10.15607\/RSS.2020.XVI.088"},{"key":"ref_27","unstructured":"Wang, Y., Zhan, S.S., Jiao, R., Wang, Z., Jin, W., Yang, Z., Wang, Z., Huang, C., and Zhu, Q. (2023, January 23\u201329). Enforcing hard constraints with soft barriers: Safe reinforcement learning in unknown stochastic environments. Proceedings of the International Conference on Machine Learning, Honolulu, HI, USA."},{"key":"ref_28","unstructured":"Cheng, R., Orosz, G., Murray, R.M., and Burdick, J.W. (February, January 27). End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA."},{"key":"ref_29","unstructured":"Dawson, C., Qin, Z., Gao, S., and Fan, C. (2022, January 14\u201318). Safe nonlinear control using robust neural lyapunov-barrier functions. Proceedings of the Conference on Robot Learning, Auckland, New Zealand."},{"key":"ref_30","unstructured":"Ferlez, J., Elnaggar, M., Shoukry, Y., and Fleming, C. (2020). Shieldnn: A provably safe nn filter for unsafe nn controllers. arXiv."},{"key":"ref_31","doi-asserted-by":"crossref","first-page":"1928","DOI":"10.1109\/LCSYS.2021.3136486","article-title":"Safe and sample-efficient reinforcement learning for clustered dynamic environments","volume":"6","author":"Chen","year":"2021","journal-title":"IEEE Control Syst. Lett."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1214\/18-SS121","article-title":"A review of dynamic network models with latent variables","volume":"12","author":"Kim","year":"2018","journal-title":"Stat. Surv."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1646","DOI":"10.1080\/01621459.2014.988214","article-title":"Latent space models for dynamic networks","volume":"110","author":"Sewell","year":"2015","journal-title":"J. Am. Stat. Assoc."},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1145\/1117454.1117459","article-title":"Dynamic social network analysis using latent space models","volume":"7","author":"Sarkar","year":"2005","journal-title":"ACM Sigkdd Explor. Newsl."},{"key":"ref_35","first-page":"1","article-title":"A survey of reinforcement learning algorithms for dynamically varying environments","volume":"54","author":"Padakandla","year":"2021","journal-title":"ACM Comput. Surv. CSUR"},{"key":"ref_36","unstructured":"Levine, S. (2018). Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv."},{"key":"ref_37","unstructured":"Lee, K., Seo, Y., Lee, S., Lee, H., and Shin, J. (2020, January 13\u201318). Context-aware dynamics model for generalization in model-based reinforcement learning. Proceedings of the International Conference on Machine Learning, Virtual."},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Hao, Z., Zhu, H., Chen, W., and Cai, R. (2023, January 20\u201323). Latent Causal Dynamics Model for Model-Based Reinforcement Learning. Proceedings of the International Conference on Neural Information Processing, Changsha, China.","DOI":"10.1007\/978-981-99-8082-6_17"},{"key":"ref_39","unstructured":"Li, Y., Song, J., and Ermon, S. (2017). Inferring the latent structure of human decision-making from raw visual inputs. arXiv."},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"8707","DOI":"10.1109\/TVT.2021.3098321","article-title":"Interpretable decision-making for autonomous vehicles at highway on-ramps with latent space reinforcement learning","volume":"70","author":"Wang","year":"2021","journal-title":"IEEE Trans. Veh. Technol."},{"key":"ref_41","unstructured":"Qin, Z., Zhang, K., Chen, Y., Chen, J., and Fan, C. (2021). Learning safe multi-agent control with decentralized neural barrier certificates. arXiv."},{"key":"ref_42","first-page":"1","article-title":"CARLA: An open urban driving simulator","volume":"78","author":"Dosovitskiy","year":"2017","journal-title":"Proc. Mach. Learn. Res."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Wen, L., Duan, J., Li, S.E., Xu, S., and Peng, H. (2020, January 20\u201323). Safe reinforcement learning for autonomous vehicles through parallel constrained policy optimization. Proceedings of the 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece.","DOI":"10.1109\/ITSC45102.2020.9294262"},{"key":"ref_44","unstructured":"Shalev-Shwartz, S., Shammah, S., and Shashua, A. (2017). On a formal model of safe and scalable self-driving cars. arXiv."},{"key":"ref_45","unstructured":"Bouton, M., Karlsson, J., Nakhaei, A., Fujimura, K., Kochenderfer, M.J., and Tumova, J. (2019). Reinforcement learning with probabilistic guarantees for autonomous driving. arXiv."},{"key":"ref_46","unstructured":"Achiam, J., Held, D., Tamar, A., and Abbeel, P. (2017, January 6\u201311). Constrained policy optimization. Proceedings of the International Conference on Machine Learning, Sydney, Australia."},{"key":"ref_47","unstructured":"(2024, March 02). Carla Autonomous Driving Leaderboard. Available online: https:\/\/leaderboard.carla.org\/."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/10\/3139\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T14:42:42Z","timestamp":1760107362000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/10\/3139"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,5,15]]},"references-count":47,"journal-issue":{"issue":"10","published-online":{"date-parts":[[2024,5]]}},"alternative-id":["s24103139"],"URL":"https:\/\/doi.org\/10.3390\/s24103139","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,5,15]]}}}