{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T19:45:05Z","timestamp":1782848705234,"version":"3.54.5"},"reference-count":43,"publisher":"MDPI AG","issue":"21","license":[{"start":{"date-parts":[[2022,11,1]],"date-time":"2022-11-01T00:00:00Z","timestamp":1667260800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Artificial Intelligence based modular Architecture Implementation and Validation for Autonomous Driving (AIVATAR) project","award":["PID2021-126623OB-I00"],"award-info":[{"award-number":["PID2021-126623OB-I00"]}]},{"name":"Artificial Intelligence based modular Architecture Implementation and Validation for Autonomous Driving (AIVATAR) project","award":["P2018\/NMT-4331"],"award-info":[{"award-number":["P2018\/NMT-4331"]}]},{"name":"RoboCity2030-DIH-CM project","award":["PID2021-126623OB-I00"],"award-info":[{"award-number":["PID2021-126623OB-I00"]}]},{"name":"RoboCity2030-DIH-CM project","award":["P2018\/NMT-4331"],"award-info":[{"award-number":["P2018\/NMT-4331"]}]},{"name":"Programas de actividades I+D (CAM)","award":["PID2021-126623OB-I00"],"award-info":[{"award-number":["PID2021-126623OB-I00"]}]},{"name":"Programas de actividades I+D (CAM)","award":["P2018\/NMT-4331"],"award-info":[{"award-number":["P2018\/NMT-4331"]}]},{"name":"EU Structural Funds and Scholarship","award":["PID2021-126623OB-I00"],"award-info":[{"award-number":["PID2021-126623OB-I00"]}]},{"name":"EU Structural Funds and Scholarship","award":["P2018\/NMT-4331"],"award-info":[{"award-number":["P2018\/NMT-4331"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>Intersections are considered one of the most complex scenarios in a self-driving framework due to the uncertainty in the behaviors of surrounding vehicles and the different types of scenarios that can be found. To deal with this problem, we provide a Deep Reinforcement Learning approach for intersection handling, which is combined with Curriculum Learning to improve the training process. The state space is defined by two vectors, containing adversaries and ego vehicle information. We define a features extractor module and an actor\u2013critic approach combined with Curriculum Learning techniques, adding complexity to the environment by increasing the number of vehicles. In order to address a complete autonomous driving system, a hybrid architecture is proposed. The operative level generates the driving commands, the strategy level defines the trajectory and the tactical level executes the high-level decisions. This high-level decision system is the main goal of this research. To address realistic experiments, we set up three scenarios: intersections with traffic lights, intersections with traffic signs and uncontrolled intersections. The results of this paper show that a Proximal Policy Optimization algorithm can infer ego vehicle-desired behavior for different intersection scenarios based only on the behavior of adversarial vehicles.<\/jats:p>","DOI":"10.3390\/s22218373","type":"journal-article","created":{"date-parts":[[2022,11,2]],"date-time":"2022-11-02T08:15:12Z","timestamp":1667376912000},"page":"8373","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":64,"title":["Reinforcement Learning-Based Autonomous Driving at Intersections in CARLA Simulator"],"prefix":"10.3390","volume":"22","author":[{"given":"Rodrigo","family":"Guti\u00e9rrez-Moreno","sequence":"first","affiliation":[{"name":"Department of Electronics, University of Alcal\u00e1, 28805 Alcal\u00e1 de Henares, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4179-6100","authenticated-orcid":false,"given":"Rafael","family":"Barea","sequence":"additional","affiliation":[{"name":"Department of Electronics, University of Alcal\u00e1, 28805 Alcal\u00e1 de Henares, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8145-9045","authenticated-orcid":false,"given":"Elena","family":"L\u00f3pez-Guill\u00e9n","sequence":"additional","affiliation":[{"name":"Department of Electronics, University of Alcal\u00e1, 28805 Alcal\u00e1 de Henares, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-5101-0485","authenticated-orcid":false,"given":"Javier","family":"Araluce","sequence":"additional","affiliation":[{"name":"Department of Electronics, University of Alcal\u00e1, 28805 Alcal\u00e1 de Henares, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-0087-3077","authenticated-orcid":false,"given":"Luis M.","family":"Bergasa","sequence":"additional","affiliation":[{"name":"Department of Electronics, University of Alcal\u00e1, 28805 Alcal\u00e1 de Henares, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"1968","published-online":{"date-parts":[[2022,11,1]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1007\/s10111-013-0254-y","article-title":"How do environmental characteristics at intersections change in their relevance for drivers before entering an intersection: Analysis of drivers\u2019 gaze and driving behavior in a driving simulator study","volume":"16","author":"Werneke","year":"2014","journal-title":"Cogn. Technol."},{"key":"ref_2","unstructured":"NHTSA (2019). Traffic Safety Facts 2019."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"374","DOI":"10.1007\/s42154-020-00113-1","article-title":"Deep Reinforcement Learning Enabled Decision-Making for Autonomous Driving at Intersections","volume":"3","author":"Li","year":"2020","journal-title":"Automot. Innov."},{"key":"ref_4","doi-asserted-by":"crossref","unstructured":"Qiao, Z., Muelling, K., Dolan, J.M., Palanisamy, P., and Mudalige, P. (2018, January 26\u201330). Automatically Generated Curriculum based Reinforcement Learning for Autonomous Vehicles in Urban Environment. Proceedings of the 2018 IEEE Intelligent Vehicles Symposium (IV), Changshu, China.","DOI":"10.1109\/IVS.2018.8500603"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"Aoki, S., and Rajkumar, R. (2019, January 18\u201321). V2V-based Synchronous Intersection Protocols for Mixed Traffic of Human-Driven and Self-Driving Vehicles. Proceedings of the 2019 IEEE 25th International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA), Hangzhou, China.","DOI":"10.1109\/RTCSA.2019.8864572"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.23919\/JCC.2021.07.001","article-title":"V2I based environment perception for autonomous vehicles at intersections","volume":"18","author":"Duan","year":"2021","journal-title":"China Commun."},{"key":"ref_7","doi-asserted-by":"crossref","unstructured":"Isele, D., Rahimi, R., Cosgun, A., Subramanian, K., and Fujimura, K. (2018, January 21\u201325). Navigating Occluded Intersections with Autonomous Vehicles Using Deep Reinforcement Learning. Proceedings of the 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia.","DOI":"10.1109\/ICRA.2018.8461233"},{"key":"ref_8","unstructured":"Zhang, W.B., de La Fortelle, A., Acarman, T., and Yang, M. (2017, January 11\u201314). Towards full automated drive in urban environments: A demonstration in GoMentum Station, California. Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV 2017), Los Angeles, CA, USA."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Xu, H., Gao, Y., Yu, F., and Darrell, T. (2016). End-to-end Learning of Driving Models from Large-scale Video Datasets. arXiv.","DOI":"10.1109\/CVPR.2017.376"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Kendall, A., Hawke, J., Janz, D., Mazur, P., Reda, D., Allen, J., Lam, V., Bewley, A., and Shah, A. (2018). Learning to Drive in a Day. arXiv.","DOI":"10.1109\/ICRA.2019.8793742"},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Anzalone, L., Barra, S., and Nappi, M. (2021, January 19\u201322). Reinforced Curriculum Learning For Autonomous Driving In Carla. Proceedings of the 2021 IEEE International Conference on Image Processing (ICIP), Anchorage, AK, USA.","DOI":"10.1109\/ICIP42928.2021.9506673"},{"key":"ref_12","unstructured":"Behrisch, M., Bieker, L., Erdmann, J., and Krajzewicz, D. (2011, January 23\u201328). SUMO\u2014Simulation of Urban MObility: An overview. Proceedings of the SIMUL 2011, Third International Conference on Advances in System Simulation, Barcelona, Spain."},{"key":"ref_13","unstructured":"Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. (2017, January 13\u201315). CARLA: An Open Urban Driving Simulator. Proceedings of the 1st Annual Conference on Robot Learning, Mountain View, CA, USA."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Wang, P., Li, H., and Chan, C. (2019). Continuous Control for Automated Lane Change Behavior Based on Deep Deterministic Policy Gradient Algorithm. arXiv.","DOI":"10.1109\/IVS.2019.8813903"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Paden, B., C\u00e1p, M., Yong, S.Z., Yershov, D.S., and Frazzoli, E. (2016). A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles. arXiv.","DOI":"10.1109\/TIV.2016.2578706"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"1135","DOI":"10.1109\/TITS.2015.2498841","article-title":"A Review of Motion Planning Techniques for Automated Vehicles","volume":"17","author":"Nashashibi","year":"2016","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Mirchevska, B., Pek, C., Werling, M., Althoff, M., and Boedecker, J. (2018, January 4\u20137). High-level Decision Making for Safe and Reasonable Autonomous Lane Changing using Reinforcement Learning. Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA.","DOI":"10.1109\/ITSC.2018.8569448"},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Ye, F., Zhang, S., Wang, P., and Chan, C. (2021). A Survey of Deep Reinforcement Learning Algorithms for Motion Planning and Control of Autonomous Vehicles. arXiv.","DOI":"10.1109\/IV48863.2021.9575880"},{"key":"ref_19","unstructured":"Bojarski, M., Testa, D.D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L.D., Monfort, M., Muller, U., and Zhang, J. (2016). End to End Learning for Self-Driving Cars. arXiv."},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Moghadam, M., Alizadeh, A., Tekin, E., and Elkaim, G.H. (2020). An End-to-end Deep Reinforcement Learning Approach for the Long-term Short-term Planning on the Frenet Space. arXiv.","DOI":"10.1109\/CASE49439.2021.9551598"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Chopra, R., and Roy, S. (2020). End-to-End Reinforcement Learning for Self-driving Car. Advanced Computing and Intelligent Engineering, Springer.","DOI":"10.1007\/978-981-15-1081-6_5"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Tram, T., Jansson, A., Gr\u00f6nberg, R., Ali, M., and Sj\u00f6berg, J. (2018, January 4\u20137). Learning Negotiating Behavior Between Cars in Intersections using Deep Q-Learning. Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA.","DOI":"10.1109\/ITSC.2018.8569316"},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Tram, T., Batkovic, I., Ali, M., and Sj\u00f6berg, J. (2019, January 27\u201330). Learning When to Drive in Intersections by Combining Reinforcement Learning and Model Predictive Control. Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand.","DOI":"10.1109\/ITSC.2019.8916922"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Kamran, D., Lopez, C.F., Lauer, M., and Stiller, C. (2020). Risk-Aware High-level Decisions for Automated Driving at Occluded Intersections with Reinforcement Learning. arXiv.","DOI":"10.1109\/IV47402.2020.9304606"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Bouton, M., Nakhaei, A., Fujimura, K., and Kochenderfer, M.J. (2019, January 9\u201312). Safe Reinforcement Learning with Scene Decomposition for Navigating Complex Urban Environments. Proceedings of the 2019 IEEE Intelligent Vehicles Symposium (IV), Paris, France.","DOI":"10.1109\/IVS.2019.8813803"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Bouton, M., Cosgun, A., and Kochenderfer, M.J. (2017, January 11\u201314). Belief state planning for autonomously navigating urban intersections. Proceedings of the 2017 IEEE Intelligent Vehicles Symposium (IV), Los Angeles, CA, USA.","DOI":"10.1109\/IVS.2017.7995818"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Bouton, M., Nakhaei, A., Fujimura, K., and Kochenderfer, M.J. (2019, January 27\u201330). Cooperation-Aware Reinforcement Learning for Merging in Dense Traffic. Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand.","DOI":"10.1109\/ITSC.2019.8916924"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Shu, K., Yu, H., Chen, X., Chen, L., Wang, Q., Li, L., and Cao, D. (2020). Autonomous Driving at Intersections: A Critical-Turning-Point Approach for Left Turns. arXiv.","DOI":"10.1109\/ITSC45102.2020.9294754"},{"key":"ref_30","doi-asserted-by":"crossref","unstructured":"Kurzer, K., Sch\u00f6rner, P., Albers, A., Thomsen, H., Daaboul, K., and Z\u00f6llner, J.M. (2021). Generalizing Decision Making for Automated Driving with an Invariant Environment Representation using Deep Reinforcement Learning. arXiv.","DOI":"10.1109\/IV48863.2021.9575669"},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Soviany, P., Ionescu, R.T., Rota, P., and Sebe, N. (2021). Curriculum Learning: A Survey. arXiv.","DOI":"10.1007\/s11263-022-01611-x"},{"key":"ref_32","first-page":"4555","article-title":"A Survey on Curriculum Learning","volume":"44","author":"Wang","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell."},{"key":"ref_33","doi-asserted-by":"crossref","unstructured":"Diaz-Diaz, A., Oca\u00f1a, M., Llamazares, A., G\u00f3mez-Hu\u00e9lamo, C., Revenga, P., and Bergasa, L.M. (2022, January 5\u20139). HD maps: Exploiting OpenDRIVE potential for Path Planning and Map Monitoring. Proceedings of the 2022 IEEE Intelligent Vehicles Symposium (IV), Aachen, Germany.","DOI":"10.1109\/IV51971.2022.9827297"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., and Ng, A.Y. (2009, January 12\u201317). ROS: An open-source Robot Operating System. Proceedings of the ICRA Workshop on Open Source Software, Kobe, Japan.","DOI":"10.1109\/MRA.2010.936956"},{"key":"ref_35","doi-asserted-by":"crossref","unstructured":"Guti\u00e9rrez, R., L\u00f3pez-Guill\u00e9n, E., Bergasa, L.M., Barea, R., P\u00e9rez, \u00d3., G\u00f3mez Hu\u00e9lamo, C., Arango, J.F., del Egido, J., and L\u00f3pez, J. (2020). A Waypoint Tracking Controller for Autonomous Road Vehicles Using ROS Framework. Sensors, 20.","DOI":"10.3390\/s20144062"},{"key":"ref_36","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv."},{"key":"ref_37","unstructured":"van Hasselt, H., Guez, A., Hessel, M., Mnih, V., and Silver, D. (2016). Learning values across many orders of magnitude. arXiv."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1805","DOI":"10.1103\/PhysRevE.62.1805","article-title":"Congested traffic states in empirical observations and microscopic simulations","volume":"62","author":"Treiber","year":"2000","journal-title":"Phys. Rev. E"},{"key":"ref_39","unstructured":"Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym. arXiv."},{"key":"ref_40","doi-asserted-by":"crossref","unstructured":"Wegener, A., Pi\u00f3rkowski, M., Raya, M., Hellbr\u00fcck, H., Fischer, S., and Hubaux, J.P. (2008, January 14\u201317). TraCI: An Interface for Coupling Road Traffic and Network Simulators. Proceedings of the 11th Communications and Networking Simulation Symposium, Ottawa, ON, Canada.","DOI":"10.1145\/1400713.1400740"},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"1231","DOI":"10.1177\/0278364913491297","article-title":"Vision meets Robotics: The KITTI Dataset","volume":"32","author":"Geiger","year":"2013","journal-title":"Int. J. Robot. Res."},{"key":"ref_42","doi-asserted-by":"crossref","unstructured":"Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., and Beijbom, O. (2018). PointPillars: Fast Encoders for Object Detection from Point Clouds. arXiv.","DOI":"10.1109\/CVPR.2019.01298"},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Arango, J.F., Bergasa, L.M., Revenga, P., Barea, R., L\u00f3pez-Guill\u00e9n, E., G\u00f3mez-Hu\u00e9lamo, C., Araluce, J., and Guti\u00e9rrez, R. (2020). Drive-By-Wire Development Process Based on ROS for an Autonomous Electric Vehicle. Sensors, 20.","DOI":"10.3390\/s20216121"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/21\/8373\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T01:07:14Z","timestamp":1760144834000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/22\/21\/8373"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,11,1]]},"references-count":43,"journal-issue":{"issue":"21","published-online":{"date-parts":[[2022,11]]}},"alternative-id":["s22218373"],"URL":"https:\/\/doi.org\/10.3390\/s22218373","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,11,1]]}}}