{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T03:03:51Z","timestamp":1760151831288,"version":"build-2065373602"},"reference-count":24,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2022,4,15]],"date-time":"2022-04-15T00:00:00Z","timestamp":1649980800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Informatics"],"abstract":"<jats:p>Autonomous driving is a research field that has received attention in recent years, with increasing applications of reinforcement learning (RL) algorithms. It is impractical to train an autonomous vehicle thoroughly in the physical space, i.e., the so-called \u2019real world\u2019; therefore, simulators are used in almost all training of autonomous driving algorithms. There are numerous autonomous driving simulators, very few of which are specifically targeted at RL. RL-based cars are challenging due to the variety of reward functions available. There is a lack of simulators addressing many central RL research tasks within autonomous driving, such as scene understanding, localization and mapping, planning and driving policies, and control, which have diverse requirements and goals. It is, therefore, challenging to prototype new RL projects with different simulators, especially when there is a need to examine several reward functions at once. This paper introduces a modified simulator based on the Udacity simulator, made for autonomous cars using RL. It creates reward functions, along with sensors to create a baseline implementation for RL-based vehicles. The modified simulator also resets the vehicle when it gets stuck or is in a non-terminating loop, making it more reliable. Overall, the paper seeks to make the prototyping of new systems simple, with the testing of different RL-based systems.<\/jats:p>","DOI":"10.3390\/informatics9020033","type":"journal-article","created":{"date-parts":[[2022,4,18]],"date-time":"2022-04-18T04:21:28Z","timestamp":1650255688000},"page":"33","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":2,"title":["Development of a Simulator for Prototyping Reinforcement Learning-Based Autonomous Cars"],"prefix":"10.3390","volume":"9","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-0967-221X","authenticated-orcid":false,"given":"Martin","family":"Holen","sequence":"first","affiliation":[{"name":"Centre for Artificial Intelligence Research, University of Agder, 4879 Grimstad, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4088-1642","authenticated-orcid":false,"given":"Kristian Muri","family":"Knausg\u00e5rd","sequence":"additional","affiliation":[{"name":"Top Research Centre Mechatronics, University of Agder, 4879 Grimstad, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6331-702X","authenticated-orcid":false,"given":"Morten","family":"Goodwin","sequence":"additional","affiliation":[{"name":"Centre for Artificial Intelligence Research, University of Agder, 4879 Grimstad, Norway"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,4,15]]},"reference":[{"key":"ref_1","unstructured":"Dosovitskiy, A., Ros, G., Codevilla, F., L\u00f3pez, A., and Koltun, V. (2017, January 13\u201315). CARLA: An Open Urban Driving Simulator. Proceedings of the 1st Annual Conference on Robot Learning, Mountain View, CA, USA."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Balaji, B., Mallya, S., Genc, S., Gupta, S., Dirac, L., Khare, V., Roy, G., Sun, T., Tao, Y., and Townsend, B. (2020). DeepRacer: Educational Autonomous Racing Platform for Experimentation with Sim2Real Reinforcement Learning. IEEE Int. Conf. Robot. Autom., 2746\u20132754.","DOI":"10.1109\/ICRA40945.2020.9197465"},{"key":"ref_3","first-page":"1879","article-title":"CoInCar-Sim: An Open-Source Simulation Framework for Cooperatively Interacting Automobiles","volume":"2018","author":"Naumann","year":"2018","journal-title":"IEEE Intell. Veh. Symp. Proc."},{"key":"ref_4","unstructured":"Quiter, C., and Rehn, A. (2022, February 01). Deepdrive\/Deepdrive: Deepdrive is a Simulator That Allows Anyone with a PC to Push the State-of-the-Art in Self-Driving. Available online: https:\/\/github.com\/deepdrive\/deepdrive."},{"key":"ref_5","unstructured":"(2022, February 01). Vector Informatik GmbH. DYNA4|Virtual Test Driving|Vector. Available online: https:\/\/www.vector.com\/int\/en\/products\/products-a-z\/software\/dyna4\/."},{"key":"ref_6","unstructured":"Li, Q., Peng, Z., Zhang, Q., Liu, C., and Zhou, B. (2020). Improving the Generalization of End-to-End Driving through Procedural Generation. arXiv."},{"key":"ref_7","unstructured":"Cai, P., Lee, Y., Luo, Y., and Hsu, D. (August, January 31). SUMMIT: A Simulator for Urban Driving in Massive Mixed Traffic. Proceedings of the IEEE International Conference on Robotics and Automation, Paris, France."},{"key":"ref_8","unstructured":"Berner, C., Brockman, G., Chan, B., Cheung, V., D\u0229biak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., and Hesse, C. (2019). Dota 2 with Large Scale Deep Reinforcement Learning. arXiv."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"350","DOI":"10.1038\/s41586-019-1724-z","article-title":"Grandmaster level in StarCraft II using multi-agent reinforcement learning","volume":"575","author":"Vinyals","year":"2019","journal-title":"Nature"},{"key":"ref_11","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1038\/nature16961","article-title":"Mastering the game of Go with deep neural networks and tree search","volume":"529","author":"Silver","year":"2016","journal-title":"Nature"},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"354","DOI":"10.1038\/nature24270","article-title":"Mastering the game of Go without human knowledge","volume":"550","author":"Silver","year":"2017","journal-title":"Nature"},{"key":"ref_13","unstructured":"Sharifzadeh, S., Chiotellis, I., Triebel, R., and Cremers, D. (2016). Learning to Drive using Inverse Reinforcement Learning and Deep Q-Networks. NeurIPS Workshop on Deep Learning for Action and Interaction. arXiv."},{"key":"ref_14","doi-asserted-by":"crossref","unstructured":"Liang, X., Wang, T., Yang, L., and Xing, E. (2018, January 8\u201314). CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving. Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany.","DOI":"10.1007\/978-3-030-01234-2_36"},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Kendall, A., Hawke, J., Janz, D., Mazur, P., Reda, D., Allen, J.M., Lam, V.D., Bewley, A., and Shah, A. (2019, January 20\u201324). Learning to drive in a day. Proceedings of the IEEE International Conference on Robotics and Automation, Montreal, QC, Canada.","DOI":"10.1109\/ICRA.2019.8793742"},{"key":"ref_16","doi-asserted-by":"crossref","first-page":"297","DOI":"10.1049\/iet-its.2019.0317","article-title":"Hierarchical Reinforcement Learning for Self-Driving Decision-Making without Reliance on Labeled Driving Data","volume":"14","author":"Duan","year":"2020","journal-title":"IET Intell. Transp. Syst."},{"key":"ref_17","unstructured":"Kiran, B.R., Sobh, I., Talpaert, V., Mannion, P., Sallab, A.A.A., Yogamani, S., and P\u00e9rez, P. (2020). Deep Reinforcement Learning for Autonomous Driving: A Survey. IEEE Trans. Intell. Transp. Syst., 1\u201318."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"108","DOI":"10.1109\/MRA.2006.1678144","article-title":"Simultaneous localization and mapping (SLAM): Part II","volume":"13","author":"Bailey","year":"2006","journal-title":"IEEE Robot. Autom. Mag."},{"key":"ref_19","unstructured":"rFpro (2022, February 01). Driving Simulation for Autonomous Driving, ADAS, Vehicle Dynamics and Motorsport. Available online: https:\/\/www.rfpro.com\/."},{"key":"ref_20","unstructured":"CARLA Community (2022, February 01). CARLA Simulator. Available online: https:\/\/carla.readthedocs.io\/en\/latest\/."},{"key":"ref_21","unstructured":"CARLA Community (2022, February 01). Sensors Reference-CARLA Simulator. Available online: https:\/\/carla.readthedocs.io\/en\/latest\/core_sensors\/."},{"key":"ref_22","unstructured":"CARLA Community (2022, February 01). 2nd-Actors and Blueprints-CARLA Simulator. Available online: https:\/\/carla.readthedocs.io\/en\/latest\/core_actors\/."},{"key":"ref_23","unstructured":"The MathWorks, I. (2022, February 01). Automated Driving Toolbox-MATLAB. Available online: https:\/\/se.mathworks.com\/products\/automated-driving.html."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Li, Q., Peng, Z., Zhang, Q., Liu, C., and Zhou, B. (2022, February 01). Vehicle Configuration\u2014PGDrive 0.1.1 Documentation. Available online: https:\/\/github.com\/decisionforce\/pgdrive.","DOI":"10.1186\/s13638-022-02139-x"}],"container-title":["Informatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2227-9709\/9\/2\/33\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:54:36Z","timestamp":1760136876000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2227-9709\/9\/2\/33"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,15]]},"references-count":24,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2022,6]]}},"alternative-id":["informatics9020033"],"URL":"https:\/\/doi.org\/10.3390\/informatics9020033","relation":{},"ISSN":["2227-9709"],"issn-type":[{"type":"electronic","value":"2227-9709"}],"subject":[],"published":{"date-parts":[[2022,4,15]]}}}