{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T14:59:27Z","timestamp":1769525967511,"version":"3.49.0"},"reference-count":45,"publisher":"MDPI AG","issue":"7","license":[{"start":{"date-parts":[[2023,6,30]],"date-time":"2023-06-30T00:00:00Z","timestamp":1688083200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"national funds","award":["UIDB\/05549\/2020"],"award-info":[{"award-number":["UIDB\/05549\/2020"]}]},{"name":"national funds","award":["UIDP\/05549\/2020"],"award-info":[{"award-number":["UIDP\/05549\/2020"]}]},{"name":"national funds","award":["NORTE-01-0145-FEDER-000045"],"award-info":[{"award-number":["NORTE-01-0145-FEDER-000045"]}]},{"name":"Norte Portugal Regional Operational Program (NORTE 2020)","award":["UIDB\/05549\/2020"],"award-info":[{"award-number":["UIDB\/05549\/2020"]}]},{"name":"Norte Portugal Regional Operational Program (NORTE 2020)","award":["UIDP\/05549\/2020"],"award-info":[{"award-number":["UIDP\/05549\/2020"]}]},{"name":"Norte Portugal Regional Operational Program (NORTE 2020)","award":["NORTE-01-0145-FEDER-000045"],"award-info":[{"award-number":["NORTE-01-0145-FEDER-000045"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Reinforcement Learning is one of the many machine learning paradigms. With no labelled data, it is concerned with balancing the exploration and exploitation of an environment with one or more agents present in it. Recently, many breakthroughs have been made in the creation of these agents for video game machine learning development, especially in first-person shooters with platforms such as ViZDoom, DeepMind Lab, and Unity\u2019s ML-Agents. In this paper, we review the state-of-the-art of creation of Reinforcement Learning agents for use in multiplayer deathmatch first-person shooters. We selected various platforms, frameworks, and training architectures from various papers and examined each of them, analysing their uses. We compared each platform and training architecture, and then concluded whether machine learning agents can now face off against humans and whether they make for better gameplay than traditional Artificial Intelligence. In the end, we thought about future research and what researchers should keep in mind when exploring and testing this area.<\/jats:p>","DOI":"10.3390\/a16070323","type":"journal-article","created":{"date-parts":[[2023,6,30]],"date-time":"2023-06-30T01:02:41Z","timestamp":1688086961000},"page":"323","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Reinforcement Learning Applied to AI Bots in First-Person Shooters: A Systematic Review"],"prefix":"10.3390","volume":"16","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1274-5045","authenticated-orcid":false,"given":"Pedro","family":"Almeida","sequence":"first","affiliation":[{"name":"2AI, School of Technology, Polytechnic Institute of C\u00e1vado and Ave, 4750 Barcelos, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4658-5844","authenticated-orcid":false,"given":"Vitor","family":"Carvalho","sequence":"additional","affiliation":[{"name":"2AI, School of Technology, Polytechnic Institute of C\u00e1vado and Ave, 4750 Barcelos, Portugal"},{"name":"LASI\u2014Associate Laboratory of Intelligent Systems, 4800 Guimar\u00e3es, Portugal"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6961-2660","authenticated-orcid":false,"given":"Alberto","family":"Sim\u00f5es","sequence":"additional","affiliation":[{"name":"2AI, School of Technology, Polytechnic Institute of C\u00e1vado and Ave, 4750 Barcelos, Portugal"},{"name":"LASI\u2014Associate Laboratory of Intelligent Systems, 4800 Guimar\u00e3es, Portugal"}]}],"member":"1968","published-online":{"date-parts":[[2023,6,30]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"484","DOI":"10.1038\/nature16961","article-title":"Mastering the game of Go with deep neural networks and tree search","volume":"529","author":"Silver","year":"2016","journal-title":"Nature"},{"key":"ref_2","doi-asserted-by":"crossref","first-page":"1140","DOI":"10.1126\/science.aar6404","article-title":"Mastering chess and shogi by self-play with a general reinforcement learning algorithm","volume":"362","author":"Silver","year":"2018","journal-title":"Science"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1109\/TCIAIG.2010.2100395","article-title":"Reinforcement Learning in First Person Shooter Games","volume":"3","author":"McPartland","year":"2011","journal-title":"IEEE Trans. Comput. Intell. AI Games"},{"key":"ref_5","doi-asserted-by":"crossref","unstructured":"ElDahshan, K., Farouk, H., and Mofreh, E. (2022, January 8\u20139). Deep Reinforcement Learning based Video Games: A Review. Proceedings of the 2022 2nd International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC), Cairo, Egypt.","DOI":"10.1109\/MIUCC55081.2022.9781752"},{"key":"ref_6","unstructured":"G\u00e9ron, A. (2017). Hands-On Machine Learning with Scikit-Learn and TensorFlow Concepts, Tools, and Techniques to Build Intelligent Systems, O\u2019Reilly Media, Inc."},{"key":"ref_7","unstructured":"Mitchel, T. (1997). Machine Learning, McGraw-Hill."},{"key":"ref_8","unstructured":"Sutton, R., and Barto, A. (2018). Reinforcement Learning\u2014An Introduction, The MIT Press. [2nd ed.]."},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Page, M., McKenzie, J., Bossuyt, P., Boutron, I., Hoffmann, T., Mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., and Brennan, S. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. PLoS Med., 18.","DOI":"10.1371\/journal.pmed.1003583"},{"key":"ref_10","first-page":"1122","article-title":"AI Soldier using Reinforcement Learning","volume":"4","author":"Bhojwani","year":"2019","journal-title":"Int. J. Innov. Sci. Res. Technol."},{"key":"ref_11","unstructured":"Ulrich, C., Salameh, H., and Wu, M. (2020). Training a Game AI with Machine Learning. [Bachelor\u2019s Thesis, IT University of Copenhagen]."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Shao, K., Zhao, D., and Zhu, Y. (2018, January 14\u201317). Learning Battles in ViZDoom via deep reinforcement Learning. Proceedings of the 2018 IEEE Conference on Computational Intelligence and Games (CIG), Maastricht, The Netherlands.","DOI":"10.1109\/CIG.2018.8490423"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Sun, Y., Khan, A., Yand, K., Fend, J., and Liu, S. (2019, January 17\u201319). Playing First-Person-Shooter Games with A3C-Anticipator Network Based Agents Using Reinforcement Learning. Proceedings of the International Conference on Artificial Intelligence and Security, Berlin, Germany.","DOI":"10.1007\/978-3-030-24268-8_43"},{"key":"ref_14","unstructured":"Adamsson, M. (2018). Curriculum Learning for Increasing the Performance of a Reinforcement Learning Agent in a Static First-Person Shooter Game. [Master\u2019s Thesis, KTH University]."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Piergigli, D., Ripamonti, L., Maggiorini, D., and Gadia, D. (2019, January 20\u201323). Deep Reinforcement Learning to train agents in a multiplayer First Person Shooter some preliminary results. Proceedings of the 2019 IEEE Conference on Games (CoG), London, UK.","DOI":"10.1109\/CIG.2019.8848061"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Serafim, P., Nogueira, Y., Vidal, C., and Neto, J. (November, January 29). Evaluating competition in training of Deep Reinforcement Learning agents in First-Person Shooter games. Proceedings of the 2018 17th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames), Foz Do Igua\u00e7u, Brazil.","DOI":"10.1109\/SBGAMES.2018.00023"},{"key":"ref_17","doi-asserted-by":"crossref","unstructured":"Wydmuch, M., Kempka, M., and Jasjiwski, W. (2018). ViZDoom Competitions Playing Doom from Pixels. arXiv.","DOI":"10.1109\/TG.2018.2877047"},{"key":"ref_18","unstructured":"Huang, S., Su, H., Zhu, J., and Chen, T. (February, January 27). Combo-Action Training Agent for FPS Game with Auxiliary Tasks. Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19), Honolulu, HI, USA."},{"key":"ref_19","unstructured":"Tensorflow (2023, June 20). TensorFlow\u2019s Official Website. Available online: https:\/\/www.tensorflow.org\/about."},{"key":"ref_20","unstructured":"Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., and Isard, M. (2016, January 2\u20134). TensorFlow A System for Large-Scale Machine Learning. Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI \u201916), Savannah, GA, USA."},{"key":"ref_21","unstructured":"Abrahams, S., Hafner, D., Erwitt, E., and Scarpinelli, A. (2016). TensorFlow for Machine Intelligence, Bleeding Edge Press."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Ketkar, N., and Moolayil, J. (2021). Deep Learning with Python, Apress. [2nd ed.].","DOI":"10.1007\/978-1-4842-5364-9"},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Prakash, K.B., and Kanagachidambaresan, G.R. (2021). Programming with TensorFlow, Springer.","DOI":"10.1007\/978-3-030-57077-4"},{"key":"ref_24","unstructured":"Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. (2017, January 4\u20139). Automatic differentiation in PyTorch. Proceedings of the 31st Conference on Neural Information Processing System, Long Beach, CA, USA."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Yilmaz, O., and Celikcan, U. (2018, January 28\u201330). Q-learning with Na\u00efve Bayes Approach Towards More Engaging Game Agents. Proceedings of the 2018 International Conference on Artificial Intelligence and Data Processing (IDAP), Malatya, Turkey.","DOI":"10.1109\/IDAP.2018.8620897"},{"key":"ref_26","unstructured":"Mnih, V., Badia, A., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous Methods for Deep Reinforcement Learning. arXiv."},{"key":"ref_27","unstructured":"Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Sammut, C., and Webb, G. (2017). Encyclopedia of Machine Learning and Data Mining, Springer. [2nd ed.].","DOI":"10.1007\/978-1-4899-7687-1"},{"key":"ref_29","doi-asserted-by":"crossref","unstructured":"Soviany, P., Ionescu, R., Rota, P., and Sebe, N. (2022). Curriculum Learning: A Survey. arXiv.","DOI":"10.1007\/s11263-022-01611-x"},{"key":"ref_30","unstructured":"Juliani, A. (2023, June 20). Introducing ML-Agents Toolkit v0.2: Curriculum Learning, New Environments, and More\u2014Unity Blog. (8 December 2018). Available online: https:\/\/blog.unity.com\/community\/introducing-ml-agents-v0-2-curriculum-learning-new-environments-and-more."},{"key":"ref_31","unstructured":"Unity Team (2023, June 20). Unity Engine\u2019s Official Site. Available online: https:\/\/unity.com\/."},{"key":"ref_32","unstructured":"Beattie, C., Leibo, J., Teplyashin, D., Ward, T., Wainwright, M., K\u00fcttler, H., Lefrancq, A., Green, S., Vald\u00e9s, V., and Sadik, A. (2016). DeepMind Lab. arXiv."},{"key":"ref_33","unstructured":"Jaderberg, M., Czarnecki, W., Dunning, I., Marris, L., Lever, G., Castaneda, A., Beattie, C., Rabinowitz, N., Morcos, A., and Ruderman, A. (2018). Human-level performance in first-person multiplayer. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Kempka, M., Wydmuch, M., Runc, G., Toczek, J., and Jaskowski, W. (2016, January 20\u201323). ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning. Proceedings of the 2016 IEEE Conference on Computational Intelligence and Games (CIG), Santorini, Greece.","DOI":"10.1109\/CIG.2016.7860433"},{"key":"ref_35","unstructured":"Farama Foundation (2023, June 20). ViZDoom\u2019s Github Scenario Page. Available online: https:\/\/github.com\/Farama-Foundation\/ViZDoom\/tree\/master\/scenarios."},{"key":"ref_36","unstructured":"Unity Team (2023, June 20). The ML-Agent\u2019s Github Page. Available online: https:\/\/github.com\/Unity-Technologies\/ml-agents."},{"key":"ref_37","unstructured":"Juliani, A., Berges, V., Teng, E., Cohen, A., Harper, J., Elion, C., Goy, C., Gao, Y., Henry, H., and Mattar, M. (2020). Unity: A General Platform for Intelligent Agents. arXiv."},{"key":"ref_38","unstructured":"Hagen, J. (2022). Agent Participation in First Person Shooter Games Using Reinforcement Learning and Behaviour Cloning. [Master\u2019s Thesis, Breda University]."},{"key":"ref_39","first-page":"5703","article-title":"Creating Intelligent Agents in Game Using Reinforcement Learning","volume":"7","author":"Akansha","year":"2020","journal-title":"Int. Res. J. Eng. Technol. (IRJET)"},{"key":"ref_40","unstructured":"Berner, C., Brockman, G., Chan, B., Cheung, V., D\u0119biak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., and Hesse, C. (2021). Dota 2 with Large Scale Deep Reinforcement Learning. arXiv."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"350","DOI":"10.1038\/s41586-019-1724-z","article-title":"Grandmaster level in StarCraft II using multi-agent reinforcement learning","volume":"575","author":"Vinyals","year":"2019","journal-title":"Nature"},{"key":"ref_42","unstructured":"Raman, R., Gupta, N., and Jeppu, Y. (2011, January 20\u201323). Framework for Formal Verification of Machine Learning Based Complex System-of-System. Proceedings of the 2011, INCOSE International Symposium, Denver, CO, USA."},{"key":"ref_43","doi-asserted-by":"crossref","unstructured":"Krichen, M., Mihoub, A., Alzahrani, M., Adoni, W., and Nahhal, T. (2022, January 9\u201311). Are Formal Methods Applicable To Machine Learning And Artificial Intelligence?. Proceedings of the 2022 2nd International Conference of Smart Systems and Emerging Technologies (SMARTTECH), Riyadh, Saudi Arabia.","DOI":"10.1109\/SMARTTECH54121.2022.00025"},{"key":"ref_44","doi-asserted-by":"crossref","first-page":"626","DOI":"10.1145\/242223.242257","article-title":"Formal methods: State of the art and future directions","volume":"28","author":"Clarke","year":"1996","journal-title":"ACM Comput. Surv."},{"key":"ref_45","doi-asserted-by":"crossref","first-page":"389","DOI":"10.1038\/s42256-019-0088-2","article-title":"The global landscape of AI ethics guidelines","volume":"1","author":"Jobin","year":"2019","journal-title":"Nat. Mach. Intell."}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/16\/7\/323\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T20:03:26Z","timestamp":1760126606000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/16\/7\/323"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,6,30]]},"references-count":45,"journal-issue":{"issue":"7","published-online":{"date-parts":[[2023,7]]}},"alternative-id":["a16070323"],"URL":"https:\/\/doi.org\/10.3390\/a16070323","relation":{},"ISSN":["1999-4893"],"issn-type":[{"value":"1999-4893","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,6,30]]}}}