{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T07:02:21Z","timestamp":1760598141443,"version":"build-2065373602"},"reference-count":34,"publisher":"MDPI AG","issue":"3","license":[{"start":{"date-parts":[[2022,3,9]],"date-time":"2022-03-09T00:00:00Z","timestamp":1646784000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Algorithms"],"abstract":"<jats:p>Currently, artificial intelligence is in an important period of growth. Due to the technology boom, it is now possible to solve problems that could not be resolved previously. For example, through goal-driven learning, it is possible that intelligent machines or agents may be able to perform tasks without human intervention. However, this also leads to the problem of understanding the agent\u2019s decision making. Therefore, explainable goal-driven learning attempts to eliminate this gap. This work focuses on the adaptability of two explainability methods in continuous environments. The methods based on learning and introspection proposed a probability value for success to explain the agent\u2019s behavior. These had already been tested in discrete environments. The continuous environment used in this study is the car-racing problem. This is a simulated car racing game that forms part of the Python Open AI Gym Library. The agents in this environment were trained with the Deep Q-Network algorithm, and in parallel the explainability methods were implemented. This research included a proposal for carrying out the adaptation and implementation of these methods in continuous states. The adaptation of the learning method produced major changes, implemented through an artificial neural network. The obtained probabilities of both methods were consistent throughout the experiments. The probability result was greater in the learning method. In terms of computational resources, the introspection method was slightly better than its counterpart.<\/jats:p>","DOI":"10.3390\/a15030091","type":"journal-article","created":{"date-parts":[[2022,3,9]],"date-time":"2022-03-09T11:30:38Z","timestamp":1646825438000},"page":"91","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":6,"title":["Analysis of Explainable Goal-Driven Reinforcement Learning in a Continuous Simulated Environment"],"prefix":"10.3390","volume":"15","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-2558-9646","authenticated-orcid":false,"given":"Ernesto","family":"Portugal","sequence":"first","affiliation":[{"name":"Escuela de Ingenier\u00eda, Universidad Central de Chile, Santiago 8330601, Chile"},{"name":"Escola Polit\u00e9cnica de Pernambuco, Universidade de Pernambuco, Recife 50720-001, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1131-3382","authenticated-orcid":false,"given":"Francisco","family":"Cruz","sequence":"additional","affiliation":[{"name":"Escuela de Ingenier\u00eda, Universidad Central de Chile, Santiago 8330601, Chile"},{"name":"School of Information Technology, Deakin University, Geelong 3220, Australia"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-0732-7017","authenticated-orcid":false,"given":"Angel","family":"Ayala","sequence":"additional","affiliation":[{"name":"Escola Polit\u00e9cnica de Pernambuco, Universidade de Pernambuco, Recife 50720-001, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6001-3925","authenticated-orcid":false,"given":"Bruno","family":"Fernandes","sequence":"additional","affiliation":[{"name":"Escola Polit\u00e9cnica de Pernambuco, Universidade de Pernambuco, Recife 50720-001, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2022,3,9]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","unstructured":"Singhal, A., Sinha, P., and Pant, R. (2017). Use of deep learning in modern recommendation system: A summary of recent works. arXiv.","DOI":"10.5120\/ijca2017916055"},{"key":"ref_2","first-page":"21","article-title":"A survey of existing e-mail spam filtering methods considering machine learning techniques","volume":"18","author":"Bhuiyan","year":"2018","journal-title":"Glob. J. Comput. Sci. Technol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"102805","DOI":"10.1016\/j.cviu.2019.102805","article-title":"A survey on deep learning based face recognition","volume":"189","author":"Guo","year":"2019","journal-title":"Comput. Vis. Image Underst."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1007\/s10916-017-0715-6","article-title":"A critical review for developing accurate and dynamic predictive models using machine learning methods in medicine and health care","volume":"41","author":"Alanazi","year":"2017","journal-title":"J. Med. Syst."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"740","DOI":"10.1109\/TITS.2020.3024655","article-title":"Survey of deep reinforcement learning for motion planning of autonomous vehicles","volume":"23","author":"Aradi","year":"2020","journal-title":"IEEE Trans. Intell. Transp. Syst."},{"key":"ref_6","unstructured":"Das, A., and Rad, P. (2020). Opportunities and challenges in explainable artificial intelligence (xai): A survey. arXiv."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"103525","DOI":"10.1016\/j.artint.2021.103525","article-title":"Levels of explainable artificial intelligence for human-aligned conversational explanations","volume":"299","author":"Dazeley","year":"2021","journal-title":"Artif. Intell."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Lim, B.Y., Dey, A.K., and Avrahami, D. (2009, January 4\u20139). Why and why not explanations improve the intelligibility of context-aware intelligent systems. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Boston, MA, USA.","DOI":"10.1145\/1518701.1519023"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Cruz, F., Acu\u00f1a, G., Cubillos, F., Moreno, V., and Bassi, D. (2007). Indirect training of grey-box models: Application to a bioprocess. International Symposium on Neural Networks, Springer.","DOI":"10.1007\/978-3-540-72393-6_47"},{"key":"ref_10","doi-asserted-by":"crossref","unstructured":"Naranjo, F.C., and Leiva, G.A. (2010, January 15\u201319). Indirect training with error backpropagation in Gray-Box Neural Model: Application to a chemical process. Proceedings of the 2010 XXIX International Conference of the Chilean Computer Science Society, Antofagasta, Chile.","DOI":"10.1109\/SCCC.2010.41"},{"key":"ref_11","unstructured":"Ayala, A., Cruz, F., Fernandes, B., and Dazeley, R. (2021). Explainable Deep Reinforcement Learning Using Introspection in a Non-episodic Task. arXiv."},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Barros, P., Tanevska, A., Cruz, F., and Sciutti, A. (2020, January 7\u201311). Moody Learners-Explaining Competitive Behaviour of Reinforcement Learning Agents. Proceedings of the 2020 Joint IEEE 10th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), Valparaiso, Chile.","DOI":"10.1109\/ICDL-EpiRob48136.2020.9278125"},{"key":"ref_13","unstructured":"Dazeley, R., Vamplew, P., and Cruz, F. (2021). Explainable reinforcement learning for Broad-XAI: A conceptual framework and survey. arXiv."},{"key":"ref_14","first-page":"44","article-title":"DARPA\u2019s Explainable Artificial Intelligence (XAI) Program","volume":"40","author":"Gunning","year":"2019","journal-title":"AI Mag."},{"key":"ref_15","unstructured":"Sado, F., Loo, C.K., Liew, W.S., Kerzel, M., and Wermter, S. (2020). Explainable Goal-Driven Agents and Robots\u2014A Comprehensive Review. arXiv."},{"key":"ref_16","unstructured":"Sutton, R.S., and Barto, A.G. (2018). Reinforcement Learning: An Introduction, MIT Press."},{"key":"ref_17","unstructured":"Goodrich, M.A., and Schultz, A.C. (2022, January 30). Human-Robot Interaction: A Survey, Foundations and Trends in Human-Computer Interaction. Available online: https:\/\/www.researchgate.net\/publication\/220613473_Human-Robot_Interaction_A_Survey."},{"key":"ref_18","unstructured":"Mill\u00e1n, C., Fernandes, B.J., and Cruz, F. (2019, January 24\u201326). Human feedback in Continuous Actor-Critic Reinforcement Learning. Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning ESANN, Bruges, Belgium."},{"key":"ref_19","doi-asserted-by":"crossref","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","article-title":"Peeking inside the black-box: A survey on explainable artificial intelligence (XAI)","volume":"6","author":"Adadi","year":"2018","journal-title":"IEEE Access"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1016\/j.artmed.2019.01.001","article-title":"Explainable artificial intelligence for breast cancer: A visual case-based reasoning approach","volume":"94","author":"Lamy","year":"2019","journal-title":"Artif. Intell. Med."},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Wang, X., Chen, Y., Yang, J., Wu, L., Wu, Z., and Xie, X. (2018, January 17\u201320). A reinforcement learning framework for explainable recommendation. Proceedings of the 2018 IEEE International Conference on Data Mining (ICDM), Singapore.","DOI":"10.1109\/ICDM.2018.00074"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"107052","DOI":"10.1016\/j.ast.2021.107052","article-title":"Explainable Deep Reinforcement Learning for UAV autonomous path planning","volume":"118","author":"He","year":"2021","journal-title":"Aerosp. Sci. Technol."},{"key":"ref_23","doi-asserted-by":"crossref","unstructured":"Madumal, P., Miller, T., Sonenberg, L., and Vetere, F. (2020, January 7\u201312). Explainable reinforcement learning through a causal lens. Proceedings of the AAAI Conference on Artificial Intelligence, New York, NY, USA.","DOI":"10.1609\/aaai.v34i03.5631"},{"key":"ref_24","doi-asserted-by":"crossref","first-page":"103367","DOI":"10.1016\/j.artint.2020.103367","article-title":"Interestingness elements for explainable reinforcement learning: Understanding agents\u2019 capabilities and limitations","volume":"288","author":"Sequeira","year":"2020","journal-title":"Artif. Intell."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Cruz, F., Dazeley, R., and Vamplew, P. (2019, January 2\u20135). Memory-based explainable reinforcement learning. Proceedings of the Australasian Joint Conference on Artificial Intelligence, Adelaide, SA, Australia.","DOI":"10.1007\/978-3-030-35288-2_6"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Cruz, F., Dazeley, R., and Vamplew, P. (2021). Explainable robotic systems: Understanding goal-driven actions in a reinforcement learning scenario. Neural Comput. Appl.","DOI":"10.1007\/s00521-021-06425-5"},{"key":"ref_27","unstructured":"Milani, S., Topin, N., Veloso, M., and Fang, F. (2022). A Survey of Explainable Reinforcement Learning. arXiv."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"106685","DOI":"10.1016\/j.knosys.2020.106685","article-title":"Explainability in deep reinforcement learning","volume":"214","author":"Heuillet","year":"2021","journal-title":"Knowl.-Based Syst."},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"529","DOI":"10.1038\/nature14236","article-title":"Human-level control through deep reinforcement learning","volume":"518","author":"Mnih","year":"2015","journal-title":"Nature"},{"key":"ref_30","unstructured":"Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym. arXiv."},{"key":"ref_31","unstructured":"Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018, January 10\u201315). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. Proceedings of the International Conference on Machine Learning, Stockholm, Sweden."},{"key":"ref_32","doi-asserted-by":"crossref","unstructured":"Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. (2018, January 2\u20137). Rainbow: Combining improvements in deep reinforcement learning. Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, LA, USA.","DOI":"10.1609\/aaai.v32i1.11796"},{"key":"ref_33","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Gupta, J.K., Egorov, M., and Kochenderfer, M. (2017, January 8\u201312). Cooperative multi-agent control using deep reinforcement learning. Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, S\u00e3o Paulo, Brazil.","DOI":"10.1007\/978-3-319-71682-4_5"}],"container-title":["Algorithms"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1999-4893\/15\/3\/91\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T22:33:31Z","timestamp":1760135611000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1999-4893\/15\/3\/91"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,9]]},"references-count":34,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2022,3]]}},"alternative-id":["a15030091"],"URL":"https:\/\/doi.org\/10.3390\/a15030091","relation":{},"ISSN":["1999-4893"],"issn-type":[{"type":"electronic","value":"1999-4893"}],"subject":[],"published":{"date-parts":[[2022,3,9]]}}}