{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,10]],"date-time":"2026-02-10T20:49:42Z","timestamp":1770756582669,"version":"3.50.0"},"publisher-location":"Cham","reference-count":41,"publisher":"Springer Nature Switzerland","isbn-type":[{"value":"9783032083234","type":"print"},{"value":"9783032083241","type":"electronic"}],"license":[{"start":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T00:00:00Z","timestamp":1760572800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2025,10,16]],"date-time":"2025-10-16T00:00:00Z","timestamp":1760572800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2026]]},"abstract":"<jats:title>Abstract<\/jats:title>\n          <jats:p>This article presents a new method for leveraging explainable reinforcement learning (XRL) knowledge to enhance the performance of reinforcement learning (RL) agents. Although current XRL approaches are mainly focused on improving interpretability and user trust by providing explanations for agent actions, their ability to guide and optimise RL agent\u2019s training is under-explored. To address this gap, we extend an existing introspective analysis framework by integrating XRL metrics directly into the training pipelines of model-free RL algorithms. This integration allows dynamic adjustments of algorithm-specific parameters based on real-time feedback from XRL metrics. The proposed methodology is validated across diverse OpenAI Gym environments (CartPole and Taxi). By evaluating both on-policy and off-policy approaches, we demonstrate that incorporating XRL insights leads to significant improvements in agent performance. The analysis of the results highlights the benefits regarding enhanced explainability and optimised decision-making. This work contributes in XRL research area by aligning interpretability with actionable performance gains, paving the way for more reliable and transparent RL systems in complex, real-world applications.<\/jats:p>","DOI":"10.1007\/978-3-032-08324-1_11","type":"book-chapter","created":{"date-parts":[[2025,10,15]],"date-time":"2025-10-15T08:49:21Z","timestamp":1760518161000},"page":"247-270","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":1,"title":["Explain to\u00a0Gain: Introspective Reinforcement Learning for\u00a0Enhanced Performance"],"prefix":"10.1007","author":[{"given":"Santiago","family":"Quintana-Amate","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Delaney","family":"Stevens","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Varniethan","family":"Ketheeswaran","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Patrick","family":"Capaldo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dylan","family":"Sheldon","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Mark","family":"Hall","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2025,10,16]]},"reference":[{"key":"11_CR1","unstructured":"Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, 2nd edn (2018)"},{"issue":"6419","key":"11_CR2","doi-asserted-by":"publisher","first-page":"1140","DOI":"10.1126\/science.aar6404","volume":"362","author":"D Silver","year":"2018","unstructured":"Silver, D., et al.: A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science 362(6419), 1140\u20131144 (2018)","journal-title":"Science"},{"key":"11_CR3","doi-asserted-by":"crossref","unstructured":"Vinyals, O., et al.: Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575(7782), 350\u2013354 (2019)","DOI":"10.1038\/s41586-019-1724-z"},{"key":"11_CR4","doi-asserted-by":"crossref","unstructured":"Puiutta, E., Veith, E.M.S.P.: Explainable reinforcement learning: A survey. In: Holzinger, A., Kieseberg, P., Tjoa, A.M., Weippl, E. (eds.) Machine Learning and Knowledge Extraction. LNCS, vol. 12279, pp. 77\u201395. Springer, Cham (2020)","DOI":"10.1007\/978-3-030-57321-8_5"},{"key":"11_CR5","doi-asserted-by":"crossref","unstructured":"Heuillet, A., Couthouis, F., Daz-Rodriguez, N.: Explainability in deep reinforcement learning. Knowl. Based Syst. 226, 107152 (2021)","DOI":"10.1016\/j.knosys.2020.106685"},{"key":"11_CR6","doi-asserted-by":"publisher","first-page":"1746","DOI":"10.1109\/TETC.2022.3171314","volume":"10","author":"Y Jia","year":"2021","unstructured":"Jia, Y., Mcdermid, J., Lawton, T., Habli, I.: The role of explainability in assuring safety of machine learning in healthcare. IEEE Trans. Emerg. Top. Comput. 10, 1746\u20131760 (2021)","journal-title":"IEEE Trans. Emerg. Top. Comput."},{"key":"11_CR7","doi-asserted-by":"crossref","unstructured":"Sequeira, P., Gervasio, M.: IxDRL: A novel explainable deep reinforcement learning toolkit based on analyses of interestingness. In: Ras, Z.W., Kak, S., Pizzi, N. (eds.) Explainable Artificial Intelligence. LNAI, vol. 14221, pp. 373\u2013396. Springer, Cham (2023)","DOI":"10.1007\/978-3-031-44064-9_20"},{"key":"11_CR8","unstructured":"Lu, W., Zhao, X., Fryen, T., Lee, J.H., Li, M., Magg, S., Wermter, S.: Causal state distillation for explainable reinforcement learning. In: Proceedings of the 3rd Conference on Causal Learning and Reasoning, vol. 236, pp. 106\u2013142. PMLR (2024)"},{"key":"11_CR9","unstructured":"Finkelstein, M., et al.: Explainable Reinforcement Learning via Model Transforms. In: Advances in Neural Information Processing Systems, vol. 35, pp. 12345\u201312358 (2022)"},{"key":"11_CR10","unstructured":"Guan, L., Verma, M., Guo, S., Zhang, R., Kambhampati, S.: Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation. In: Advances in Neural Information Processing Systems, vol. 34, pp. 20047\u201320059 (2021)"},{"key":"11_CR11","doi-asserted-by":"crossref","unstructured":"Milani, S., Topin, N., Veloso, M., Fang, F.: Explainable reinforcement learning: a survey and comparative review. ACM Comput. Surv. 56(7), 1\u201336, Article 168 (2024)","DOI":"10.1145\/3616864"},{"key":"11_CR12","unstructured":"Puri, N., et al.: Explain your move: understanding agent actions using specific and relevant feature attribution. In: International Conference on Learning Representations (2020)"},{"key":"11_CR13","doi-asserted-by":"crossref","unstructured":"Yan, D.: Research on reinforcement learning explainable strategies based on advantage saliency. Front. omput. Intel. Syst. 3(1), 124\u2013129 (2022)","DOI":"10.54097\/fcis.v3i1.6348"},{"key":"11_CR14","unstructured":"Atrey, A., Clary, K., Jensen, D.: Exploratory not explanatory: counterfactual analysis of saliency maps for deep reinforcement learning. In: International Conference on Learning Representations (2020)"},{"key":"11_CR15","doi-asserted-by":"crossref","unstructured":"Topin, N., Veloso, M.: Generation of policy-level explanations for reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 2514\u20132521 (2019)","DOI":"10.1609\/aaai.v33i01.33012514"},{"key":"11_CR16","unstructured":"Koul, A., Greydanus, S., Fern, A.: Learning finite state representations of recurrent policy networks. In: International Conference on Learning Representations (2019)"},{"key":"11_CR17","unstructured":"Amir, D., Amir, O.: Highlights: summarizing agent behavior to people. In: Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems, pp. 1168\u20131176. IFAAMAS, Richland (2018)"},{"key":"11_CR18","doi-asserted-by":"crossref","unstructured":"Sreedharan, S., Srivastava, S., Kambhampati, S.: TLdR: Policy summarization for factored SSP problems using temporal abstractions. In: Proceedings of the International Conference on Automated Planning and Scheduling, vol. 30, pp. 272\u2013280 (2020)","DOI":"10.1609\/icaps.v30i1.6671"},{"key":"11_CR19","unstructured":"Lin, Z., Lam, K.H., Fern, A.: Contrastive explanations for reinforcement learning via embedded self predictions. In: International Conference on Learning Representations (2021)"},{"key":"11_CR20","doi-asserted-by":"crossref","unstructured":"Olson, M.L., Khanna, R., Neal, L., Li, F., Wong, W.K.: Counterfactual state explanations for reinforcement learning agents via generative deep learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 8839\u20138847 (2021)","DOI":"10.1016\/j.artint.2021.103455"},{"key":"11_CR21","unstructured":"Sequeira, P., Hostetler, J., Gervasio, M.: Global and local analysis of interestingness for competency-aware deep reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 6126\u20136134 (2023)"},{"key":"11_CR22","unstructured":"Schroeter, N., Cruz, F., Wermter, S.: Introspection-based explainable reinforcement learning in episodic and non-episodic scenarios. IEEE Access 11, 37615\u201337628 (2023)"},{"key":"11_CR23","unstructured":"Cruz, F., Dazeley, R., Vamplew, P., Moreira, I.: Explainable robotic systems: understanding goal-driven actions in a reinforcement learning scenario. Neural Comput. Appl. 33, 11527\u201311546 (2021)"},{"key":"11_CR24","unstructured":"Ayala, A., Cruz, F., Dazeley, R., Vamplew, P., Fernandes, B.: Explainable deep reinforcement learning using introspection in a non-episodic task. IEEE Access 9, 152419\u2013152432 (2021)"},{"key":"11_CR25","unstructured":"Sequeira, P., Hostetler, J., Gervasio, M., Yeh, E.: Outcome-guided counterfactuals for reinforcement learning agents from a jointly trained generative latent space. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 6135\u20136143 (2023)"},{"key":"11_CR26","unstructured":"Ng, A.Y., Harada, D., Russell, S.J.: Policy invariance under reward transformations: theory and application to reward shaping. In: Proceedings of the 16th International Conference on Machine Learning, pp. 278\u2013287. Morgan Kaufmann, San Francisco (1999)"},{"key":"11_CR27","unstructured":"Devlin, S.M., Kudenko, D.: Dynamic potential-based reward shaping. In: Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems, pp. 433\u2013440. IFAAMAS, Richland (2012)"},{"key":"11_CR28","doi-asserted-by":"crossref","unstructured":"Wang, Z., Taylor, M.E.: Improving reinforcement learning with confidence-based demonstrations. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 3027\u20133033 (2017)","DOI":"10.24963\/ijcai.2017\/422"},{"key":"11_CR29","unstructured":"Ma, H., Luo, Z., Vo, T.V., Sima, K., Leong, T.Y.: Highly efficient self-adaptive reward shaping for reinforcement Learning. arXiv preprint arXiv:2401.00989 (2024)"},{"key":"11_CR30","unstructured":"Jacq, A., Ferret, J., Pietquin, O., Geist, M.: Lazy-MDPs: towards interpretable RL by learning when to act. In: Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pp. 669\u2013677. IFAAMAS, Richland (2022)"},{"key":"11_CR31","doi-asserted-by":"crossref","unstructured":"Mallick, R., et al.: A hybrid transformer with domain adaptation using interpretability techniques for the application to the detection of risk situations. Multimedia Tools Appl. 83(35), 83339\u201383356 (2024)","DOI":"10.1007\/s11042-024-18687-x"},{"key":"11_CR32","doi-asserted-by":"crossref","unstructured":"Ayyar, M.P., Benois-Pineau, J., Zemmari, A.: ESL: explain to improve streaming learning for transformers. In: Pattern Recognition. LNCS, vol. 14995, pp. 160\u2013175. Springer, Cham (2024)","DOI":"10.1007\/978-3-031-78189-6_11"},{"issue":"32","key":"11_CR33","doi-asserted-by":"publisher","first-page":"23429","DOI":"10.1007\/s00521-021-06375-y","volume":"35","author":"ME Taylor","year":"2023","unstructured":"Taylor, M.E., Yun, S., Hollinger, G.A., Borowczak, M., Nguyen, V., Osoba, O.A.: Improving reinforcement learning with human assistance: an argument for human subject studies with HippoGym. Neural Comput. Appl. 35(32), 23429\u201323439 (2023)","journal-title":"Neural Comput. Appl."},{"key":"11_CR34","doi-asserted-by":"crossref","unstructured":"Liu, Y., Wang, X., Chang, Y., Jiang, C.: Towards explainable reinforcement learning using scoring mechanism augmented agents. In: Knowledge Science, Engineering and Management. LNCS, vol. 13463, pp. 547\u2013558. Springer, Cham (2022)","DOI":"10.1007\/978-3-031-10986-7_44"},{"key":"11_CR35","doi-asserted-by":"crossref","unstructured":"Sequeira, P., Gervasio, M.: Interestingness elements for explainable reinforcement learning: understanding agents\u2019 capabilities and limitations. Artif. Intell. 288, 103367 (2020)","DOI":"10.1016\/j.artint.2020.103367"},{"key":"11_CR36","unstructured":"Finkelstein, M., et al.: Explainable Reinforcement Learning via Model Transforms. In: Advances in Neural Information Processing Systems, vol. 35, pp. 12345\u201312358 (2022)"},{"key":"11_CR37","doi-asserted-by":"crossref","unstructured":"Madumal, P., Miller, T., Sonenberg, L., Vetere, F.: Explainable reinforcement learning through a causal lens. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 2493\u20132500 (2020)","DOI":"10.1609\/aaai.v34i03.5631"},{"key":"11_CR38","doi-asserted-by":"crossref","unstructured":"Mulder, C.P.H., Bazeley-White, E., Dimitrakopoulos, P.G., Hector, A., Scherer-Lorenzen, M., Schmid, B.: Species evenness and productivity in experimental plant communities. Oikos 107(1), 50\u201363 (2004)","DOI":"10.1111\/j.0030-1299.2004.13110.x"},{"key":"11_CR39","doi-asserted-by":"crossref","unstructured":"Xu, M., Chen, X., Wang, J.: Policy correction and state-conditioned action evaluation for few-shot lifelong deep reinforcement learning. IEEE Trans. Neural Netw. Learn. Syst. 36(4), 6843\u20136857 (2025)","DOI":"10.1109\/TNNLS.2024.3385570"},{"key":"11_CR40","doi-asserted-by":"crossref","unstructured":"Retzlaff, C., et al.: Human-in-the-loop reinforcement learning: a survey and position on requirements, challenges, and opportunities. J. Artif. Intell. Res. 79, 359\u2013415 (2024)","DOI":"10.1613\/jair.1.15348"},{"key":"11_CR41","unstructured":"Brockman, G., et al.: OpenAI Gym. arXiv preprint arXiv:1606.01540 (2016)"}],"container-title":["Communications in Computer and Information Science","Explainable Artificial Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/978-3-032-08324-1_11","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,15]],"date-time":"2025-10-15T08:49:48Z","timestamp":1760518188000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/978-3-032-08324-1_11"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,16]]},"ISBN":["9783032083234","9783032083241"],"references-count":41,"URL":"https:\/\/doi.org\/10.1007\/978-3-032-08324-1_11","relation":{},"ISSN":["1865-0929","1865-0937"],"issn-type":[{"value":"1865-0929","type":"print"},{"value":"1865-0937","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,10,16]]},"assertion":[{"value":"16 October 2025","order":1,"name":"first_online","label":"First Online","group":{"name":"ChapterHistory","label":"Chapter History"}},{"value":"xAI","order":1,"name":"conference_acronym","label":"Conference Acronym","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"World Conference on Explainable Artificial Intelligence","order":2,"name":"conference_name","label":"Conference Name","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"Istanbul","order":3,"name":"conference_city","label":"Conference City","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"T\u00fcrkiye","order":4,"name":"conference_country","label":"Conference Country","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"2025","order":5,"name":"conference_year","label":"Conference Year","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"9 July 2025","order":7,"name":"conference_start_date","label":"Conference Start Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"11 July 2025","order":8,"name":"conference_end_date","label":"Conference End Date","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"3","order":9,"name":"conference_number","label":"Conference Number","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"xai2025","order":10,"name":"conference_id","label":"Conference ID","group":{"name":"ConferenceInfo","label":"Conference Information"}},{"value":"https:\/\/xaiworldconference.com\/2025\/","order":11,"name":"conference_url","label":"Conference URL","group":{"name":"ConferenceInfo","label":"Conference Information"}}]}}