{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,6,28]],"date-time":"2025-06-28T11:26:42Z","timestamp":1751110002256,"version":"3.37.3"},"reference-count":77,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2021,12,18]],"date-time":"2021-12-18T00:00:00Z","timestamp":1639785600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2021,12,18]],"date-time":"2021-12-18T00:00:00Z","timestamp":1639785600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Softw Syst Model"],"published-print":{"date-parts":[[2022,6]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>Modern software systems are increasingly expected to show higher degrees of autonomy and self-management to cope with uncertain and diverse situations. As a consequence, autonomous systems can exhibit unexpected and surprising behaviours. This is exacerbated due to the ubiquity and complexity of Artificial Intelligence (AI)-based systems. This is the case of Reinforcement Learning (RL), where autonomous agents learn through trial-and-error how to find good solutions to a problem. Thus, the underlying decision-making criteria may become opaque to users that interact with the system and who may require explanations about the system\u2019s reasoning. Available work for eXplainable Reinforcement Learning (XRL) offers different trade-offs: e.g. for runtime explanations, the approaches are model-specific or can only analyse results after-the-fact. Different from these approaches, this paper aims to provide an <jats:italic>online model-agnostic<\/jats:italic> approach for XRL towards trustworthy and understandable AI. We present ETeMoX, an architecture based on temporal models to keep track of the decision-making processes of RL systems. In cases where the resources are limited (e.g. storage capacity or time to response), the architecture also integrates complex event processing, an event-driven approach, for detecting matches to event patterns that need to be stored, instead of keeping the entire history. The approach is applied to a mobile communications case study that uses RL for its decision-making. In order to test the generalisability of our approach, three variants of the underlying RL algorithms are used: Q-Learning, SARSA and DQN. The encouraging results show that using the proposed configurable architecture, RL developers are able to obtain explanations about the evolution of a metric, relationships between metrics, and were able to track situations of interest happening over time windows.\n<\/jats:p>","DOI":"10.1007\/s10270-021-00952-4","type":"journal-article","created":{"date-parts":[[2021,12,18]],"date-time":"2021-12-18T07:02:37Z","timestamp":1639810957000},"page":"1091-1113","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":7,"title":["Event-driven temporal models for explanations - ETeMoX: explaining reinforcement learning"],"prefix":"10.1007","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1801-3494","authenticated-orcid":false,"given":"Juan Marcelo","family":"Parra-Ullauri","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Antonio","family":"Garc\u00eda-Dom\u00ednguez","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Nelly","family":"Bencomo","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Changgang","family":"Zheng","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chen","family":"Zhen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Juan","family":"Boubeta-Puig","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Guadalupe","family":"Ortiz","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shufan","family":"Yang","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2021,12,18]]},"reference":[{"key":"952_CR1","doi-asserted-by":"publisher","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","volume":"6","author":"A Adadi","year":"2018","unstructured":"Adadi, A., Berrada, M.: Peeking inside the black-box: a survey on explainable artificial intelligence (xai). IEEE Access 6, 52138\u201352160 (2018)","journal-title":"IEEE Access"},{"key":"952_CR2","unstructured":"Anjomshoae, S., Najjar, A., Calvaresi, D., Fr\u00e4mling, K.: Explainable agents and robots: results from a systematic literature review. In: 18th International conference on autonomous agents and multiagent systems (AAMAS 2019), Montreal, Canada, May 13\u201317, 2019, pp. 1078\u20131088. International Foundation for Autonomous Agents and Multiagent Systems (2019)"},{"key":"952_CR3","doi-asserted-by":"publisher","unstructured":"Asim, M., Llewellyn-Jones, D., Lempereur, B., Zhou, B., Shi, Q., Merabti, M.: Event Driven Monitoring of Composite Services. In: 2013 International conference on social computing, pp. 550\u2013557 (2013). https:\/\/doi.org\/10.1109\/SocialCom.2013.83","DOI":"10.1109\/SocialCom.2013.83"},{"key":"952_CR4","doi-asserted-by":"publisher","unstructured":"Barquero, G., Burgue\u00f1o, L., Troya, J., Vallecillo, A.: Extending Complex Event Processing to Graph-structured Information. In: Proceedings of MoDELS 2018, pp. 166\u2013175. ACM, New York, NY, USA (2018). https:\/\/doi.org\/10.1145\/3239372.3239402","DOI":"10.1145\/3239372.3239402"},{"issue":"5","key":"952_CR5","doi-asserted-by":"publisher","first-page":"3049","DOI":"10.1007\/s10270-018-00712-x","volume":"18","author":"N Bencomo","year":"2019","unstructured":"Bencomo, N., G\u00f6tz, S., Song, H.: Models@run.time: a guided tour of the state-of-the-art and research challenges. Softw. Syst. Model. 18(5), 3049\u20133082 (2019). https:\/\/doi.org\/10.1007\/s10270-018-00712-x","journal-title":"Softw. Syst. Model."},{"issue":"10","key":"952_CR6","doi-asserted-by":"publisher","first-page":"22","DOI":"10.1109\/MC.2009.326","volume":"42","author":"G Blair","year":"2009","unstructured":"Blair, G., Bencomo, N., France, R.B.: Models@run.time. Computer 42(10), 22\u201327 (2009). https:\/\/doi.org\/10.1109\/MC.2009.326","journal-title":"Computer"},{"key":"952_CR7","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1016\/j.knosys.2015.06.021","volume":"89","author":"J Boubeta-Puig","year":"2015","unstructured":"Boubeta-Puig, J., Ortiz, G., Medina-Bulo, I.: MEdit4CEP: a model-driven solution for real-time decision making in SOA 2.0. Knowledge-Based Syst. 89, 97\u2013112 (2015). https:\/\/doi.org\/10.1016\/j.knosys.2015.06.021","journal-title":"Knowledge-Based Syst."},{"issue":"1","key":"952_CR8","doi-asserted-by":"publisher","first-page":"5","DOI":"10.1007\/s10270-019-00773-6","volume":"19","author":"A Bucchiarone","year":"2020","unstructured":"Bucchiarone, A., Cabot, J., Paige, R.F., Pierantonio, A.: Grand challenges in model-driven engineering: an analysis of the state of the research. Softw. Syst. Model. 19(1), 5\u201313 (2020)","journal-title":"Softw. Syst. Model."},{"key":"952_CR9","doi-asserted-by":"crossref","unstructured":"Camacho, A., Icarte, R.T., Klassen, T.Q., Valenzano, R.A., McIlraith, S.A.: Ltl and beyond: Formal languages for reward function specification in reinforcement learning. In: IJCAI 19, 6065\u20136073 (2019)","DOI":"10.24963\/ijcai.2019\/840"},{"key":"952_CR10","volume-title":"Data Protection: A Practical Guide To UK and EU Law","author":"P Carey","year":"2018","unstructured":"Carey, P.: Data Protection: A Practical Guide To UK and EU Law. Oxford University Press Inc., Oxford (2018)"},{"key":"952_CR11","unstructured":"Cashmore, M., Collins, A., Krarup, B., Krivic, S., Magazzeni, D., Smith, D.: Towards explainable ai planning as a service. arXiv preprint arXiv:1908.05059 (2019)"},{"issue":"7623","key":"952_CR12","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1038\/538020a","volume":"538","author":"D Castelvecchi","year":"2016","unstructured":"Castelvecchi, D.: Can we open the black box of ai? Nat. News 538(7623), 20 (2016)","journal-title":"Nat. News"},{"key":"952_CR13","doi-asserted-by":"crossref","unstructured":"Cicotti, G., Coppolino, L., Cristaldi, R., et al.: QoS Monitoring in a cloud services environment: The SRT-15 Approach. In: Euro-Par 2011: Parallel processing workshops. LNCS, pp. 15\u201324. Springer, Berlin, Heidelberg (2011)","DOI":"10.1007\/978-3-642-29737-3_3"},{"key":"952_CR14","unstructured":"Coggan, M.: Exploration and exploitation in reinforcement learning. CRA-W DMP Project at McGill University, Research supervised by Prof. Doina Precup (2004)"},{"key":"952_CR15","doi-asserted-by":"publisher","first-page":"103426","DOI":"10.1016\/j.csi.2020.103426","volume":"70","author":"D Corral-Plaza","year":"2020","unstructured":"Corral-Plaza, D., Medina-Bulo, I., Ortiz, G., Boubeta-Puig, J.: A stream processing architecture for heterogeneous data sources in the Internet of Things. Comput. Standards Interfaces 70, 103426 (2020). https:\/\/doi.org\/10.1016\/j.csi.2020.103426","journal-title":"Comput. Standards Interfaces"},{"key":"952_CR16","volume-title":"Metareasoning, monitoring, and self-explanation","author":"MT Cox","year":"2011","unstructured":"Cox, M.T.: Metareasoning, monitoring, and self-explanation. Thinking about thinking, Metareasoning (2011)"},{"key":"952_CR17","unstructured":"Davis, F.D.: A technology acceptance model for empirically testing new end-user information systems: theory and results. Ph.D. thesis, Massachusetts Institute of Technology (1985)"},{"key":"952_CR18","doi-asserted-by":"crossref","unstructured":"De\u00a0Freitas, E.P., Heimfarth, T., Netto, I.F., Lino, C.E., Pereira, C.E., Ferreira, A.M., Wagner, F.R., Larsson, T.: Uav relay network to support wsn connectivity. In: international congress on ultra modern telecommunications and control systems, pp. 309\u2013314. IEEE (2010)","DOI":"10.1109\/ICUMT.2010.5676621"},{"issue":"3","key":"952_CR19","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/2513564","volume":"3","author":"T Dodson","year":"2013","unstructured":"Dodson, T., Mattei, N., Guerin, J.T., Goldsmith, J.: An english-language argumentation interface for explanation generation with markov decision processes in the domain of academic advising. ACM Trans. Interact. Intell. Syst. 3(3), 1\u201330 (2013)","journal-title":"ACM Trans. Interact. Intell. Syst."},{"key":"952_CR20","unstructured":"Elastic: Introducting machine learning for the Elastic stack (2017). Last checked: 2020-05-15"},{"issue":"1145\/2379776","key":"952_CR21","first-page":"2379788","volume":"10","author":"P Esling","year":"2012","unstructured":"Esling, P., Agon, C.: Time-series data mining. ACM Comput. Surv. 10(1145\/2379776), 2379788 (2012)","journal-title":"ACM Comput. Surv."},{"key":"952_CR22","doi-asserted-by":"publisher","unstructured":"Feltrin, G., Popovic, N., Wojtera, M (2019) A sentinel node for event-driven structural monitoring of road bridges using wireless sensor networks. https:\/\/doi.org\/10.1155\/2019\/8652527","DOI":"10.1155\/2019\/8652527"},{"key":"952_CR23","unstructured":"Fowler, M.: Event sourcing. Online, Dec p.\u00a018 (2005)"},{"key":"952_CR24","unstructured":"Fridman, L., Brown, D.E., Glazer, M., Angell, W., Dodd, S., Jenik, B., Terwilliger, J., Kindelsberger, J., Ding, L., Seaman, S., et\u00a0al.: MIT autonomous vehicle technology study: Large-scale deep learning based analysis of driver behavior and interaction with automation. arXiv preprint arXiv:1711.069761 (2017)"},{"key":"952_CR25","doi-asserted-by":"crossref","unstructured":"Garcia-Dominguez, A., Bencomo, N., Parra-Ullauri, J.M., Garc\u00eda-Paucar, L.H.: Querying and annotating model histories with time-aware patterns. In: 2019 ACM\/IEEE 22nd International conference on model driven engineering languages and systems (MODELS), pp. 194\u2013204. IEEE (2019)","DOI":"10.1109\/MODELS.2019.000-2"},{"key":"952_CR26","doi-asserted-by":"publisher","unstructured":"G\u00f3mez, A., Cabot, J., Wimmer, M.: TemporalEMF: A Temporal Metamodeling Framework. In: J.C. Trujillo, K.C. Davis, X.\u00a0Du, Z.\u00a0Li, T.W. Ling, G.\u00a0Li, M.L. Lee (eds.) Conceptual Modeling, Lecture Notes in Computer Science, pp. 365\u2013381. Springer International Publishing, Cham (2018). https:\/\/doi.org\/10.1007\/978-3-030-00847-5_26","DOI":"10.1007\/978-3-030-00847-5_26"},{"key":"952_CR27","doi-asserted-by":"crossref","unstructured":"Haeusler, M., Trojer, T., Kessler, J., et\u00a0al.: ChronoSphere: a graph-based EMF model repository for IT landscape models. Softw. Syst. Model. (2019)","DOI":"10.1007\/s10270-019-00725-0"},{"key":"952_CR28","doi-asserted-by":"crossref","unstructured":"Hartmann, T., Fouquet, F., et\u00a0al.: Analyzing complex data in motion at scale with temporal graphs. In: Proceedings of SEKE\u201917 (2017)","DOI":"10.18293\/SEKE2017-048"},{"key":"952_CR29","doi-asserted-by":"publisher","first-page":"106685","DOI":"10.1016\/j.knosys.2020.106685","volume":"214","author":"A Heuillet","year":"2021","unstructured":"Heuillet, A., Couthouis, F., D\u00edaz-Rodr\u00edguez, N.: Explainability in deep reinforcement learning. Knowl. Based Syst. 214, 106685 (2021)","journal-title":"Knowl. Based Syst."},{"key":"952_CR30","unstructured":"Juozapaitis, Z., Koul, A., Fern, A., Erwig, M., Doshi-Velez, F.: Explainable reinforcement learning via reward decomposition. In: IJCAI\/ECAI Workshop on explainable artificial intelligence (2019)"},{"key":"952_CR31","doi-asserted-by":"publisher","unstructured":"Konno, S., D\u00e9fago, X.: Approximate QoS Rule Derivation Based on Root Cause Analysis for Cloud Computing. In: 2019 IEEE 24th Pacific Rim international symposium on dependable computing (PRDC), pp. 33\u20133309 (2019). https:\/\/doi.org\/10.1109\/PRDC47002.2019.00020. ISSN: 2473-3105","DOI":"10.1109\/PRDC47002.2019.00020"},{"key":"952_CR32","doi-asserted-by":"crossref","unstructured":"Le\u00a0Bras, P., Robb, D.A., Methven, T.S., Padilla, S., Chantler, M.J.: Improving user confidence in concept maps: exploring data driven explanations. In: Proceedings of CHI 2018. ACM (2018)","DOI":"10.1145\/3173574.3173978"},{"issue":"3","key":"952_CR33","doi-asserted-by":"publisher","first-page":"1350","DOI":"10.1214\/15-AOAS848","volume":"9","author":"B Letham","year":"2015","unstructured":"Letham, B., Rudin, C., McCormick, T.H., Madigan, D., et al.: Interpretable classifiers using rules and bayesian analysis: building a better stroke prediction model. Ann. Appl. Stat 9(3), 1350\u20131371 (2015)","journal-title":"Ann. Appl. Stat"},{"key":"952_CR34","doi-asserted-by":"publisher","first-page":"265","DOI":"10.21105\/joss.00265","volume":"2","author":"R Light","year":"2017","unstructured":"Light, R.: Mosquitto: server and client implementation of the MQTT protocol. J. Open Source Softw. 2, 265 (2017)","journal-title":"J. Open Source Softw."},{"key":"952_CR35","doi-asserted-by":"crossref","unstructured":"Lim, B.Y., Dey, A.K., Avrahami, D.: Why and why not explanations improve the intelligibility of context-aware intelligent systems. In: Proceedings of CHI 2009. ACM (2009)","DOI":"10.1145\/1518701.1519023"},{"issue":"3","key":"952_CR36","doi-asserted-by":"publisher","first-page":"31","DOI":"10.1145\/3236386.3241340","volume":"16","author":"ZC Lipton","year":"2018","unstructured":"Lipton, Z.C.: The mythos of model interpretability: in machine learning, the concept of interpretability is both important and slippery. Queue 16(3), 31\u201357 (2018)","journal-title":"Queue"},{"key":"952_CR37","unstructured":"Luckham, D.C., Frasca, B.: Complex event processing in distributed systems. Computer systems laboratory technical report CSL-TR-98-754. Stanford University, Stanford 28, 16 (1998)"},{"issue":"6","key":"952_CR38","doi-asserted-by":"publisher","first-page":"2063","DOI":"10.1109\/TNNLS.2018.2790388","volume":"29","author":"M Mahmud","year":"2018","unstructured":"Mahmud, M., Kaiser, M.S., Hussain, A., Vassanelli, S.: Applications of deep learning and reinforcement learning to biological data. IEEE Trans. Neural Netw. Learn. Syst. 29(6), 2063\u20132079 (2018). https:\/\/doi.org\/10.1109\/TNNLS.2018.2790388","journal-title":"IEEE Trans. Neural Netw. Learn. Syst."},{"issue":"3","key":"952_CR39","doi-asserted-by":"publisher","first-page":"3:1","DOI":"10.5381\/jot.2020.19.3.a14","volume":"19","author":"A Mazak","year":"2020","unstructured":"Mazak, A., Wolny, S., G\u00f3mez, A., Cabot, J., Wimmer, M., Kappel, G.: Temporal models on time series databases. J. Object Technol. 19(3), 3:1 (2020). https:\/\/doi.org\/10.5381\/jot.2020.19.3.a14","journal-title":"J. Object Technol."},{"key":"952_CR40","doi-asserted-by":"crossref","unstructured":"Merwaday, A., Guvenc, I.: Uav assisted heterogeneous networks for public safety communications. In: 2015 IEEE wireless communications and networking conference workshops (WCNCW), pp. 329\u2013334. IEEE (2015)","DOI":"10.1109\/WCNCW.2015.7122576"},{"key":"952_CR41","volume-title":"An Artificial Intelligence Approach","author":"R Mitchell","year":"2013","unstructured":"Mitchell, R., Michalski, J., Carbonell, T.: An Artificial Intelligence Approach. Springer, Berlin (2013)"},{"issue":"7540","key":"952_CR42","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1038\/nature14236","volume":"518","author":"V Mnih","year":"2015","unstructured":"Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529 (2015)","journal-title":"Nature"},{"key":"952_CR43","doi-asserted-by":"publisher","unstructured":"Moser, O., Rosenberg, F., Dustdar, S.: Event Driven Monitoring for Service Composition Infrastructures. In: L.\u00a0Chen, P.\u00a0Triantafillou, T.\u00a0Suel (eds.) Web Information Systems Engineering\u2014WISE 2010, Lecture Notes in Computer Science, pp. 38\u201351. Springer, Berlin, Heidelberg (2010). https:\/\/doi.org\/10.1007\/978-3-642-17616-6_6","DOI":"10.1007\/978-3-642-17616-6_6"},{"key":"952_CR44","unstructured":"Oh, J., Hessel, M., Czarnecki, W.M., Xu, Z., van Hasselt, H., Singh, S., Silver, D.: Discovering reinforcement learning algorithms. arXiv preprint arXiv:2007.08794 (2020)"},{"key":"952_CR45","doi-asserted-by":"crossref","unstructured":"Overeem, M., Spoor, M., Jansen, S.: The dark side of event sourcing: Managing data conversion. In: 2017 IEEE 24th international conference on software analysis, evolution and reengineering (SANER), pp. 193\u2013204. IEEE (2017)","DOI":"10.1109\/SANER.2017.7884621"},{"key":"952_CR46","doi-asserted-by":"crossref","unstructured":"Panch, T., Szolovits, P., Atun, R.: Artificial intelligence, machine learning and health systems. J. Glob. Health 8(2) (2018)","DOI":"10.7189\/jogh.08.020303"},{"key":"952_CR47","unstructured":"Parra-Ullauri, J.M.: ETeMoX event-driven temporal models for explanations (2021). https:\/\/gitlab.com\/sea-aston\/etemox"},{"key":"952_CR48","doi-asserted-by":"crossref","unstructured":"Parra-Ullauri, J.M., Garc\u00eda-Dom\u00ednguez, A., Boubeta-Puig, J., Bencomo, N., Ortiz, G.: Towards an architecture integrating complex event processing and temporal graphs for service monitoring (2021)","DOI":"10.1145\/3412841.3441923"},{"key":"952_CR49","doi-asserted-by":"crossref","unstructured":"Parra-Ullauri, J.M., Garc\u00eda-Dom\u00ednguez, A., Garc\u00eda-Paucar, L.H., Bencomo, N.: Temporal models for history-aware explainability. In: Proceedings of the 12th system analysis and modelling conference, pp. 155\u2013164 (2020)","DOI":"10.1145\/3419804.3420276"},{"key":"952_CR50","unstructured":"Paul, L.C.: Handoff\/handover mechanism for mobility improvement in wireless communication. Glob. J. Res. Eng. (2014)"},{"issue":"10","key":"952_CR51","doi-asserted-by":"publisher","first-page":"911","DOI":"10.1016\/j.tics.2018.08.004","volume":"22","author":"EA Petter","year":"2018","unstructured":"Petter, E.A., Gershman, S.J., Meck, W.H.: Integrating models of interval timing and reinforcement learning. Trends Cogn. Sci. 22(10), 911\u2013922 (2018)","journal-title":"Trends Cogn. Sci."},{"key":"952_CR52","unstructured":"Poole\u00a0David Mackworth\u00a0Alan, G.R.: Computational intelligence: a logical approach.(1998). Google scholar google scholar digital library digital library (1998)"},{"issue":"7","key":"952_CR53","doi-asserted-by":"publisher","first-page":"e1934","DOI":"10.1002\/stc.1934","volume":"24","author":"N Popovic","year":"2017","unstructured":"Popovic, N., Feltrin, G., Jalsan, K.E., Wojtera, M.: Event-driven strain cycle monitoring of railway bridges using a wireless sensor network with sentinel nodes. Struct. Control. Health Monit. 24(7), e1934 (2017). https:\/\/doi.org\/10.1002\/stc.1934","journal-title":"Struct. Control. Health Monit."},{"key":"952_CR54","doi-asserted-by":"crossref","unstructured":"Puiutta, E., Veith, E.M.: Explainable reinforcement learning: a survey. In: International cross-domain conference for machine learning and knowledge extraction, pp. 77\u201395. Springer (2020)","DOI":"10.1007\/978-3-030-57321-8_5"},{"key":"952_CR55","doi-asserted-by":"crossref","unstructured":"Ras, G., van Gerven, M., Haselager, P.: Explanation methods in deep learning: users, values, concerns and challenges. In: Explainable and interpretable models in computer vision and machine learning, pp. 19\u201336. Springer (2018)","DOI":"10.1007\/978-3-319-98131-4_2"},{"key":"952_CR56","unstructured":"Ricciato, F., Widhalm, P., Craglia, M., Pantisano, F.: Estimating population density distribution from network-based mobile phone data. Publications Office of the European Union (2015)"},{"key":"952_CR57","unstructured":"Robinson, I., Webber, J., Eifrem, E.: Graph databases, second edn. O\u2019Reilly (2015). ISBN 978-1-4919-3089-2"},{"key":"952_CR58","doi-asserted-by":"publisher","DOI":"10.1016\/j.eswa.2020.113251","author":"J Rold\u00e1n","year":"2020","unstructured":"Rold\u00e1n, J., Boubeta-Puig, J., Mart\u00ednez, J.L., Ortiz, G.: Integrating complex event processing and machine learning: an intelligent architecture for detecting IoT security attacks. Expert Syst. Appl. (2020). https:\/\/doi.org\/10.1016\/j.eswa.2020.113251","journal-title":"Expert Syst. Appl."},{"key":"952_CR59","doi-asserted-by":"crossref","unstructured":"Romano, L., De\u00a0Mari, D., Jerzak, Z., Fetzer, C.: A novel approach to QoS Monitoring in the Cloud. In: 2011 First international conference on data compression, communications and processing, pp. 45\u201351 (2011)","DOI":"10.1109\/CCP.2011.49"},{"key":"952_CR60","unstructured":"Rosenfeld, A.: Better metrics for evaluating explainable artificial intelligence. In: Proceedings of the 20th international conference on autonomous agents and multiagent systems, pp. 45\u201350 (2021)"},{"key":"952_CR61","unstructured":"Samek, W., Wiegand, T., M\u00fcller, K.R.: Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models. arXiv preprint arXiv:1708.08296 (2017)"},{"key":"952_CR62","doi-asserted-by":"publisher","first-page":"103367","DOI":"10.1016\/j.artint.2020.103367","volume":"288","author":"P Sequeira","year":"2020","unstructured":"Sequeira, P., Gervasio, M.: Interestingness elements for explainable reinforcement learning: understanding agents capabilities and limitations. Artif. Intell. 288, 103367 (2020)","journal-title":"Artif. Intell."},{"issue":"1","key":"952_CR63","first-page":"1064","volume":"26","author":"T Spinner","year":"2019","unstructured":"Spinner, T., Schlegel, U., Sch\u00e4fer, H., El-Assady, M.: Explainer: a visual analytics framework for interactive and explainable machine learning. IEEE Trans. Visual Comput. Graphics 26(1), 1064\u20131074 (2019)","journal-title":"IEEE Trans. Visual Comput. Graphics"},{"key":"952_CR64","doi-asserted-by":"publisher","unstructured":"Student: The probable error of a mean. Biometrika 6(1), 1\u201325 (1908). https:\/\/doi.org\/10.2307\/2331554","DOI":"10.2307\/2331554"},{"key":"952_CR65","unstructured":"Such, F.P., Madhavan, V., Liu, R., Wang, R., Castro, P.S., Li, Y., Zhi, J., Schubert, L., Bellemare, M.G., Clune, J., et\u00a0al.: An Atari model zoo for analyzing, visualizing, and comparing deep reinforcement learning agents. arXiv preprint arXiv:1812.07069 (2018)"},{"key":"952_CR66","unstructured":"Sutton, R.S.: Generalization in reinforcement learning: Successful examples using sparse coarse coding. In: Advances in neural information processing systems, pp. 1038\u20131044 (1996)"},{"key":"952_CR67","volume-title":"Reinforcement Learning: An Introduction","author":"RS Sutton","year":"2018","unstructured":"Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT press, Cambridge (2018)"},{"key":"952_CR68","doi-asserted-by":"crossref","unstructured":"Theodoridis, T., Solachidis, V., Dimitropoulos, K., Gymnopoulos, L., Daras, P.: A survey on ai nutrition recommender systems. In: Proceedings of the 12th ACM International Conference on PErvasive technologies related to assistive environments, pp. 540\u2013546 (2019)","DOI":"10.1145\/3316782.3322760"},{"issue":"2","key":"952_CR69","doi-asserted-by":"publisher","first-page":"105","DOI":"10.1007\/s10676-009-9187-9","volume":"11","author":"M Turilli","year":"2009","unstructured":"Turilli, M., Floridi, L.: The ethics of information transparency. Ethics Inf. Technol. 11(2), 105\u2013112 (2009)","journal-title":"Ethics Inf. Technol."},{"key":"952_CR70","unstructured":"Valente\u00a0Klaine, P.H.: Self-organization for 5g and beyond mobile networks using reinforcement learning. Ph.D. thesis, University of Glasgow (2019)"},{"issue":"2","key":"952_CR71","doi-asserted-by":"publisher","first-page":"186","DOI":"10.1287\/mnsc.46.2.186.11926","volume":"46","author":"V Venkatesh","year":"2000","unstructured":"Venkatesh, V., Davis, F.D.: A theoretical extension of the technology acceptance model: four longitudinal field studies. Manage. Sci. 46(2), 186\u2013204 (2000)","journal-title":"Manage. Sci."},{"key":"952_CR72","doi-asserted-by":"crossref","unstructured":"Vijayakumar, K., Arun, C.: Automated risk identification using NLP in cloud based development environments. J. Ambient Intell. Hum. Comput. 1\u201313 (2017)","DOI":"10.1007\/s12652-017-0503-7"},{"key":"952_CR73","unstructured":"van\u00a0der Waa, J., van Diggelen, J., Bosch, K.v.d., Neerincx, M.: Contrastive explanations for reinforcement learning in terms of expected consequences. arXiv preprint arXiv:1807.08706 (2018)"},{"issue":"3\u20134","key":"952_CR74","first-page":"279","volume":"8","author":"CJ Watkins","year":"1992","unstructured":"Watkins, C.J., Dayan, P.: Q-learning. Mach. Learn. 8(3\u20134), 279\u2013292 (1992)","journal-title":"Mach. Learn."},{"key":"952_CR75","unstructured":"Watkins, C.J.C.H.: Learning from delayed rewards. Ph.D. thesis, King\u2019s College, Cambridge (1989)"},{"issue":"5","key":"952_CR76","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1109\/MCOM.2016.7470933","volume":"54","author":"Y Zeng","year":"2016","unstructured":"Zeng, Y., Zhang, R., Lim, T.J.: Wireless communications with unmanned aerial vehicles: opportunities and challenges. IEEE Commun. Mag. 54(5), 36\u201342 (2016)","journal-title":"IEEE Commun. Mag."},{"key":"952_CR77","doi-asserted-by":"crossref","unstructured":"Zheng, C., Yang, S., Parra-Ullauri, J.M., Garcia-Dominguez, A., Bencomo, N.: Reward-reinforced generative adversarial networks for multi-agent systems. IEEE Trans. Emerg. Top. Comput. Intell. (2021)","DOI":"10.1109\/TETCI.2021.3082204"}],"container-title":["Software and Systems Modeling"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10270-021-00952-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s10270-021-00952-4\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s10270-021-00952-4.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,25]],"date-time":"2022-04-25T09:23:17Z","timestamp":1650878597000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s10270-021-00952-4"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,12,18]]},"references-count":77,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,6]]}},"alternative-id":["952"],"URL":"https:\/\/doi.org\/10.1007\/s10270-021-00952-4","relation":{},"ISSN":["1619-1366","1619-1374"],"issn-type":[{"type":"print","value":"1619-1366"},{"type":"electronic","value":"1619-1374"}],"subject":[],"published":{"date-parts":[[2021,12,18]]},"assertion":[{"value":"2 April 2021","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"27 September 2021","order":2,"name":"revised","label":"Revised","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"4 November 2021","order":3,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"18 December 2021","order":4,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}}]}}