{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,8]],"date-time":"2026-01-08T00:28:43Z","timestamp":1767832123265,"version":"3.49.0"},"reference-count":60,"publisher":"MDPI AG","issue":"2","license":[{"start":{"date-parts":[[2023,3,24]],"date-time":"2023-03-24T00:00:00Z","timestamp":1679616000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"National Council for Scientific and Technological Development (CNPq)","award":["PRIN 2020TL3X8X"],"award-info":[{"award-number":["PRIN 2020TL3X8X"]}]},{"name":"Coordination for the Improvement of Higher Education Personnel (CAPES)","award":["PRIN 2020TL3X8X"],"award-info":[{"award-number":["PRIN 2020TL3X8X"]}]},{"name":"MUR project \u201cT-LADIES\u201d","award":["PRIN 2020TL3X8X"],"award-info":[{"award-number":["PRIN 2020TL3X8X"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Robotics"],"abstract":"<jats:p>This paper presents a Runtime Verification (RV) approach for Multi-Agent Systems (MAS) using the JaCaMo framework. Our objective is to bring a layer of security to the MAS. This is achieved keeping in mind possible safety-critical uses of the MAS, such as robotic applications. This layer is capable of controlling events during the execution of the system without needing a specific implementation in the behaviour of each agent to recognise the events. In this paper, we mainly focus on MAS when used in the context of hybrid intelligence. This use requires communication between software agents and human beings. In some cases, communication takes place via natural language dialogues. However, this kind of communication brings us to a concern related to controlling the flow of dialogue so that agents can prevent any change in the topic of discussion that could impair their reasoning. The latter may be a problem and undermine the development of the software agents. In this paper, we tackle this problem by proposing and demonstrating the implementation of a framework that aims to control the dialogue flow in a MAS; especially when the MAS communicates with the user through natural language to aid decision-making in a hospital bed allocation scenario.<\/jats:p>","DOI":"10.3390\/robotics12020049","type":"journal-article","created":{"date-parts":[[2023,3,24]],"date-time":"2023-03-24T08:43:03Z","timestamp":1679647383000},"page":"49","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":9,"title":["RV4JaCa\u2014Towards Runtime Verification of Multi-Agent Systems and Robotic Applications"],"prefix":"10.3390","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6090-8294","authenticated-orcid":false,"given":"Debora C.","family":"Engelmann","sequence":"first","affiliation":[{"name":"School of Technology, Pontifical Catholic University of Rio Grande do Sul, Porto Alegre 90619-900, Brazil"},{"name":"Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genoa, 16145 Genova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8711-4670","authenticated-orcid":false,"given":"Angelo","family":"Ferrando","sequence":"additional","affiliation":[{"name":"Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genoa, 16145 Genova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9438-5508","authenticated-orcid":false,"given":"Alison R.","family":"Panisson","sequence":"additional","affiliation":[{"name":"Department of Computing, Federal University of Santa Catarina, Ararangu\u00e1 88906-072, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6297-2011","authenticated-orcid":false,"given":"Davide","family":"Ancona","sequence":"additional","affiliation":[{"name":"Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genoa, 16145 Genova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8688-9901","authenticated-orcid":false,"given":"Rafael H.","family":"Bordini","sequence":"additional","affiliation":[{"name":"School of Technology, Pontifical Catholic University of Rio Grande do Sul, Porto Alegre 90619-900, Brazil"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2261-9926","authenticated-orcid":false,"given":"Viviana","family":"Mascardi","sequence":"additional","affiliation":[{"name":"Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genoa, 16145 Genova, Italy"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1968","published-online":{"date-parts":[[2023,3,24]]},"reference":[{"key":"ref_1","unstructured":"Bordini, R.H., Dastani, M., Dix, J., and Seghrouchni, A.E.F. (2009). Multi-Agent Programming, Languages, Tools and Applications, Springer."},{"key":"ref_2","unstructured":"Markov, Z., and Russell, I. (2016, January 16\u201318). An Ontology-Based Mobile Application for Task Managing in Collaborative Groups. Proceedings of the Twenty-Ninth International Florida Artificial Intelligence Research Society Conference, FLAIRS 2016, Key Largo, FL, USA."},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Cardoso, R.C., Ferrando, A., Briola, D., Menghi, C., and Ahlbrecht, T. (2021). Agents and Robots for Reliable Engineered Autonomy: A Perspective from the Organisers of AREA 2020. J. Sens. Actuator Netw., 10.","DOI":"10.3390\/jsan10020033"},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"1094","DOI":"10.1007\/s10458-016-9356-2","article-title":"BDI agent testability revisited","volume":"31","author":"Winikoff","year":"2017","journal-title":"Auton. Agents Multi Agent Syst."},{"key":"ref_5","unstructured":"Larson, K., Winikoff, M., Das, S., and Durfee, E.H. (2017, January 8\u201312). Debugging Agent Programs with Why?: Questions. Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2017, S\u00e3o Paulo, Brazil."},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"5","DOI":"10.1007\/s10515-011-0088-x","article-title":"Model checking agent programming languages","volume":"19","author":"Dennis","year":"2012","journal-title":"Autom. Softw. Eng."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/978-3-319-75632-5_1","article-title":"Introduction to Runtime Verification","volume":"Volume 10457","author":"Bartocci","year":"2018","journal-title":"Lectures on Runtime Verification\u2014Introductory and Advanced Topics"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"293","DOI":"10.1016\/j.jlap.2008.08.004","article-title":"A brief account of runtime verification","volume":"78","author":"Leucker","year":"2009","journal-title":"J. Log. Algebr. Methods Program."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"200","DOI":"10.1007\/s10703-017-0274-y","article-title":"Verifying data- and control-oriented properties combining static and runtime verification: Theory and tools","volume":"51","author":"Ahrendt","year":"2017","journal-title":"Form. Methods Syst. Des."},{"key":"ref_10","doi-asserted-by":"crossref","first-page":"747","DOI":"10.1016\/j.scico.2011.10.004","article-title":"Multi-agent oriented programming with JaCaMo","volume":"78","author":"Boissier","year":"2013","journal-title":"Sci. Comput. Program."},{"key":"ref_11","unstructured":"Engelmann, D.C., Ferrando, A., Panisson, A.R., Ancona, D., Bordini, R.H., and Mascardi, V. (2022, January 24). RV4JaCa\u2013Runtime Verification for Multi-Agent Systems. Proceedings of the Second Workshop on Agents and Robots for Reliable Engineered Autonomy, AREA@IJCAI-ECAI 2022, Vienna, Austria."},{"key":"ref_12","unstructured":"Wooldridge, M. (2002). An Introduction to MultiAgent Systems, John Wiley & Sons Ltd."},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Bordini, R.H., H\u00fcbner, J.F., and Wooldridge, M. (2007). Programming Multi-Agent Systems in AgentSpeak Using Jason, John Wiley & Sons.","DOI":"10.1002\/9780470061848"},{"key":"ref_14","first-page":"214","article-title":"From Agents to Organizations: An Organizational View of Multi-agent Systems","volume":"Volume 2935","author":"Giorgini","year":"2003","journal-title":"Proceedings of the Agent-Oriented Software Engineering IV, 4th International Workshop, AOSE 2003"},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"281","DOI":"10.1017\/S0269888905000317","article-title":"A survey of multi-agent organizational paradigms","volume":"19","author":"Horling","year":"2004","journal-title":"Knowl. Eng. Rev."},{"key":"ref_16","unstructured":"Anjomshoae, S., Najjar, A., Calvaresi, D., and Fr\u00e4mling, K. (2019, January 13\u201317). Explainable agents and robots: Results from a systematic literature review. Proceedings of the 18th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2019), Montreal, QC, Canada."},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"52138","DOI":"10.1109\/ACCESS.2018.2870052","article-title":"Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI)","volume":"6","author":"Adadi","year":"2018","journal-title":"IEEE Access"},{"key":"ref_18","unstructured":"Donadello, I., Dragoni, M., and Eccher, C. (April, January 30). Explaining reasoning algorithms with persuasiveness: A case study for a behavioural change system. Proceedings of the 35th Annual ACM Symposium on Applied Computing, Brno, Czech Republic."},{"key":"ref_19","unstructured":"Rao, A.S., and Georgeff, M.P. (1995, January 12\u201314). BDI agents: From theory to practice. Proceedings of the First International Conference on Multiagent Systems, San Francisco, CA, USA."},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"370","DOI":"10.1504\/IJAOSE.2007.016266","article-title":"Developing organised multiagent systems using the MOISE+ model: Programming issues at the system and agent levels","volume":"1","author":"Hubner","year":"2007","journal-title":"Int. J. Agent Oriented Softw. Eng."},{"key":"ref_21","unstructured":"Ricci, A., Piunti, M., Viroli, M., and Omicini, A. (2009). Multi-Agent Programming, Springer."},{"key":"ref_22","doi-asserted-by":"crossref","unstructured":"Cardoso, R.C., and Ferrando, A. (2021). A Review of Agent-Based Programming for Multi-Agent Systems. Computers, 10.","DOI":"10.3390\/computers10020016"},{"key":"ref_23","unstructured":"Clarke, E.M. Model checking. Proceedings of the International Conference on Foundations of Software Technology and Theoretical Computer Science."},{"key":"ref_24","unstructured":"Loveland, D.W. (1978). Automated Theorem Proving: A Logical Basis, North-Holland. Fundamental Studies in Computer Science."},{"key":"ref_25","doi-asserted-by":"crossref","first-page":"8","DOI":"10.1007\/s10458-020-09487-2","article-title":"Towards a framework for certification of reliable autonomous systems","volume":"35","author":"Fisher","year":"2021","journal-title":"Auton. Agents Multi Agent Syst."},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"102610","DOI":"10.1016\/j.scico.2021.102610","article-title":"RML: Theory and Practice of a Domain Specific Language for Runtime Verification","volume":"205","author":"Ancona","year":"2021","journal-title":"Sci. Comput. Program."},{"key":"ref_27","first-page":"47","article-title":"Comparing Trace Expressions and Linear Temporal Logic for Runtime Verification","volume":"Volume 9660","author":"Bonsangue","year":"2016","journal-title":"Proceedings of the Theory and Practice of Formal Methods\u2014Essays Dedicated to Frank de Boer on the Occasion of His 60th Birthday"},{"key":"ref_28","unstructured":"Ancona, D., Ferrando, A., and Mascardi, V. (2017, January 8\u201312). Parametric Runtime Verification of Multiagent Systems. Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2017, S\u00e3o Paulo, Brazil."},{"key":"ref_29","first-page":"141","article-title":"A Tutorial on Runtime Verification","volume":"Volume 34","author":"Broy","year":"2013","journal-title":"Engineering Dependable Software Systems"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"189","DOI":"10.1023\/B:FORM.0000017721.39909.4b","article-title":"An Overview of the Runtime Verification Tool Java PathExplorer","volume":"24","author":"Havelund","year":"2004","journal-title":"Form. Methods Syst. Des."},{"key":"ref_31","unstructured":"Ferrando, A., Ancona, D., and Mascardi, V. (2017, January 8\u201312). Decentralizing MAS Monitoring with DecAMon. Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2017, S\u00e3o Paulo, Brazil."},{"key":"ref_32","unstructured":"Ancona, D., Briola, D., Ferrando, A., and Mascardi, V. (2015, January 4\u20138). Global Protocols as First Class Entities for Self-Adaptive Agents. Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2015, Istanbul, Turkey."},{"key":"ref_33","first-page":"1","article-title":"Queueing Problems in Emergency Departments: A Review of Practical Approaches and Research Methodologies","volume":"Volume 3","author":"Elalouf","year":"2022","journal-title":"Proceedings of the Operations Research Forum"},{"key":"ref_34","doi-asserted-by":"crossref","first-page":"249","DOI":"10.1097\/CIN.0000000000000421","article-title":"A hospital bed allocation hybrid model based on situation awareness","volume":"36","author":"Righi","year":"2018","journal-title":"Comput. Inform. Nurs."},{"key":"ref_35","unstructured":"Matos, J., and Rodrigues, P.P. (2011, January 26\u201329). Modeling decisions for hospital bed management\u2014A review. Proceedings of the 4th International Conference on Health Informatics, Rome, Italy."},{"key":"ref_36","first-page":"100222","article-title":"Brazil\u2019s health system functionality amidst of the COVID-19 pandemic: An analysis of resilience","volume":"10","author":"Bigoni","year":"2022","journal-title":"Lancet Reg. Health Am."},{"key":"ref_37","doi-asserted-by":"crossref","first-page":"108320","DOI":"10.1016\/j.ijpe.2021.108320","article-title":"Cope with the COVID-19 pandemic: Dynamic bed allocation and patient subsidization in a public healthcare system","volume":"243","author":"Ma","year":"2022","journal-title":"Int. J. Prod. Econ."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"1873","DOI":"10.1007\/s10916-010-9646-1","article-title":"Intelligent Analysis of Acute Bed Overflow in a Tertiary Hospital in Singapore","volume":"36","author":"Teow","year":"2012","journal-title":"J. Med. Syst."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"e054545","DOI":"10.1136\/bmjopen-2021-054545","article-title":"Effects of patient-level risk factors, departmental allocation and seasonality on intrahospital patient transfer patterns: Network analysis applied on a Norwegian single-centre data set","volume":"12","author":"Zhang","year":"2022","journal-title":"BMJ Open"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"149","DOI":"10.1136\/emj.20.2.149","article-title":"Can good bed management solve the overcrowding in accident and emergency departments?","volume":"20","author":"Proudlove","year":"2003","journal-title":"Emerg. Med. J."},{"key":"ref_41","unstructured":"Engelmann, D.C., Panisson, A.R., Vieira, R., H\u00fcbner, J.F., Mascardi, V., and Bordini, R.H. (June, January 29). MAIDS\u2014A Framework for the Development of Multi-Agent Intentional Dialogue Systems. Proceedings of the 22st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2023, London, UK."},{"key":"ref_42","unstructured":"Gombolay, M.C., Yang, X.J., Hayes, B., Seo, N., Liu, Z., Wadhwania, S., Yu, T., Shah, N., Golen, T., and Shah, J.A. (2016, January 12\u201314). Robotic assistance in coordination of patient care. Proceedings of the 12nd Robotics: Science and Systems Conference, Cambridge, MA, USA."},{"key":"ref_43","first-page":"77","article-title":"Dial4JaCa\u2014A Communication Interface Between Multi-agent Systems and Chatbots","volume":"Volume 12946","author":"Dignum","year":"2021","journal-title":"Proceedings of the Advances in Practical Applications of Agents, Multi-Agent Systems, and Social Good. The PAAMS Collection\u201419th International Conference, PAAMS 2021, Salamanca, Spain, 6\u20138 October 2021"},{"key":"ref_44","first-page":"273","article-title":"Engineering Explainable Agents: An Argumentation-Based Approach","volume":"Volume 13190","author":"Alechina","year":"2021","journal-title":"Proceedings of the Engineering Multi-Agent Systems\u20149th International Workshop, EMAS 2021"},{"key":"ref_45","unstructured":"Panisson, A.R., Meneguzzi, F., Vieira, R., and Bordini, R.H. (2014, January 5). An approach for argumentation-based reasoning using defeasible logic in multi-agent programming languages. Proceedings of the 11th International Workshop on Argumentation in Multiagent Systems, Paris, France."},{"key":"ref_46","doi-asserted-by":"crossref","first-page":"51","DOI":"10.1016\/j.websem.2007.03.004","article-title":"Pellet: A practical OWL-DL reasoner","volume":"5","author":"Sirin","year":"2007","journal-title":"J. Web Semant."},{"key":"ref_47","first-page":"1","article-title":"SWRL: A semantic web rule language combining OWL and RuleML","volume":"21","author":"Horrocks","year":"2004","journal-title":"W3C Memb. Submiss."},{"key":"ref_48","first-page":"3","article-title":"A Conversational Agent to Support Hospital Bed Allocation","volume":"Volume 13073","author":"Britto","year":"2021","journal-title":"Proceedings of the Intelligent Systems\u201410th Brazilian Conference, BRACIS 2021"},{"key":"ref_49","unstructured":"Cohen, P. Foundations of Collaborative Task-Oriented Dialogue: What\u2019s in a Slot?. Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue."},{"key":"ref_50","unstructured":"Bakar, N.A., and Selamat, A. (2013, January 18\u201320). Runtime Verification of Multi-agent Systems Interaction Quality. Proceedings of the Intelligent Information and Database Systems\u20145th Asian Conference, ACIIDS 2013, Kuala Lumpur, Malaysia."},{"key":"ref_51","unstructured":"Roungroongsom, C., and Pradubsuwun, D. (2015). Recent Advances in Information and Communication Technology 2015, Springer."},{"key":"ref_52","doi-asserted-by":"crossref","unstructured":"Lim, Y.J., Hong, G., Shin, D., Jee, E., and Bae, D. (2016, January 18\u201320). A runtime verification framework for dynamically adaptive multi-agent systems. Proceedings of the 2016 International Conference on Big Data and Smart Computing, BigComp 2016, Hong Kong, China.","DOI":"10.1109\/BIGCOMP.2016.7425981"},{"key":"ref_53","doi-asserted-by":"crossref","unstructured":"Alberti, M., Gavanelli, M., Lamma, E., Mello, P., and Torroni, P. (2005, January 21\u201323). The SCIFF Abductive Proof-Procedure. Proceedings of the AI*IA 2005: Advances in Artificial Intelligence, 9th Congress of the Italian Association for Artificial Intelligence, Milan, Italy.","DOI":"10.1007\/11558590_14"},{"key":"ref_54","doi-asserted-by":"crossref","unstructured":"Dignum, V. (2009). Handbook of Research on Multi-Agent Systems\u2014Semantics and Dynamics of Organizational Models, IGI Global.","DOI":"10.4018\/978-1-60566-256-5"},{"key":"ref_55","unstructured":"Chesani, F., Mello, P., Montali, M., and Torroni, P. (2009, January 11\u201317). Commitment Tracking via the Reactive Event Calculus. Proceedings of the 21st International Joint Conference on Artificial Intelligence, Pasadena, CA, USA."},{"key":"ref_56","doi-asserted-by":"crossref","unstructured":"Kyrarini, M., Lygerakis, F., Rajavenkatanarayanan, A., Sevastopoulos, C., Nambiappan, H.R., Chaitanya, K.K., Babu, A.R., Mathew, J., and Makedon, F. (2021). A survey of robots in healthcare. Technologies, 9.","DOI":"10.3390\/technologies9010008"},{"key":"ref_57","doi-asserted-by":"crossref","unstructured":"Holland, J., Kingston, L., McCarthy, C., Armstrong, E., O\u2019Dwyer, P., Merz, F., and McConnell, M. (2021). Service robots in the healthcare sector. Robotics, 10.","DOI":"10.3390\/robotics10010047"},{"key":"ref_58","unstructured":"Khan, A., and Anwar, Y. Robots in healthcare: A survey. Proceedings of the Science and Information Conference."},{"key":"ref_59","doi-asserted-by":"crossref","first-page":"68","DOI":"10.1145\/3127874","article-title":"Healthcare robotics","volume":"60","author":"Riek","year":"2017","journal-title":"Commun. ACM"},{"key":"ref_60","unstructured":"Cardoso, R.C., Ferrando, A., Dennis, L.A., and Fisher, M. (2020, January 14\u201315). An Interface for Programming Verifiable Autonomous Agents in ROS. Proceedings of the Multi-Agent Systems and Agreement Technologies\u201417th European Conference, EUMAS 2020, and 7th International Conference, AT 2020, Thessaloniki, Greece."}],"container-title":["Robotics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/2218-6581\/12\/2\/49\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T19:02:22Z","timestamp":1760122942000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/2218-6581\/12\/2\/49"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,24]]},"references-count":60,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2023,4]]}},"alternative-id":["robotics12020049"],"URL":"https:\/\/doi.org\/10.3390\/robotics12020049","relation":{},"ISSN":["2218-6581"],"issn-type":[{"value":"2218-6581","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,24]]}}}