{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,31]],"date-time":"2025-10-31T08:02:06Z","timestamp":1761897726496,"version":"3.41.0"},"reference-count":64,"publisher":"Association for Computing Machinery (ACM)","issue":"3-4","license":[{"start":{"date-parts":[[2022,12,15]],"date-time":"2022-12-15T00:00:00Z","timestamp":1671062400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Auton. Adapt. Syst."],"published-print":{"date-parts":[[2022,12,31]]},"abstract":"<jats:p>\n            Many self-adaptive systems benefit from human involvement and oversight, where a human operator can provide expertise not available to the system and detect problems that the system is unaware of. One way of achieving this synergy is by placing the human operator\n            <jats:italic>on the loop<\/jats:italic>\n            \u2014i.e., providing supervisory oversight and intervening in the case of questionable adaptation decisions. To make such interaction effective, an\n            <jats:italic>explanation<\/jats:italic>\n            can play an important role in allowing the human operator to understand why the system is making certain decisions and improve the level of knowledge that the operator has about the system. This, in turn, may improve the operator\u2019s capability to intervene and, if necessary, override the decisions being made by the system. However, explanations may incur costs, in terms of delay in actions and the possibility that a human may make a bad judgment. Hence, it is not always obvious whether an explanation will improve overall utility and, if so, then what kind of explanation should be provided to the operator. In this work, we define a formal framework for reasoning about explanations of adaptive system behaviors and the conditions under which they are warranted. Specifically, we characterize explanations in terms of explanation\n            <jats:italic>content<\/jats:italic>\n            ,\n            <jats:italic>effect<\/jats:italic>\n            , and\n            <jats:italic>cost<\/jats:italic>\n            . We then present a dynamic system adaptation approach that leverages a probabilistic reasoning technique to determine when an explanation should be used to improve overall system utility. We evaluate our explanation framework in the context of a realistic industrial control system with adaptive behaviors.\n          <\/jats:p>","DOI":"10.1145\/3557898","type":"journal-article","created":{"date-parts":[[2022,8,17]],"date-time":"2022-08-17T12:03:53Z","timestamp":1660737833000},"page":"1-26","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":3,"title":["Modeling and Analysis of Explanation for Secure Industrial Control Systems"],"prefix":"10.1145","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0045-2811","authenticated-orcid":false,"given":"Sridhar","family":"Adepu","sequence":"first","affiliation":[{"name":"University of Bristol, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9379-9717","authenticated-orcid":false,"given":"Nianyu","family":"Li","sequence":"additional","affiliation":[{"name":"Peking University, China"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7891-6885","authenticated-orcid":false,"given":"Eunsuk","family":"Kang","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6735-8301","authenticated-orcid":false,"given":"David","family":"Garlan","sequence":"additional","affiliation":[{"name":"Carnegie Mellon University, USA"}]}],"member":"320","published-online":{"date-parts":[[2022,12,15]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-02161-9_1"},{"key":"e_1_3_2_3_2","first-page":"1","volume-title":"International Symposium on Software Engineering for Self-adaptive Systems","author":"Lemos Rog\u00e9rio de","year":"2010","unstructured":"Rog\u00e9rio de Lemoset al.2010. Software engineering for self-adaptive systems: A second research roadmap. In International Symposium on Software Engineering for Self-adaptive Systems. 1\u201332."},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-22969-0_21"},{"key":"e_1_3_2_5_2","first-page":"146","volume-title":"10th IEEE\/ACM International Symposium on Software Engineering for Adaptive and Self-managing Systems","author":"C\u00e1mara Javier","year":"2015","unstructured":"Javier C\u00e1mara, Gabriel A. Moreno, and David Garlan. 2015. Reasoning about human participation in self-adaptive systems. In 10th IEEE\/ACM International Symposium on Software Engineering for Adaptive and Self-managing Systems. 146\u2013156."},{"key":"e_1_3_2_6_2","first-page":"163","volume-title":"12th IEEE\/ACM International Symposium on Software Engineering for Adaptive and Self-managing Systems","author":"Lloyd Eric","year":"2017","unstructured":"Eric Lloyd, Shihong Huang, and Emmanuelle Tognoli. 2017. Improving human-in-the-loop adaptive systems using brain-computer interaction. In 12th IEEE\/ACM International Symposium on Software Engineering for Adaptive and Self-managing Systems. 163\u2013174."},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1002\/cpe.4082"},{"key":"e_1_3_2_8_2","unstructured":"Department of Homeland Security. 2022. ICS-CERT Advisories. Retrieved from https:\/\/ics-cert.us-cert.gov\/advisories."},{"key":"e_1_3_2_9_2","first-page":"8","volume-title":"IJCAI-17 Workshop on Explainable AI (XAI)","author":"Biran Or","year":"2017","unstructured":"Or Biran and Courtenay Cotton. 2017. Explanation and justification in machine learning: A survey. In IJCAI-17 Workshop on Explainable AI (XAI), Vol. 8. 8\u201313."},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1109\/WI-IAT.2011.17"},{"volume-title":"15th International Symposium on Software Engineering for Adaptive and Self-managing Systems (SEAMS)","author":"Li Nianyu","key":"e_1_3_2_11_2","unstructured":"Nianyu Li, Sridhar Adepu, Eunsuk Kang, and David Garlan. Explanations for human-on-the-loop: A probabilistic model checking approach. In 15th International Symposium on Software Engineering for Adaptive and Self-managing Systems (SEAMS)."},{"key":"e_1_3_2_12_2","first-page":"73","volume-title":"Probabilistic Model Checking: Advances and Applications","author":"Kwiatkowska Marta","year":"2018","unstructured":"Marta Kwiatkowska, Gethin Norman, and David Parker. 2018. Probabilistic Model Checking: Advances and Applications. Springer International Publishing, Cham, 73\u2013121."},{"key":"e_1_3_2_13_2","unstructured":"Singapore University of Technology and Design. 2022. Secure Water Treatment (SWaT). Retrieved from https:\/\/itrust.sutd.edu.sg\/itrust-labs-home\/itrust-labs_swat\/."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.artint.2018.07.007"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1109\/64.21896"},{"key":"e_1_3_2_16_2","unstructured":"T. R. Fennel and James D. Johannes. 1990. An architecture for rule based system explanation. In Fifth Conference on Artificial Intelligence for Space Applications vol. 3073. NASA 113."},{"key":"e_1_3_2_17_2","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1007\/978-1-4757-5945-7_2","volume-title":"Natural Language Generation in Artificial Intelligence and Computational Linguistics","author":"Paris C\u00e9cile L.","year":"1991","unstructured":"C\u00e9cile L. Paris. 1991. Generation and explanation: Building an explanation facility for the explainable expert systems framework. In Natural Language Generation in Artificial Intelligence and Computational Linguistics. Springer, 49\u201382."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1145\/2909824.3020233"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1109\/SMC.2017.8123160"},{"key":"e_1_3_2_20_2","volume-title":"Workshops at the 32nd AAAI Conference on Artificial Intelligence","author":"Molineaux Matthew","year":"2018","unstructured":"Matthew Molineaux, Dustin Dannenhauer, and David W. Aha. 2018. Towards explainable NPCs: A relational exploration learning agent. In Workshops at the 32nd AAAI Conference on Artificial Intelligence."},{"key":"e_1_3_2_21_2","first-page":"168","volume-title":"International Workshop on Languages, Methodologies and Development Tools for Multi-Agent Systems","author":"Harbers Maaike","year":"2009","unstructured":"Maaike Harbers, Karel Van Den Bosch, and John-Jules Meyer. 2009. A methodology for developing self-explaining agents for virtual training. In International Workshop on Languages, Methodologies and Development Tools for Multi-Agent Systems. Springer, 168\u2013182."},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACIIW.2017.8272595"},{"key":"e_1_3_2_23_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1109\/VLHCC.2013.6645235","volume-title":"IEEE Symposium on Visual Languages and Human Centric Computing","author":"Kulesza Todd","year":"2013","unstructured":"Todd Kulesza, Simone Stumpf, Margaret Burnett, Sherry Yang, Irwin Kwan, and Weng-Keen Wong. 2013. Too much, too little, or just right? Ways explanations impact end users\u2019 mental models. In IEEE Symposium on Visual Languages and Human Centric Computing. IEEE, 3\u201310."},{"key":"e_1_3_2_24_2","volume-title":"Data Protection: A Practical Guide to UK and EU Law.","author":"Carey. Peter","year":"2018","unstructured":"Peter Carey.2018. Data Protection: A Practical Guide to UK and EU Law.Oxford University Press, Inc."},{"key":"e_1_3_2_25_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-91122-9_18"},{"key":"e_1_3_2_26_2","article-title":"Improving Transparency and Understandability of Multi-objective Probabilistic Planning","author":"Sukkerd Roykrong","year":"2018","unstructured":"Roykrong Sukkerd. 2018. Improving Transparency and Understandability of Multi-objective Probabilistic Planning. Thesis Proposal. School of Computer Science Institute for Software Research Software Engineering, Carnegie Mellon University.","journal-title":"Thesis Proposal. School of Computer Science Institute for Software Research Software Engineering, Carnegie Mellon University"},{"key":"e_1_3_2_27_2","doi-asserted-by":"publisher","DOI":"10.1515\/pjbr-2018-0009"},{"key":"e_1_3_2_28_2","unstructured":"Cindy L. Bethel. 2009. Robots without faces: Non-verbal social human-robot Interaction. Doctoral dissertation dissertation\/Ph. D.\u2019s thesis. University of South Florida."},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-16178-0_5"},{"key":"e_1_3_2_30_2","doi-asserted-by":"publisher","DOI":"10.1145\/1518701.1519023"},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.24963\/ijcai.2017\/23"},{"key":"e_1_3_2_32_2","first-page":"97","volume-title":"European Workshop on Probabilistic Graphical Models (PGM)","author":"Elizalde Francisco","year":"2008","unstructured":"Francisco Elizalde, L. Enrique Sucar, Manuel Luque, J. Diez, and Alberto Reyes. 2008. Policy explanation in factored Markov decision processes. In European Workshop on Probabilistic Graphical Models (PGM). 97\u2013104."},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1609\/icaps.v19i1.13365"},{"key":"e_1_3_2_34_2","first-page":"19","volume-title":"4th International Workshop on Software Engineering for Smart Cyber-physical Systems","author":"Sukkerd Roykrong","year":"2018","unstructured":"Roykrong Sukkerd, Reid G. Simmons, and David Garlan. 2018. Towards explainable multi-objective probabilistic planning. In 4th International Workshop on Software Engineering for Smart Cyber-physical Systems. 19\u201325."},{"key":"e_1_3_2_35_2","first-page":"355","volume-title":"6th ACM Symposium on Information, Computer and Communications Security","author":"C\u00e1rdenas Alvaro A.","year":"2011","unstructured":"Alvaro A. C\u00e1rdenas, Saurabh Amin, Zong-Syun Lin, Yu-Lun Huang, Chi-Yen Huang, and Shankar Sastry. 2011. Attacks against process control systems: Risk assessment, detection, and response. In 6th ACM Symposium on Information, Computer and Communications Security. 355\u2013366."},{"key":"e_1_3_2_36_2","article-title":"Assessing the effectiveness of attack detection at a hackfest on industrial control systems","author":"Adepu Sridhar","year":"2018","unstructured":"Sridhar Adepu and Aditya Mathur. 2018. Assessing the effectiveness of attack detection at a hackfest on industrial control systems. IEEE Trans. Sustain. Comput. 6, 2 (2018), 231\u2013244.","journal-title":"IEEE Trans. Sustain. Comput."},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1145\/1952982.1952995"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1145\/2566468.2566484"},{"key":"e_1_3_2_39_2","first-page":"37","volume-title":"Computer Security","author":"Adepu Sridhar","year":"2018","unstructured":"Sridhar Adepu, Nandha Kumar Kandasamy, and Aditya Mathur. 2018. EPIC: An electric power testbed for research and training in cyber physical systems security. In Computer Security. Springer, 37\u201352."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2018.12.006"},{"key":"e_1_3_2_41_2","first-page":"41","volume-title":"IEEE 18th International Symposium on High Assurance Systems Engineering (HASE)","author":"Sabaliauskaite Giedre","year":"2017","unstructured":"Giedre Sabaliauskaite and Sridhar Adepu. 2017. Integrating six-step model with information flow diagrams for comprehensive analysis of cyber-physical system safety and security. In IEEE 18th International Symposium on High Assurance Systems Engineering (HASE). IEEE, 41\u201348."},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASEW.2019.00030"},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1080\/17512549.2017.1325401"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1109\/JPROC.2017.2725642"},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICII.2019.00038"},{"key":"e_1_3_2_46_2","article-title":"Explainable software for cyber-physical systems (ES4CPS): Report from the GI Dagstuhl Seminar 19023, January 6-11 2019, Schloss Dagstuhl","author":"Greenyer Joel","year":"2019","unstructured":"Joel Greenyer, Malte Lochau, and Thomas Vogel. 2019. Explainable software for cyber-physical systems (ES4CPS): Report from the GI Dagstuhl Seminar 19023, January 6-11 2019, Schloss Dagstuhl. arXiv preprint arXiv:1904.11851.","journal-title":"arXiv preprint arXiv:1904.11851"},{"key":"e_1_3_2_47_2","unstructured":"Tesla\u2019s Trouble with Semi Trucks & Another Shakeup of the Autopilot Team \u2013 Is There a Connection? Retrieved 10 Oct. 2021 from https:\/\/cleantechnica.com\/2019\/05\/21\/teslas-trouble-with-trucks-and-another-shakeup-of-the-autopilot-team-is-there-a-connection\/."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/TII.2021.3071405"},{"key":"e_1_3_2_49_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1145\/2070821.2070822","volume-title":"International Workshop on Machine Learning Technologies in Software Engineering","author":"Comb\u00e9fis S\u00e9bastien","year":"2011","unstructured":"S\u00e9bastien Comb\u00e9fis, Dimitra Giannakopoulou, Charles Pecheur, and Michael Feary. 2011. Learning system abstractions for human operators. In International Workshop on Machine Learning Technologies in Software Engineering. 3\u201310."},{"key":"e_1_3_2_50_2","first-page":"227","volume-title":"8th International Symposium on Aviation Psychology","author":"Palmer E.","year":"1996","unstructured":"E. Palmer. 1996. Oops, it didn\u2019t arm.\u2014A case study of two automation surprises. In 8th International Symposium on Aviation Psychology. 227\u2013232."},{"key":"e_1_3_2_51_2","first-page":"1","volume-title":"10th Joint Meeting on Foundations of Software Engineering, ESEC\/FSE","author":"Moreno Gabriel A.","year":"2015","unstructured":"Gabriel A. Moreno, Javier C\u00e1mara, David Garlan, and Bradley R. Schmerl. 2015. Proactive self-adaptation under uncertainty: A probabilistic model checking approach. In 10th Joint Meeting on Foundations of Software Engineering, ESEC\/FSE. 1\u201312."},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.5555\/646833.708042"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.5555\/2032305.2032352"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1002\/9780470316887"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10703-012-0177-x"},{"key":"e_1_3_2_56_2","unstructured":"Karline Soetaert. 2018. plot3D: Tools for Plotting 3-D and 2-D Data. Retrieved from https:\/\/cran.r-project.org\/web\/packages\/plot3D\/vignettes\/plot3D.pdf."},{"key":"e_1_3_2_57_2","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1109\/CySWater.2016.7469060","volume-title":"International Workshop on Cyber-physical Systems for Smart Water Networks (CySWater)","author":"Mathur A. P.","year":"2016","unstructured":"A. P. Mathur and N. O. Tippenhauer. 2016. SWaT: A water treatment testbed for research and training on ICS security. In International Workshop on Cyber-physical Systems for Smart Water Networks (CySWater). 31\u201336."},{"key":"e_1_3_2_58_2","first-page":"22","volume-title":"IEEE\/ACM 2nd International Workshop on Software Engineering for Smart Cyber-Physical Systems (SEsCPS)","author":"Kang Eunsuk","year":"2016","unstructured":"Eunsuk Kang, Sridhar Adepu, Daniel Jackson, and Aditya P. Mathur. 2016. Model-based security analysis of a water treatment system. In IEEE\/ACM 2nd International Workshop on Software Engineering for Smart Cyber-Physical Systems (SEsCPS). IEEE, 22\u201328."},{"key":"e_1_3_2_59_2","doi-asserted-by":"crossref","first-page":"141","DOI":"10.1109\/HASE.2016.14","volume-title":"IEEE 17th International Symposium on High Assurance Systems Engineering (HASE)","author":"Adepu Sridhar","year":"2016","unstructured":"Sridhar Adepu and Aditya Mathur. 2016. An investigation into the response of a water treatment system to cyber attacks. In IEEE 17th International Symposium on High Assurance Systems Engineering (HASE). IEEE, 141\u2013148."},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.pmcj.2019.101048"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/ASE.2019.00093"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1109\/ICCPS48487.2020.00011"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cose.2020.101935"},{"key":"e_1_3_2_64_2","first-page":"131","volume-title":"Workshop on Cyber-Physical Systems Security and PrivaCy","author":"Umer Muhammad Azmi","year":"2017","unstructured":"Muhammad Azmi Umer, Aditya Mathur, Khurum Nazir Junejo, and Sridhar Adepu. 2017. Integrating design and data centric approaches to generate invariants for distributed attack detection. In Workshop on Cyber-Physical Systems Security and PrivaCy. 131\u2013136."},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2017.03.013"}],"container-title":["ACM Transactions on Autonomous and Adaptive Systems"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3557898","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3557898","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:02:40Z","timestamp":1750186960000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3557898"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,12,15]]},"references-count":64,"journal-issue":{"issue":"3-4","published-print":{"date-parts":[[2022,12,31]]}},"alternative-id":["10.1145\/3557898"],"URL":"https:\/\/doi.org\/10.1145\/3557898","relation":{},"ISSN":["1556-4665","1556-4703"],"issn-type":[{"type":"print","value":"1556-4665"},{"type":"electronic","value":"1556-4703"}],"subject":[],"published":{"date-parts":[[2022,12,15]]},"assertion":[{"value":"2021-02-11","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-06-29","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-12-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}