{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T11:20:04Z","timestamp":1773141604987,"version":"3.50.1"},"reference-count":59,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2022,3,4]],"date-time":"2022-03-04T00:00:00Z","timestamp":1646352000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"DOI":"10.13039\/100000181","name":"United States Air Force Office of Scientific Research","doi-asserted-by":"crossref","award":["FA9550-20-1-0089"],"award-info":[{"award-number":["FA9550-20-1-0089"]}],"id":[{"id":"10.13039\/100000181","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2022,6,30]]},"abstract":"<jats:p>The design of current natural language-oriented robot architectures enables certain architectural components to circumvent moral reasoning capabilities. One example of this is reflexive generation of clarification requests as soon as referential ambiguity is detected in a human utterance. As shown in previous research, this can lead robots to (1) miscommunicate their moral dispositions and (2) weaken human perception or application of moral norms within their current context. We present a solution to these problems by performing moral reasoning on each potential disambiguation of an ambiguous human utterance and responding accordingly, rather than immediately and naively requesting clarification. We implement our solution in the Distributed Integrated Cognition Affect and Reflection robot architecture, which, to our knowledge, is the only current robot architecture with both moral reasoning and clarification request generation capabilities. We then evaluate our method with a human subjects experiment, the results of which indicate that our approach successfully ameliorates the two identified concerns.<\/jats:p>","DOI":"10.1145\/3503795","type":"journal-article","created":{"date-parts":[[2022,3,4]],"date-time":"2022-03-04T10:14:11Z","timestamp":1646388851000},"page":"1-18","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":13,"title":["Enabling Morally Sensitive Robotic Clarification Requests"],"prefix":"10.1145","volume":"11","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-6335-1743","authenticated-orcid":false,"given":"Ryan Blake","family":"Jackson","sequence":"first","affiliation":[{"name":"Colorado School of Mines, Golden, Colorado, USA"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7921-771X","authenticated-orcid":false,"given":"Tom","family":"Williams","sequence":"additional","affiliation":[{"name":"Colorado School of Mines, Golden, Colorado, USA"}]}],"member":"320","published-online":{"date-parts":[[2022,3,4]]},"reference":[{"key":"e_1_3_2_2_2","doi-asserted-by":"publisher","DOI":"10.1145\/1349822.1349839"},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1075\/slcs.80.03bac"},{"key":"e_1_3_2_4_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-008-0001-3"},{"key":"e_1_3_2_5_2","article-title":"Polite interactions with robots","author":"Benotti Luciana","year":"2016","unstructured":"Luciana Benotti and Patrick Blackburn. 2016. Polite interactions with robots. What Social Robots Can and Should Do: Proceedings of Robophilosophy 2016\/TRANSOR 2016 290 (2016), 293.","journal-title":"What Social Robots Can and Should Do: Proceedings of Robophilosophy 2016\/TRANSOR"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1145\/1228716.1228752"},{"key":"e_1_3_2_7_2","volume-title":"Proceedings of the Symposium on Linguistic and Cognitive Approaches to Dialogue Agents","author":"Briggs Gordon","year":"2012","unstructured":"Gordon Briggs and Matthias Scheutz. 2012. Multi-modal belief updates in multi-robot human-robot dialogue interaction. In Proceedings of the Symposium on Linguistic and Cognitive Approaches to Dialogue Agents."},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-014-0235-1"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.5898\/JHRI.6.1.Briggs"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1017\/CBO9780511813085"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1177\/1745691610393980"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0057410"},{"key":"e_1_3_2_13_2","article-title":"Sharing a life with harvey: Exploring the acceptance of and relationship-building with a social robot","author":"Graaf Maartje Ma De","year":"2015","unstructured":"Maartje Ma De Graaf, Somaya Ben Allouch, and Tineke Klamer. 2015. Sharing a life with harvey: Exploring the acceptance of and relationship-building with a social robot. Comput. Hum. Behav. (2015).","journal-title":"Comput. Hum. Behav."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROBOT.2009.5152776"},{"key":"e_1_3_2_15_2","doi-asserted-by":"publisher","DOI":"10.1111\/j.2044-8309.2011.02082.x"},{"key":"e_1_3_2_16_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/N18-1177"},{"key":"e_1_3_2_17_2","volume-title":"Proceedings of the Annual Meeting of the Cognitive Science Society","author":"Gervits Felix","year":"2017","unstructured":"Felix Gervits, Gordon Briggs, and Matthias Scheutz. 2017. The pragmatic parliament: A framework for socially-appropriate utterance selection in artificial agents. In Proceedings of the Annual Meeting of the Cognitive Science Society."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cobeha.2015.03.001"},{"key":"e_1_3_2_19_2","volume-title":"Syntax and Semantics","author":"Grice Paul","year":"1975","unstructured":"Paul Grice. 1975. Logic and conversation. In Syntax and Semantics."},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.3758\/s13428-015-0642-8"},{"key":"e_1_3_2_21_2","volume-title":"Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS\u201921)","author":"Jackson Ryan Blake","year":"2021","unstructured":"Ryan Blake Jackson, Sihui Li, Santosh Balajee Banisetty, Sriram Siva, Hao Zhang, Neil Dantam, and Tom Williams. 2021. An integrated approach to context-sensitive moral cognition in robot cognitive architectures. In Proceedings of the IEEE\/RSJ International Conference on Intelligent Robots and Systems (IROS\u201921)."},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1145\/3306618.3314241"},{"key":"e_1_3_2_23_2","volume-title":"Proceedings of the International Conference on Robot Ethics and Standards (ICRES\u201918)","author":"Jackson Ryan Blake","year":"2018","unstructured":"Ryan Blake Jackson and Tom Williams. 2018. Robot: Asker of questions and changer of norms? In Proceedings of the International Conference on Robot Ethics and Standards (ICRES\u201918)."},{"key":"e_1_3_2_24_2","doi-asserted-by":"publisher","DOI":"10.1109\/HRI.2019.8673123"},{"key":"e_1_3_2_25_2","volume-title":"Proceedings of the HRI Workshop on The Dark Side of Human-Robot Interaction","author":"Jackson Ryan Blake","year":"2019","unstructured":"Ryan Blake Jackson and Tom Williams. 2019. On perceived social and moral agency in natural language capable robots. In Proceedings of the HRI Workshop on The Dark Side of Human-Robot Interaction."},{"key":"e_1_3_2_26_2","article-title":"What are the odds? a practical guide to computing and reporting bayes factors","volume":"7","author":"Jarosz Andrew F.","year":"2014","unstructured":"Andrew F. Jarosz and Jennifer Wiley. 2014. What are the odds? a practical guide to computing and reporting bayes factors. J. Probl. Solv. 7 (2014).","journal-title":"J. Probl. Solv."},{"key":"e_1_3_2_27_2","article-title":"Jasp","author":"Team JASP","year":"2016","unstructured":"JASP Team et\u00a0al. 2016. Jasp. Version 0.8. 0.0. software.","journal-title":"Version 0.8. 0.0. software"},{"key":"e_1_3_2_28_2","doi-asserted-by":"publisher","DOI":"10.1145\/2157689.2157696"},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1145\/2559636.2559820"},{"key":"e_1_3_2_30_2","volume-title":"Proceedings of the RO-MAN Workshop: Toward a Framework for Joint Action, What about Common Ground","author":"Knepper Ross A.","year":"2016","unstructured":"Ross A. Knepper. 2016. On the communicative aspect of human-robot joint action. In Proceedings of the RO-MAN Workshop: Toward a Framework for Joint Action, What about Common Ground."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.5555\/519369"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.21236\/ADA534697"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1145\/2696454.2696458"},{"key":"e_1_3_2_34_2","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/W15-4604"},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1145\/2701973.2701977"},{"key":"e_1_3_2_36_2","volume-title":"The Theory and Use of Clarification Requests in Dialogue","author":"Purver Matthew Richard John","year":"2004","unstructured":"Matthew Richard John Purver. 2004. The Theory and Use of Clarification Requests in Dialogue. Ph.D. Dissertation. University of London."},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-bioeng-071811-150036"},{"key":"e_1_3_2_38_2","volume-title":"Proceedings of the AAAI Workshop on Intelligent Robotic Systems","author":"Scheutz Matthias","year":"2013","unstructured":"Matthias Scheutz, Gordon Briggs, Rehj Cantrell, Evan Krause, Tom Williams, and Richard Veale. 2013. Novel mechanisms for natural human-robot interactions in the DIARC architecture. In Proceedings of the AAAI Workshop on Intelligent Robotic Systems."},{"key":"e_1_3_2_39_2","volume-title":"Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS\u201917)","author":"Scheutz Matthias","year":"2017","unstructured":"Matthias Scheutz, Evan Krause, Brad Oosterveld, Tyler Frasca, and Robert Platt. 2017. Spoken instruction-based one-shot object and action learning in a cognitive robotic architecture. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS\u201917)."},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2015.7333661"},{"key":"e_1_3_2_41_2","volume-title":"Cognitive Architectures","author":"Scheutz Matthias","year":"2018","unstructured":"Matthias Scheutz, Thomas Williams, Evan Krause, Bradley Oosterveld, Vasanth Sarathy, and Tyler Frasca. 2018. An overview of the distributed integrated cognition affect and reflection DIARC architecture. In Cognitive Architectures."},{"key":"e_1_3_2_42_2","first-page":"59","article-title":"Indirect speech acts","volume":"3","author":"Searle John R.","year":"1975","unstructured":"John R. Searle. 1975. Indirect speech acts. Syntax Semant. 3 (1975), 59\u201382.","journal-title":"Syntax Semant."},{"key":"e_1_3_2_43_2","doi-asserted-by":"publisher","DOI":"10.1075\/is.11.2.01sha"},{"issue":"4","key":"e_1_3_2_44_2","article-title":"Believable robot characters","author":"Simmons Reid","year":"2011","unstructured":"Reid Simmons, Maxim Makatchev, Rachel Kirby, Min Kyung Lee, et\u00a0al. 2011. Believable robot characters. AI Mag.4 (2011).","journal-title":"AI Mag."},{"key":"e_1_3_2_45_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2017.06.007"},{"key":"e_1_3_2_46_2","doi-asserted-by":"crossref","first-page":"409","DOI":"10.7551\/mitpress\/9816.003.0057","article-title":"Toward information theoretic human-robot dialog","volume":"32","author":"Tellex Stefanie","year":"2013","unstructured":"Stefanie Tellex, Pratiksha Thaker, Robin Deits, Dimitar Simeonov, Thomas Kollar, and Nicholas Roy. 2013. Toward information theoretic human-robot dialog. Robot.: Sci. Syst. 32 (2013), 409\u2013417.","journal-title":"Robot.: Sci. Syst."},{"key":"e_1_3_2_47_2","volume-title":"Proceedings of the AAAI Fall Symposium Series on Artificial Intelligence for Human-Robot Interaction","author":"Trott Sean","year":"2017","unstructured":"Sean Trott and Benjamin Bergen. 2017. A theoretical model of indirect request comprehension. In Proceedings of the AAAI Fall Symposium Series on Artificial Intelligence for Human-Robot Interaction."},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.7208\/chicago\/9780226852904.001.0001"},{"key":"e_1_3_2_49_2","doi-asserted-by":"publisher","DOI":"10.1109\/TRO.2007.906261"},{"key":"e_1_3_2_50_2","unstructured":"Willie Walker Paul Lamere Philip Kwok Bhiksha Raj Rita Singh Evandro Gouvea Peter Wolf and Joe Woelfel. 2004. Sphinx-4: A flexible open source framework for speech recognition."},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1145\/3183654.3183703"},{"key":"e_1_3_2_52_2","article-title":"A consultant framework for natural language processing in integrated robot architectures","author":"Williams Tom","year":"2017","unstructured":"Tom Williams. 2017. A consultant framework for natural language processing in integrated robot architectures. IEEE Intell. Inf. Bull. (2017).","journal-title":"IEEE Intell. Inf. Bull."},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1109\/HRI.2016.7451767"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v29i1.9377"},{"key":"e_1_3_2_55_2","volume-title":"Proceedings of the Annual Meeting of the Cognitive Science Society (COGSCI\u201918)","author":"Williams Tom","year":"2018","unstructured":"Tom Williams, Ryan Blake Jackson, and Jane Lockshin. 2018. A bayesian analysis of moral norm malleability during clarification dialogues. In Proceedings of the Annual Meeting of the Cognitive Science Society (COGSCI\u201918). Cognitive Science Society."},{"key":"e_1_3_2_56_2","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v30i1.9916"},{"key":"e_1_3_2_57_2","volume-title":"The Oxford Handbook of Reference","author":"Williams Tom","year":"2018","unstructured":"Tom Williams and Matthias Scheutz. 2018. Reference in robotics: A givenness hierarchy theoretic approach. In The Oxford Handbook of Reference, Jeanette Gundel and Barbara Abbott (Eds.)."},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/3171221.3171246"},{"key":"e_1_3_2_59_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-030-05204-1_17"},{"key":"e_1_3_2_60_2","article-title":"Dempster-Shafer theoretic resolution of referential ambiguity","author":"Williams Tom","year":"2018","unstructured":"Tom Williams, Fereshta Yazdani, Prasanth Suresh, Matthias Scheutz, and Michael Beetz. 2018. Dempster-Shafer theoretic resolution of referential ambiguity. Auton. Robots (2018).","journal-title":"Auton. Robots"}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3503795","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3503795","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T19:02:19Z","timestamp":1750186939000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3503795"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,3,4]]},"references-count":59,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2022,6,30]]}},"alternative-id":["10.1145\/3503795"],"URL":"https:\/\/doi.org\/10.1145\/3503795","relation":{},"ISSN":["2573-9522","2573-9522"],"issn-type":[{"value":"2573-9522","type":"print"},{"value":"2573-9522","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,3,4]]},"assertion":[{"value":"2021-01-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2021-11-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-03-04","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}