{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,16]],"date-time":"2026-04-16T03:38:31Z","timestamp":1776310711910,"version":"3.50.1"},"reference-count":86,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2023,3,15]],"date-time":"2023-03-15T00:00:00Z","timestamp":1678838400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["J. Hum.-Robot Interact."],"published-print":{"date-parts":[[2023,6,30]]},"abstract":"<jats:p>The term<jats:italic>multimodality<\/jats:italic>has come to take on several somewhat different meanings depending on the underlying theoretical paradigms and traditions along with the purpose and context of use. The term is closely related to<jats:italic>embodiment<\/jats:italic>, which, in turn, is also used in several different ways. In this article, we elaborate on this connection and propose that a pragmatic and pluralistic stance is appropriate for multimodality. We further propose a distinction between first- and second-order effects of multimodality\u2014what is achieved by multiple modalities in isolation and the opportunities that emerge when several modalities are entangled. This highlights questions regarding ways to cluster or interchange different modalities, for example, through redundancy or degeneracy. Apart from discussing multimodality with respect to an individual agent, we further look to more distributed agents and situations in which social aspects become relevant.<\/jats:p><jats:p>In robotics, understanding the various uses and interpretations of these terms can prevent miscommunication when designing robots as well as increase awareness of the underlying theoretical concepts. Given the complexity of the different ways in which multimodality is relevant in social robotics, this can provide the basis for negotiating appropriate meanings of the term on a case-by-case basis.<\/jats:p><jats:p\/>","DOI":"10.1145\/3549955","type":"journal-article","created":{"date-parts":[[2022,7,20]],"date-time":"2022-07-20T20:49:56Z","timestamp":1658350196000},"page":"1-13","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":4,"title":["Multiple Roles of Multimodality Among Interacting Agents"],"prefix":"10.1145","volume":"12","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-8937-8063","authenticated-orcid":false,"given":"Erik","family":"Lagerstedt","sequence":"first","affiliation":[{"name":"University of Sk\u00f6vde, V\u00e4stra G\u00f6taland, Sweden"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1177-4119","authenticated-orcid":false,"given":"Serge","family":"Thill","sequence":"additional","affiliation":[{"name":"Donders Institute for Brain, Cognition, and Behaviour, Radboud University, Gelderland, The Netherlands"}]}],"member":"320","published-online":{"date-parts":[[2023,3,15]]},"reference":[{"key":"e_1_3_2_2_2","first-page":"117","volume-title":"The Oxford Handbook of 4E Cognition","author":"Aizawa Ken","year":"2018","unstructured":"Ken Aizawa. 2018. Critical note: So, what again is 4E cognition? In The Oxford Handbook of 4E Cognition, Albert Newen, Leon De Brun, and Shaun Gallager (Eds.). Oxford University Press Oxford, Oxford, UK, Chapter 6, 117\u20131126."},{"key":"e_1_3_2_3_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.jss.2017.08.042"},{"key":"e_1_3_2_4_2","volume-title":"Beyond the brain: How Body and Environment Shape Animal and Human Minds","author":"Barrett Louise","year":"2011","unstructured":"Louise Barrett. 2011. Beyond the brain: How Body and Environment Shape Animal and Human Minds. Princeton University Press, Princeton and Oxford."},{"key":"e_1_3_2_5_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0106172"},{"key":"e_1_3_2_6_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cognition.2011.05.011"},{"key":"e_1_3_2_7_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2007.11.057"},{"key":"e_1_3_2_8_2","doi-asserted-by":"publisher","DOI":"10.2307\/2185537"},{"key":"e_1_3_2_9_2","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCC.2004.826268"},{"key":"e_1_3_2_10_2","doi-asserted-by":"publisher","DOI":"10.1016\/S0921-8890(05)80025-9"},{"key":"e_1_3_2_11_2","doi-asserted-by":"publisher","DOI":"10.1162\/089892904322755601"},{"key":"e_1_3_2_12_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1005766107"},{"key":"e_1_3_2_13_2","volume-title":"Radical Embodied Cognitive Science","author":"Chemero Anthony","year":"2011","unstructured":"Anthony Chemero. 2011. Radical Embodied Cognitive Science. MIT Press, Cambridge, Massachusetts."},{"key":"e_1_3_2_14_2","doi-asserted-by":"publisher","DOI":"10.1098\/rstb.2007.2054"},{"key":"e_1_3_2_15_2","volume-title":"Mindware: An Introduction to the Philosophy of Cognitive Science.","author":"Clark Andy","year":"2000","unstructured":"Andy Clark. 2000. Mindware: An Introduction to the Philosophy of Cognitive Science.Oxford University Press, New York."},{"key":"e_1_3_2_16_2","first-page":"453","volume-title":"The Oxford Handbook of 4E Cognition","author":"Jaegher Hanne De","year":"2018","unstructured":"Hanne De Jaegher. 2018. The intersubjective turn. In The Oxford Handbook of 4E Cognition, Albert Newen, Leon De Brun, and Shaun Gallager (Eds.). Oxford University Press, Oxford, 453\u2013467."},{"key":"e_1_3_2_17_2","volume-title":"The Intentional Stance","author":"Dennett Daniel Clement","year":"1987","unstructured":"Daniel Clement Dennett. 1987. The Intentional Stance. MIT Press, Cambridge, Massachusetts."},{"key":"e_1_3_2_18_2","doi-asserted-by":"publisher","DOI":"10.4324\/9781315134543"},{"key":"e_1_3_2_19_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00230027"},{"key":"e_1_3_2_20_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.231499798"},{"key":"e_1_3_2_21_2","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2017.2776209"},{"key":"e_1_3_2_22_2","doi-asserted-by":"publisher","DOI":"10.1037\/0033-295X.114.4.864"},{"key":"e_1_3_2_23_2","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2020.00480"},{"key":"e_1_3_2_24_2","doi-asserted-by":"crossref","first-page":"133","DOI":"10.1515\/9781438466224-007","volume-title":"Body\/Self\/Other: The Phenomenology of Scial Encounters","author":"Gallagher Shaun","year":"2017","unstructured":"Shaun Gallagher. 2017. Social interaction, autonomy and recognition. In Body\/Self\/Other: The Phenomenology of Scial Encounters, Luna Dolezal and Danielle Petherbridge (Eds.). Routledge, London, Chapter 5, 133\u2013160."},{"key":"e_1_3_2_25_2","first-page":"417","volume-title":"The Oxford Handbook of 4E Cognition","author":"Gallese Vittorio","year":"2018","unstructured":"Vittorio Gallese and Corrado Sinigaglia. 2018. Embodied resonance. In The Oxford Handbook of 4E Cognition, Albert Newen, Leon De Brun, and Shaun Gallager (Eds.). Oxford University Press, Oxford, 417\u2013432."},{"key":"e_1_3_2_26_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuroimage.2007.02.003"},{"key":"e_1_3_2_27_2","volume-title":"The Perception of the Visual World","author":"Gibson James J.","year":"1950","unstructured":"James J. Gibson. 1950. The Perception of the Visual World. Houghton Mifflin, Boston."},{"key":"e_1_3_2_28_2","volume-title":"The Senses Considered as Perceptual Systems","author":"Gibson James J.","year":"1966","unstructured":"James J. Gibson. 1966. The Senses Considered as Perceptual Systems. Houghton Mifflin, London."},{"key":"e_1_3_2_29_2","doi-asserted-by":"publisher","DOI":"10.1075\/is.8.3.10gro"},{"key":"e_1_3_2_30_2","volume-title":"The Organization of Behaviour","author":"Hebb Donald O.","year":"1949","unstructured":"Donald O. Hebb. 1949. The Organization of Behaviour. Wiley & Sons, New York."},{"key":"e_1_3_2_31_2","doi-asserted-by":"publisher","DOI":"10.2307\/1416950"},{"key":"e_1_3_2_32_2","doi-asserted-by":"publisher","DOI":"10.1016\/S1364-6613(02)01913-7"},{"key":"e_1_3_2_33_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0140525X07003123"},{"key":"e_1_3_2_34_2","doi-asserted-by":"crossref","DOI":"10.7551\/mitpress\/1881.001.0001","volume-title":"Cognition in the Wild","author":"Hutchins Edwin","year":"1995","unstructured":"Edwin Hutchins. 1995. Cognition in the Wild. MIT Press, Cambridge, Massachusetts."},{"key":"e_1_3_2_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10339-007-0170-2"},{"key":"e_1_3_2_36_2","doi-asserted-by":"publisher","DOI":"10.1080\/21693277.2019.1645628"},{"key":"e_1_3_2_37_2","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2015.2480801"},{"key":"e_1_3_2_38_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2010.5598620"},{"key":"e_1_3_2_39_2","doi-asserted-by":"publisher","DOI":"10.7208\/chicago\/9780226458144.001.0001"},{"key":"e_1_3_2_40_2","doi-asserted-by":"publisher","DOI":"10.1145\/3125739.3125768"},{"key":"e_1_3_2_41_2","doi-asserted-by":"publisher","DOI":"10.1518\/hfes.46.1.50.30392"},{"key":"e_1_3_2_42_2","doi-asserted-by":"publisher","DOI":"10.1109\/ROMAN.2011.6005252"},{"key":"e_1_3_2_43_2","first-page":"136","article-title":"Law as a social system","volume":"83","author":"Luhmann Niklas","year":"1988","unstructured":"Niklas Luhmann. 1988. Law as a social system. Northwestern University Law Reviews 83 (1988), 136\u2013150.","journal-title":"Northwestern University Law Reviews"},{"key":"e_1_3_2_44_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-94-009-8947-4"},{"key":"e_1_3_2_45_2","volume-title":"Ascribing Mental Qualities to Machines","author":"McCarthy John","year":"1979","unstructured":"John McCarthy. 1979. Ascribing Mental Qualities to Machines. Technical Report. Department of Computer Science, Stanford University, Stanford, California."},{"key":"e_1_3_2_46_2","doi-asserted-by":"publisher","DOI":"10.7208\/chicago\/9780226532653.001.0001"},{"key":"e_1_3_2_47_2","doi-asserted-by":"publisher","DOI":"10.1007\/978-0-585-29605-0_19"},{"key":"e_1_3_2_48_2","doi-asserted-by":"publisher","DOI":"10.1109\/HRI53351.2022.9889620"},{"key":"e_1_3_2_49_2","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1093\/oxfordhb\/9780198735410.001.0001","volume-title":"The Oxford handbook of 4E cognition","author":"Newen Albert","year":"2018","unstructured":"Albert Newen, Leon De Bruin, and Shaun Gallagher. 2018. 4E cognition: Historical roots, key concepts, and central issues. In The Oxford handbook of 4E cognition, (Eds.). Oxford University Press, Oxford, UK, Chapter 1, 3\u201315."},{"key":"e_1_3_2_50_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0140525X01000115"},{"key":"e_1_3_2_51_2","doi-asserted-by":"publisher","DOI":"10.1080\/17470919.2016.1207702"},{"key":"e_1_3_2_52_2","doi-asserted-by":"publisher","DOI":"10.1515\/pjbr-2021-0002"},{"key":"e_1_3_2_53_2","doi-asserted-by":"publisher","DOI":"10.1037\/0735-7036.116.2.116"},{"key":"e_1_3_2_54_2","doi-asserted-by":"publisher","DOI":"10.3758\/s13415-013-0227-x"},{"key":"e_1_3_2_55_2","doi-asserted-by":"publisher","DOI":"10.5555\/1208449"},{"key":"e_1_3_2_56_2","volume-title":"Proceedings of \u20194. Fachtagung der Gesellschaft f\u00fcr Kognitionswissenschaft\u2019","author":"Quick Tom","year":"1999","unstructured":"Tom Quick and Kerstin Dautenhahn. 1999. Making embodiment measurable. In Proceedings of \u20194. Fachtagung der Gesellschaft f\u00fcr Kognitionswissenschaft\u2019. http:\/\/supergoodtech.com\/tomquick\/phd\/kogwis\/webtext.html. Infix-Verl., Bielefeld, Germany."},{"key":"e_1_3_2_57_2","volume-title":"The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places","author":"Reeves Byron","year":"1996","unstructured":"Byron Reeves and Clifford Ivar Nass. 1996. The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places. Cambridge University Press, Cambridge."},{"key":"e_1_3_2_58_2","doi-asserted-by":"publisher","DOI":"10.1145\/2157689.2157814"},{"key":"e_1_3_2_59_2","first-page":"86","volume-title":"Proceedings of the 18th International Conference on Autonomous Agents and Multiagent Systems","author":"Ritschel Hannes","year":"2019","unstructured":"Hannes Ritschel, Ilhan Aslan, David Sedlbauer, and Elisabeth Andr\u00e9. 2019. Irony man: Augmenting a social robot with the ability to use irony in multimodal communication with humans. In Proceedings of the 18th International Conference on Autonomous Agents and Multiagent Systems. ACM, Montreal, Canada, 86\u201394."},{"key":"e_1_3_2_60_2","doi-asserted-by":"publisher","DOI":"10.1007\/BF00248742"},{"key":"e_1_3_2_61_2","doi-asserted-by":"publisher","DOI":"10.1109\/THMS.2017.2648849"},{"key":"e_1_3_2_62_2","doi-asserted-by":"publisher","DOI":"10.1007\/s12369-012-0143-1"},{"key":"e_1_3_2_63_2","doi-asserted-by":"publisher","DOI":"10.1075\/is.14.3.02sci"},{"key":"e_1_3_2_64_2","doi-asserted-by":"publisher","DOI":"10.3389\/fnbot.2019.00053"},{"key":"e_1_3_2_65_2","doi-asserted-by":"publisher","DOI":"10.1109\/WCSP.2011.6096958"},{"key":"e_1_3_2_66_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cogsys.2012.05.001"},{"key":"e_1_3_2_67_2","doi-asserted-by":"publisher","DOI":"10.1080\/00140139.2013.858777"},{"key":"e_1_3_2_68_2","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/9780262014601.001.0001"},{"key":"e_1_3_2_69_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cogsys.2016.06.003"},{"key":"e_1_3_2_70_2","doi-asserted-by":"publisher","DOI":"10.3389\/fpsyg.2017.01962"},{"key":"e_1_3_2_71_2","doi-asserted-by":"publisher","DOI":"10.1145\/3461781"},{"key":"e_1_3_2_72_2","doi-asserted-by":"crossref","first-page":"43","DOI":"10.1007\/978-3-319-97550-4_4","volume-title":"Cognitive Architectures","author":"Thill Serge","year":"2019","unstructured":"Serge Thill. 2019. What we need from an embodied cognitive architecture. In Cognitive Architectures, Aldinhas Ferreira, Maria Isabel, Jo\u00e3o Silva Sequeira, and Rodrigo Ventura (Eds.). Springer, Berlin, 43\u201357."},{"key":"e_1_3_2_73_2","doi-asserted-by":"publisher","DOI":"10.1109\/CogSIMA.2014.6816540"},{"key":"e_1_3_2_74_2","first-page":"25","volume-title":"Cognition: A Bridge Between Robotics and Interaction","author":"Thill Serge","year":"2015","unstructured":"Serge Thill and Tom Ziemke. 2015. Interaction as a bridge between cognition and robotics. In Cognition: A Bridge Between Robotics and Interaction. IEEE, Oregon, Portland, USA, 25\u201331."},{"key":"e_1_3_2_75_2","doi-asserted-by":"publisher","DOI":"10.1017\/S0140525X05000129"},{"key":"e_1_3_2_76_2","doi-asserted-by":"publisher","DOI":"10.1109\/ACII52823.2021.9597423"},{"key":"e_1_3_2_77_2","doi-asserted-by":"publisher","DOI":"10.1207\/s15326969eco0302_2"},{"key":"e_1_3_2_78_2","doi-asserted-by":"publisher","DOI":"10.2478\/s13230-010-0016-y"},{"key":"e_1_3_2_79_2","doi-asserted-by":"publisher","DOI":"10.5555\/2717060"},{"key":"e_1_3_2_80_2","doi-asserted-by":"publisher","DOI":"10.1109\/DevLrn.2013.6652521"},{"key":"e_1_3_2_81_2","first-page":"5","volume-title":"Instinctive Behavior \u2014 The Development of a Modern Concept","author":"Uexk\u00fcll Jakob von","year":"1957","unstructured":"Jakob von Uexk\u00fcll. 1957. A stroll through the worlds of animals and men: A picture book of invisible worlds. In Instinctive Behavior \u2014 The Development of a Modern Concept, Claire H. Schiller (Ed.). International University Press, Inc, New York, 5\u201380. Original work published 1934."},{"key":"e_1_3_2_82_2","doi-asserted-by":"publisher","DOI":"10.1145\/3434073.3444644"},{"key":"e_1_3_2_83_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.biosystems.2018.07.007"},{"key":"e_1_3_2_84_2","first-page":"701","volume-title":"First International Workshop on Epigenetic Robotics Modeling Cognitive Development in Robotic Systems","author":"Ziemke Tom","year":"2001","unstructured":"Tom Ziemke. 2001. Are robots embodied? In First International Workshop on Epigenetic Robotics Modeling Cognitive Development in Robotic Systems, Vol. 85. Citeseer, Lund, Sweden, 701\u2013746."},{"key":"e_1_3_2_85_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.biosystems.2007.05.015"},{"key":"e_1_3_2_86_2","doi-asserted-by":"publisher","DOI":"10.1515\/semi.2001.050"},{"key":"e_1_3_2_87_2","first-page":"1","volume-title":"Cognition: A Bridge Between Robotics and Interaction. Workshop at HRI\u201915","author":"Ziemke Tom","year":"2015","unstructured":"Tom Ziemke, Serge Thill, and David Vernon. 2015. Embodiment is a double-edged sword in human-robot interaction: Ascribed vs. intrinsic intentionality. In Cognition: A Bridge Between Robotics and Interaction. Workshop at HRI\u201915. ACM, New York, 1\u20132."}],"container-title":["ACM Transactions on Human-Robot Interaction"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3549955","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3549955","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T18:08:14Z","timestamp":1750183694000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3549955"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,3,15]]},"references-count":86,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2023,6,30]]}},"alternative-id":["10.1145\/3549955"],"URL":"https:\/\/doi.org\/10.1145\/3549955","relation":{},"ISSN":["2573-9522","2573-9522"],"issn-type":[{"value":"2573-9522","type":"print"},{"value":"2573-9522","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,3,15]]},"assertion":[{"value":"2021-03-08","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2022-06-30","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2023-03-15","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}