{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,26]],"date-time":"2026-06-26T22:09:13Z","timestamp":1782511753477,"version":"3.54.5"},"reference-count":32,"publisher":"Springer Science and Business Media LLC","issue":"4","license":[{"start":{"date-parts":[[2023,9,9]],"date-time":"2023-09-09T00:00:00Z","timestamp":1694217600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,9,9]],"date-time":"2023-09-09T00:00:00Z","timestamp":1694217600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100004359","name":"Vetenskapsr\u00e5det","doi-asserted-by":"publisher","award":["2017-03979"],"award-info":[{"award-number":["2017-03979"]}],"id":[{"id":"10.13039\/501100004359","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004785","name":"NordForsk","doi-asserted-by":"publisher","award":["86892"],"award-info":[{"award-number":["86892"]}],"id":[{"id":"10.13039\/501100004785","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["J Multimodal User Interfaces"],"published-print":{"date-parts":[[2023,12]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This paper presents the design and development of PepperOSC, an interface that connects Pepper and NAO robots with sound production tools to enable the development of interactive sonification in human-robot interaction (HRI). The interface uses Open Sound Control (OSC) messages to stream kinematic data from robots to various sound design and music production tools. The goals of PepperOSC are twofold: (i) to provide a tool for HRI researchers in developing multimodal user interfaces through sonification, and (ii) to lower the barrier for sound designers to contribute to HRI. To demonstrate the potential use of PepperOSC, this paper also presents two applications we have conducted: (i) a course project by two master\u2019s students who created a robot sound model in Pure Data, and (ii) a museum installation of Pepper robot, employing sound models developed by a sound designer and a composer\/researcher in music technology using MaxMSP and SuperCollider respectively. Furthermore, we discuss the potential use cases of PepperOSC in social robotics and artistic contexts. These applications demonstrate the versatility of PepperOSC and its ability to explore diverse aesthetic strategies for robot movement sonification, offering a promising approach to enhance the effectiveness and appeal of human-robot interactions.<\/jats:p>","DOI":"10.1007\/s12193-023-00414-9","type":"journal-article","created":{"date-parts":[[2023,9,9]],"date-time":"2023-09-09T12:01:56Z","timestamp":1694260916000},"page":"231-239","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["PepperOSC: enabling interactive sonification of a robot\u2019s expressive movement"],"prefix":"10.1007","volume":"17","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-3572-6429","authenticated-orcid":false,"given":"Adrian B.","family":"Latupeirissa","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Roberto","family":"Bresin","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,9,9]]},"reference":[{"key":"414_CR1","doi-asserted-by":"crossref","unstructured":"Alexanderson S, Henter GE, Kucherenko T, et\u00a0al (2020) Style-controllable speech-driven gesture synthesis using normalising flows. In: Computer Graphics Forum, Wiley Online Library, pp 487\u2013496","DOI":"10.1111\/cgf.13946"},{"key":"414_CR2","doi-asserted-by":"publisher","first-page":"255","DOI":"10.1016\/j.softx.2017.06.003","volume":"6","author":"S Baldan","year":"2017","unstructured":"Baldan S, Delle Monache S, Rocchesso D (2017) The sound design toolkit. SoftwareX 6:255\u2013260. https:\/\/doi.org\/10.1016\/j.softx.2017.06.003","journal-title":"SoftwareX"},{"key":"414_CR3","doi-asserted-by":"publisher","unstructured":"Bellona J, Bai L, Dahl L, et\u00a0al (2017) Empirically informed sound synthesis application for enhancing the perception of expressive robotic movement. In: Proceedings of the 23rd International Conference on Auditory Display - ICAD 2017. The International Community for Auditory Display, University Park Campus, pp 73\u201380, https:\/\/doi.org\/10.21785\/icad2017.049","DOI":"10.21785\/icad2017.049"},{"key":"414_CR4","unstructured":"B\u00f6kesoy S (2014) A Recursive Mapping System for Motion and Sound in a Robot between Human Interaction Design. In: ICMC"},{"key":"414_CR5","unstructured":"Bresin R, Frid E, Latupeirissa A, et\u00a0al (2021) Robust non-verbal expression in humanoid robots: New methods for augmenting expressive movements with sound. In: Workshop on Sound in Human-Robot Interaction, ACM\/IEEE International Conference on Human-Robot Interaction"},{"key":"414_CR6","doi-asserted-by":"crossref","unstructured":"Eriksson S, Unander-Scharin \u00c5, Trichon V, et\u00a0al (2019) Dancing with drones: Crafting novel artistic expressions through intercorporeality. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp 1\u201312","DOI":"10.1145\/3290605.3300847"},{"issue":"2","key":"414_CR7","doi-asserted-by":"publisher","first-page":"357","DOI":"10.1007\/s12369-021-00788-4","volume":"14","author":"E Frid","year":"2022","unstructured":"Frid E, Bresin R (2022) Perceptual evaluation of blended sonification of mechanical robot sounds produced by emotionally expressive gestures: augmenting consequential sounds to improve non-verbal robot communication. Int J Soc Robot 14(2):357\u2013372. https:\/\/doi.org\/10.1007\/s12369-021-00788-4","journal-title":"Int J Soc Robot"},{"key":"414_CR8","unstructured":"Frid E, Bresin R, Alexanderson S (2018) Perception of Mechanical Sounds Inherent to Expressive Gestures of a NAO Robot - Implications for Movement Sonification of Humanoids. Sound Music Computd"},{"key":"414_CR9","unstructured":"Herman T, Hunt A, Neuhoff JG (2011) Introduction. In: Hermann T, Hunt A, Neuhoff JG (eds) The sonification handbook, vol\u00a01. Logos Verlag, Berlin, p 1\u20136, https:\/\/sonification.de\/handbook"},{"key":"414_CR10","unstructured":"Hermann T (2008) Taxonomy and definitions for sonification and auditory display. In: International Community for Auditory Display http:\/\/hdl.handle.net\/1853\/49960"},{"key":"414_CR11","unstructured":"Hermann T (2011) Model-based sonification. In: Hermann T, Hunt A, Neuhoff JG (eds) The sonification handbook, vol\u00a01. Logos Verlag, Berlin, p 399\u2013427, https:\/\/sonification.de\/handbook"},{"key":"414_CR12","unstructured":"Hermann T, Niehus C, Ritter H (2003) Interactive visualization and sonification for monitoring complex processes. In: International Community for Auditory Display"},{"key":"414_CR13","unstructured":"Hoffman G (2011) On stage: robots as performers. In: RSS 2011 Workshop on Human-Robot Interaction: Perspectives and Contributions to Robotics from the Human Sciences. Los Angeles, CA, p\u00a021"},{"key":"414_CR14","unstructured":"Hunt A, Hermann T (2011) Interactive sonification. In: Hermann T, Hunt A, Neuhoff JG (eds) The sonification handbook, vol\u00a01. Logos Verlag, Berlin, p 273\u2013298, https:\/\/sonification.de\/handbook"},{"key":"414_CR15","unstructured":"Johannsen G (2002) Auditory display of directions and states for mobile systems. In: Proceedings of the International Conference on Auditory Display (ICAD), doi:1853\/51337"},{"key":"414_CR16","doi-asserted-by":"publisher","first-page":"42","DOI":"10.1007\/978-3-642-25504-5_5","volume-title":"Social robotics","author":"H Knight","year":"2011","unstructured":"Knight H (2011) Eight lessons learned about non-verbal interactions through robot theater. In: Mutlu B, Bartneck C, Ham J et al (eds) Social robotics. Springer, Berlin, pp 42\u201351"},{"key":"414_CR17","unstructured":"Kramer G, Walker B, Bonebright T, et\u00a0al (1997) Sonification report: status of the field and research Agenda. In: International Community for Auditory Display (ICAD), Santa Fe, NM"},{"key":"414_CR18","volume-title":"Effort; economy of human movement","author":"L Rv","year":"1974","unstructured":"Rv L, Lawrence FC (1974) Effort; economy of human movement, 2nd edn. Macdonald & Evans, London","edition":"2"},{"key":"414_CR19","unstructured":"Latupeirissa AB, Panariello C, Bresin R (2020) Exploring emotion perception in sonic HRI. In: Sound and Music Computing Conference, Torino, 24-26 June 2020, Zenodo, pp 434\u2013441"},{"key":"414_CR20","doi-asserted-by":"publisher","DOI":"10.1145\/3585277","author":"AB Latupeirissa","year":"2023","unstructured":"Latupeirissa AB, Panariello C, Bresin R (2023) probing aesthetics strategies for robot sound: complexity and materiality in movement sonification. ACM Trans Hum-Robot Interact. https:\/\/doi.org\/10.1145\/3585277","journal-title":"ACM Trans Hum-Robot Interact"},{"issue":"2","key":"414_CR21","doi-asserted-by":"publisher","first-page":"291","DOI":"10.1007\/s12369-013-0178-y","volume":"5","author":"I Leite","year":"2013","unstructured":"Leite I, Martinho C, Paiva A (2013) Social robots for long-term interaction: a survey. Int J Soc Robot 5(2):291\u2013308. https:\/\/doi.org\/10.1007\/s12369-013-0178-y","journal-title":"Int J Soc Robot"},{"key":"414_CR22","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1162\/lmj_a_01092","volume":"30","author":"C Panariello","year":"2020","unstructured":"Panariello C (2020) Study in three phases\u202f: an adaptive sound installation. Leonardo Music J 30:44\u201349. https:\/\/doi.org\/10.1162\/lmj_a_01092","journal-title":"Leonardo Music J"},{"issue":"1","key":"414_CR23","doi-asserted-by":"publisher","first-page":"119","DOI":"10.5898\/JHRI.1.1.Riek","volume":"1","author":"LD Riek","year":"2012","unstructured":"Riek LD (2012) Wizard of oz studies in HRI: a systematic review and new reporting guidelines. J Hum-Robot Interact 1(1):119\u2013136. https:\/\/doi.org\/10.5898\/JHRI.1.1.Riek","journal-title":"J Hum-Robot Interact"},{"issue":"6","key":"414_CR24","doi-asserted-by":"publisher","first-page":"1507","DOI":"10.1007\/s12369-022-00891-0","volume":"14","author":"FA Robinson","year":"2022","unstructured":"Robinson FA, Bown O, Velonaki M (2022) Designing sound for social robots: candidate design principles. Int J Soc Robot 14(6):1507\u20131525. https:\/\/doi.org\/10.1007\/s12369-022-00891-0","journal-title":"Int J Soc Robot"},{"key":"414_CR25","doi-asserted-by":"publisher","unstructured":"Schwenk M, Arras KO (2014) R2-D2 Reloaded: a flexible sound synthesis system for sonic human-robot interaction design. In: The 23rd IEEE International Symposium on Robot and Human Interactive Communication. IEEE, Edinburgh, UK, pp 161\u2013167, https:\/\/doi.org\/10.1109\/ROMAN.2014.6926247","DOI":"10.1109\/ROMAN.2014.6926247"},{"key":"414_CR26","unstructured":"Snyder J, Johns R, Avis C, et\u00a0al (2015) Machine yearning: an industrial robotic arm as a performance instrument. In: Proceedings of the International Conference on New Interfaces for Musical Expression. The School of Music and the Center for Computation and Technology (CCT), Louisiana State University, Baton Rouge, Louisiana, USA, p 184\u2013186"},{"key":"414_CR27","unstructured":"Walker BN, Nees MA (2011) Theory of sonification. In: Hermann T, Hunt A, Neuhoff JG (eds) The sonification handbook, vol\u00a01. Logos Verlag, Berlin, p 9\u201339, https:\/\/sonification.de\/handbook"},{"key":"414_CR28","unstructured":"Wright M, Freed A (1997) Open SoundControl: a new protocol for communicating with sound synthesizers. In: International Computer Music Conference (ICMC) pp 101\u2013104. http:\/\/hdl.handle.net\/2027\/spo.bbp2372.1997.033"},{"key":"414_CR29","doi-asserted-by":"publisher","unstructured":"Zahray L, Savery R, Syrkett L, et\u00a0al (2020) Robot gesture sonification to enhance awareness of robot status and enjoyment of interaction. In: 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN). IEEE, Naples, Italy, pp 978\u2013985, https:\/\/doi.org\/10.1109\/RO-MAN47096.2020.9223452","DOI":"10.1109\/RO-MAN47096.2020.9223452"},{"key":"414_CR30","doi-asserted-by":"publisher","unstructured":"Zhang BJ, Fitter NT (2023) Nonverbal sound in human-robot interaction: a systematic review. ACM Trans Hum-Robot Interact. https:\/\/doi.org\/10.1145\/3583743","DOI":"10.1145\/3583743"},{"issue":"4","key":"414_CR31","doi-asserted-by":"publisher","first-page":"10566","DOI":"10.1109\/LRA.2022.3193228","volume":"7","author":"BJ Zhang","year":"2022","unstructured":"Zhang BJ, Sigafoos N, Moffit R et al (2022) SonifyIt: towards transformative sound for all robots. IEEE Robot Autom Lett 7(4):10566\u201310572. https:\/\/doi.org\/10.1109\/LRA.2022.3193228","journal-title":"IEEE Robot Autom Lett"},{"key":"414_CR32","doi-asserted-by":"publisher","unstructured":"Zhang R, Barnes J, Ryan J, et\u00a0al (2016) Musical robots for children with ASD using a client-server architecture. In: Proceedings of the 22nd International Conference on Auditory Display - ICAD 2016. The International Community for Auditory Display, Canberra, Australia, pp 83\u201389, https:\/\/doi.org\/10.21785\/icad2016.007","DOI":"10.21785\/icad2016.007"}],"updated-by":[{"DOI":"10.1007\/s12193-023-00424-7","type":"correction","label":"Correction","source":"publisher","updated":{"date-parts":[[2023,10,30]],"date-time":"2023-10-30T00:00:00Z","timestamp":1698624000000}}],"container-title":["Journal on Multimodal User Interfaces"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12193-023-00414-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s12193-023-00414-9\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s12193-023-00414-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,14]],"date-time":"2023-11-14T11:09:52Z","timestamp":1699960192000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s12193-023-00414-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,9,9]]},"references-count":32,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2023,12]]}},"alternative-id":["414"],"URL":"https:\/\/doi.org\/10.1007\/s12193-023-00414-9","relation":{"correction":[{"id-type":"doi","id":"10.1007\/s12193-023-00424-7","asserted-by":"object"}]},"ISSN":["1783-7677","1783-8738"],"issn-type":[{"value":"1783-7677","type":"print"},{"value":"1783-8738","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,9,9]]},"assertion":[{"value":"8 May 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"11 August 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 September 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"30 October 2023","order":4,"name":"change_date","label":"Change Date","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Correction","order":5,"name":"change_type","label":"Change Type","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"A Correction to this paper has been published:","order":6,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"https:\/\/doi.org\/10.1007\/s12193-023-00424-7","URL":"https:\/\/doi.org\/10.1007\/s12193-023-00424-7","order":7,"name":"change_details","label":"Change Details","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no conflicts of interest that are relevant to the content of this article.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflicts of interest"}}]}}