{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,2,21]],"date-time":"2025-02-21T11:13:02Z","timestamp":1740136382518,"version":"3.37.3"},"reference-count":49,"publisher":"Springer Science and Business Media LLC","issue":"5","license":[{"start":{"date-parts":[[2023,10,1]],"date-time":"2023-10-01T00:00:00Z","timestamp":1696118400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,10,3]],"date-time":"2023-10-03T00:00:00Z","timestamp":1696291200000},"content-version":"vor","delay-in-days":2,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100001537","name":"University of Auckland","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100001537","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Pers Ubiquit Comput"],"published-print":{"date-parts":[[2023,10]]},"abstract":"<jats:title>Abstract<\/jats:title><jats:p>This article presents an empirical investigation into spectator interpretation of multimodal artworks. Specifically, this article explores perception of the artwork in two different sensorial modalities \u2014 sight and sound \u2014 and the effect of their interaction. We selected four abstract paintings and created four acousmatic musical pieces that were composed to reflect the content of the artwork. We then ran a between-subjects experimental study in three conditions: visual only, music only, and a combination of both. A total of 48 participants completed an online survey in which they were asked to report their interpretations of the shown artworks. Following a thematic analysis on the collected data, we clustered participants\u2019 interpretation into two main categories:<jats:italic>reflective<\/jats:italic>and<jats:italic>perceptive<\/jats:italic>. The combination increased spectators\u2019 attention to the artworks, affected the temporality of the artworks, and created richer understandings of the multimodal works. The study provides knowledge for the inclusion of multimodal experiences in the presentation of art expanding the possibilities for inter-sensory dialogue in the arts.<\/jats:p>","DOI":"10.1007\/s00779-023-01744-x","type":"journal-article","created":{"date-parts":[[2023,10,3]],"date-time":"2023-10-03T09:02:49Z","timestamp":1696323769000},"page":"1965-1975","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Perceptive vs reflective: spectator interpretation of multimodal artworks"],"prefix":"10.1007","volume":"27","author":[{"given":"Oliver","family":"Bramah","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Sue","family":"Adams","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4512-4897","authenticated-orcid":false,"given":"Fabio","family":"Morreale","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2023,10,3]]},"reference":[{"key":"1744_CR1","doi-asserted-by":"crossref","unstructured":"Leppert R (2013) In: Leondard A (ed) Shephard T. Seeing music Routledge, London, pp 25\u201330","DOI":"10.4324\/9780203629987-10"},{"key":"1744_CR2","volume-title":"The life of imagination: revealing and making the world","author":"JA Gosetti-Ferencei","year":"2018","unstructured":"Gosetti-Ferencei JA (2018) The life of imagination: revealing and making the world, 1st edn. Columbia University Press, New York","edition":"1"},{"issue":"4","key":"1744_CR3","doi-asserted-by":"publisher","first-page":"353","DOI":"10.1093\/aesthj\/ays042","volume":"52","author":"B Nanay","year":"2012","unstructured":"Nanay B (2012) The multimodal experience of art. Br J Aesthet 52(4):353\u2013363","journal-title":"Br J Aesthet"},{"issue":"7","key":"1744_CR4","doi-asserted-by":"publisher","first-page":"119","DOI":"10.3390\/bs10070119","volume":"10","author":"M Reybrouck","year":"2020","unstructured":"Reybrouck M, Podlipniak P, Welch D (2020) Music listening as coping behavior: From reactive response to sense-making. Behav Sci 10(7):119","journal-title":"Behav Sci"},{"issue":"2","key":"1744_CR5","doi-asserted-by":"publisher","first-page":"103","DOI":"10.1017\/S1355771812000027","volume":"17","author":"D Garro","year":"2012","unstructured":"Garro D (2012) From sonic art to visual music: Divergences, convergences, intersections. Organised Sound. 17(2):103\u2013113","journal-title":"Organised Sound."},{"key":"1744_CR6","doi-asserted-by":"publisher","DOI":"10.7312\/chio18588","volume-title":"Audio-vision: Sound on Screen","author":"M Chion","year":"2019","unstructured":"Chion M (2019) Audio-vision: Sound on Screen. Columbia University Press, New York"},{"issue":"2","key":"1744_CR7","doi-asserted-by":"publisher","first-page":"635","DOI":"10.2466\/pms.1982.54.2.635","volume":"54","author":"AC Parrott","year":"1982","unstructured":"Parrott AC (1982) Effects of paintings and music, both alone and in combination, on emotional judgments. Percept Mot Skills 54(2):635\u2013641","journal-title":"Percept Mot Skills"},{"issue":"1","key":"1744_CR8","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1525\/mp.2009.27.1.43","volume":"27","author":"MG Boltz","year":"2009","unstructured":"Boltz MG, Ebendorf B, Field B (2009) Audiovisual interactions: the impact of visual information on music perception and memory. Music Percept Interdiscip J 27(1):43\u201359","journal-title":"Music Percept Interdiscip J"},{"key":"1744_CR9","doi-asserted-by":"crossref","unstructured":"Bramah O, Cheng X, Morreale F (2021) The singing gallery: Combining music and static visual artworks. In: Audio Mostly 2021, pp 117\u2013120","DOI":"10.1145\/3478384.3478396"},{"key":"1744_CR10","unstructured":"Mazzola G (1997) Semiotics of music. A Handbook on the Sign-Theoretic Foundations of Nature and Culture"},{"key":"1744_CR11","unstructured":"Descartes R (1653) Renatus Des-Cartes Excellent Compendium of Musick: With Necessary and Judicious Animadversions Thereupon. by a Person of Honour. Printed by Thomas Harper, for Humphrey Moseley, and are to bee sold at his shop at the signe of the Princes Armes in S. Pauls Church-Yard, and by Thomas Heath in Coven Garden., London"},{"key":"1744_CR12","unstructured":"Bashwiner DM (2010) Musical emotion: Toward a biologically grounded theory. PhD thesis, The University of Chicago, Chicago"},{"key":"1744_CR13","unstructured":"Dolan T (2013) In: Shephard T, Leondard A (eds) Painting and Music. Routledge, London, pp 127\u2013134"},{"key":"1744_CR14","doi-asserted-by":"publisher","DOI":"10.4324\/9780429274015","volume-title":"Musical Sense-Making: Enaction","author":"M Reybrouck","year":"2020","unstructured":"Reybrouck M (2020) Musical Sense-Making: Enaction. Experience and Computation, Routledge, London"},{"key":"1744_CR15","doi-asserted-by":"crossref","unstructured":"Morreale F, Masu R, De\u00a0Angeli A (2019) The influence of coauthorship in the interpretation of multimodal interfaces. Wirel Commun Mob Comput 2019","DOI":"10.1155\/2019\/9234812"},{"key":"1744_CR16","volume-title":"Music and semiotics: An experiential approach to musical sense-making","author":"M Reybrouck","year":"2017","unstructured":"Reybrouck M (2017) Music and semiotics: An experiential approach to musical sense-making, 1st edn. IntechOpen, London","edition":"1"},{"issue":"2","key":"1744_CR17","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1108\/13673279810249369","volume":"2","author":"B Dervin","year":"1998","unstructured":"Dervin B (1998) Sense making theory and practice: an overview of user interests in knowledge seeking and use. J Knowl Manag 2(2):36\u201346","journal-title":"J Knowl Manag"},{"issue":"2","key":"1744_CR18","doi-asserted-by":"publisher","first-page":"101","DOI":"10.1080\/10632921.2014.905813","volume":"44","author":"B Dervin","year":"2014","unstructured":"Dervin B, Funk C (2014) Standing in two worlds looking at an art exhibition: Sense-making in the millennial generation. J Arts Manag Law Soc 44(2):101\u2013117","journal-title":"J Arts Manag Law Soc"},{"issue":"3\u20134","key":"1744_CR19","doi-asserted-by":"publisher","first-page":"213","DOI":"10.1016\/S0360-1315(99)00046-9","volume":"34","author":"A Antonietti","year":"2000","unstructured":"Antonietti A, Cantoia M (2000) To see a painting versus to walk in a painting: an experiment on sense-making through virtual reality. Comput Educ 34(3\u20134):213\u2013223","journal-title":"Comput Educ"},{"issue":"2","key":"1744_CR20","doi-asserted-by":"publisher","first-page":"171","DOI":"10.1111\/j.2151-6952.2008.tb00303.x","volume":"51","author":"P Carter-Birken","year":"2008","unstructured":"Carter-Birken P (2008) Interpretation and the role of the viewer in museums of modern and contemporary art. Curator Mus J 51(2):171\u2013178","journal-title":"Curator Mus J"},{"key":"1744_CR21","volume-title":"Nature of Abstract Art","author":"M Schapiro","year":"1937","unstructured":"Schapiro M (1937) Nature of Abstract Art. American Marxist Association, USA"},{"issue":"3","key":"1744_CR22","doi-asserted-by":"publisher","first-page":"222","DOI":"10.1080\/00043249.1988.10792416","volume":"47","author":"FV O\u2019Connor","year":"1988","unstructured":"O\u2019Connor FV (1988) Two methodologies for the interpretation of abstract expressionism. Art J 47(3):222\u2013228","journal-title":"Art J"},{"issue":"10","key":"1744_CR23","doi-asserted-by":"publisher","first-page":"1272","DOI":"10.1097\/ACM.0b013e31822c161d","volume":"86","author":"PB Schaff","year":"2011","unstructured":"Schaff PB, Isken S, Tager RM (2011) From contemporary art to core clinical skills: Observation, interpretation, and meaning-making in a complex environment. Acad Med 86(10):1272\u20131276","journal-title":"Acad Med"},{"issue":"1","key":"1744_CR24","doi-asserted-by":"publisher","first-page":"77","DOI":"10.2190\/Y7CA-TBY6-V7LR-76GK","volume":"22","author":"GJ Feist","year":"2004","unstructured":"Feist GJ, Brady TR (2004) Openness to experience, non-conformity, and the preference for abstract art. Empir Stud Arts 22(1):77\u201389","journal-title":"Empir Stud Arts"},{"issue":"1","key":"1744_CR25","doi-asserted-by":"crossref","first-page":"1","DOI":"10.3402\/jac.v8.29934","volume":"8","author":"P Giladi","year":"2016","unstructured":"Giladi P (2016) Embodied meaning and art as sense-making: a critique of beiser\u2019s interpretation of the \u201cend of art thesis\u2019\u2019. J Aesthet Cult 8(1):1\u201311","journal-title":"J Aesthet Cult"},{"issue":"2","key":"1744_CR26","doi-asserted-by":"publisher","first-page":"224","DOI":"10.1177\/0741088304274170","volume":"22","author":"GA Hull","year":"2005","unstructured":"Hull GA, Nelson ME (2005) Locating the semiotic power of multimodality. Writ Commun 22(2):224\u2013261","journal-title":"Writ Commun"},{"issue":"3","key":"1744_CR27","doi-asserted-by":"publisher","first-page":"285","DOI":"10.1162\/leon.2007.40.3.285","volume":"40","author":"BM Galeyev","year":"2007","unstructured":"Galeyev BM (2007) The nature and functions of synesthesia in music. Leonardo 40(3):285\u2013288","journal-title":"Leonardo"},{"issue":"32","key":"1744_CR28","first-page":"1","volume":"2","author":"S Day","year":"1996","unstructured":"Day S (1996) Synaesthesia and synaesthetic metaphors. Psyche 2(32):1\u201316","journal-title":"Psyche"},{"issue":"1","key":"1744_CR29","doi-asserted-by":"publisher","first-page":"58","DOI":"10.1080\/09647040490885493","volume":"13","author":"A Ione","year":"2010","unstructured":"Ione A, Tyler C (2010) Neuroscience, history and the arts synesthesia: Is f-sharp colored violet? J Hist Neurosci 13(1):58\u201365","journal-title":"J Hist Neurosci"},{"key":"1744_CR30","unstructured":"Shaw-Miller S (2013) In: Shephard T, Leonard A (eds) Synaesthesia. Routledge, London, pp 13\u201323"},{"issue":"1","key":"1744_CR31","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1080\/17458927.2017.1268811","volume":"12","author":"S Casini","year":"2017","unstructured":"Casini S (2017) Synesthesia, transformation and synthesis: toward a multi-sensory pedagogy of the image. Sense Soc 12(1):1\u201317","journal-title":"Sense Soc"},{"issue":"3","key":"1744_CR32","doi-asserted-by":"publisher","first-page":"679","DOI":"10.1093\/cercor\/bhk016","volume":"17","author":"D Talsma","year":"2007","unstructured":"Talsma D, Doty TJ, Woldorff MG (2007) Selective attention and audiovisual integration: Is attending to both modalities a prerequisite for early integration? Cereb Cortex 17(3):679\u2013690","journal-title":"Cereb Cortex"},{"key":"1744_CR33","unstructured":"Rogers H (2013) In: Shephard T, Leondard A (eds) Multimedia Art. Routledge, London, pp 367\u2013375"},{"key":"1744_CR34","first-page":"1","volume":"31","author":"T Lee","year":"2016","unstructured":"Lee T, Lim H, Kim D-W, Hwang S, Yoon K (2016) System for matching paintings with music based on emotions. SIGGRAPH ASIA 31:1\u20134","journal-title":"SIGGRAPH ASIA"},{"key":"1744_CR35","doi-asserted-by":"crossref","unstructured":"McDonnell M (2020) The question of form in visual music, 1st edn. Focal Press, New York, pp 30\u201348","DOI":"10.4324\/9780429295102-3"},{"key":"1744_CR36","doi-asserted-by":"crossref","unstructured":"Coulter J (2010) Electroacoustic music with moving images: The art of media pairing. Organised Sound 15(1):26\u201334","DOI":"10.1017\/S1355771809990239"},{"key":"1744_CR37","doi-asserted-by":"crossref","unstructured":"Battey B (2005) Autarkeia Aggregatum. Audiovisual Composition. https:\/\/vimeo.com\/14923910","DOI":"10.1145\/1179849.1179964"},{"issue":"6","key":"1744_CR38","doi-asserted-by":"publisher","first-page":"984","DOI":"10.1108\/EL-11-2018-0219","volume":"37","author":"C Molina Beltr\u00e1n","year":"2018","unstructured":"Molina Beltr\u00e1n C, Segura Navarrete AA, Vidal-Castro C, Rubio-Manzano C, Mart\u00ednez-Araneda C (2018) Improving the affective analysis in texts: Automatic method to detect affective intensity in lexicons based on Plutchik\u2019s wheel of emotions. Electron Libr 37(6):984\u20131006","journal-title":"Electron Libr"},{"key":"1744_CR39","unstructured":"Plutchik R (1991) The Emotions. Elsevier Science & Technology, Lanham, MD & London"},{"key":"1744_CR40","unstructured":"Plutchik R, Kellerman H (1980) Theories of Emotion. Elsevier Science & Technology, New York & London"},{"issue":"2","key":"1744_CR41","doi-asserted-by":"publisher","first-page":"77","DOI":"10.1191\/1478088706qp063oa","volume":"3","author":"V Braun","year":"2006","unstructured":"Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3(2):77\u2013101","journal-title":"Qual Res Psychol"},{"issue":"1","key":"1744_CR42","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1017\/S1355771809990240","volume":"15","author":"DL Barreiro","year":"2010","unstructured":"Barreiro DL (2010) Sonic image and acousmatic listening. Organised Sound 15(1):35\u201342","journal-title":"Organised Sound"},{"issue":"4","key":"1744_CR43","doi-asserted-by":"publisher","first-page":"749","DOI":"10.1007\/s00779-020-01395-2","volume":"25","author":"L Turchet","year":"2021","unstructured":"Turchet L, West T, Wanderley MM (2021) Touching the audience: musical haptic wearables for augmented and participatory live music performances. Pers Ubiquit Comput 25(4):749-769","journal-title":"Pers Ubiquit Comput"},{"issue":"2","key":"1744_CR44","doi-asserted-by":"publisher","first-page":"183","DOI":"10.1109\/THMS.2018.2885408","volume":"49","author":"L Turchet","year":"2018","unstructured":"Turchet L, Barthet M (2018) Co-design of musical haptic wearables for electronic music performer\u2019s communication. IEEE Trans Hum Mach Sys 49(2):183\u2013193","journal-title":"IEEE Trans Hum Mach Sys"},{"key":"1744_CR45","doi-asserted-by":"crossref","unstructured":"Chamberlain A, Malizia A, Dix AJ (2014) Visual and tactile engagement: designing projected touch-surfaces for community use in a rural context. In: Proceedings of the 2014 International Working Conference on Advanced Visual Interfaces, pp 137\u2013140","DOI":"10.1145\/2598153.2598202"},{"key":"1744_CR46","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1016\/j.ijhcs.2017.06.004","volume":"108","author":"CT Vi","year":"2017","unstructured":"Vi CT, Ablart D, Gatti E, Velasco C, Obrist M (2017) Not just seeing, but also feeling art: Mid-air haptic experiences integrated in a multisensory art exhibition. Int J Hum Comput Stud 108:1-14","journal-title":"Int J Hum Comput Stud"},{"key":"1744_CR47","doi-asserted-by":"crossref","unstructured":"Sengers P, Gaver B (2006) Staying open to interpretation: engaging multiple meanings in design and evaluation. In: Proceedings of the 6th Conference on Designing Interactive Systems, pp 99\u2013108","DOI":"10.1145\/1142405.1142422"},{"key":"1744_CR48","volume-title":"Teaching in the Art Museum: Interpretation as Experience","author":"R Burnham","year":"2011","unstructured":"Burnham R, Kai-Kee E (2011) Teaching in the Art Museum: Interpretation as Experience. Getty Publications, Los Angeles"},{"issue":"2","key":"1744_CR49","doi-asserted-by":"publisher","first-page":"9","DOI":"10.1109\/MMUL.2017.33","volume":"24","author":"M Obrist","year":"2017","unstructured":"Obrist M, Gatti E, Maggioni E, Vi CT, Velasco C (2017) Multisensory experiences in hci. IEEE MultiMedia 24(2):9\u201313","journal-title":"IEEE MultiMedia"}],"container-title":["Personal and Ubiquitous Computing"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00779-023-01744-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1007\/s00779-023-01744-x\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1007\/s00779-023-01744-x.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,12,23]],"date-time":"2023-12-23T07:29:32Z","timestamp":1703316572000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1007\/s00779-023-01744-x"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,10]]},"references-count":49,"journal-issue":{"issue":"5","published-print":{"date-parts":[[2023,10]]}},"alternative-id":["1744"],"URL":"https:\/\/doi.org\/10.1007\/s00779-023-01744-x","relation":{},"ISSN":["1617-4909","1617-4917"],"issn-type":[{"type":"print","value":"1617-4909"},{"type":"electronic","value":"1617-4917"}],"subject":[],"published":{"date-parts":[[2023,10]]},"assertion":[{"value":"31 March 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"20 August 2023","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"3 October 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Conflict of interest"}}]}}