{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,26]],"date-time":"2026-02-26T22:53:29Z","timestamp":1772146409784,"version":"3.50.1"},"reference-count":62,"publisher":"Association for Computing Machinery (ACM)","issue":"2","license":[{"start":{"date-parts":[[2019,5,31]],"date-time":"2019-05-31T00:00:00Z","timestamp":1559260800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.acm.org\/publications\/policies\/copyright_policy#Background"}],"funder":[{"name":"European Union's Horizon 2020 Research and Innovation programme","award":["688503"],"award-info":[{"award-number":["688503"]}]}],"content-domain":{"domain":["dl.acm.org"],"crossmark-restriction":true},"short-container-title":["ACM Trans. Multimedia Comput. Commun. Appl."],"published-print":{"date-parts":[[2019,5,31]]},"abstract":"<jats:p>Different senses provide us with information of various levels of precision and enable us to construct a more precise representation of the world. Rich multisensory simulations are thus beneficial for comprehension, memory reinforcement, or retention of information. Crossmodal mappings refer to the systematic associations often made between different sensory modalities (e.g., high pitch is matched with angular shapes) and govern multisensory processing. A great deal of research effort has been put into exploring cross-modal correspondences in the field of cognitive science. However, the possibilities they open in the digital world have been relatively unexplored. Multiple sensorial media (mulsemedia) provides a highly immersive experience to the users and enhances their Quality of Experience (QoE) in the digital world. Thus, we consider that studying the plasticity and the effects of cross-modal correspondences in a mulsemedia setup can bring interesting insights about improving the human computer dialogue and experience. In our experiments, we exposed users to videos with certain visual dimensions (brightness, color, and shape), and we investigated whether the pairing with a cross-modal matching sound (high and low pitch) and the corresponding auto-generated vibrotactile effects (produced by a haptic vest) lead to an enhanced QoE. For this, we captured the eye gaze and the heart rate of users while experiencing mulsemedia, and we asked them to fill in a set of questions targeting their enjoyment and perception at the end of the experiment. Results showed differences in eye-gaze patterns and heart rate between the experimental and the control group, indicating changes in participants\u2019 engagement when videos were accompanied by matching cross-modal sounds (this effect was the strongest for the video displaying angular shapes and high-pitch audio) and transitively generated cross-modal vibrotactile effects.&lt;?vsp -1pt?&gt;<\/jats:p>","DOI":"10.1145\/3303080","type":"journal-article","created":{"date-parts":[[2019,6,6]],"date-time":"2019-06-06T12:28:42Z","timestamp":1559824122000},"page":"1-22","update-policy":"https:\/\/doi.org\/10.1145\/crossmark-policy","source":"Crossref","is-referenced-by-count":18,"title":["Using Eye Tracking and Heart-Rate Activity to Examine Crossmodal Correspondences QoE in Mulsemedia"],"prefix":"10.1145","volume":"15","author":[{"given":"Gebremariam","family":"Mesfin","sequence":"first","affiliation":[{"name":"Brunel University London, Uxbridge, United Kingdom"}]},{"given":"Nadia","family":"Hussain","sequence":"additional","affiliation":[{"name":"Brunel University London, Uxbridge, United Kingdom"}]},{"given":"Alexandra","family":"Covaci","sequence":"additional","affiliation":[{"name":"University of Kent, Kent, United Kingdom"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-2578-5580","authenticated-orcid":false,"given":"Gheorghita","family":"Ghinea","sequence":"additional","affiliation":[{"name":"Brunel University London, Uxbridge, United Kingdom"}]}],"member":"320","published-online":{"date-parts":[[2019,6,5]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","DOI":"10.1145\/2487268.2487270"},{"key":"e_1_2_1_2_1","doi-asserted-by":"publisher","DOI":"10.1145\/2957753"},{"key":"e_1_2_1_3_1","doi-asserted-by":"publisher","DOI":"10.1145\/1980462.1980494"},{"key":"e_1_2_1_4_1","volume-title":"Katrien De Moor, Ann Dooms, Sebastian Egger, Marie-Neige Garcia, Tobias Hossfeld, Satu Jumisko-Pyykk\u00f6, Christian Keimel, Mohamed-Chaker Larabi, et al.","author":"Brunnstr\u00f6m Kjell","year":"2013"},{"key":"e_1_2_1_5_1","doi-asserted-by":"publisher","DOI":"10.1145\/2998181.2998240"},{"key":"e_1_2_1_6_1","first-page":"39","article-title":"Implicit association between basic tastes and pitch. Neurosci","volume":"464","author":"Crisinel Anne-Sylvie","year":"2009","journal-title":"Lett."},{"key":"e_1_2_1_7_1","volume-title":"Is the low-cost EyeTribe eye tracker any good for research? No. e585v1. PeerJ PrePrints","author":"Dalmaijer Edwin","year":"2014"},{"key":"e_1_2_1_8_1","doi-asserted-by":"publisher","DOI":"10.1109\/TOH.2012.70"},{"key":"e_1_2_1_9_1","doi-asserted-by":"publisher","DOI":"10.1145\/1836135.1836143"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.1145\/1840784.1840812"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1109\/QoMEX.2016.7498964"},{"key":"e_1_2_1_12_1","doi-asserted-by":"publisher","DOI":"10.1109\/VECIMS.2008.4592743"},{"key":"e_1_2_1_13_1","doi-asserted-by":"publisher","DOI":"10.1109\/JSTSP.2016.2609843"},{"key":"e_1_2_1_14_1","unstructured":"David Gal S. Christian Wheeler and Baba Shiv. 2007. Cross-modal influences on gustatory perception. Available at SSRN 1030197.  David Gal S. Christian Wheeler and Baba Shiv. 2007. Cross-modal influences on gustatory perception. Available at SSRN 1030197."},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.1145\/2071396.2071398"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.1145\/2379790.2379794"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.1145\/2617994"},{"key":"e_1_2_1_18_1","volume-title":"Proceedings of DiGRA 2005: Changing Views--Worlds in Play.","author":"Gilleade Kiel","year":"2005"},{"key":"e_1_2_1_19_1","doi-asserted-by":"publisher","DOI":"10.1109\/TSMCA.2004.826309"},{"key":"e_1_2_1_20_1","doi-asserted-by":"publisher","DOI":"10.1509\/jmr.14.0414"},{"key":"e_1_2_1_21_1","volume-title":"Proceedings of the 1st International Workshop on Haptic and Audio Interaction Design (HAID\u201906)","author":"Hoggan E."},{"key":"e_1_2_1_22_1","doi-asserted-by":"publisher","DOI":"10.1145\/3001773.3001776"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/1980462.1980493"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1109\/VS-GAMES.2010.20"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1109\/QoMEX.2017.7965656"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.1109\/T-AFFC.2011.15"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/2071423.2071449"},{"key":"e_1_2_1_28_1","volume-title":"The influence of new colored light stimulation methods on heart rate variability, temperature, and well-being: Results of a pilot study in humans. Evidence-Based Complementary and Alternative Medicine","author":"Litscher Daniela","year":"2013"},{"key":"e_1_2_1_29_1","unstructured":"Lawrence E. Marks. 1974. On associations of light and sound: The mediation of brightness pitch and loudness. Am. J. Psychol. (1974) 173--188.  Lawrence E. Marks. 1974. On associations of light and sound: The mediation of brightness pitch and loudness. Am. J. Psychol. (1974) 173--188."},{"key":"e_1_2_1_30_1","doi-asserted-by":"publisher","DOI":"10.1037\/0096-1523.13.3.384"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.1145\/1101389.1101462"},{"key":"e_1_2_1_33_1","first-page":"303","article-title":"Next mega tech theme is virtual reality","volume":"612","author":"Munster Gene","year":"2015","journal-title":"Gene"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1145\/3108243"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","DOI":"10.1145\/2508244.2508256"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1145\/3173574.3174151"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/2540930.2540939"},{"key":"e_1_2_1_38_1","doi-asserted-by":"publisher","DOI":"10.1145\/3123266.3123440"},{"key":"e_1_2_1_39_1","first-page":"175","article-title":"Cross-modal integration between odors and abstract symbols. Neurosci","volume":"478","author":"Seo Han-Seok","year":"2010","journal-title":"Lett."},{"key":"e_1_2_1_40_1","volume-title":"Illusions: What you see is what you hear. Nature 408, 6814","author":"Shams Ladan","year":"2000"},{"key":"e_1_2_1_41_1","volume-title":"Proceedings of the 3rd International Conference of Synaesthesia and Art.","author":"Simner J."},{"key":"e_1_2_1_42_1","doi-asserted-by":"publisher","DOI":"10.1080\/00221325.1956.10532990"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuropsychologia.2015.07.011"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.1109\/WIAMIS.2012.6226776"},{"key":"e_1_2_1_45_1","first-page":"122","article-title":"The color of wine--part 1","volume":"28","author":"Spence Charles","year":"2010","journal-title":"Wrld. Fine Wine"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13414-010-0073-7"},{"key":"e_1_2_1_47_1","volume-title":"Simple statistics for library and information professionals","author":"Stephen Peter","year":"1997"},{"key":"e_1_2_1_48_1","volume-title":"Can multisensory cues in VR help train pattern recognition to citizen scientists? arXiv preprint arXiv:1804.00229","author":"Striner Alina","year":"2018"},{"key":"e_1_2_1_49_1","doi-asserted-by":"publisher","DOI":"10.1109\/IWSSIP.2016.7502696"},{"key":"e_1_2_1_50_1","volume-title":"An extended research of crossmodal correspondence between color and sound in psychology and cognitive ergonomics. PeerJ 6 (Mar","author":"Sun Xiuwen","year":"2018"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.1145\/3123024.3123190"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/2858036.2858304"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.5555\/3227248.3227583"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.14236\/ewic\/EVA2017.39"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.1109\/APSIPA.2014.7041631"},{"key":"e_1_2_1_56_1","volume-title":"Proceedings of the 2013 IEEE International Conference on Consumer Electronics (ICCE\u201913)","author":"Valenti Giulio"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.3758\/s13423-016-1097-3"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.3758\/BF03204939"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1037\/0033-2909.88.3.638"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2009.02.013"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.1145\/2661329"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1109\/TMM.2014.2371240"},{"key":"e_1_2_1_63_1","doi-asserted-by":"publisher","DOI":"10.1145\/3083187.3084014"}],"container-title":["ACM Transactions on Multimedia Computing, Communications, and Applications"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3303080","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3303080","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,17]],"date-time":"2025-06-17T23:53:39Z","timestamp":1750204419000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3303080"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2019,5,31]]},"references-count":62,"journal-issue":{"issue":"2","published-print":{"date-parts":[[2019,5,31]]}},"alternative-id":["10.1145\/3303080"],"URL":"https:\/\/doi.org\/10.1145\/3303080","relation":{},"ISSN":["1551-6857","1551-6865"],"issn-type":[{"value":"1551-6857","type":"print"},{"value":"1551-6865","type":"electronic"}],"subject":[],"published":{"date-parts":[[2019,5,31]]},"assertion":[{"value":"2018-07-01","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2018-12-01","order":1,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2019-06-05","order":2,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}]}}