{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,8]],"date-time":"2025-11-08T13:56:25Z","timestamp":1762610185384,"version":"3.41.2"},"reference-count":57,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2025,6,10]],"date-time":"2025-06-10T00:00:00Z","timestamp":1749513600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100001659","name":"Deutsche Forschungsgemeinschaft","doi-asserted-by":"publisher","award":["438822823","444697733"],"award-info":[{"award-number":["438822823","444697733"]}],"id":[{"id":"10.13039\/501100001659","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Psychol."],"abstract":"<jats:p>This study explores how audiovisual immersive virtual environments (IVEs) can assess cognitive performance in classroom-like settings, addressing limitations in simpler acoustic and visual representations. This study examines the potential of a test paradigm using speaker-story mapping, called \u201caudiovisual scene analysis (AV-SA),\u201d originally developed for virtual reality (VR) hearing research, as a method to evaluate audiovisual scene analysis in a virtual classroom scenario. Factors affecting acoustic and visual scene representation were varied to investigate their impact on audiovisual scene analysis. Two acoustic representations were used: a simple \u201cdiotic\u201d presentation where the same signal is presented to both ears, as well as a dynamically live-rendered binaural synthesis (\u201cbinaural\u201d). Two visual representations were used: 360\u00b0\/omnidirectional video with intrinsic lip-sync and computer-generated imagery (CGI) without lip-sync. Three subjective experiments were conducted with different combinations of the two acoustic and visual conditions: The first experiment, involving 36 participants, used 360\u00b0 video with \u201cbinaural\u201d audio. The second experiment, with 24 participants, combined 360\u00b0 video with \u201cdiotic\u201d audio. The third experiment, with 34 participants, used the CGI environment with \u201cbinaural\u201d audio. Each environment presented 20 different speakers in a classroom-like circle of 20 chairs, with the number of simultaneously active speakers ranging from 2 to 10, while the remaining speakers kept silent and were always shown. During the experiments, the subjects' task was to correctly map the stories' topics to the corresponding speakers. The primary dependent variable was the number of correct assignments during a fixed period of 2 min, followed by two questionnaires on mental load after each trial. In addition, before and\/or after the experiments, subjects needed to complete questionnaires about simulator sickness, noise sensitivity, and presence. Results indicate that the experimental condition significantly influenced task performance, mental load, and user behavior but did not affect perceived simulator sickness and presence. Performance decreased when comparing the 360\u00b0 video and \u201cbinaural\u201d audio experiment with either the experiment using \u201cdiotic\u201d audio and 360\u00b0, or using \u201cbinaural\u201d audio with CGI-based, showing the usefulness of the test method in investigating influences on cognitive audiovisual scene analysis performance.<\/jats:p>","DOI":"10.3389\/fpsyg.2025.1520630","type":"journal-article","created":{"date-parts":[[2025,6,10]],"date-time":"2025-06-10T05:22:53Z","timestamp":1749532973000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":1,"title":["Speaker-story mapping as a method to evaluate audiovisual scene analysis in a virtual classroom scenario"],"prefix":"10.3389","volume":"16","author":[{"given":"Stephan","family":"Fremerey","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Carolin","family":"Breuer","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Larissa","family":"Leist","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Maria","family":"Klatte","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Janina","family":"Fels","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alexander","family":"Raake","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2025,6,10]]},"reference":[{"key":"B1","doi-asserted-by":"crossref","first-page":"1007","DOI":"10.1145\/3343031.3350947","article-title":"\u201c360-degree video gaze behaviour: a ground-truth data set and a classification algorithm for eye movements,\u201d","volume-title":"Proceedings of the 27th ACM International Conference on Multimedia","author":"Agtzidis","year":"2019"},{"key":"B2","doi-asserted-by":"publisher","first-page":"1586","DOI":"10.1121\/10.0013991","article-title":"Auditory spatial analysis in reverberant multi-talker environments with congruent and incongruent audio-visual room information","volume":"152","author":"Ahrens","year":"2022","journal-title":"J. Acoust. Soc. Am"},{"key":"B3","first-page":"3890","article-title":"\u201cAudio-visual scene analysis in reverberant multi-talker environments,\u201d","volume-title":"23rd International Congress on Acoustics","author":"Ahrens","year":"2019"},{"key":"B4","doi-asserted-by":"publisher","first-page":"1021","DOI":"10.1007\/s00221-020-06014-3","article-title":"Top-down task-specific determinants of multisensory motor reaction time enhancements and sensory switch costs","volume":"239","author":"Barutchu","year":"2021","journal-title":"Exp. Brain Res"},{"key":"B5","first-page":"117","article-title":"The cocktail party phenomenon: a review of research on speech intelligibility in multiple-talker conditions","volume":"86","author":"Bronkhorst","year":"2000","journal-title":"Acta Acust. United Acust"},{"key":"B6","doi-asserted-by":"publisher","first-page":"1465","DOI":"10.3758\/s13414-015-0882-9","article-title":"The cocktail-party problem revisited: early processing and selection of multi-talker speech","volume":"77","author":"Bronkhorst","year":"2015","journal-title":"Attent. Percept. Psychophys"},{"key":"B7","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2306.10175","article-title":"SSE: a metric for evaluating search system explainability","author":"Chen","year":"2023","journal-title":"arXiv"},{"key":"B8","doi-asserted-by":"publisher","first-page":"975","DOI":"10.1121\/1.1907229","article-title":"Some experiments on the recognition of speech, with one and two ears","volume":"25","author":"Cherry","year":"1953","journal-title":"J. Acoust. Soc. Am"},{"key":"B9","doi-asserted-by":"publisher","first-page":"100","DOI":"10.1016\/0022-0965(73)90134-3","article-title":"Listening to distraction: a developmental study of selective attention","volume":"15","author":"Doyle","year":"1973","journal-title":"J. Exp. Child Psychol"},{"volume-title":"Loudness Normalisation and Permitted Maximum Level of Audio Signals","year":"2023","key":"B10"},{"key":"B11","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/I3DA48870.2021.9610916","article-title":"\u201cEffect of acoustic scene complexity and visual scene representation on auditory perception in virtual audio-visual environments,\u201d","volume-title":"2021 Immersive and 3D Audio: from Architecture to Automotive (I3DA)","author":"Fichna","year":"2021"},{"key":"B12","doi-asserted-by":"publisher","first-page":"90","DOI":"10.1016\/j.zefq.2019.10.003","article-title":"The \u201cnational aeronautics and space administration-task load index\u201d (NASA-TLX)-an instrument for measuring consultation workload within general practice: evaluation of psychometric properties","volume":"147","author":"Fl\u00e4gel","year":"2019","journal-title":"Z. Evid. Fortbild. Qual. Gesundhwes"},{"key":"B13","first-page":"225","article-title":"Virtual reality in psychology","volume":"2","author":"Foreman","year":"2010","journal-title":"Themes Sci. Technol. Educ"},{"key":"B14","doi-asserted-by":"crossref","first-page":"207","DOI":"10.1109\/QoMEX61742.2024.10598262","article-title":"\u201cAVT-ECOCLASS-VR: an open-source audiovisual 360\u00b0 video and immersive CGI multi-talker dataset to evaluate cognitive performance,\u201d","volume-title":"2024 16th International Conference on Quality of Multimedia Experience (QoMEX)","author":"Fremerey","year":"2024"},{"key":"B15","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1145\/3204949.3208134","article-title":"\u201cAvtrack360: an open dataset and software recording people's head rotations watching 360 videos on an HMD,\u201d","volume-title":"Proceedings of the 9th ACM Multimedia Systems Conference","author":"Fremerey","year":"2018"},{"key":"B16","doi-asserted-by":"publisher","first-page":"3907","DOI":"10.1121\/1.412407","article-title":"Hrtf measurements of a kemar","volume":"97","author":"Gardner","year":"1995","journal-title":"J. Acoust. Soc. Am"},{"key":"B17","first-page":"185","article-title":"\u201cNASA RTLX as a novel assessment for determining cognitive load and user acceptance of expert and user-based evaluation methods exemplified through a mhealth diabetes self-management application evaluation,\u201d","volume-title":"pHealth 2019","author":"Georgsson","year":"2019"},{"key":"B18","doi-asserted-by":"publisher","first-page":"3817","DOI":"10.1038\/s41598-017-04201-x","article-title":"Concurrent talking in immersive virtual reality: on the dominance of visual speech cues","volume":"7","author":"Gonzalez-Franco","year":"2017","journal-title":"Sci. Rep"},{"key":"B19","first-page":"1","article-title":"\u201cAVrate Voyager: an open source online testing platform,\u201d","volume-title":"2021 IEEE 23st International Workshop on Multimedia Signal Processing (MMSP)","author":"G\u00f6ring","year":"2021"},{"key":"B20","first-page":"1245","article-title":"\u201cThe national aviation and space agency task load index (NASA-TLX): does it need updating?\u201d","volume-title":"Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Volume 66","author":"Helton","year":"2022"},{"key":"B21","doi-asserted-by":"publisher","first-page":"83","DOI":"10.1016\/j.heares.2016.04.007","article-title":"Eeg activity evoked in preparation for multi-talker listening by adults and children","volume":"336","author":"Holmes","year":"2016","journal-title":"Hear. Res"},{"key":"B22","unstructured":"Important Safety Information\n          \n          2021"},{"key":"B23","unstructured":"Virtual Acoustics-A Real-Time Auralization Framework for Scientific Research\n          \n          2024"},{"volume-title":"The Series of Plates Designed as a Test for Color Deficiency","year":"2009","author":"Ishihara","key":"B24"},{"key":"B25","doi-asserted-by":"publisher","first-page":"35","DOI":"10.1121\/1.4990375","article-title":"Modeling speech localization, talker identification, and word recognition in a multi-talker setting","volume":"142","author":"Josupeit","year":"2017","journal-title":"J. Acoust. Soc. Am"},{"key":"B26","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.594511","author":"Kay","year":"2021","journal-title":"Artool: Aligned Rank Transform for Nonparametric Factorial ANOVAs, r package version 0.11.0"},{"key":"B27","doi-asserted-by":"publisher","first-page":"203","DOI":"10.1207\/s15327108ijap0303_3","article-title":"Simulator sickness questionnaire: an enhanced method for quantifying simulator sickness","volume":"3","author":"Kennedy","year":"1993","journal-title":"Int. J. Aviat. Psychol"},{"key":"B28","first-page":"425","article-title":"Head movements made by listeners in experimental and real-life listening activities","volume":"61","author":"Kim","year":"2013","journal-title":"J. Audio Eng. Soc"},{"key":"B29","doi-asserted-by":"publisher","first-page":"492","DOI":"10.1121\/10.0001670","article-title":"A multimedia speech corpus for audio visual research in virtual reality (l)","volume":"148","author":"Kishline","year":"2020","journal-title":"J. Acoust. Soc. Am"},{"key":"B30","doi-asserted-by":"publisher","first-page":"1140","DOI":"10.1037\/a0022189","article-title":"Switching in the cocktail party: exploring intentional control of auditory selective attention","volume":"37","author":"Koch","year":"2011","journal-title":"J. Exp. Psychol.: Hum. Percept. Perform"},{"key":"B31","doi-asserted-by":"publisher","first-page":"2010","DOI":"10.1080\/17470218.2014.898079","article-title":"Intentional attention switching in dichotic listening: exploring the efficiency of nonspatial and spatial selection","volume":"67","author":"Lawo","year":"2014","journal-title":"Q. J. Expe. Psychol"},{"key":"B32","doi-asserted-by":"crossref","first-page":"164","DOI":"10.1109\/VR.2002.996519","article-title":"\u201cEffects of field of view on presence, enjoyment, memory, and simulator sickness in a virtual environment,\u201d","volume-title":"Proceedings IEEE Virtual Reality 2002","author":"Lin","year":"2002"},{"key":"B33","doi-asserted-by":"publisher","first-page":"36","DOI":"10.1016\/j.actpsy.2017.04.008","article-title":"Intentional switching in auditory selective attention: exploring age-related effects in a spatial setup requiring speech perception","volume":"177","author":"Oberem","year":"2017","journal-title":"Acta Psychol"},{"key":"B34","doi-asserted-by":"publisher","first-page":"1139","DOI":"10.3813\/AAA.918793","article-title":"Intentional switching in auditory selective attention: exploring different binaural reproduction methods in an anechoic chamber","volume":"100","author":"Oberem","year":"2014","journal-title":"Acta Acust. United Acust"},{"key":"B35","doi-asserted-by":"publisher","first-page":"32","DOI":"10.1016\/j.heares.2017.12.013","article-title":"Intentional switching in auditory selective attention: exploring attention shifts with different reverberation times","volume":"359","author":"Oberem","year":"2018","journal-title":"Hear. Res"},{"key":"B36","first-page":"631","article-title":"\u201cAudio-visual scene analysis with self-supervised multisensory features,\u201d","volume-title":"Proceedings of the European conference on computer vision (ECCV)","author":"Owens","year":"2018"},{"key":"B37","doi-asserted-by":"crossref","first-page":"21","DOI":"10.1007\/978-3-319-29938-9_2","article-title":"\u201cAssessing the user experience of password reset policies in a university,\u201d","volume-title":"Technology and Practice of Passwords: 9th International Conference, PASSWORDS 2015, Cambridge, UK, December 7-9, 2015, Proceedings 9","author":"Parkin","year":"2016"},{"key":"B38","unstructured":"Tests\n          \n          2023"},{"key":"B39","doi-asserted-by":"publisher","first-page":"41","DOI":"10.1016\/j.jecp.2018.02.005","article-title":"Equivalent auditory distraction in children and adults","volume":"172","author":"R\u00f6er","year":"2018","journal-title":"J. Exp. Child Psychol"},{"key":"B40","doi-asserted-by":"publisher","first-page":"1810","DOI":"10.3390\/app14051810","article-title":"Analysis of head micromovements and body posture for vigilance decrement assessment","volume":"14","author":"Rossi","year":"2024","journal-title":"Appl. Sci"},{"key":"B41","doi-asserted-by":"publisher","first-page":"1029","DOI":"10.1016\/j.neuropsychologia.2003.12.012","article-title":"Development of attentional networks in childhood","volume":"42","author":"Rueda","year":"2004","journal-title":"Neuropsychologia"},{"key":"B42","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3225153.3225166","article-title":"\u201cEffects of virtual acoustics on target-word identification performance in multi-talker environments,\u201d","volume-title":"Proceedings of the 15th ACM Symposium on Applied Perception","author":"Rungta","year":"2018"},{"key":"B43","doi-asserted-by":"publisher","first-page":"e19472","DOI":"10.2196\/19472","article-title":"Validation of the raw national aeronautics and space administration task load index (NASA-TLX) questionnaire to assess perceived workload in patient monitoring tasks: pooled analysis study using mixed models","volume":"22","author":"Said","year":"2020","journal-title":"J. Med. Internet Res"},{"key":"B44","doi-asserted-by":"publisher","DOI":"10.1155\/2009\/121494","article-title":"Cognitive load in ecommerce applications-measurement and effects on user satisfaction","author":"Schmutz","year":"2009","journal-title":"Adv. Hum. Comput. Interact"},{"key":"B45","doi-asserted-by":"publisher","first-page":"561","DOI":"10.1016\/j.ijhcs.2012.04.001","article-title":"The immersive virtual environment of the digital fulldome: considerations of relevant psychological processes","volume":"70","author":"Schnall","year":"2012","journal-title":"Int. J. Hum. Comput. Stud"},{"key":"B46","doi-asserted-by":"publisher","first-page":"266","DOI":"10.1162\/105474601300343603","article-title":"The experience of presence: factor analytic insights","volume":"10","author":"Schubert","year":"2001","journal-title":"Presence"},{"key":"B47","doi-asserted-by":"publisher","first-page":"102958","DOI":"10.1016\/j.apergo.2019.102958","article-title":"Psychometric evaluation of simulator sickness questionnaire and its variants as a measure of cybersickness in consumer virtual environments","volume":"82","author":"Sevinc","year":"2020","journal-title":"Appl. Ergon"},{"key":"B48","doi-asserted-by":"publisher","first-page":"182","DOI":"10.1016\/j.tics.2008.02.003","article-title":"Object-based auditory and visual attention","volume":"12","author":"Shinn-Cunningham","year":"2008","journal-title":"Trends Cogn. Sci"},{"key":"B49","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/QoMEX.2017.7965658","article-title":"\u201cMeasuring and comparing qoe and simulator sickness of omnidirectional videos in different head mounted displays,\u201d","volume-title":"2017 Ninth international conference on quality of multimedia experience (QoMEX)","author":"Singla","year":"2017"},{"key":"B50","doi-asserted-by":"publisher","first-page":"19028","DOI":"10.1038\/s41598-024-70026-0","article-title":"Acoustic scene complexity affects motion behavior during speech perception in audio-visual multi-talker virtual environments","volume":"14","author":"Slomianka","year":"2024","journal-title":"Sci. Rep"},{"key":"B51","doi-asserted-by":"publisher","first-page":"389","DOI":"10.3758\/BF03211906","article-title":"On measuring selective attention to an expected sensory modality","volume":"59","author":"Spence","year":"1997","journal-title":"Percept. Psychophys"},{"key":"B52","doi-asserted-by":"publisher","first-page":"20","DOI":"10.1097\/01.HJ.0000558464.75151.52","article-title":"Using virtual reality to assess auditory performance","volume":"72","author":"Stecker","year":"2019","journal-title":"Hear. J"},{"key":"B53","article-title":"\u201cToward objective measures of auditory co-immersion in virtual and augmented reality,\u201d","volume-title":"Audio Engineering Society Conference: 2018 AES International Conference on Audio for Virtual and Augmented Reality","author":"Stecker","year":"2018"},{"key":"B54","doi-asserted-by":"publisher","first-page":"305","DOI":"10.3109\/14992027.2012.759665","article-title":"Evaluation of the preliminary auditory profile test battery in an international multi-centre study","volume":"52","author":"Van Esch","year":"2013","journal-title":"Int. J. Audiol"},{"key":"B55","first-page":"5017","article-title":"\u201cTemporal mismatch effects in short-term memory of audio-visually presented spoken digits,\u201d","author":"Vollmer","year":"2023"},{"key":"B56","doi-asserted-by":"crossref","first-page":"143","DOI":"10.1145\/1978942.1978963","article-title":"\u201cThe aligned rank transform for nonparametric factorial analyses using only ANOVA procedures,\u201d","volume-title":"Proceedings of the SIGCHI conference on human factors in computing systems","author":"Wobbrock","year":"2011"},{"key":"B57","first-page":"107","article-title":"A German version of weinstein's noise sensitivity scale","volume":"44","author":"Zimmer","year":"1997","journal-title":"Z L\u00e4rmbek\u00e4mpfung"}],"container-title":["Frontiers in Psychology"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyg.2025.1520630\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,10]],"date-time":"2025-06-10T05:23:01Z","timestamp":1749532981000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fpsyg.2025.1520630\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,6,10]]},"references-count":57,"alternative-id":["10.3389\/fpsyg.2025.1520630"],"URL":"https:\/\/doi.org\/10.3389\/fpsyg.2025.1520630","relation":{},"ISSN":["1664-1078"],"issn-type":[{"type":"electronic","value":"1664-1078"}],"subject":[],"published":{"date-parts":[[2025,6,10]]},"article-number":"1520630"}}