{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,27]],"date-time":"2026-02-27T06:24:56Z","timestamp":1772173496469,"version":"3.50.1"},"reference-count":25,"publisher":"Public Library of Science (PLoS)","issue":"6","license":[{"start":{"date-parts":[[2025,6,6]],"date-time":"2025-06-06T00:00:00Z","timestamp":1749168000000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Landesgraduiertenf\u00f6rderung Abschlussstipendium"},{"name":"Eric and Wendy Schmidt AI in Science Postdoctoral Fellowship"}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Animals continuously combine information across sensory modalities and time, and use these combined signals to guide their behaviour. Picture a predator watching their prey sprint and screech through a field. To date, a range of multisensory algorithms have been proposed to model this process including linear and nonlinear fusion, which combine the inputs from multiple sensory channels via either a sum or nonlinear function. However, many multisensory algorithms treat successive observations independently, and so cannot leverage the temporal structure inherent to naturalistic stimuli. To investigate this, we introduce a novel multisensory task in which we provide the same number of task-relevant signals per trial but vary how this information is presented: from many short bursts to a few long sequences. We demonstrate that multisensory algorithms that treat different time steps as independent, perform sub-optimally on this task. However, simply augmenting these algorithms to integrate across sensory channels and short temporal windows allows them to perform surprisingly well, and comparably to fully recurrent neural networks. Overall, our work: highlights the benefits of fusing multisensory information across channels and time, shows that small increases in circuit\/model complexity can lead to significant gains in performance, and provides a novel multisensory task for testing the relevance of this in biological systems.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1013125","type":"journal-article","created":{"date-parts":[[2025,6,6]],"date-time":"2025-06-06T13:29:30Z","timestamp":1749216570000},"page":"e1013125","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Fusing multisensory signals across channels and time"],"prefix":"10.1371","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-5976-1717","authenticated-orcid":true,"given":"Swathi","family":"Anil","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1007-6474","authenticated-orcid":true,"given":"Dan F. M.","family":"Goodman","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2428-4605","authenticated-orcid":true,"given":"Marcus","family":"Ghosh","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"340","published-online":{"date-parts":[[2025,6,6]]},"reference":[{"key":"pcbi.1013125.ref001","doi-asserted-by":"crossref","DOI":"10.1093\/acprof:oso\/9780195387247.001.0001","volume-title":"Sensory cue integration","author":"J Trommershauser","year":"2011"},{"issue":"6","key":"pcbi.1013125.ref002","doi-asserted-by":"crossref","first-page":"429","DOI":"10.1038\/nrn3503","article-title":"Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons","volume":"14","author":"CR Fetsch","year":"2013","journal-title":"Nat Rev Neurosci"},{"key":"pcbi.1013125.ref003","doi-asserted-by":"crossref","first-page":"117","DOI":"10.1016\/j.jmp.2016.04.006","article-title":"A tutorial on cue combination and signal detection theory: using changes in sensitivity to evaluate how observers integrate sensory information","volume":"73","author":"PR Jones","year":"2016","journal-title":"J Math Psychol"},{"issue":"4","key":"pcbi.1013125.ref004","doi-asserted-by":"crossref","first-page":"1003","DOI":"10.1037\/0033-295X.111.4.1003","article-title":"A theory of interactive parallel processing: new capacity measures and predictions for a response time inequality series","volume":"111","author":"JT Townsend","year":"2004","journal-title":"Psychol Rev"},{"issue":"15","key":"pcbi.1013125.ref005","doi-asserted-by":"crossref","first-page":"1391","DOI":"10.1016\/j.cub.2012.05.031","article-title":"Noise and correlations in parallel perceptual decision making","volume":"22","author":"TU Otto","year":"2012","journal-title":"Curr Biol"},{"issue":"1","key":"pcbi.1013125.ref006","doi-asserted-by":"crossref","first-page":"146","DOI":"10.1038\/nn.2983","article-title":"Neural correlates of reliability-based cue weighting during multisensory integration","volume":"15","author":"CR Fetsch","year":"2011","journal-title":"Nat Neurosci"},{"key":"pcbi.1013125.ref007","doi-asserted-by":"crossref","DOI":"10.7554\/eLife.03005","article-title":"Optimal multisensory decision-making in a reaction-time task","volume":"3","author":"J Drugowitsch","year":"2014","journal-title":"Elife"},{"issue":"5","key":"pcbi.1013125.ref008","article-title":"Neural correlates of optimal multisensory decision making under time-varying reliabilities with an invariant linear probabilistic population code","volume":"104","author":"H Hou","year":"2019","journal-title":"Neuron"},{"issue":"15","key":"pcbi.1013125.ref009","doi-asserted-by":"crossref","DOI":"10.1016\/j.neuron.2023.05.008","article-title":"Mouse frontal cortex mediates additive multisensory decisions","volume":"111","author":"P Coen","year":"2023","journal-title":"Neuron"},{"key":"pcbi.1013125.ref010","doi-asserted-by":"crossref","first-page":"11543","DOI":"10.1038\/ncomms11543","article-title":"Correlation detection as a general mechanism for multisensory integration","volume":"7","author":"CV Parise","year":"2016","journal-title":"Nat Commun"},{"issue":"7","key":"pcbi.1013125.ref011","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1012246","article-title":"Nonlinear fusion is optimal for a wide class of multisensory tasks","volume":"20","author":"M Ghosh","year":"2024","journal-title":"PLoS Comput Biol"},{"issue":"1","key":"pcbi.1013125.ref012","doi-asserted-by":"crossref","first-page":"9","DOI":"10.1167\/jov.23.1.9","article-title":"Serial dependence in visual perception: a review","volume":"23","author":"D Pascucci","year":"2023","journal-title":"J Vis"},{"issue":"11","key":"pcbi.1013125.ref013","doi-asserted-by":"crossref","first-page":"819","DOI":"10.1016\/j.tics.2017.08.002","article-title":"Serial dependence in audition: Free, fast, and featureless?","volume":"21","author":"BJ Dyson","year":"2017","journal-title":"Trends Cogn Sci"},{"issue":"12","key":"pcbi.1013125.ref014","doi-asserted-by":"crossref","first-page":"4607","DOI":"10.1523\/JNEUROSCI.6079-10.2011","article-title":"Recalibration of auditory space following milliseconds of cross-modal discrepancy","volume":"31","author":"DR Wozny","year":"2011","journal-title":"J Neurosci"},{"issue":"1","key":"pcbi.1013125.ref015","doi-asserted-by":"crossref","first-page":"3742","DOI":"10.1038\/s41598-018-22137-8","article-title":"Trial by trial dependencies in multisensory perception and their correlates in dynamic brain activity","volume":"8","author":"SJ Kayser","year":"2018","journal-title":"Sci Rep"},{"issue":"13","key":"pcbi.1013125.ref016","doi-asserted-by":"crossref","first-page":"20","DOI":"10.1167\/19.13.20","article-title":"Visual serial dependence in an audiovisual stimulus","volume":"19","author":"WK Lau","year":"2019","journal-title":"J Vis"},{"key":"pcbi.1013125.ref017","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1016\/j.neuroscience.2020.04.013","article-title":"Operating in a multisensory context: assessing the interplay between multisensory reaction time facilitation and inter-sensory task-switching effects","volume":"436","author":"LH Shaw","year":"2020","journal-title":"Neuroscience"},{"issue":"5","key":"pcbi.1013125.ref018","doi-asserted-by":"crossref","first-page":"738","DOI":"10.1038\/nn.3689","article-title":"Serial dependence in visual perception","volume":"17","author":"J Fischer","year":"2014","journal-title":"Nat Neurosci"},{"issue":"7","key":"pcbi.1013125.ref019","doi-asserted-by":"crossref","first-page":"493","DOI":"10.1016\/j.tics.2017.04.011","article-title":"Serial dependence across perception, attention, and memory","volume":"21","author":"A Kiyonaga","year":"2017","journal-title":"Trends Cogn Sci"},{"issue":"6756","key":"pcbi.1013125.ref020","doi-asserted-by":"crossref","first-page":"911","DOI":"10.1038\/44831","article-title":"Optimizing the success of random searches","volume":"401","author":"GM Viswanathan","year":"1999","journal-title":"Nature"},{"key":"pcbi.1013125.ref021","volume-title":"Bayesian models of perception and action: An introduction","author":"WJ Ma","year":"2023"},{"key":"pcbi.1013125.ref022","first-page":"2024","article-title":"Multisensory integration across reference frames with additive feed-forward networks","author":"A Farahmandi","year":"2024","journal-title":"bioRxiv"},{"key":"pcbi.1013125.ref023","first-page":"2825","article-title":"Scikit-learn: machine learning in Python","volume":"12","author":"F Pedregosa","year":"2011","journal-title":"J Mach Learn Res"},{"key":"pcbi.1013125.ref024","volume-title":"PyTorch: an imperative style, high-performance deep learning library","author":"A Paszke","year":"2019"},{"key":"pcbi.1013125.ref025","unstructured":"Kingma D, Ba J. Adam: a method for stochastic optimization. In: International Conference on Learning Representations. 2014."}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1013125","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,6,6]],"date-time":"2025-06-06T13:29:39Z","timestamp":1749216579000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1013125"}},"subtitle":[],"editor":[{"given":"Tianming","family":"Yang","sequence":"first","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2025,6,6]]},"references-count":25,"journal-issue":{"issue":"6","published-online":{"date-parts":[[2025,6,6]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1013125","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2024.12.19.629348","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,6,6]]}}}