{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,10]],"date-time":"2026-03-10T14:20:56Z","timestamp":1773152456097,"version":"3.50.1"},"reference-count":29,"publisher":"Frontiers Media SA","license":[{"start":{"date-parts":[[2023,5,2]],"date-time":"2023-05-02T00:00:00Z","timestamp":1682985600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":["frontiersin.org"],"crossmark-restriction":true},"short-container-title":["Front. Neurosci."],"abstract":"<jats:p>The visual system provides a valuable model for studying the working mechanisms of sensory processing and high-level consciousness. A significant challenge in this field is the reconstruction of images from decoded neural activity, which could not only test the accuracy of our understanding of the visual system but also provide a practical tool for solving real-world problems. Although recent advances in deep learning have improved the decoding of neural spike trains, little attention has been paid to the underlying mechanisms of the visual system. To address this issue, we propose a deep learning neural network architecture that incorporates the biological properties of the visual system, such as receptive fields, to reconstruct visual images from spike trains. Our model outperforms current models and has been evaluated on different datasets from both retinal ganglion cells (RGCs) and the primary visual cortex (V1) neural spikes. Our model demonstrated the great potential of brain-inspired algorithms to solve a challenge that our brain solves.<\/jats:p>","DOI":"10.3389\/fnins.2023.1130606","type":"journal-article","created":{"date-parts":[[2023,5,2]],"date-time":"2023-05-02T04:30:56Z","timestamp":1683001856000},"update-policy":"https:\/\/doi.org\/10.3389\/crossmark-policy","source":"Crossref","is-referenced-by-count":10,"title":["The brain-inspired decoder for natural visual image reconstruction"],"prefix":"10.3389","volume":"17","author":[{"given":"Wenyi","family":"Li","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Shengjie","family":"Zheng","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Yufan","family":"Liao","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Rongqi","family":"Hong","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chenggang","family":"He","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Weiliang","family":"Chen","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chunshan","family":"Deng","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Xiaojian","family":"Li","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"1965","published-online":{"date-parts":[[2023,5,2]]},"reference":[{"key":"B1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.1910.09120","article-title":"Unsupervised decoding of spinal motor neuron spike trains for estimating hand kinematics following targeted muscle reinnervation","author":"Andalib","year":"2019","journal-title":"arXiv preprint arXiv:1910.09120"},{"key":"B2","doi-asserted-by":"publisher","first-page":"137","DOI":"10.1016\/S0020-7373(81)80001-6","article-title":"Mechanisms of human facial recognition","volume":"15","author":"Baron","year":"1981","journal-title":"Int. J. Man Machine Stud."},{"key":"B3","doi-asserted-by":"publisher","first-page":"022501","DOI":"10.1088\/2634-4386\/ac4a83","article-title":"2022 roadmap on neuromorphic computing and engineering","volume":"2","author":"Christensen","year":"2022","journal-title":"Neuromorphic Comput. Eng."},{"key":"B4","doi-asserted-by":"publisher","first-page":"1648","DOI":"10.1038\/nn.4128","article-title":"Flexible gating of contextual influences in natural vision","volume":"18","author":"Coen-Cagli","year":"2015","journal-title":"Nat. Neurosci."},{"key":"B5","doi-asserted-by":"publisher","first-page":"776","DOI":"10.1093\/cercor\/bhr133","article-title":"Representation of surface luminance and contrast in primary visual cortex","volume":"22","author":"Dai","year":"2012","journal-title":"Cereb. Cortex"},{"key":"B6","doi-asserted-by":"publisher","first-page":"119121","DOI":"10.1016\/j.neuroimage.2022.119121","article-title":"Self-supervised natural image reconstruction and large-scale semantic classification from brain activity","volume":"254","author":"Gaziv","year":"2022","journal-title":"Neuroimage"},{"key":"B7","doi-asserted-by":"publisher","first-page":"649","DOI":"10.1146\/annurev.neuro.27.070203.144220","article-title":"The human visual cortex","volume":"27","author":"Grill-Spector","year":"2004","journal-title":"Annu. Rev. Neurosci."},{"key":"B8","doi-asserted-by":"publisher","first-page":"127","DOI":"10.1016\/j.inffus.2011.08.002","article-title":"A new image fusion performance metric based on visual information fidelity","volume":"14","author":"Han","year":"2013","journal-title":"Inform. Fusion"},{"key":"B9","doi-asserted-by":"crossref","first-page":"105","DOI":"10.1109\/SMC.2018.00028","article-title":"\u201cImage reconstruction from neural activity recorded from monkey inferior temporal cortex using generative adversarial networks,\u201d","volume-title":"2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC)","author":"Hayashi","year":"2018"},{"key":"B10","doi-asserted-by":"publisher","first-page":"504","DOI":"10.1126\/science.1127647","article-title":"Reducing the dimensionality of data with neural networks","volume":"313","author":"Hinton","year":"2006","journal-title":"Science"},{"key":"B11","doi-asserted-by":"publisher","first-page":"352","DOI":"10.1038\/nature06713","article-title":"Identifying natural images from human brain activity","volume":"452","author":"Kay","year":"2008","journal-title":"Nature"},{"key":"B12","doi-asserted-by":"publisher","first-page":"15164","DOI":"10.1007\/s10489-022-03402-w","article-title":"Fusion of ANNs as decoder of retinal spike trains for scene reconstruction","volume":"52","author":"Li","year":"2022","journal-title":"Appl. Intell."},{"key":"B13","doi-asserted-by":"publisher","first-page":"4357","DOI":"10.1109\/TIP.2018.2835143","article-title":"Gabor convolutional networks","volume":"27","author":"Luan","year":"2018","journal-title":"IEEE Trans. Image Process."},{"key":"B14","unstructured":"\u201cUnderstanding the effective receptive field in deep convolutional neural networks,\u201d49054913\n            LuoW.\n            LiY.\n            UrtasunR.\n            ZemelR.\n          Advances in Neural Information Processing Systems2016"},{"key":"B15","doi-asserted-by":"publisher","first-page":"915","DOI":"10.1016\/j.neuron.2008.11.004","article-title":"Visual image reconstruction from human brain activity using a combination of multiscale local image decoders","volume":"60","author":"Miyawaki","year":"2008","journal-title":"Neuron"},{"key":"B16","doi-asserted-by":"crossref","first-page":"157","DOI":"10.1109\/EAIT.2012.6407885","article-title":"\u201cRetinal vessel segmentation using gabor filter and artificial neural network,\u201d","volume-title":"2012 Third International Conference on Emerging Applications of Information Technology","author":"Nandy","year":"2012"},{"key":"B17","doi-asserted-by":"publisher","first-page":"1641","DOI":"10.1016\/j.cub.2011.08.031","article-title":"Reconstructing visual experiences from brain activity evoked by natural movies","volume":"21","author":"Nishimoto","year":"2011","journal-title":"Curr. Biol."},{"key":"B18","doi-asserted-by":"publisher","first-page":"e1005189","DOI":"10.1371\/journal.pcbi.1005189","article-title":"Using matrix and tensor factorizations for the single-trial analysis of population spike trains","volume":"12","author":"Onken","year":"2016","journal-title":"PLoS Comput. Biol."},{"key":"B19","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2111.15309","article-title":"Deep auto-encoder with neural response","author":"Ran","year":"2021","journal-title":"arXiv preprint arXiv:2111.15309"},{"key":"B20","doi-asserted-by":"publisher","first-page":"e1005649","DOI":"10.1371\/journal.pcbi.1005649","article-title":"Decoding brain activity using a large-scale probabilistic functional-anatomical atlas of human cognition","volume":"13","author":"Rubin","year":"2017","journal-title":"PLoS Comput. Biol."},{"key":"B21","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1111\/1467-9280.01411","article-title":"Emotional facilitation of sensory processing in the visual cortex","volume":"14","author":"Schupp","year":"2003","journal-title":"Psychol. Sci."},{"key":"B22","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1007\/978-1-4614-7320-6_345-1","article-title":"Hierarchical models of the visual system","volume":"6","author":"Serre","year":"2014","journal-title":"Encyclop. Comput. Neurosci."},{"key":"B23","doi-asserted-by":"publisher","first-page":"1193","DOI":"10.1146\/annurev.neuro.24.1.1193","article-title":"Natural image statistics and neural representation","volume":"24","author":"Simoncelli","year":"2001","journal-title":"Annu. Rev. Neurosci."},{"key":"B24","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1038\/s41467-022-29999-7","article-title":"On\/off domains shape receptive field structure in mouse visual cortex","volume":"13","author":"Tring","year":"2022","journal-title":"Nat. Commun."},{"key":"B25","doi-asserted-by":"publisher","DOI":"10.1101\/2021.12.30.474591","article-title":"V1 receptive field structure contributes to neuronal response latency","author":"Vafaei","year":"2022","journal-title":"bioRxiv"},{"key":"B26","doi-asserted-by":"publisher","first-page":"600","DOI":"10.1109\/TIP.2003.819861","article-title":"Image quality assessment: from error visibility to structural similarity","volume":"13","author":"Wang","year":"2004","journal-title":"IEEE Trans. Image Process."},{"key":"B27","doi-asserted-by":"publisher","first-page":"477","DOI":"10.1146\/annurev.neuro.29.051605.113024","article-title":"Complete functional characterization of sensory neurons by system identification","volume":"29","author":"Wu","year":"2006","journal-title":"Annu. Rev. Neurosci."},{"key":"B28","doi-asserted-by":"publisher","first-page":"772","DOI":"10.1126\/science.277.5327.772a","article-title":"Spikes: exploring the neural code","volume":"277","author":"Zador","year":"1997","journal-title":"Science"},{"key":"B29","doi-asserted-by":"publisher","first-page":"19","DOI":"10.1016\/j.neunet.2020.01.033","article-title":"Reconstruction of natural visual scenes from neural spikes with deep neural networks","volume":"125","author":"Zhang","year":"2020","journal-title":"Neural Netw."}],"container-title":["Frontiers in Neuroscience"],"original-title":[],"link":[{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnins.2023.1130606\/full","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,5,2]],"date-time":"2023-05-02T04:31:03Z","timestamp":1683001863000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.frontiersin.org\/articles\/10.3389\/fnins.2023.1130606\/full"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,5,2]]},"references-count":29,"alternative-id":["10.3389\/fnins.2023.1130606"],"URL":"https:\/\/doi.org\/10.3389\/fnins.2023.1130606","relation":{},"ISSN":["1662-453X"],"issn-type":[{"value":"1662-453X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,5,2]]},"article-number":"1130606"}}