{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,11,11]],"date-time":"2025-11-11T02:12:34Z","timestamp":1762827154505,"version":"build-2065373602"},"reference-count":36,"publisher":"World Scientific Pub Co Pte Ltd","issue":"15","funder":[{"name":"National Key Research and Development Program of China","award":["2022YFF1202903"],"award-info":[{"award-number":["2022YFF1202903"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Int. J. Patt. Recogn. Artif. Intell."],"published-print":{"date-parts":[[2025,12,15]]},"abstract":"<jats:p>Visual information is first encoded into spike trains by retinal ganglion cells (RGCs), forming the foundation of biological vision. Decoding the semantic and perceptual content embedded in these spikes remains a key challenge in neuroscience. We introduce a novel framework that leverages recent advances in artificial intelligence to interpret spike-based visual signals. Our framework consists of spike-driven multimodal large language model (MLLM) and a spike-driven ControlNet to emulate two core functions of the visual cortex: semantic understanding and image reconstruction. By bridging biological and computational vision, our framework enables effective decoding of spike signals generated by RGCs. We further validate the model on the spike-driven MLLM on the MSCOCO test set. The experiment results demonstrate that our framework not only enhances interpretability and usability of spike-driven outputs but also holds promise as a tool to support neuroscience research and aid visually impaired individuals. Participant feedback highlights its potential to facilitate future advances in neural decoding and vision restoration technologies.<\/jats:p>","DOI":"10.1142\/s021800142552024x","type":"journal-article","created":{"date-parts":[[2025,8,13]],"date-time":"2025-08-13T03:26:38Z","timestamp":1755055598000},"source":"Crossref","is-referenced-by-count":0,"title":["Toward a Deep Understanding of Retinal Spike Trains via MLLMs and Diffusion Models"],"prefix":"10.1142","volume":"39","author":[{"ORCID":"https:\/\/orcid.org\/0009-0004-9888-9983","authenticated-orcid":false,"given":"Shanshan","family":"Zhao","sequence":"first","affiliation":[{"name":"Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, Guangdong 518055, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Chenxi","family":"Qin","sequence":"additional","affiliation":[{"name":"Academy of Medical Engineering and Translational Medicine, Medical College, Tianjin University, Tianjin 300072, P. R. China"},{"name":"Haihe Laboratory of Brain-Computer Interaction and Human-Machine, Integration Tianjin 300051, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Jie","family":"Wu","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering at Southern, University of Science and Technology, Sustech P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Pengxiang","family":"Li","sequence":"additional","affiliation":[{"name":"Academy of Medical Engineering and Translational Medicine, Medical College, Tianjin University, Tianjin 300072, P. R. China"},{"name":"Haihe Laboratory of Brain-Computer Interaction and Human-Machine, Integration Tianjin 300051, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Liqun","family":"Chen","sequence":"additional","affiliation":[{"name":"Academy of Medical Engineering and Translational Medicine, Medical College, Tianjin University, Tianjin 300072, P. R. China"},{"name":"Haihe Laboratory of Brain-Computer Interaction and Human-Machine, Integration Tianjin 300051, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Wenwei","family":"Shao","sequence":"additional","affiliation":[{"name":"Academy of Medical Engineering and Translational Medicine, Medical College, Tianjin University, Tianjin 300072, P. R. China"},{"name":"Haihe Laboratory of Brain-Computer Interaction and Human-Machine, Integration Tianjin 300051, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-1701-9141","authenticated-orcid":false,"given":"Feng","family":"Zheng","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, Guangdong 518055, P. R. China"}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"219","published-online":{"date-parts":[[2025,10,22]]},"reference":[{"key":"S021800142552024XBIB002","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-319-46454-1_24"},{"key":"S021800142552024XBIB003","doi-asserted-by":"publisher","DOI":"10.1016\/j.clinph.2019.11.029"},{"key":"S021800142552024XBIB006","first-page":"251584141881750","volume":"11","author":"Bloch E.","year":"2019","journal-title":"Ther. Adv. Ophthalmol."},{"key":"S021800142552024XBIB007","doi-asserted-by":"publisher","DOI":"10.3389\/fninf.2017.00049"},{"key":"S021800142552024XBIB009","first-page":"24841","volume":"36","author":"Chen Z.","year":"2024","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"S021800142552024XBIB010","doi-asserted-by":"publisher","DOI":"10.1088\/1741-2552\/ac0679"},{"key":"S021800142552024XBIB011","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2016.06.049"},{"key":"S021800142552024XBIB013","doi-asserted-by":"publisher","DOI":"10.3115\/v1\/W14-3348"},{"key":"S021800142552024XBIB014","first-page":"9907","volume":"36","author":"Duan Y.","year":"2024","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"S021800142552024XBIB015","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2317773121"},{"key":"S021800142552024XBIB016","doi-asserted-by":"publisher","DOI":"10.1038\/nrn3783"},{"key":"S021800142552024XBIB017","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2015.04.032"},{"key":"S021800142552024XBIB018","first-page":"6840","volume-title":"Proc. 34th Int. Conf. Neural Information Processing Systems","volume":"33","author":"Ho J.","year":"2020"},{"key":"S021800142552024XBIB021","doi-asserted-by":"publisher","DOI":"10.1016\/j.ophtha.2011.09.028"},{"key":"S021800142552024XBIB022","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-vision-100820-114239"},{"key":"S021800142552024XBIB023","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7298932"},{"key":"S021800142552024XBIB024","doi-asserted-by":"publisher","DOI":"10.1007\/s10489-022-03402-w"},{"key":"S021800142552024XBIB026","first-page":"74","volume-title":"Text Summarization Branches Out","author":"Lin C.-Y.","year":"2004"},{"key":"S021800142552024XBIB027","first-page":"34892","volume-title":"Proc. 37th Int. Conf. Neural Information Processing Systems","author":"Liu H.","year":"2023"},{"key":"S021800142552024XBIB028","first-page":"75740","volume":"36","author":"Luo A.","year":"2024","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"S021800142552024XBIB029","doi-asserted-by":"publisher","DOI":"10.1007\/s40747-023-01333-8"},{"key":"S021800142552024XBIB030","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2023.06.007"},{"key":"S021800142552024XBIB031","first-page":"311","volume-title":"Proc. 40th Annual Meeting of the Association for Computational Linguistics","author":"Papineni K.","year":"2002"},{"key":"S021800142552024XBIB032","volume":"30","author":"Parthasarathy N.","year":"2017","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"S021800142552024XBIB033","first-page":"7","volume-title":"Proc. Twelfth Int. Conf. Learning Representations","author":"Prince J. S.","year":"2024"},{"key":"S021800142552024XBIB034","first-page":"8748","volume-title":"Proc. Int. Conf. Machine Learning","author":"Radford A.","year":"2021"},{"key":"S021800142552024XBIB035","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-023-06031-6"},{"key":"S021800142552024XBIB036","first-page":"24705\u201324728","volume":"36","author":"Scotti P.","year":"2024","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"S021800142552024XBIB037","doi-asserted-by":"publisher","DOI":"10.1109\/WACV57701.2024.00738"},{"key":"S021800142552024XBIB039","doi-asserted-by":"publisher","DOI":"10.1038\/s41593-023-01304-9"},{"key":"S021800142552024XBIB040","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR.2015.7299087"},{"key":"S021800142552024XBIB041","doi-asserted-by":"publisher","DOI":"10.1093\/cercor\/bhab498"},{"key":"S021800142552024XBIB042","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.acl-long.741"},{"key":"S021800142552024XBIB043","doi-asserted-by":"publisher","DOI":"10.1109\/CVPR52733.2024.02173"},{"key":"S021800142552024XBIB044","doi-asserted-by":"publisher","DOI":"10.1109\/TNNLS.2024.3351120"},{"key":"S021800142552024XBIB045","doi-asserted-by":"publisher","DOI":"10.1109\/ICCV51070.2023.00355"}],"container-title":["International Journal of Pattern Recognition and Artificial Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.worldscientific.com\/doi\/pdf\/10.1142\/S021800142552024X","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,11]],"date-time":"2025-11-11T02:05:56Z","timestamp":1762826756000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.worldscientific.com\/doi\/10.1142\/S021800142552024X"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,10,22]]},"references-count":36,"journal-issue":{"issue":"15","published-print":{"date-parts":[[2025,12,15]]}},"alternative-id":["10.1142\/S021800142552024X"],"URL":"https:\/\/doi.org\/10.1142\/s021800142552024x","relation":{},"ISSN":["0218-0014","1793-6381"],"issn-type":[{"type":"print","value":"0218-0014"},{"type":"electronic","value":"1793-6381"}],"subject":[],"published":{"date-parts":[[2025,10,22]]},"article-number":"2552024"}}