{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T21:51:38Z","timestamp":1787003498162,"version":"build-2736575974"},"reference-count":35,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2023,1,12]],"date-time":"2023-01-12T00:00:00Z","timestamp":1673481600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2023,1,12]],"date-time":"2023-01-12T00:00:00Z","timestamp":1673481600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100000065","name":"National Institute of Neurological Disorders and Stroke","doi-asserted-by":"publisher","award":["NS033221"],"award-info":[{"award-number":["NS033221"]}],"id":[{"id":"10.13039\/100000065","id-type":"DOI","asserted-by":"publisher"}]},{"name":"NSF Center for Brains, Minds and Machines"},{"DOI":"10.13039\/100005270","name":"McKnight Foundation","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100005270","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Sci Rep"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Extracting meaning from a dynamic and variable flow of incoming information is a major goal of both natural and artificial intelligence. Computer vision (CV) guided by deep learning (DL) has made significant strides in recognizing a specific identity despite highly variable attributes. This is the same challenge faced by the nervous system and partially addressed by the concept cells\u2014neurons exhibiting selective firing in response to specific persons\/places, described in the human medial temporal lobe (MTL) \u2060. Yet, access to neurons representing a particular concept is limited due to these neurons\u2019 sparse coding. It is conceivable, however, that the information required for such decoding is present in relatively small neuronal populations. To evaluate how well neuronal populations encode identity information in natural settings, we recorded neuronal activity from multiple brain regions of nine neurosurgical epilepsy patients implanted with depth electrodes, while the subjects watched an episode of the TV series \u201c24\u201d. First, we devised a minimally supervised CV algorithm (with comparable performance against manually-labeled data) to detect the most prevalent characters (above 1% overall appearance) in each frame. Next, we implemented DL models that used the time-varying population neural data as inputs and decoded the visual presence of the four main characters throughout the episode. This methodology allowed us to compare \u201ccomputer vision\u201d with \u201cneuronal vision\u201d\u2014footprints associated with each character present in the activity of a subset of neurons\u2014and identify the brain regions that contributed to this decoding process. We then tested the DL models during a recognition memory task following movie viewing where subjects were asked to recognize clip segments from the presented episode. DL model activations were not only modulated by the presence of the corresponding characters but also by participants\u2019 subjective memory of whether they had seen the clip segment, and by the associative strengths of the characters in the narrative plot. The described approach can offer novel ways to probe the representation of concepts in time-evolving dynamic behavioral tasks. Further, the results suggest that the information required to robustly decode concepts is present in the population activity of only tens of neurons even in brain regions beyond MTL.<\/jats:p>","DOI":"10.1038\/s41598-022-26946-w","type":"journal-article","created":{"date-parts":[[2023,1,12]],"date-time":"2023-01-12T09:13:01Z","timestamp":1673514781000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":18,"title":["Decoding of human identity by computer vision and neuronal vision"],"prefix":"10.1038","volume":"13","author":[{"given":"Yipeng","family":"Zhang","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zahra M.","family":"Aghajan","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Matias","family":"Ison","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Qiujing","family":"Lu","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Hanlin","family":"Tang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Guldamla","family":"Kalender","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Tonmoy","family":"Monsoor","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jie","family":"Zheng","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gabriel","family":"Kreiman","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Vwani","family":"Roychowdhury","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Itzhak","family":"Fried","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2023,1,12]]},"reference":[{"key":"26946_CR1","doi-asserted-by":"publisher","first-page":"39","DOI":"10.1007\/s10462-018-9633-3","volume":"52","author":"Q Abbas","year":"2019","unstructured":"Abbas, Q., Ibrahim, M. E. A. & Jaffar, M. A. A comprehensive review of recent advances on deep vision systems. Artif. Intell. Rev. 52, 39\u201376 (2019).","journal-title":"Artif. Intell. Rev."},{"key":"26946_CR2","doi-asserted-by":"crossref","unstructured":"Ranjan, R. et al. Deep learning for understanding faces: machines may be just as good, or better, than humans. IEEE Signal Process. Mag. 35(1), 66\u201383 (2018).","DOI":"10.1109\/MSP.2017.2764116"},{"key":"26946_CR3","doi-asserted-by":"publisher","first-page":"1102","DOI":"10.1038\/nature03687","volume":"435","author":"RQ Quiroga","year":"2005","unstructured":"Quiroga, R. Q., Reddy, L., Kreiman, G., Koch, C. & Fried, I. Invariant visual representation by single neurons in the human brain. Nature 435, 1102\u20131107 (2005).","journal-title":"Nature"},{"key":"26946_CR4","doi-asserted-by":"publisher","first-page":"1997","DOI":"10.1152\/jn.00125.2007","volume":"98","author":"RQ Quiroga","year":"2007","unstructured":"Quiroga, R. Q., Reddy, L., Koch, C. & Fried, I. Decoding visual inputs from multiple neurons in the human temporal lobe. J. Neurophysiol. 98, 1997\u20132007 (2007).","journal-title":"J. Neurophysiol."},{"key":"26946_CR5","doi-asserted-by":"publisher","first-page":"96","DOI":"10.1126\/science.1164685","volume":"322","author":"H Gelbard-Sagiv","year":"2008","unstructured":"Gelbard-Sagiv, H., Mukamel, R., Harel, M., Malach, R. & Fried, I. Internally generated reactivation of single neurons in human hippocampus during free recall. Science 322, 96\u2013101 (2008).","journal-title":"Science"},{"key":"26946_CR6","doi-asserted-by":"publisher","first-page":"587","DOI":"10.1038\/nrn3251","volume":"13","author":"RQ Quiroga","year":"2012","unstructured":"Quiroga, R. Q. Concept cells: The building blocks of declarative memory functions. Nat. Rev. Neurosci. 13, 587\u2013597 (2012).","journal-title":"Nat. Rev. Neurosci."},{"key":"26946_CR7","doi-asserted-by":"publisher","first-page":"30175","DOI":"10.1038\/srep30175","volume":"6","author":"H Tang","year":"2016","unstructured":"Tang, H. et al. Predicting episodic memory formation for movie events. Sci. Rep. 6, 30175 (2016).","journal-title":"Sci. Rep."},{"key":"26946_CR8","doi-asserted-by":"publisher","first-page":"1661","DOI":"10.1162\/089976604774201631","volume":"16","author":"RQ Quiroga","year":"2004","unstructured":"Quiroga, R. Q., Nadasdy, Z. & Ben-Shaul, Y. Unsupervised spike detection and sorting with wavelets and superparamagnetic clustering. Neural Comput. 16, 1661\u20131687 (2004).","journal-title":"Neural Comput."},{"key":"26946_CR9","doi-asserted-by":"crossref","unstructured":"Ison, M. J. J., Quian\u00a0Quiroga, R., Fried, I., Quian Quiroga, R. & Fried, I. Rapid encoding of new memories by individual neurons in the human brain. Neuron 87, 220\u2013230 (2015).","DOI":"10.1016\/j.neuron.2015.06.016"},{"key":"26946_CR10","doi-asserted-by":"publisher","first-page":"10503","DOI":"10.1073\/pnas.1423036112","volume":"112","author":"NA Suthana","year":"2015","unstructured":"Suthana, N. A. et al. Specific responses of human hippocampal neurons are associated with better memory. Proc. Natl. Acad. Sci. U. S. A. 112, 10503\u201310508 (2015).","journal-title":"Proc. Natl. Acad. Sci. U. S. A."},{"key":"26946_CR11","unstructured":"Redmon, J. & Farhadi, A. YOLOv3: An Incremental Improvement. (2018). http:\/\/arxiv.org\/abs\/1804.02767."},{"key":"26946_CR12","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","volume":"9","author":"S Hochreiter","year":"1997","unstructured":"Hochreiter, S. & Schmidhuber, J. Long Short-Term Memory. Neural Comput. 9, 1735\u20131780 (1997).","journal-title":"Neural Comput."},{"key":"26946_CR13","doi-asserted-by":"crossref","unstructured":"Zeiler, M. D. & Fergus, R. Visualizing and understanding convolutional networks. In European Conference on Computer Vision (ECCV) 818\u2013833 (2014).","DOI":"10.1007\/978-3-319-10590-1_53"},{"key":"26946_CR14","doi-asserted-by":"crossref","unstructured":"Davachi, L. Item, context and relational episodic encoding in humans. Curr. Opin. Neurol. 16(6), 693\u2013700.(2006).","DOI":"10.1016\/j.conb.2006.10.012"},{"key":"26946_CR15","doi-asserted-by":"crossref","unstructured":"Eichenbaum, H., Yonelinas, A. P. & Ranganath, C. The medial temporal lobe and recognition memory. Ann. Rev. Neurosci. 30, 123 (2007).","DOI":"10.1146\/annurev.neuro.30.051606.094328"},{"key":"26946_CR16","doi-asserted-by":"publisher","first-page":"279","DOI":"10.1146\/annurev.neuro.27.070203.144130","volume":"27","author":"LR Squire","year":"2004","unstructured":"Squire, L. R., Stark, C. E. L. & Clark, R. E. The medial temporal lobe. Annu. Rev. Neurosci. 27, 279\u2013306 (2004).","journal-title":"Annu. Rev. Neurosci."},{"key":"26946_CR17","doi-asserted-by":"crossref","unstructured":"Bettencourt, L. M. The rules of information aggregation and emergence of collective intelligent behavior. Top. Cogn. Sci. 1(4), 598\u2013620 (2009).","DOI":"10.1111\/j.1756-8765.2009.01047.x"},{"key":"26946_CR18","doi-asserted-by":"publisher","DOI":"10.1016\/J.TINS.2020.09.006","author":"SE Favila","year":"2020","unstructured":"Favila, S. E., Lee, H. & Kuhl, B. A. Transforming the concept of memory reactivation. Trends Neurosci. https:\/\/doi.org\/10.1016\/J.TINS.2020.09.006 (2020).","journal-title":"Trends Neurosci."},{"key":"26946_CR19","doi-asserted-by":"publisher","first-page":"8871","DOI":"10.1523\/JNEUROSCI.5321-11.2012","volume":"32","author":"JR Manning","year":"2012","unstructured":"Manning, J. R., Sperling, M. R., Sharan, A., Rosenberg, E. A. & Kahana, M. J. Spontaneously reactivated patterns in frontal and temporal lobe predict semantic clustering during memory search. J. Neurosci. 32, 8871\u20138878 (2012).","journal-title":"J. Neurosci."},{"key":"26946_CR20","doi-asserted-by":"publisher","first-page":"1111","DOI":"10.1126\/science.1244056","volume":"342","author":"JF Miller","year":"2013","unstructured":"Miller, J. F. et al. Neural activity in human hippocampal formation reveals the spatial context of retrieved memories. Science 342, 1111\u20131114 (2013).","journal-title":"Science"},{"key":"26946_CR21","doi-asserted-by":"publisher","first-page":"2000","DOI":"10.1162\/jocn_a_00839","volume":"27","author":"M St-Laurent","year":"2015","unstructured":"St-Laurent, M., Abdi, H. & Buchsbaum, B. R. Distributed patterns of reactivation predict vividness of recollection. J. Cogn. Neurosci. 27, 2000\u20132018 (2015).","journal-title":"J. Cogn. Neurosci."},{"key":"26946_CR22","doi-asserted-by":"publisher","first-page":"3350","DOI":"10.1093\/cercor\/bht194","volume":"24","author":"AM Gordon","year":"2014","unstructured":"Gordon, A. M., Rissman, J., Kiani, R. & Wagner, A. D. Cortical reinstatement mediates the relationship between content-specific encoding activity and subsequent recollection decisions. Cereb. Cortex 24, 3350\u20133364 (2014).","journal-title":"Cereb. Cortex"},{"key":"26946_CR23","doi-asserted-by":"crossref","unstructured":"Kahana, M. J., Howard, M. W. & Polyn, S. M. Associative retrieval processes in episodic memory. (2008).","DOI":"10.1016\/B978-012370509-9.00185-6"},{"issue":"10","key":"26946_CR24","doi-asserted-by":"publisher","first-page":"2017","DOI":"10.1162\/jocn_a_01544","volume":"33","author":"GW Lindsay","year":"2021","unstructured":"Lindsay, G. W. Convolutional neural networks as a model of the visual system: Past, present, and future. J. Cogn. Neurosci. 33(10), 2017\u20132031 (2021).","journal-title":"J. Cogn. Neurosci."},{"issue":"1","key":"26946_CR25","doi-asserted-by":"publisher","first-page":"222","DOI":"10.1111\/nyas.14320","volume":"1464","author":"G Kreiman","year":"2020","unstructured":"Kreiman, G. & Serre, T. Beyond the feedforward sweep: Feedback computations in the visual cortex. Ann. N. Y. Acad. Sci. 1464(1), 222\u2013241 (2020).","journal-title":"Ann. N. Y. Acad. Sci."},{"issue":"1","key":"26946_CR26","doi-asserted-by":"publisher","first-page":"399","DOI":"10.1146\/annurev-vision-091718-014951","volume":"5","author":"T Serre","year":"2019","unstructured":"Serre, T. Deep learning: The good, the bad, and the ugly. Ann. Rev. Vis. Sci. 5(1), 399\u2013426 (2019).","journal-title":"Ann. Rev. Vis. Sci."},{"key":"26946_CR27","doi-asserted-by":"crossref","unstructured":"Goh, G., et al. Multimodal neurons in artificial neural networks. Distill 6.3, e30 (2021).","DOI":"10.23915\/distill.00030"},{"key":"26946_CR28","unstructured":"Radford, A., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR (2021)."},{"key":"26946_CR29","doi-asserted-by":"publisher","first-page":"3743","DOI":"10.1016\/j.cub.2017.10.062","volume":"27","author":"ZM Aghajan","year":"2017","unstructured":"Aghajan, Z. M. et al. Theta oscillations in the human medial temporal lobe during real-world ambulatory movement. Curr. Biol. 27, 3743-3751e3 (2017).","journal-title":"Curr. Biol."},{"key":"26946_CR30","unstructured":"Castellano, B. PySceneDetect v0.5.5 Manual \u2014 PySceneDetect v0.5.5 documentation. Available at: https:\/\/pyscenedetect-manual.readthedocs.io\/en\/latest\/."},{"key":"26946_CR31","doi-asserted-by":"crossref","unstructured":"Wojke, N., Bewley, A. & Paulus, D. Simple online and realtime tracking with a deep association metric. In 2017 IEEE International Conference on Image Processing (ICIP) 3645\u20133649 (2017).","DOI":"10.1109\/ICIP.2017.8296962"},{"key":"26946_CR32","doi-asserted-by":"crossref","unstructured":"Schroff, F., Kalenichenko, D. & Philbin, J. FaceNet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 815\u2013823 (2015).","DOI":"10.1109\/CVPR.2015.7298682"},{"key":"26946_CR33","doi-asserted-by":"crossref","unstructured":"He, K., Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition, In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 770\u2013778 (2016).","DOI":"10.1109\/CVPR.2016.90"},{"issue":"2","key":"26946_CR34","first-page":"11045","volume":"4","author":"T Arnold","year":"2019","unstructured":"Arnold, T., Tilton, L. & Berke, A. Visual style in two network era sitcoms. J. Cult. Anal. 4(2), 11045 (2019).","journal-title":"J. Cult. Anal."},{"key":"26946_CR35","doi-asserted-by":"crossref","unstructured":"Tampuu, A., Matiisen, T., \u00d3lafsd\u00f3ttir, H. F., Barry, C. & Vicente, R. Efficient neural decoding of self-location with a deep recurrent network. PLoS Comput. Biol. 15(2), e1006822 (2019).","DOI":"10.1371\/journal.pcbi.1006822"}],"container-title":["Scientific Reports"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-26946-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-26946-w","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-26946-w.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,12]],"date-time":"2023-01-12T09:18:54Z","timestamp":1673515134000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s41598-022-26946-w"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,1,12]]},"references-count":35,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2023,12]]}},"alternative-id":["26946"],"URL":"https:\/\/doi.org\/10.1038\/s41598-022-26946-w","relation":{},"ISSN":["2045-2322"],"issn-type":[{"value":"2045-2322","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,1,12]]},"assertion":[{"value":"8 February 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"22 December 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 January 2023","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"651"}}