{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T18:25:41Z","timestamp":1787250341316,"version":"build-2736575974"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1014656","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T00:00:00Z","timestamp":1787184000000}}],"reference-count":59,"publisher":"Public Library of Science (PLoS)","issue":"8","license":[{"start":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T00:00:00Z","timestamp":1786924800000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100000006","name":"Office of Naval Research","doi-asserted-by":"publisher","award":["N00014-24-1-2259"],"award-info":[{"award-number":["N00014-24-1-2259"]}],"id":[{"id":"10.13039\/100000006","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100002369","name":"University of Delaware Research Foundation","doi-asserted-by":"publisher","id":[{"id":"10.13039\/100002369","id-type":"DOI","asserted-by":"publisher"}]},{"name":"University of Delaware Electrical and Computer Engineering Department","award":["None"],"award-info":[{"award-number":["None"]}]},{"name":"University of Delaware College of Engineering","award":["None"],"award-info":[{"award-number":["None"]}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Predicting the neural response to natural images in the visual cortex requires extracting relevant features from the images and relating those feature to the observed responses. In this work, we optimize the feature extraction in order to maximize the information shared between the image features and the neural response across voxels in a given region of interest (ROI) extracted from the BOLD signal measured by functional magnetic resonance imaging (fMRI). We adapt contrastive learning (CL) to fine-tune a convolutional neural network, which was pretrained for image classification, such that a mapping of a given image\u2019s features are more similar to the corresponding fMRI response than to the responses to other images. We exploit the Natural Scenes Dataset as organized for the Algonauts Project, which contains the high-resolution fMRI responses of eight subjects to tens of thousands of naturalistic images. We show that CL fine-tuning creates feature extraction models that enable higher encoding accuracy in both early and higher visual ROIs as compared to the features from the pretrained network. Quantitatively, the performance is similar to a baseline approach that directly uses a regression loss at the output of the network to tune it for fMRI response encoding. We investigate inter-subject transfer of the CL fine-tuned models, including subjects from the Natural Object Dataset, another lower-resolution dataset with 9 subjects. We also pool subjects for fine-tuning, which further improves encoding performance in early ROIs. Finally, we examine the performance of the fine-tuned models on common image classification tasks, explore the landscape of ROI-specific models by applying dimensionality reduction on the Bhattacharya dissimilarity matrix created using the predictions on those tasks, and show that these landscapes match those based on representational similarity analysis. Finally, we generate images via Stable Diffusion based on vector-space prompts created by aligning the CL-tuned models embeddings for different ROIs, showing that generated images have similar embeddings to the original but that estimates of the intrinsic dimension are lower for generated versus original representations.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1014656","type":"journal-article","created":{"date-parts":[[2026,8,17]],"date-time":"2026-08-17T17:37:39Z","timestamp":1786988259000},"page":"e1014656","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":0,"title":["Contrastive learning to fine-tune feature extraction models for the visual cortex"],"prefix":"10.1371","volume":"22","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-3086-3307","authenticated-orcid":true,"given":"Alex","family":"Mulrooney","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhi","family":"Li","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Austin J.","family":"Brockmeier","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2026,8,17]]},"reference":[{"key":"pcbi.1014656.ref001","doi-asserted-by":"crossref","unstructured":"Scotti PS, Banerjee A, Goode J, Shabalin S, Nguyen A, Cohen E, et al. Reconstructing the Mind\u2019s Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors. 2023.","DOI":"10.52202\/075280-1073"},{"key":"pcbi.1014656.ref002","doi-asserted-by":"crossref","first-page":"29654","DOI":"10.52202\/075280-1290","article-title":"Brain encoding models based on multimodal transformers can transfer across language and vision","volume":"36","author":"J Tang","year":"2023","journal-title":"Adv Neural Inf Process Syst"},{"issue":"1","key":"pcbi.1014656.ref003","doi-asserted-by":"crossref","first-page":"116","DOI":"10.1038\/s41593-021-00962-x","article-title":"A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence","volume":"25","author":"EJ Allen","year":"2022","journal-title":"Nat Neurosci"},{"key":"pcbi.1014656.ref004","unstructured":"Chen T, Kornblith S, Norouzi M, Hinton G. A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning. PMLR; 2020. p. 1597\u2013607."},{"key":"pcbi.1014656.ref005","unstructured":"Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, et al. Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning. PMLR; 2021. p. 8748\u20138763."},{"issue":"7185","key":"pcbi.1014656.ref006","doi-asserted-by":"crossref","first-page":"352","DOI":"10.1038\/nature06713","article-title":"Identifying natural images from human brain activity","volume":"452","author":"KN Kay","year":"2008","journal-title":"Nature"},{"issue":"5880","key":"pcbi.1014656.ref007","doi-asserted-by":"crossref","first-page":"1191","DOI":"10.1126\/science.1152876","article-title":"Predicting human brain activity associated with the meanings of nouns","volume":"320","author":"TM Mitchell","year":"2008","journal-title":"Science"},{"issue":"12","key":"pcbi.1014656.ref008","article-title":"The training process of many deep networks explores the same low-dimensional manifold","volume":"121","author":"J Mao","year":"2024","journal-title":"Proc Natl Acad Sci U S A"},{"key":"pcbi.1014656.ref009","article-title":"Imagenet classification with deep convolutional neural networks","volume":"25","author":"A Krizhevsky","year":"2012","journal-title":"Adv Neural Inform Process Syst"},{"key":"pcbi.1014656.ref010","unstructured":"Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai X, Unterthiner T, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:201011929. 2020."},{"issue":"1","key":"pcbi.1014656.ref011","doi-asserted-by":"crossref","first-page":"559","DOI":"10.1038\/s41597-023-02471-x","article-title":"A large-scale fMRI dataset for the visual processing of naturalistic scenes","volume":"10","author":"Z Gong","year":"2023","journal-title":"Sci Data"},{"key":"pcbi.1014656.ref012","unstructured":"Ge J, Tang S, Fan J, Jin C. On the Provable Advantage of Unsupervised Pretraining. In: The Twelfth International Conference on Learning Representations. 2024. Available from: https:\/\/openreview.net\/forum?id=rmXXKxQpOR"},{"key":"pcbi.1014656.ref013","doi-asserted-by":"crossref","unstructured":"Deng J, Dong W, Socher R, Li LJ, Li K, Fei-Fei L. Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. IEEE; 2009. p. 248\u201355.","DOI":"10.1109\/CVPR.2009.5206848"},{"key":"pcbi.1014656.ref014","unstructured":"Griffin G, Holub A, Perona P. Caltech 256. 2022."},{"issue":"6","key":"pcbi.1014656.ref015","doi-asserted-by":"crossref","first-page":"1452","DOI":"10.1109\/TPAMI.2017.2723009","article-title":"Places: A 10 Million Image Database for Scene Recognition","volume":"40","author":"B Zhou","year":"2018","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"pcbi.1014656.ref016","first-page":"4","article-title":"Representational similarity analysis - connecting the branches of systems neuroscience","volume":"2","author":"N Kriegeskorte","year":"2008","journal-title":"Front Syst Neurosci"},{"key":"pcbi.1014656.ref017","doi-asserted-by":"crossref","unstructured":"Rombach R, Blattmann A, Lorenz D, Esser P, Ommer B. High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE\/CVF conference on computer vision and pattern recognition. 2022. p. 10684\u201395.","DOI":"10.1109\/CVPR52688.2022.01042"},{"issue":"1","key":"pcbi.1014656.ref018","doi-asserted-by":"crossref","first-page":"12140","DOI":"10.1038\/s41598-017-11873-y","article-title":"Estimating the intrinsic dimension of datasets by a minimal neighborhood information","volume":"7","author":"E Facco","year":"2017","journal-title":"Sci Rep"},{"key":"pcbi.1014656.ref019","doi-asserted-by":"crossref","first-page":"119728","DOI":"10.1016\/j.neuroimage.2022.119728","article-title":"Feature-space selection with banded ridge regression","volume":"264","author":"T Dupr\u00e9 la Tour","year":"2022","journal-title":"Neuroimage"},{"issue":"1","key":"pcbi.1014656.ref020","doi-asserted-by":"crossref","first-page":"69","DOI":"10.1080\/00401706.1970.10488635","article-title":"Ridge Regression: Applications to Nonorthogonal Problems","volume":"12","author":"AE Hoerl","year":"1970","journal-title":"Technometrics"},{"key":"pcbi.1014656.ref021","doi-asserted-by":"crossref","first-page":"135","DOI":"10.3389\/fncom.2015.00135","article-title":"Fourier power, subjective distance, and object categories all provide plausible models of BOLD responses in scene-selective visual areas","volume":"9","author":"MD Lescroart","year":"2015","journal-title":"Front Comput Neurosci"},{"issue":"20","key":"pcbi.1014656.ref022","doi-asserted-by":"crossref","first-page":"7322","DOI":"10.1523\/JNEUROSCI.4588-10.2011","article-title":"Real-world scene representations in high-level visual cortex: it\u2019s the spaces more than the places","volume":"31","author":"DJ Kravitz","year":"2011","journal-title":"J Neurosci"},{"key":"pcbi.1014656.ref023","doi-asserted-by":"crossref","unstructured":"Li DSP, Bonner MF. Emergent selectivity for scenes, object properties, and contour statistics in feedforward models of scene-preferring cortex. bioRxiv. 2022. doi: 10.1101\/2021.09.24.461733","DOI":"10.1101\/2021.09.24.461733"},{"key":"pcbi.1014656.ref024","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1016\/j.neuroimage.2016.10.001","article-title":"Seeing it all: Convolutional network layers map the function of the human visual system","volume":"152","author":"M Eickenberg","year":"2017","journal-title":"Neuroimage"},{"issue":"27","key":"pcbi.1014656.ref025","doi-asserted-by":"crossref","first-page":"10005","DOI":"10.1523\/JNEUROSCI.5023-14.2015","article-title":"Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream","volume":"35","author":"U G\u00fc\u00e7l\u00fc","year":"2015","journal-title":"J Neurosci"},{"key":"pcbi.1014656.ref026","doi-asserted-by":"crossref","first-page":"108318","DOI":"10.1016\/j.jneumeth.2019.108318","article-title":"A visual encoding model based on deep neural networks and transfer learning for brain activity measured by functional magnetic resonance imaging","volume":"325","author":"C Zhang","year":"2019","journal-title":"J Neurosci Methods"},{"key":"pcbi.1014656.ref027","doi-asserted-by":"crossref","first-page":"125","DOI":"10.1016\/j.neuroimage.2019.05.039","article-title":"Variational autoencoder: An unsupervised model for encoding and decoding fMRI activity in visual cortex","volume":"198","author":"K Han","year":"2019","journal-title":"Neuroimage"},{"issue":"8","key":"pcbi.1014656.ref028","doi-asserted-by":"crossref","DOI":"10.1371\/journal.pcbi.1009267","article-title":"Unveiling functions of the visual cortex using task-specific deep neural networks","volume":"17","author":"K Dwivedi","year":"2021","journal-title":"PLoS Comput Biol"},{"issue":"1","key":"pcbi.1014656.ref029","doi-asserted-by":"crossref","first-page":"3329","DOI":"10.1038\/s41467-023-38674-4","article-title":"Brain-optimized deep neural network models of human visual areas learn non-hierarchical representations","volume":"14","author":"G St-Yves","year":"2023","journal-title":"Nat Commun"},{"key":"pcbi.1014656.ref030","unstructured":"Gutmann M, Hyv\u00e4rinen A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics. JMLR Workshop and Conference Proceedings. 2010. p. 297\u2013304."},{"key":"pcbi.1014656.ref031","doi-asserted-by":"crossref","unstructured":"Ma Z, Collins M. Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018. p. 3698\u2013707.","DOI":"10.18653\/v1\/D18-1405"},{"key":"pcbi.1014656.ref032","first-page":"3407","article-title":"Demystifying contrastive self-supervised learning: invariances, augmentations and dataset biases","volume":"33","author":"S Purushwalkam","year":"2020","journal-title":"Adv Neural Inform Process Syst"},{"key":"pcbi.1014656.ref033","first-page":"5171","article-title":"On variational bounds of mutual information.","volume-title":"International Conference on Machine Learning","author":"B Poole","year":"2019"},{"key":"pcbi.1014656.ref034","first-page":"875","article-title":"Formal limitations on the measurement of mutual information.","volume-title":"International Conference on Artificial Intelligence and Statistics","author":"D McAllester","year":"2020"},{"key":"pcbi.1014656.ref035","first-page":"1179","article-title":"Contrastive learning, multi-view redundancy, and linear models","volume-title":"Algorithmic Learning Theory","author":"C Tosh","year":"2021"},{"issue":"1","key":"pcbi.1014656.ref036","doi-asserted-by":"crossref","first-page":"15666","DOI":"10.1038\/s41598-023-42891-8","article-title":"Natural scene reconstruction from fMRI signals using generative latent diffusion","volume":"13","author":"F Ozcelik","year":"2023","journal-title":"Sci Rep"},{"issue":"3","key":"pcbi.1014656.ref037","doi-asserted-by":"crossref","DOI":"10.1073\/pnas.2014196118","article-title":"Unsupervised neural network models of the ventral visual stream","volume":"118","author":"C Zhuang","year":"2021","journal-title":"Proc Natl Acad Sci U S A"},{"issue":"8","key":"pcbi.1014656.ref038","doi-asserted-by":"crossref","first-page":"1004","DOI":"10.3390\/brainsci11081004","article-title":"A Visual Encoding Model Based on Contrastive Self-Supervised Learning for Human Brain Activity along the Ventral Visual Stream","volume":"11","author":"J Li","year":"2021","journal-title":"Brain Sci"},{"key":"pcbi.1014656.ref039","doi-asserted-by":"crossref","unstructured":"He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. p. 770\u20138.","DOI":"10.1109\/CVPR.2016.90"},{"issue":"12","key":"pcbi.1014656.ref040","doi-asserted-by":"crossref","first-page":"1415","DOI":"10.1038\/s42256-023-00753-y","article-title":"Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset","volume":"5","author":"AY Wang","year":"2023","journal-title":"Nat Mach Intell"},{"key":"pcbi.1014656.ref041","doi-asserted-by":"crossref","unstructured":"Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, et al. Microsoft coco: Common objects in context. In: Computer Vision\u2013ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer; 2014. p. 740\u201355.","DOI":"10.1007\/978-3-319-10602-1_48"},{"key":"pcbi.1014656.ref042","unstructured":"Gifford AT, Lahner B, Saba-Sadiya S, Vilas MG, Lascelles A, Oliva A, et al. The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenes. 2023."},{"issue":"3","key":"pcbi.1014656.ref043","doi-asserted-by":"crossref","first-page":"211","DOI":"10.1007\/s11263-015-0816-y","article-title":"ImageNet Large Scale Visual Recognition Challenge","volume":"115","author":"O Russakovsky","year":"2015","journal-title":"Int J Comput Vis"},{"key":"pcbi.1014656.ref044","unstructured":"Bordes F, Balestriero R, Garrido Q, Bardes A, Vincent P. Guillotine Regularization: Why removing layers is needed to improve generalization in Self-Supervised Learning. Trans Mach Learn Res. 2023."},{"key":"pcbi.1014656.ref045","first-page":"2825","article-title":"Scikit-learn: Machine learning in Python","volume":"12","author":"F Pedregosa","year":"2011","journal-title":"J Mach Learn Res"},{"key":"pcbi.1014656.ref046","article-title":"Pytorch: An imperative style, high-performance deep learning library","volume":"32","author":"A Paszke","year":"2019","journal-title":"Adv Neural Inform Process Syst"},{"issue":"7615","key":"pcbi.1014656.ref047","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1038\/nature18933","article-title":"A multi-modal parcellation of human cerebral cortex","volume":"536","author":"MF Glasser","year":"2016","journal-title":"Nature"},{"key":"pcbi.1014656.ref048","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/0377-0427(87)90125-7","article-title":"Silhouettes: A graphical aid to the interpretation and validation of cluster analysis","volume":"20","author":"PJ Rousseeuw","year":"1987","journal-title":"J Computat Appl Math"},{"key":"pcbi.1014656.ref049","doi-asserted-by":"crossref","unstructured":"Ding Y, Tian C, Ding H, Liu L. The CLIP Model is Secretly an Image-to-Prompt Converter. In: Thirty-seventh Conference on Neural Information Processing Systems. 2023.Available from: https:\/\/openreview.net\/forum?id=lHa7gFbmvS","DOI":"10.52202\/075280-2456"},{"issue":"1","key":"pcbi.1014656.ref050","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1007\/BF02289451","article-title":"A Generalized Solution of the Orthogonal Procrustes Problem","volume":"31","author":"PH Sch\u00f6nemann","year":"1966","journal-title":"Psychometrika"},{"key":"pcbi.1014656.ref051","doi-asserted-by":"crossref","first-page":"417","DOI":"10.1146\/annurev-vision-082114-035447","article-title":"Deep Neural Networks: A New Framework for Modeling Biological Vision and Brain Information Processing","volume":"1","author":"N Kriegeskorte","year":"2015","journal-title":"Annu Rev Vis Sci"},{"issue":"2","key":"pcbi.1014656.ref052","doi-asserted-by":"crossref","first-page":"774","DOI":"10.1016\/j.neuroimage.2012.01.021","article-title":"FreeSurfer","volume":"62","author":"B Fischl","year":"2012","journal-title":"Neuroimage"},{"issue":"10","key":"pcbi.1014656.ref053","doi-asserted-by":"crossref","first-page":"1368","DOI":"10.3390\/e23101368","article-title":"Scikit-Dimension: A Python Package for Intrinsic Dimension Estimation","volume":"23","author":"J Bac","year":"2021","journal-title":"Entropy (Basel)"},{"key":"pcbi.1014656.ref054","doi-asserted-by":"crossref","first-page":"109","DOI":"10.3389\/fnsys.2016.00109","article-title":"Models of neuronal stimulus-response functions: elaboration, estimation, and evaluation","volume":"10","author":"AF Meyer","year":"2017","journal-title":"Front Syst Neurosci"},{"key":"pcbi.1014656.ref055","unstructured":"Mulrooney A, Brockmeier AJ. Contrastive Learning to Fine-Tune Feature Extraction Models for the Visual Cortex. 2024. Available from: https:\/\/arxiv.org\/abs\/2410.06067"},{"issue":"10","key":"pcbi.1014656.ref056","doi-asserted-by":"crossref","first-page":"3911","DOI":"10.1093\/cercor\/bhu277","article-title":"Probabilistic Maps of Visual Topography in Human Cortex","volume":"25","author":"L Wang","year":"2015","journal-title":"Cereb Cortex"},{"issue":"4","key":"pcbi.1014656.ref057","doi-asserted-by":"crossref","first-page":"046027","DOI":"10.1088\/1741-2552\/ad6184","article-title":"Structurally-constrained encoding framework using a multi-voxel reduced-rank latent model for human natural vision","volume":"21","author":"A Ranjbar","year":"2024","journal-title":"J Neural Eng"},{"issue":"2","key":"pcbi.1014656.ref058","doi-asserted-by":"crossref","first-page":"455","DOI":"10.1016\/j.neuroimage.2010.07.034","article-title":"Partial Least Squares (PLS) methods for neuroimaging: a tutorial and review","volume":"56","author":"A Krishnan","year":"2011","journal-title":"Neuroimage"},{"key":"pcbi.1014656.ref059","unstructured":"Oquab M, Darcet T, Moutakanni T, Vo H, Szafraniec M, Khalidov V, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint. arXiv:230407193. 2023."}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1014656","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T00:00:00Z","timestamp":1787184000000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1014656","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,8,20]],"date-time":"2026-08-20T17:49:49Z","timestamp":1787248189000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1014656"}},"subtitle":[],"editor":[{"given":"Amy","family":"Kuceyeski","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2026,8,17]]},"references-count":59,"journal-issue":{"issue":"8","published-online":{"date-parts":[[2026,8,17]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1014656","relation":{},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,8,17]]}}}