{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,9,24]],"date-time":"2026-09-24T06:18:23Z","timestamp":1790230703701,"version":"4.1.0"},"reference-count":74,"publisher":"National Academy of Sciences","issue":"28","license":[{"start":{"date-parts":[[2025,7,8]],"date-time":"2025-07-08T00:00:00Z","timestamp":1751932800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by-nc-nd\/4.0\/"}],"content-domain":{"domain":["www.pnas.org"],"crossmark-restriction":true},"short-container-title":["Proc. Natl. Acad. Sci. U.S.A."],"published-print":{"date-parts":[[2025,7,15]]},"abstract":"<jats:p>Stimulus-driven, multiarea processing in the inferotemporal (IT) cortex is thought to be critical for transforming sensory inputs into useful representations of the world. What are the formats of these neural representations and how are they computed across the nodes of the IT networks? A growing literature in computational neuroscience focuses on the computational-level objective of acquiring high-level image statistics that supports useful distinctions, including between object identities or categories. Here, inspired by classic theories of vision, we suggest an alternative possibility. We show that inferring 3D objects may be a distinct computational-level objective of IT, implemented via an algorithm analogous to graphics-based generative models of how 3D scenes form and project to images, but in the reverse order. Using perception of bodies as a case study, we show that inverse graphics spontaneously emerges in inference networks trained to map images to 3D objects. Remarkably, this correspondence to the reverse of a graphics-based generative model also holds across the body processing network of the macaque IT cortex. Finally, inference networks recapitulate the feedforward progression across the stages of this IT network and do so better than the currently dominant vision models, including both supervised and unsupervised variants, none of which aligns with the reverse of graphics. This work suggests inverse graphics as a multiarea neural algorithm implemented within IT, and points to ways for replicating primate vision capabilities in machines.<\/jats:p>","DOI":"10.1073\/pnas.2420287122","type":"journal-article","created":{"date-parts":[[2025,7,8]],"date-time":"2025-07-08T11:39:52Z","timestamp":1751974792000},"update-policy":"https:\/\/doi.org\/10.1073\/pnas.cm10313","source":"Crossref","is-referenced-by-count":2,"title":["Multiarea processing in body patches of the primate inferotemporal cortex implements inverse graphics"],"prefix":"10.1073","volume":"122","author":[{"ORCID":"https:\/\/orcid.org\/0009-0003-4470-9871","authenticated-orcid":false,"given":"Hakan","family":"Yilmaz","sequence":"first","affiliation":[{"name":"Department of Psychology","place":["New Haven"]},{"name":"Yale University","place":["New Haven"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Aalap D.","family":"Shah","sequence":"additional","affiliation":[{"name":"Department of Psychology","place":["New Haven"]},{"name":"Yale University","place":["New Haven"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Ariadne","family":"Letrou","sequence":"additional","affiliation":[{"name":"Department of Psychology","place":["Princeton"]},{"name":"Princeton University","place":["Princeton"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9054-2123","authenticated-orcid":false,"given":"Satwant","family":"Kumar","sequence":"additional","affiliation":[{"name":"Neuroreef Labs","place":["Mountain View"]},{"name":"Department of Neurosciences","place":["Leuven, Belgium"]},{"name":"KU Leuven","place":["Leuven, Belgium"]},{"name":"Leuven Brain Institute","place":["Leuven, Belgium"]},{"name":"KU Leuven","place":["Leuven, Belgium"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-8778-835X","authenticated-orcid":false,"given":"Rufin","family":"Vogels","sequence":"additional","affiliation":[{"name":"Department of Neurosciences","place":["Leuven, Belgium"]},{"name":"KU Leuven","place":["Leuven, Belgium"]},{"name":"Leuven Brain Institute","place":["Leuven, Belgium"]},{"name":"KU Leuven","place":["Leuven, Belgium"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9072-0938","authenticated-orcid":false,"given":"Ilker","family":"Yildirim","sequence":"additional","affiliation":[{"name":"Department of Psychology","place":["New Haven"]},{"name":"Yale University","place":["New Haven"]},{"name":"Department of Statistics and Data Science","place":["New Haven"]},{"name":"Yale University","place":["New Haven"]},{"name":"Wu Tsai Institute","place":["New Haven"]},{"name":"Yale University","place":["New Haven"]},{"name":"Foundations of Data Science Institute","place":["New Haven"]},{"name":"Yale University","place":["New Haven"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"341","published-online":{"date-parts":[[2025,7,8]]},"reference":[{"key":"e_1_3_4_1_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1194908"},{"key":"e_1_3_4_2_2","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.17-11-04302.1997"},{"key":"e_1_3_4_3_2","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.2748-13.2014"},{"key":"e_1_3_4_4_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.1063414"},{"key":"e_1_3_4_5_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2013.06.015"},{"key":"e_1_3_4_6_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-020-2350-5"},{"key":"e_1_3_4_7_2","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.adl1776"},{"key":"e_1_3_4_8_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.neuron.2012.01.010"},{"key":"e_1_3_4_9_2","doi-asserted-by":"publisher","DOI":"10.1038\/nature14539"},{"key":"e_1_3_4_10_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.2014196118"},{"key":"e_1_3_4_11_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-022-28091-4"},{"key":"e_1_3_4_12_2","unstructured":"Y. Lecun Y. Bengio \u201cConvolutional networks for images speech and time-series\u201d in The Handbook of Brain Theory and Neural Networks M. Arbib Ed. (MIT Press 1995)."},{"key":"e_1_3_4_13_2","doi-asserted-by":"publisher","DOI":"10.1038\/nn.4244"},{"key":"e_1_3_4_14_2","doi-asserted-by":"crossref","unstructured":"C. Conwell J. S. Prince K. N. Kay G. A. Alvarez T. Konkle What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? bioRxiv [Preprint] (2023). https:\/\/doi.org\/10.1101\/2022.03.28.485868 (Accessed 21 September 2024).","DOI":"10.1101\/2022.03.28.485868"},{"key":"e_1_3_4_15_2","doi-asserted-by":"crossref","unstructured":"K. He \u201cMasked Autoencoders Are Scalable Vision Learners\u201d in 2022 IEEE\/CVF Conference on Computer Vision and Pattern Recognition (CVPR) K. Dana G. Hua S. Roth D. Samaras R. Singh Eds. (IEEE Computer Society 2022) pp. 15979\u201315988.","DOI":"10.1109\/CVPR52688.2022.01553"},{"key":"e_1_3_4_16_2","unstructured":"A. Bakry M. Elhoseiny T. El-Gaaly A. Elgammal Digging deep into the layers of CNNs: In search of how CNNs achieve view invariance. arXiv [Preprint] (2016). http:\/\/arxiv.org\/abs\/1508.01983 (Accessed 18 April 2025)."},{"key":"e_1_3_4_17_2","unstructured":"A. Nayebi \u201cTask-driven convolutional recurrent models of the visual system\u201d in Proceedings of the 32nd International Conference on Neural Information Processing Systems S. Bengio H.M. Wallach H. Larochelle K. Grauman N. Cesa-Bianchi Eds. (Curran Associates Inc. 2018) pp. 5295\u20135306."},{"key":"e_1_3_4_18_2","doi-asserted-by":"publisher","DOI":"10.1523\/JNEUROSCI.5023-14.2015"},{"key":"e_1_3_4_19_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev.psych.55.090902.142005"},{"key":"e_1_3_4_20_2","doi-asserted-by":"publisher","DOI":"10.1364\/JOSAA.20.001434"},{"key":"e_1_3_4_21_2","volume-title":"Pattern Theory: The Stochastic Analysis of Real-World Signals","author":"Mumford D.","year":"2011","unstructured":"D. Mumford, A. Desolneux, Pattern Theory: The Stochastic Analysis of Real-World Signals (A. K. Peters\/CRC Press, New York, 2011)."},{"key":"e_1_3_4_22_2","first-page":"3","article-title":"Recovering intrinsic scene characteristics from images","volume":"157","author":"Barrow H.","year":"1978","unstructured":"H. Barrow, J. Tenenbaum, Recovering intrinsic scene characteristics from images. Comput. Vis. Syst. 157, 3\u201326 (1978).","journal-title":"Comput. Vis. Syst."},{"key":"e_1_3_4_23_2","doi-asserted-by":"publisher","DOI":"10.3758\/BF03200564"},{"key":"e_1_3_4_24_2","volume-title":"Vision","author":"Marr D.","year":"1982","unstructured":"D. Marr, Vision (W. H. Freeman, 1982)."},{"key":"e_1_3_4_25_2","doi-asserted-by":"publisher","DOI":"10.1002\/9781444310498"},{"key":"e_1_3_4_26_2","doi-asserted-by":"crossref","unstructured":"B. A. Olshausen in Perception as an Inference Problem in The Cognitive Neurosciences M. S. Gazzaniga G. R. Mangun Eds. (The MIT Press 2014).","DOI":"10.7551\/mitpress\/9504.003.0037"},{"key":"e_1_3_4_27_2","doi-asserted-by":"publisher","DOI":"10.1126\/science.aag2612"},{"key":"e_1_3_4_28_2","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1306572110"},{"key":"e_1_3_4_29_2","unstructured":"P. A. Tsividis Human-level reinforcement learning through theory-based modeling Exploration and Planning. arXiv [Preprint] (2021). http:\/\/arxiv.org\/abs\/2107.12544 (Accessed 21 September 2024)."},{"key":"e_1_3_4_30_2","first-page":"269","article-title":"Representation and recognition of the spatial organization of three-dimensional shapes","volume":"200","author":"Marr D.","year":"1978","unstructured":"D. Marr, H. K. Nishihara, Representation and recognition of the spatial organization of three-dimensional shapes. Proc. R. Soc. Lond. Ser. B Biol. Sci. 200, 269\u2013294 (1978).","journal-title":"Proc. R. Soc. Lond. Ser. B Biol. Sci."},{"key":"e_1_3_4_31_2","unstructured":"T. Chen S. Kornblith M. Norouzi G. Hinton \u201cA simple framework for contrastive learning of visual representations\u201d in Proceedings of the 37th International Conference on Machine Learning ICML\u201920 H. Daume A. Singh Eds. (JMLR.org 2020) vol. 119 pp. 1597\u20131607."},{"key":"e_1_3_4_32_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-020-01405-z"},{"key":"e_1_3_4_33_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-vision-112823-030616"},{"key":"e_1_3_4_34_2","doi-asserted-by":"publisher","DOI":"10.1037\/rev0000178"},{"key":"e_1_3_4_35_2","doi-asserted-by":"publisher","DOI":"10.1007\/s10955-017-1836-5"},{"key":"e_1_3_4_36_2","unstructured":"A. Stuhlm\u00fcller J. Taylor N. Goodman \u201cLearning stochastic inverses\u201d in Advances in Neural Information Processing Systems C. J. Burges L. Bottou M. Welling Z. Ghahramani K. Q. Weinberger Eds. (Curran Associates Inc. 2013). vol. 26."},{"key":"e_1_3_4_37_2","doi-asserted-by":"publisher","DOI":"10.1126\/sciadv.aax5979"},{"key":"e_1_3_4_38_2","doi-asserted-by":"publisher","DOI":"10.1093\/cercor\/bhx320"},{"key":"e_1_3_4_39_2","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pcbi.1003553"},{"key":"e_1_3_4_40_2","doi-asserted-by":"crossref","unstructured":"S. J. Gershman J. M. Beck \u201cComplex probabilistic inference: From cognition to neural computation\u201d in Computational Models of Brain and Behavior A. A. Moustafa Ed. (John Wiley & Sons Ltd 2017) pp. 453\u2013466.","DOI":"10.1002\/9781119159193.ch33"},{"key":"e_1_3_4_41_2","doi-asserted-by":"crossref","unstructured":"M. Loper N. Mahmood J. Romero G. Pons-Moll M. J. Black SMPL: A skinned multi-person linear model. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 34 248:1\u2013248:16 (2015).","DOI":"10.1145\/2816795.2818013"},{"key":"e_1_3_4_42_2","doi-asserted-by":"crossref","unstructured":"A. Kanazawa M. J. Black D. W. Jacobs J. Malik \u201cEnd-to-end recovery of human shape and pose\u201d in Computer vision and pattern regognition (CVPR) D. Forsyth I. Laptev D. Ramanan A. Oliva Eds. (IEEE Computer Society 2018).","DOI":"10.1109\/CVPR.2018.00744"},{"key":"e_1_3_4_43_2","doi-asserted-by":"crossref","unstructured":"N. Kolotouros G. Pavlakos M. Black K. Daniilidis \u201cLearning to reconstruct 3D human pose and shape via model-fitting in the loop\u201d in 2019 IEEE\/CVF International Conference on Computer Vision (ICCV) I. S. Kweon N. Paragios M. H. Yang S. Lazebnik Eds. (IEEE Computer Society 2019) pp. 2252\u20132261.","DOI":"10.1109\/ICCV.2019.00234"},{"key":"e_1_3_4_44_2","unstructured":"K. He X. Zhang S. Ren J. Sun Deep residual learning for image recognition. arXiv [Preprint] (2015). http:\/\/arxiv.org\/abs\/1512.03385 (Accessed 7 July 2024)."},{"key":"e_1_3_4_45_2","unstructured":"K. Simonyan A. Zisserman Very deep convolutional networks for large-scale image recognition. arXiv [Preprint] (2015). http:\/\/arxiv.org\/abs\/1409.1556 (Accessed 8 July 2024)."},{"key":"e_1_3_4_46_2","doi-asserted-by":"publisher","DOI":"10.1146\/annurev-vision-100720-113429"},{"key":"e_1_3_4_47_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2017.05.011"},{"key":"e_1_3_4_48_2","doi-asserted-by":"crossref","unstructured":"Y. Shi Rapid concerted switching of the neural code in inferotemporal cortex. bioRxiv [Preprint] (2023). https:\/\/doi.org\/10.1101\/2023.12.06.570341. Accessed 21 September 2024.","DOI":"10.1101\/2023.12.06.570341"},{"key":"e_1_3_4_49_2","unstructured":"S. Szymanowicz Feed-forward generalisable 3D scene reconstruction from a single image. arXiv [Preprint] (2024). http:\/\/arxiv.org\/abs\/2406.04343 (Accessed 16 August 2024)."},{"key":"e_1_3_4_50_2","doi-asserted-by":"crossref","unstructured":"Y. Deng \u201cAccurate 3D face reconstruction with weakly-supervised learning: From single image to image set\u201d in 2019 IEEE\/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) A. Gupta D. Hoiem G. Hua Z. Tu Eds. (IEEE Computer Society 2019) pp. 285\u2013295.","DOI":"10.1109\/CVPRW.2019.00038"},{"key":"e_1_3_4_51_2","doi-asserted-by":"crossref","unstructured":"K. Genova \u201cUnsupervised training for 3D morphable model regression\u201d in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition D. Forsyth I. Laptev A. Oliva D. Ramanan Eds (IEEE Computer Society 2018) pp. 8377\u20138386.","DOI":"10.1109\/CVPR.2018.00874"},{"key":"e_1_3_4_52_2","unstructured":"Z. Li Learning the 3D fauna of the web. arXiv [Preprint] (2024). http:\/\/arxiv.org\/abs\/2401.02400 (Accessed 18 April 2025)."},{"key":"e_1_3_4_53_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41583-020-0277-3"},{"key":"e_1_3_4_54_2","doi-asserted-by":"publisher","DOI":"10.1007\/s11263-023-01870-2"},{"key":"e_1_3_4_55_2","doi-asserted-by":"crossref","unstructured":"H. Xu \u201cGHUM & GHUML: Generative 3D human shape and articulated pose models\u201d in IEEE\/CVF Conference on Computer Vision and Pattern Recognition (Oral) C. Liu G. Mori K. Saenko S. Savarese Eds. (IEEE Computer Society 2020) pp. 6184\u20136193.","DOI":"10.1109\/CVPR42600.2020.00622"},{"key":"e_1_3_4_56_2","doi-asserted-by":"crossref","unstructured":"G. Bouritsas S. Bokhnyak S. Ploumpis M. Bronstein S. Zafeiriou \u201cNeural 3D morphable models: spiral convolutional networks for 3D shape representation learning and generation\u201d in The IEEE International Conference on Computer Vision (ICCV) I. S. Kweon N. Paragios M. H. Yang S. Lazebnik Eds. (IEEE Computer Society 2019).","DOI":"10.1109\/ICCV.2019.00731"},{"key":"e_1_3_4_57_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-021-26751-5"},{"key":"e_1_3_4_58_2","unstructured":"C. P. Burgess Understanding disentangling in beta-VAE. arXiv [Preprint] (2018). http:\/\/arxiv.org\/abs\/1804.03599 (Accessed 18 April 2025)."},{"key":"e_1_3_4_59_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.tics.2024.02.008"},{"key":"e_1_3_4_60_2","unstructured":"M. Huh B. Cheung T. Wang P. Isola The platonic representation hypothesis. arXiv [Preprint] (2024). http:\/\/arxiv.org\/abs\/2405.07987 (Accessed 18 April 2025)."},{"key":"e_1_3_4_61_2","doi-asserted-by":"publisher","DOI":"10.1016\/j.conb.2020.11.009"},{"key":"e_1_3_4_62_2","unstructured":"Q. Liao T. Poggio Bridging the gaps between residual learning recurrent neural networks and visual cortex. arXiv [Preprint] (2020). http:\/\/arxiv.org\/abs\/1604.03640 (Accessed 21 September 2024)."},{"key":"e_1_3_4_63_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41593-019-0392-5"},{"key":"e_1_3_4_64_2","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.48764"},{"key":"e_1_3_4_65_2","doi-asserted-by":"publisher","DOI":"10.1038\/s41562-023-01759-7"},{"key":"e_1_3_4_66_2","unstructured":"Y. Hong \u201cLRM: Large reconstruction model for single image to 3D\u201d in The Twelfth International Conference on Learning Representations B. Kim M. Nickel M. Wang N. F. Chen V. Marivate Eds. (2023)."},{"key":"e_1_3_4_67_2","doi-asserted-by":"publisher","DOI":"10.1111\/cgf.14022"},{"key":"e_1_3_4_68_2","unstructured":"B. O. Community Blender - A 3D Modelling and Rendering Package (Blender Foundation Stichting Blender Foundation Amsterdam 2018)."},{"key":"e_1_3_4_69_2","unstructured":"D. P. Kingma J. Ba Adam: A method for stochastic optimization. arXiv [Preprint] (2017). http:\/\/arxiv.org\/abs\/1412.6980 (Accessed 21 September 2024)."},{"key":"e_1_3_4_70_2","unstructured":"A. Paszke \u201cPyTorch: An imperative style high-performance deep learning library\u201d in Proceedings of the 33rd International Conference on Neural Information Processing Systems H. M. Wallach H. Larochelle A. Beygelzimer F. d\u2019Alch\u00e9-Buc E. B. Fox Eds. (Curran Associates Inc. 2019) vol. 721 pp. 8026\u20138037."},{"key":"e_1_3_4_71_2","unstructured":"K. Tian \u201cDesigning BERT for convolutional networks: Sparse and hierarchical masked modeling\u201d in The Eleventh International Conference on Learning Representations Y. Liu C. Finn Y. Choi M. Deisenroth Eds. (2022)."},{"key":"e_1_3_4_72_2","unstructured":"Y. You \u201cLarge batch optimization for deep learning: Training BERT in 76 minutes\u201d in International Conference on Learning Representations A. Rush S. Levine K. Livescu S. Mohamed Eds. (2019)."},{"key":"e_1_3_4_73_2","unstructured":"F. Bordes R. Balestriero P. Vincent Towards democratizing joint-embedding self-supervised learning. arXiv [Preprint] (2023). http:\/\/arxiv.org\/abs\/2303.01986 (Accessed 21 September 2024)."},{"key":"e_1_3_4_74_2","unstructured":"I. Loshchilov F. Hutter \u201cDecoupled weight decay regularization\u201d in International Conference on Learning Representations T. Sainath I. Murray M. Ranzato O. Vinyals Eds. (2018)."}],"container-title":["Proceedings of the National Academy of Sciences"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/pnas.org\/doi\/pdf\/10.1073\/pnas.2420287122","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,8]],"date-time":"2025-07-08T11:41:23Z","timestamp":1751974883000},"score":1,"resource":{"primary":{"URL":"https:\/\/pnas.org\/doi\/10.1073\/pnas.2420287122"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,7,8]]},"references-count":74,"journal-issue":{"issue":"28","published-print":{"date-parts":[[2025,7,15]]}},"alternative-id":["10.1073\/pnas.2420287122"],"URL":"https:\/\/doi.org\/10.1073\/pnas.2420287122","relation":{},"ISSN":["0027-8424","1091-6490"],"issn-type":[{"value":"0027-8424","type":"print"},{"value":"1091-6490","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,7,8]]},"assertion":[{"value":"2024-10-02","order":0,"name":"received","label":"Received","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-05-29","order":2,"name":"accepted","label":"Accepted","group":{"name":"publication_history","label":"Publication History"}},{"value":"2025-07-08","order":3,"name":"published","label":"Published","group":{"name":"publication_history","label":"Publication History"}}],"article-number":"e2420287122"}}