{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T18:33:59Z","timestamp":1785522839474,"version":"3.56.0"},"reference-count":22,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T00:00:00Z","timestamp":1784851200000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2026,7,29]],"date-time":"2026-07-29T00:00:00Z","timestamp":1785283200000},"content-version":"vor","delay-in-days":5,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/100012940","name":"Universit\u00e4tsklinikum T\u00fcbingen","doi-asserted-by":"crossref","id":[{"id":"10.13039\/100012940","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Bioinformatics"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Background<\/jats:title>\n                    <jats:p>Single-cell foundation models such as scGPT and Geneformer learn rich representations of gene expression programs, but whether these representations encode gene regulatory relationships beyond expression-level confounds remains unclear. Attention patterns in these models have been shown to capture co-expression rather than direct regulation, leaving open the question of whether deeper representations\u2014particularly the residual stream\u2014contain genuine regulatory information.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We systematically investigated residual-stream geometry in scGPT and Geneformer across four tissue contexts from the Tabula Sapiens atlas, evaluating whether geometric proximity between gene vectors provides incremental predictive value for curated TRRUST transcription factor\u2013target edges beyond expression confounds. Under repeated stratified cross-validation, geometric features provided significant incremental signal in kidney and immune settings, validated by label-permutation and geometry-shuffle null controls; centered-cosine similarity, PCA projection and multi-layer bundling recovered comparable signal in lung tissues, and the multi-layer bundle improved every domain (kidney \u0394AUROC\u00a0=\u00a0+\u00a00.122, immune +\u00a00.042, lung +\u00a00.028, external lung +\u00a00.027; geometry-augmented AUROC 0.60\u20130.69). The effect was fully robust to leave-TF-out and leave-target-out cross-validation and to harder degree- and expression-matched negative edges, but under the stricter leave-both-out split\u2014no transcription factor and no target shared between folds\u2014it collapsed to near-zero (\u0394AUROC at most +\u00a00.003, and not statistically significant in kidney or immune), marking the ceiling of out-of-entity generalization. With a comparable per-layer residual-stream extraction applied to both models, the apparent Geneformer advantage mostly disappeared (small residual gaps remained in three of four domains), indicating it largely reflected representation-construction choices rather than a substantial architectural difference. Asymmetric geometric features predicted regulatory edge orientation (AUROC 0.80\u20130.90), and the geometric signal added incremental value on top of expression-based gene regulatory network (GRN) inference (GENIE3, co-expression).<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Conclusion<\/jats:title>\n                    <jats:p>Foundation model residual streams carry incremental, regulatory-relevant geometric signal that is distributed across layers and that complements expression-based GRN inference for retrospective edge prioritization. The signal is statistical enrichment rather than a stand-alone regulatory classifier: absolute performance is modest and out-of-entity generalization is limited, so its practical role is as an orthogonal evidence channel for edge re-ranking and hypothesis prioritization in multi-evidence frameworks.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s12859-026-06538-5","type":"journal-article","created":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T13:53:37Z","timestamp":1784901217000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":0,"title":["Residual-stream geometry of single-cell foundation models carries incremental gene-regulatory signal across tissues"],"prefix":"10.1186","volume":"27","author":[{"given":"Ihor","family":"Kendiukhov","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2026,7,24]]},"reference":[{"issue":"5560","key":"6538_CR1","doi-asserted-by":"publisher","first-page":"1669","DOI":"10.1126\/science.1069883","volume":"295","author":"EH Davidson","year":"2002","unstructured":"Davidson EH, Rast JP, Oliveri P, et al. A genomic regulatory network for development. Science. 2002;295(5560):1669\u201378.","journal-title":"Science"},{"issue":"6","key":"6538_CR2","doi-asserted-by":"publisher","first-page":"450","DOI":"10.1038\/nrg2102","volume":"8","author":"U Alon","year":"2007","unstructured":"Alon U. Network motifs: theory and experimental approaches. Nat Rev Genet. 2007;8(6):450\u201361.","journal-title":"Nat Rev Genet"},{"issue":"10","key":"6538_CR3","doi-asserted-by":"publisher","first-page":"770","DOI":"10.1038\/nrm2503","volume":"9","author":"G Karlebach","year":"2008","unstructured":"Karlebach G, Shamir R. Modelling and analysis of gene regulatory networks. Nat Rev Mol Cell Biol. 2008;9(10):770\u201380.","journal-title":"Nat Rev Mol Cell Biol"},{"issue":"9","key":"6538_CR4","doi-asserted-by":"publisher","DOI":"10.1371\/journal.pone.0012776","volume":"5","author":"VA Huynh-Thu","year":"2010","unstructured":"Huynh-Thu VA, Irrthum A, Wehenkel L, Geurts P. Inferring regulatory networks from expression data using tree-based methods. PLoS ONE. 2010;5(9):e12776.","journal-title":"PLoS ONE"},{"issue":"11","key":"6538_CR5","doi-asserted-by":"publisher","first-page":"1083","DOI":"10.1038\/nmeth.4463","volume":"14","author":"S Aibar","year":"2017","unstructured":"Aibar S, Gonz\u00e1lez-Blas CB, Moerman T, et al. SCENIC: single-cell regulatory network inference and clustering. Nat Methods. 2017;14(11):1083\u20136.","journal-title":"Nat Methods"},{"issue":"2","key":"6538_CR6","doi-asserted-by":"publisher","first-page":"147","DOI":"10.1038\/s41592-019-0690-6","volume":"17","author":"A Pratapa","year":"2020","unstructured":"Pratapa A, Jalihal AP, Law JN, Bharadwaj A, Murali TM. Benchmarking algorithms for gene regulatory network inference from single-cell transcriptomic data. Nat Methods. 2020;17(2):147\u201354.","journal-title":"Nat Methods"},{"issue":"3","key":"6538_CR7","doi-asserted-by":"publisher","first-page":"252","DOI":"10.1016\/j.cels.2020.08.003","volume":"11","author":"P Dibaeinia","year":"2020","unstructured":"Dibaeinia P, Sinha S. SERGIO: a single-cell expression simulator guided by gene regulatory networks. Cell Syst. 2020;11(3):252\u201371.","journal-title":"Cell Syst"},{"issue":"8","key":"6538_CR8","doi-asserted-by":"publisher","first-page":"1470","DOI":"10.1038\/s41592-024-02201-0","volume":"21","author":"H Cui","year":"2024","unstructured":"Cui H, Wang C, Maan H, et al. scGPT: toward building a foundation model for single-cell multi-omics using generative AI. Nat Methods. 2024;21(8):1470\u201380.","journal-title":"Nat Methods"},{"issue":"7965","key":"6538_CR9","doi-asserted-by":"publisher","first-page":"616","DOI":"10.1038\/s41586-023-06139-9","volume":"618","author":"CV Theodoris","year":"2023","unstructured":"Theodoris CV, Xiao L, Chopra A, et al. Transfer learning enables predictions in network biology. Nature. 2023;618(7965):616\u201324.","journal-title":"Nature"},{"issue":"10","key":"6538_CR10","doi-asserted-by":"publisher","first-page":"852","DOI":"10.1038\/s42256-022-00534-z","volume":"4","author":"F Yang","year":"2022","unstructured":"Yang F, Wang W, Wang F, et al. scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA-seq data. Nat Mach Intell. 2022;4(10):852\u201366.","journal-title":"Nat Mach Intell"},{"key":"6538_CR11","unstructured":"Kendiukhov I. Attention patterns in single-cell foundation models capture co-expression, not causal regulation. Under review. 2025."},{"key":"6538_CR12","first-page":"5998","volume":"30","author":"A Vaswani","year":"2017","unstructured":"Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Adv Neural Inf Process Syst. 2017;30:5998\u20136008.","journal-title":"Adv Neural Inf Process Syst"},{"key":"6538_CR13","unstructured":"Elhage N, Nanda N, Olsson C. A mathematical framework for transformer circuits. Transformer Circuits Thread. 2021."},{"key":"6538_CR14","unstructured":"Elhage N, Hume T, Olsson C, et al. Toy models of superposition. arXiv:2209.10652. 2022."},{"key":"6538_CR15","unstructured":"Park K, Choe YJ, Veitch V. The linear representation hypothesis and the geometry of large language models. arXiv:2311.03658. 2023."},{"key":"6538_CR16","doi-asserted-by":"crossref","unstructured":"Rao RM, Liu J, Verkuil R, et al. MSA Transformer. In: Proceedings of the 38th international conference on machine learning. 2021:8844\u20138856.","DOI":"10.1101\/2021.02.12.430858"},{"issue":"4","key":"6538_CR17","doi-asserted-by":"publisher","first-page":"688","DOI":"10.1016\/j.cell.2020.01.021","volume":"180","author":"JM Stokes","year":"2020","unstructured":"Stokes JM, Yang K, Swanson K, et al. A deep learning approach to antibiotic discovery. Cell. 2020;180(4):688\u2013702.","journal-title":"Cell"},{"key":"6538_CR18","first-page":"3111","volume":"26","author":"T Mikolov","year":"2013","unstructured":"Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed representations of words and phrases and their compositionality. Adv Neural Inf Process Syst. 2013;26:3111\u20139.","journal-title":"Adv Neural Inf Process Syst"},{"key":"6538_CR19","unstructured":"The Tabula Sapiens Consortium. The Tabula Sapiens: a multiple-organ, single-cell transcriptomic atlas of humans. Science. 2022;376(6594):eabl4896."},{"issue":"11","key":"6538_CR20","doi-asserted-by":"publisher","first-page":"749","DOI":"10.1038\/nri3307","volume":"12","author":"SM Kaech","year":"2012","unstructured":"Kaech SM, Cui W. Transcriptional control of effector and memory CD8+ T cell differentiation. Nat Rev Immunol. 2012;12(11):749\u201361.","journal-title":"Nat Rev Immunol"},{"key":"6538_CR21","doi-asserted-by":"crossref","unstructured":"Tenney I, Das D, Pavlick E. BERT rediscovers the classical NLP pipeline. In: Proceedings of the 57th annual meeting of the association for computational linguistics. 2019. p. 4593\u2013601.","DOI":"10.18653\/v1\/P19-1452"},{"issue":"D1","key":"6538_CR22","doi-asserted-by":"publisher","first-page":"D380","DOI":"10.1093\/nar\/gkx1013","volume":"46","author":"H Han","year":"2018","unstructured":"Han H, Cho JW, Lee S, et al. TRRUST v2: an expanded reference database of human and mouse transcriptional regulatory interactions. Nucleic Acids Res. 2018;46(D1):D380\u20136.","journal-title":"Nucleic Acids Res"}],"container-title":["BMC Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12859-026-06538-5","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-026-06538-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-026-06538-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,7,31]],"date-time":"2026-07-31T17:19:50Z","timestamp":1785518390000},"score":1,"resource":{"primary":{"URL":"https:\/\/link.springer.com\/10.1186\/s12859-026-06538-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,7,24]]},"references-count":22,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2026,12]]}},"alternative-id":["6538"],"URL":"https:\/\/doi.org\/10.1186\/s12859-026-06538-5","relation":{},"ISSN":["1471-2105"],"issn-type":[{"value":"1471-2105","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,7,24]]},"assertion":[{"value":"21 February 2026","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"8 June 2026","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"24 July 2026","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"Not applicable. This study used only publicly available datasets (Tabula Sapiens atlas, TRRUST database) and did not involve human subjects, animal experiments, or clinical data requiring ethics approval.","order":1,"name":"Ethics","label":"Ethics approval and consent to participate","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","label":"Consent for publication","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare no conflict of interest.","order":3,"name":"Ethics","label":"Competing interests","group":{"name":"EthicsHeading","label":"Declarations"}}],"article-number":"162"}}