{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,11,2]],"date-time":"2026-11-02T11:57:08Z","timestamp":1793620628028,"version":"4.5.0"},"reference-count":144,"publisher":"Springer Science and Business Media LLC","issue":"3","license":[{"start":{"date-parts":[[2025,3,13]],"date-time":"2025-03-13T00:00:00Z","timestamp":1741824000000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"},{"start":{"date-parts":[[2025,3,13]],"date-time":"2025-03-13T00:00:00Z","timestamp":1741824000000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/www.springernature.com\/gp\/researchers\/text-and-data-mining"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Nat Mach Intell"],"DOI":"10.1038\/s42256-025-01007-9","type":"journal-article","created":{"date-parts":[[2025,3,13]],"date-time":"2025-03-13T06:03:41Z","timestamp":1741845821000},"page":"346-362","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":104,"title":["Transformers and genome language models"],"prefix":"10.1038","volume":"7","author":[{"given":"Micaela E.","family":"Consens","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Cameron","family":"Dufault","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Michael","family":"Wainberg","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Duncan","family":"Forster","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Mehran","family":"Karimzadeh","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9648-8949","authenticated-orcid":false,"given":"Hani","family":"Goodarzi","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2419-1943","authenticated-orcid":false,"given":"Fabian J.","family":"Theis","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Alan","family":"Moses","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9620-3413","authenticated-orcid":false,"given":"Bo","family":"Wang","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2025,3,13]]},"reference":[{"key":"1007_CR1","doi-asserted-by":"publisher","unstructured":"Nichol, A. et al. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2112.10741 (2021).","DOI":"10.48550\/arXiv.2112.10741"},{"key":"1007_CR2","doi-asserted-by":"publisher","unstructured":"Ramesh, A., Dhariwal, P., Nichol, A., Chu, C. & Chen, M. Hierarchical text-conditional image generation with clip latents. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2204.06125 (2022).","DOI":"10.48550\/arXiv.2204.06125"},{"key":"1007_CR3","doi-asserted-by":"publisher","unstructured":"Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. Preprint at https:\/\/doi.org\/10.48550\/arXiv.1810.04805 (2018).","DOI":"10.48550\/arXiv.1810.04805"},{"key":"1007_CR4","first-page":"9","volume":"1","author":"A Radford","year":"2019","unstructured":"Radford, A. et al. Language models are unsupervised multitask learners. OpenAI Blog 1, 9 (2019).","journal-title":"OpenAI Blog"},{"key":"1007_CR5","unstructured":"Yang, Z. et al. XLNet: generalized autoregressive pretraining for language understanding. In Proc. 33rd International Conference Neural Information Prcoessing Systems 517, 5753\u20135763 (2019)."},{"key":"1007_CR6","doi-asserted-by":"publisher","unstructured":"Brown, T. B. et al. Language models are few-shot learners. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2005.14165 (2020).","DOI":"10.48550\/arXiv.2005.14165"},{"key":"1007_CR7","doi-asserted-by":"publisher","first-page":"583","DOI":"10.1038\/s41586-021-03819-2","volume":"596","author":"J Jumper","year":"2021","unstructured":"Jumper, J. et al. Highly accurate protein structure prediction with alphafold. Nature 596, 583\u2013589 (2021).","journal-title":"Nature"},{"key":"1007_CR8","unstructured":"GPT-4 Technical Report (OpenAI, 2023)."},{"key":"1007_CR9","doi-asserted-by":"publisher","first-page":"1543","DOI":"10.1534\/g3.115.018564","volume":"5","author":"A Warr","year":"2015","unstructured":"Warr, A. et al. Exome sequencing: current and future perspectives. G3 Genes Genomes Genet. 5, 1543\u20131550 (2015).","journal-title":"G3 Genes Genomes Genet."},{"key":"1007_CR10","doi-asserted-by":"crossref","unstructured":"Ng, P. C. & Kirkness, E. F. Whole genome sequencing. Genet. Var. 628, 215\u2013226 (2010).","DOI":"10.1007\/978-1-60327-367-1_12"},{"key":"1007_CR11","doi-asserted-by":"publisher","first-page":"21","DOI":"10.1002\/0471142727.mb2129s109","volume":"109","author":"JD Buenrostro","year":"2015","unstructured":"Buenrostro, J. D., Wu, B., Chang, H. Y. & Greenleaf, W. J. ATAC-seq: a method for assaying chromatin accessibility genome-wide. Curr. Protoc. Mol. Biol. 109, 21\u201329 (2015).","journal-title":"Curr. Protoc. Mol. Biol."},{"key":"1007_CR12","doi-asserted-by":"publisher","first-page":"669","DOI":"10.1038\/nrg2641","volume":"10","author":"PJ Park","year":"2009","unstructured":"Park, P. J. ChIP\u2013seq: advantages and challenges of a maturing technology. Nat. Rev. Genet. 10, 669\u2013680 (2009).","journal-title":"Nat. Rev. Genet."},{"key":"1007_CR13","doi-asserted-by":"publisher","first-page":"1280","DOI":"10.1101\/gr.266551.120","volume":"31","author":"R Vaisvila","year":"2021","unstructured":"Vaisvila, R. et al. Enzymatic methyl sequencing detects DNA methylation at single-base resolution from picograms of DNA. Genome Res. 31, 1280\u20131289 (2021).","journal-title":"Genome Res."},{"key":"1007_CR14","doi-asserted-by":"publisher","first-page":"57","DOI":"10.1038\/nrg2484","volume":"10","author":"Z Wang","year":"2009","unstructured":"Wang, Z., Gerstein, M. & Snyder, M. RNA-Seq: a revolutionary tool for transcriptomics. Nat. Rev. Genet. 10, 57\u201363 (2009).","journal-title":"Nat. Rev. Genet."},{"key":"1007_CR15","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1186\/s13073-017-0467-4","volume":"9","author":"A Haque","year":"2017","unstructured":"Haque, A., Engel, J., Teichmann, S. A. & L\u00f6nnberg, T. A practical guide to single-cell RNA-sequencing for biomedical research and clinical applications. Genome Med. 9, 75 (2017).","journal-title":"Genome Med."},{"key":"1007_CR16","doi-asserted-by":"publisher","first-page":"289","DOI":"10.1126\/science.1181369","volume":"326","author":"E Lieberman-Aiden","year":"2009","unstructured":"Lieberman-Aiden, E. et al. Comprehensive mapping of long-range interactions reveals folding principles of the human genome. Science 326, 289\u2013293 (2009).","journal-title":"Science"},{"key":"1007_CR17","doi-asserted-by":"publisher","first-page":"52","DOI":"10.1038\/489052a","volume":"489","author":"JR Ecker","year":"2012","unstructured":"Ecker, J. R. et al. ENCODE explained. Nature 489, 52\u201354 (2012).","journal-title":"Nature"},{"key":"1007_CR18","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1038\/s41588-018-0295-5","volume":"51","author":"J Zou","year":"2019","unstructured":"Zou, J. et al. A primer on deep learning in genomics. Nat. Genet. 51, 12\u201318 (2019).","journal-title":"Nat. Genet."},{"key":"1007_CR19","doi-asserted-by":"crossref","unstructured":"Quang, D., Chen, Y. & Xie, X. DANN: a deep learning approach for annotating the pathogenicity of genetic variants. Bioinforma. Oxf. Engl. 31, 761\u2013763 (2014).","DOI":"10.1093\/bioinformatics\/btu703"},{"key":"1007_CR20","doi-asserted-by":"publisher","first-page":"831","DOI":"10.1038\/nbt.3300","volume":"33","author":"B Alipanahi","year":"2015","unstructured":"Alipanahi, B., Delong, A., Weirauch, M. T. & Frey, B. J. Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning. Nat. Biotechnol. 33, 831\u2013838 (2015).","journal-title":"Nat. Biotechnol."},{"key":"1007_CR21","doi-asserted-by":"publisher","first-page":"931","DOI":"10.1038\/nmeth.3547","volume":"12","author":"J Zhou","year":"2015","unstructured":"Zhou, J. & Troyanskaya, O. G. Predicting effects of noncoding variants with deep learning\u2013based sequence model. Nat. Methods 12, 931\u2013934 (2015).","journal-title":"Nat. Methods"},{"key":"1007_CR22","doi-asserted-by":"publisher","first-page":"W131","DOI":"10.1093\/nar\/gkab429","volume":"49","author":"G Pei","year":"2021","unstructured":"Pei, G., Hu, R., Jia, P. & Zhao, Z. DeepFun: a deep learning sequence-based model to decipher non-coding variant effect in a tissue- and cell type-specific manner. Nucleic Acids Res. 49, W131\u2013W139 (2021).","journal-title":"Nucleic Acids Res."},{"key":"1007_CR23","doi-asserted-by":"crossref","unstructured":"Hassanzadeh, H. R. & Wang, M. DeeperBind: enhancing prediction of sequence specificities of DNA binding proteins. In Proc. IEEE International Conference on Bioinformatics and Biomedicine Vol. 2016, 178\u2013183 (2016).","DOI":"10.1109\/BIBM.2016.7822515"},{"key":"1007_CR24","doi-asserted-by":"publisher","DOI":"10.1186\/s13059-020-01987-4","volume":"21","author":"T Trieu","year":"2020","unstructured":"Trieu, T., Martinez-Fundichely, A. & Khurana, E. DeepMILO: a deep learning approach to predict the impact of non-coding sequence variants on 3D chromatin structure. Genome Biol. 21, 79 (2020).","journal-title":"Genome Biol."},{"key":"1007_CR25","doi-asserted-by":"publisher","first-page":"1171","DOI":"10.1038\/s41588-018-0160-6","volume":"50","author":"J Zhou","year":"2018","unstructured":"Zhou, J. et al. Deep learning sequence-based ab initio prediction of variant effects on expression and disease risk. Nat. Genet. 50, 1171\u20131179 (2018).","journal-title":"Nat. Genet."},{"key":"1007_CR26","doi-asserted-by":"publisher","first-page":"e69","DOI":"10.1093\/nar\/gky215","volume":"46","author":"M Wang","year":"2018","unstructured":"Wang, M., Tai, C., E, W. & Wei, L. Define: Deep convolutional neural networks accurately quantify intensities of transcription factor-DNA binding and facilitate evaluation of functional non-coding variants. Nucleic Acids Res. 46, e69 (2018).","journal-title":"Nucleic Acids Res."},{"key":"1007_CR27","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-018-07349-w","volume":"9","author":"Z He","year":"2018","unstructured":"He, Z., Liu, L., Wang, K. & Ionita-Laza, I. A semi-supervised approach for predicting cell-type specific functional consequences of non-coding variation using MPRAs. Nat. Commun. 9, 5199 (2018).","journal-title":"Nat. Commun."},{"key":"1007_CR28","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-019-13212-3","volume":"10","author":"A Wells","year":"2019","unstructured":"Wells, A. et al. Ranking of non-coding pathogenic variants and putative essential regions of the human genome. Nat. Commun. 10, 5241 (2019).","journal-title":"Nat. Commun."},{"key":"1007_CR29","doi-asserted-by":"crossref","unstructured":"Quang, D. & Xie, X. DanQ: a hybrid convolutional and recurrent deep neural network for quantifying the function of DNA sequences. Nucleic Acids Res. 44, e107 (2016).","DOI":"10.1093\/nar\/gkw226"},{"key":"1007_CR30","doi-asserted-by":"publisher","first-page":"2112","DOI":"10.1093\/bioinformatics\/btab083","volume":"37","author":"Y Ji","year":"2021","unstructured":"Ji, Y., Zhou, Z., Liu, H. & Davuluri, R. V. DNABERT: pre-trained bidirectional encoder representations from transformers model for DNA-language in genome. Bioinformatics 37, 2112\u20132120 (2021).","journal-title":"Bioinformatics"},{"key":"1007_CR31","doi-asserted-by":"publisher","first-page":"1196","DOI":"10.1038\/s41592-021-01252-x","volume":"18","author":"Z Avsec","year":"2021","unstructured":"Avsec, Z. et al. Effective gene expression prediction from sequence by integrating long-range interactions. Nat. Methods 18, 1196\u20131203 (2021).","journal-title":"Nat. Methods"},{"key":"1007_CR32","doi-asserted-by":"publisher","first-page":"739","DOI":"10.1101\/gr.227819.117","volume":"28","author":"DR Kelley","year":"2018","unstructured":"Kelley, D. R. et al. Sequential regulatory activity prediction across chromosomes with convolutional neural networks. Genome Res. 28, 739\u2013750 (2018).","journal-title":"Genome Res."},{"key":"1007_CR33","doi-asserted-by":"crossref","unstructured":"Kelley, D. R., Snoek, J. & Rinn, J. L. Basset: learning the regulatory code of the accessible genome with deep convolutional neural networks. Genome Res. 26, 990\u2013999 (2016).","DOI":"10.1101\/gr.200535.115"},{"key":"1007_CR34","doi-asserted-by":"publisher","first-page":"376","DOI":"10.1038\/s42256-020-0201-6","volume":"2","author":"S Tasaki","year":"2020","unstructured":"Tasaki, S., Gaiteri, C., Mostafavi, S. & Wang, Y. Deep learning decodes the principles of differential gene expression. Nat. Mach. Intell. 2, 376\u2013386 (2020).","journal-title":"Nat. Mach. Intell."},{"key":"1007_CR35","doi-asserted-by":"publisher","first-page":"1254806","DOI":"10.1126\/science.1254806","volume":"347","author":"HY Xiong","year":"2015","unstructured":"Xiong, H. Y. et al. The human splicing code reveals new insights into the genetic determinants of disease. Science 347, 1254806 (2015).","journal-title":"Science"},{"key":"1007_CR36","doi-asserted-by":"publisher","first-page":"1111","DOI":"10.1038\/s41592-020-0958-x","volume":"17","author":"G Fudenberg","year":"2020","unstructured":"Fudenberg, G., Kelley, D. R. & Pollard, K. S. Predicting 3D genome folding from DNA sequence with Akita. Nat. Methods 17, 1111\u20131117 (2020).","journal-title":"Nat. Methods"},{"key":"1007_CR37","doi-asserted-by":"publisher","first-page":"354","DOI":"10.1038\/s41588-021-00782-6","volume":"53","author":"Z Avsec","year":"2021","unstructured":"Avsec, Z. et al. Base-resolution models of transcription-factor binding reveal soft motif syntax. Nat. Genet. 53, 354\u2013366 (2021).","journal-title":"Nat. Genet."},{"key":"1007_CR38","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-021-21790-4","volume":"12","author":"D Vitsios","year":"2021","unstructured":"Vitsios, D., Dhindsa, R. S., Middleton, L., Gussow, A. B. & Petrovski, S. Prioritizing non-coding regions based on human genomic constraint and sequence context with deep learning. Nat. Commun. 12, 1504 (2021).","journal-title":"Nat. Commun."},{"key":"1007_CR39","doi-asserted-by":"publisher","unstructured":"Zhou, Z. et al. DNABERT-2: efficient foundation model and benchmark for multi-species genome. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2306.15006 (2023).","DOI":"10.48550\/arXiv.2306.15006"},{"key":"1007_CR40","doi-asserted-by":"crossref","unstructured":"Cui, H., Wang, C., Maan, H. & Wang, B. scGPT: towards building a foundation model for single-cell multi-omics using generative AI. Nat. Methods 21, 1470\u20131480 (2024).","DOI":"10.1038\/s41592-024-02201-0"},{"key":"1007_CR41","doi-asserted-by":"crossref","unstructured":"Theodoris, C. V. et al. Transfer learning enables predictions in network biology. Nature 618, 616\u2013624 (2023).","DOI":"10.1038\/s41586-023-06139-9"},{"key":"1007_CR42","doi-asserted-by":"crossref","unstructured":"Tan, J. et al. Cell-type-specific prediction of 3D chromatin organization enables high-throughput in silico genetic screening. Nat. Biotechnol. 41, 1140\u20131150 (2023).","DOI":"10.1038\/s41587-022-01612-8"},{"key":"1007_CR43","doi-asserted-by":"publisher","first-page":"287","DOI":"10.1038\/s41592-024-02523-z","volume":"22","author":"H Dalla-Torre","year":"2025","unstructured":"Dalla-Torre, H. et al. Nucleotide Transformer: building and evaluating robust foundation models for human genomics. Nat. Methods 22, 287\u2013297 (2025).","journal-title":"Nat. Methods"},{"key":"1007_CR44","doi-asserted-by":"publisher","unstructured":"Bolya, D., Fu, C.-Y., Dai, X., Zhang, P. & Hoffman, J. Hydra Attention: efficient attention with many heads. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2209.07484 (2022).","DOI":"10.48550\/arXiv.2209.07484"},{"key":"1007_CR45","doi-asserted-by":"publisher","unstructured":"Ma, X. et al. Mega: moving average equipped gated attention. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2209.10655 (2022).","DOI":"10.48550\/arXiv.2209.10655"},{"key":"1007_CR46","doi-asserted-by":"publisher","unstructured":"Nguyen, E. et al. HyenaDNA: long-range genomic sequence modeling at single nucleotide resolution. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2306.15794 (2023).","DOI":"10.48550\/arXiv.2306.15794"},{"key":"1007_CR47","doi-asserted-by":"publisher","first-page":"257","DOI":"10.1042\/ETLS20160025","volume":"1","author":"W Jones","year":"2017","unstructured":"Jones, W., Alasoo, K., Fishman, D. & Parts, L. Computational biology: deep learning. Emerg. Top. Life Sci. 1, 257\u2013274 (2017).","journal-title":"Emerg. Top. Life Sci."},{"key":"1007_CR48","doi-asserted-by":"publisher","first-page":"20170387","DOI":"10.1098\/rsif.2017.0387","volume":"15","author":"T Ching","year":"2018","unstructured":"Ching, T. et al. Opportunities and obstacles for deep learning in biology and medicine. J. R. Soc. Interface 15, 20170387 (2018).","journal-title":"J. R. Soc. Interface"},{"key":"1007_CR49","first-page":"851","volume":"18","author":"S Min","year":"2017","unstructured":"Min, S., Lee, B. & Yoon, S. Deep learning in bioinformatics. Brief. Bioinform. 18, 851\u2013869 (2017).","journal-title":"Brief. Bioinform."},{"key":"1007_CR50","doi-asserted-by":"publisher","first-page":"1761","DOI":"10.1038\/s41593-019-0520-2","volume":"22","author":"BA Richards","year":"2019","unstructured":"Richards, B. A. et al. A deep learning framework for neuroscience. Nat. Neurosci. 22, 1761\u20131770 (2019).","journal-title":"Nat. Neurosci."},{"key":"1007_CR51","doi-asserted-by":"publisher","first-page":"829","DOI":"10.1038\/nbt.4233","volume":"36","author":"M Wainberg","year":"2018","unstructured":"Wainberg, M., Merico, D., Delong, A. & Frey, B. J. Deep learning in biomedicine. Nat. Biotechnol. 36, 829\u2013838 (2018).","journal-title":"Nat. Biotechnol."},{"key":"1007_CR52","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1038\/s41576-022-00532-2","volume":"24","author":"G Novakovsky","year":"2023","unstructured":"Novakovsky, G., Dexter, N., Libbrecht, M. W., Wasserman, W. W. & Mostafavi, S. Obtaining genetics insights from deep learning via explainable artificial intelligence. Nat. Rev. Genet. 24, 125\u2013137 (2023).","journal-title":"Nat. Rev. Genet."},{"key":"1007_CR53","doi-asserted-by":"publisher","first-page":"bbaa177","DOI":"10.1093\/bib\/bbaa177","volume":"22","author":"A Talukder","year":"2021","unstructured":"Talukder, A., Barham, C., Li, X. & Hu, H. Interpretation of deep learning in genomics and epigenomics. Brief. Bioinform. 22, bbaa177 (2021).","journal-title":"Brief. Bioinform."},{"key":"1007_CR54","doi-asserted-by":"publisher","first-page":"100384","DOI":"10.1016\/j.crmeth.2022.100384","volume":"3","author":"Z Li","year":"2023","unstructured":"Li, Z. et al. Applications of deep learning in understanding gene regulation. Cell Rep. Methods 3, 100384 (2023).","journal-title":"Cell Rep. Methods"},{"key":"1007_CR55","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1038\/s41576-019-0122-6","volume":"20","author":"G Eraslan","year":"2019","unstructured":"Eraslan, G., Avsec, Z., Gagneur, J. & Theis, F. J. Deep learning: new computational modelling techniques for genomics. Nat. Rev. Genet. 20, 389\u2013403 (2019).","journal-title":"Nat. Rev. Genet."},{"key":"1007_CR56","doi-asserted-by":"publisher","first-page":"e13613","DOI":"10.7717\/peerj.13613","volume":"10","author":"E Routhier","year":"2022","unstructured":"Routhier, E. & Mozziconacci, J. Genomics enters the deep learning era. PeerJ 10, e13613 (2022).","journal-title":"PeerJ"},{"key":"1007_CR57","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-022-29268-7","volume":"13","author":"N Sapoval","year":"2022","unstructured":"Sapoval, N. et al. Current progress and open challenges for applying deep learning across the biosciences. Nat. Commun. 13, 1728 (2022).","journal-title":"Nat. Commun."},{"key":"1007_CR58","doi-asserted-by":"crossref","unstructured":"Muse, S. Introduction to Biomedical Engineering 2nd edn (eds Enderle, J. D. et al.) Ch. 13, 799\u2013831 (2005).","DOI":"10.1016\/B978-0-12-238662-6.50015-X"},{"key":"1007_CR59","doi-asserted-by":"publisher","unstructured":"Marin, F. I. et al. BEND: benchmarking DNA language models on biologically meaningful tasks. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2311.12570 (2024).","DOI":"10.48550\/arXiv.2311.12570"},{"key":"1007_CR60","doi-asserted-by":"publisher","first-page":"D36","DOI":"10.1093\/nar\/gks1195","volume":"41","author":"DA Benson","year":"2013","unstructured":"Benson, D. A. et al. GenBank. Nucleic Acids Res. 41, D36\u2013D42 (2013).","journal-title":"Nucleic Acids Res."},{"key":"1007_CR61","doi-asserted-by":"publisher","first-page":"D733","DOI":"10.1093\/nar\/gkv1189","volume":"44","author":"NA O\u2019Leary","year":"2016","unstructured":"O\u2019Leary, N. A. et al. Reference sequence (RefSeq) database at NCBI: current status, taxonomic expansion, and functional annotation. Nucleic Acids Res. 44, D733\u2013D745 (2016).","journal-title":"Nucleic Acids Res."},{"key":"1007_CR62","doi-asserted-by":"publisher","first-page":"D19","DOI":"10.1093\/nar\/gkq1019","volume":"39","author":"R Leinonen","year":"2011","unstructured":"Leinonen, R., Sugawara, H. & Shumway, M. The Sequence Read Archive. Nucleic Acids Res. 39, D19\u2013D21 (2011).","journal-title":"Nucleic Acids Res."},{"key":"1007_CR63","doi-asserted-by":"publisher","first-page":"pdb.prot5384","DOI":"10.1101\/pdb.prot5384","volume":"2010","author":"L Song","year":"2010","unstructured":"Song, L. & Crawford, G. E. DNase-seq: a high-resolution technique for mapping active gene regulatory elements across the genome from mammalian cells. Cold Spring Harb. Protoc. 2010, pdb.prot5384 (2010).","journal-title":"Cold Spring Harb. Protoc."},{"key":"1007_CR64","doi-asserted-by":"publisher","first-page":"268","DOI":"10.1016\/j.ymeth.2012.05.001","volume":"58","author":"J-M Belton","year":"2012","unstructured":"Belton, J.-M. et al. Hi-C: a comprehensive technique to capture the conformation of genomes. Methods 58, 268\u2013276 (2012).","journal-title":"Methods"},{"key":"1007_CR65","doi-asserted-by":"publisher","first-page":"723","DOI":"10.1038\/s41592-024-02216-7","volume":"21","author":"D Yao","year":"2024","unstructured":"Yao, D. et al. Multicenter integrated analysis of noncoding CRISPRi screens. Nat. Methods 21, 723\u2013734 (2024).","journal-title":"Nat. Methods"},{"key":"1007_CR66","doi-asserted-by":"publisher","first-page":"699","DOI":"10.1038\/s41586-020-2493-4","volume":"583","author":"ENCODE Project Consortium et al.","year":"2020","unstructured":"ENCODE Project Consortium et al. Expanded encyclopaedias of DNA elements in the human and mouse genomes. Nature 583, 699\u2013710 (2020).","journal-title":"Nature"},{"key":"1007_CR67","doi-asserted-by":"publisher","first-page":"eaaw6507","DOI":"10.1126\/sciadv.aaw6507","volume":"5","author":"JS Satterlee","year":"2019","unstructured":"Satterlee, J. S. et al. The NIH Common Fund\/Roadmap Epigenomics Program: successes of a comprehensive consortium. Sci. Adv. 5, eaaw6507 (2019).","journal-title":"Sci. Adv."},{"key":"1007_CR68","doi-asserted-by":"publisher","first-page":"580","DOI":"10.1038\/ng.2653","volume":"45","author":"J Lonsdale","year":"2013","unstructured":"Lonsdale, J. et al. The Genotype-Tissue Expression (GTEx) project. Nat. Genet. 45, 580\u2013585 (2013).","journal-title":"Nat. Genet."},{"key":"1007_CR69","doi-asserted-by":"publisher","first-page":"68","DOI":"10.1038\/nature15393","volume":"526","author":"A Auton","year":"2015","unstructured":"Auton, A. et al. A global reference for human genetic variation. Nature 526, 68\u201374 (2015).","journal-title":"Nature"},{"key":"1007_CR70","doi-asserted-by":"publisher","unstructured":"Sennrich, R., Haddow, B. & Birch, A. Neural machine translation of rare words with subword units. Preprint at https:\/\/doi.org\/10.48550\/arXiv.1508.07909 (2016).","DOI":"10.48550\/arXiv.1508.07909"},{"key":"1007_CR71","doi-asserted-by":"publisher","first-page":"e82819","DOI":"10.7554\/eLife.82819","volume":"12","author":"A Chandra","year":"2023","unstructured":"Chandra, A., T\u00fcnnermann, L., L\u00f6fstedt, T. & Gratz, R. Transformer-based deep learning for predicting protein properties in the life sciences. eLife 12, e82819 (2023).","journal-title":"eLife"},{"key":"1007_CR72","doi-asserted-by":"publisher","first-page":"725","DOI":"10.1038\/s41588-022-01065-4","volume":"54","author":"J Zhou","year":"2022","unstructured":"Zhou, J. Sequence-based modeling of three-dimensional genome architecture from kilobase to chromosome scale. Nat. Genet. 54, 725\u2013734 (2022).","journal-title":"Nat. Genet."},{"key":"1007_CR73","doi-asserted-by":"publisher","unstructured":"Tang, Z. & Koo, P. K. Evaluating the representational power of pre-trained DNA language models for regulatory genomics. Preprint at bioRxiv https:\/\/doi.org\/10.1101\/2024.02.29.582810 (2024).","DOI":"10.1101\/2024.02.29.582810"},{"key":"1007_CR74","unstructured":"Krizhevsky, A., Sutskever, I. & Hinton, G. ImageNet classification with deep convolutional neural networks. In NIPS'12: Proc. 26th International Conference on Neural Information Processing Systems Vol. 1, 1097\u20131105 (NIPS, 2012)."},{"key":"1007_CR75","doi-asserted-by":"publisher","first-page":"179","DOI":"10.1207\/s15516709cog1402_1","volume":"14","author":"JL Elman","year":"1990","unstructured":"Elman, J. L. Finding structure in time. Cogn. Sci. 14, 179\u2013211 (1990).","journal-title":"Cogn. Sci."},{"key":"1007_CR76","doi-asserted-by":"publisher","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","volume":"9","author":"S Hochreiter","year":"1997","unstructured":"Hochreiter, S. & Schmidhuber, J. Long short-term memory. Neural Comput. 9, 1735\u20131780 (1997).","journal-title":"Neural Comput."},{"key":"1007_CR77","doi-asserted-by":"publisher","unstructured":"Vaswani, A. et al. Attention is all you need. Preprint at https:\/\/doi.org\/10.48550\/arXiv.1706.03762 (2017).","DOI":"10.48550\/arXiv.1706.03762"},{"key":"1007_CR78","unstructured":"Wang, T. et al. What language model architecture and pretraining objective works best for zero-shot generalization? In Int. Conf. Machine Learning 22964\u201322984 (PMLR, 2022)."},{"key":"1007_CR79","doi-asserted-by":"publisher","unstructured":"Poli, M. et al. Hyena Hierarchy: towards larger convolutional language models. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2302.10866 (2023).","DOI":"10.48550\/arXiv.2302.10866"},{"key":"1007_CR80","doi-asserted-by":"publisher","unstructured":"Tay, Y. et al. Are pre-trained convolutions better than pre-trained transformers? Preprint at https:\/\/doi.org\/10.48550\/arXiv.2105.03322 (2022).","DOI":"10.48550\/arXiv.2105.03322"},{"key":"1007_CR81","doi-asserted-by":"crossref","unstructured":"Yang, K. K., Lu, A. X. & Fusi, N. Convolutions are competitive with transformers for protein sequence pretraining. Cell Syst. 15, 286\u2013294.e2 (2024).","DOI":"10.1016\/j.cels.2024.01.008"},{"key":"1007_CR82","doi-asserted-by":"publisher","DOI":"10.1126\/scitranslmed.aag3101","volume":"8","author":"CS Greene","year":"2016","unstructured":"Greene, C. S. The future is unsupervised. Sci. Transl. Med. 8, 346ec108 (2016).","journal-title":"Sci. Transl. Med."},{"key":"1007_CR83","doi-asserted-by":"crossref","unstructured":"Benegas, G., Batra, S. S. & Song, Y. S. DNA language models are powerful predictors of genome-wide variant effects. Proc. Natl Acad. Sci. USA 120, e2311219120 (2023).","DOI":"10.1073\/pnas.2311219120"},{"key":"1007_CR84","doi-asserted-by":"publisher","first-page":"eado9336","DOI":"10.1126\/science.ado9336","volume":"386","author":"E Nguyen","year":"2024","unstructured":"Nguyen, E. et al. Sequence modeling and design from molecular to genome scale with Evo. Science 386, eado9336 (2024).","journal-title":"Science"},{"key":"1007_CR85","doi-asserted-by":"crossref","unstructured":"Zhang, Y., Bai, Z. & Imoto, S. Investigation of the BERT model on nucleotide sequences with non-standard pre-training and evaluation of different k-mer embeddings. Bioinformatics 39, btad617 (2023).","DOI":"10.1093\/bioinformatics\/btad617"},{"key":"1007_CR86","doi-asserted-by":"publisher","unstructured":"Gu, A., Goel, K. & R\u00e9, C. Efficiently modeling long sequences with structured state spaces. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2111.00396 (2022).","DOI":"10.48550\/arXiv.2111.00396"},{"key":"1007_CR87","doi-asserted-by":"publisher","unstructured":"Gu, A. & Dao, T. Mamba: linear-time sequence modeling with selective state spaces. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2312.00752 (2024).","DOI":"10.48550\/arXiv.2312.00752"},{"key":"1007_CR88","doi-asserted-by":"publisher","unstructured":"Schiff, Y. et al. Caduceus: bi-directional equivariant long-range dna sequence modeling. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2403.03234 (2024).","DOI":"10.48550\/arXiv.2403.03234"},{"key":"1007_CR89","doi-asserted-by":"crossref","unstructured":"Bishop, C. M. & Bishop, H. Deep Learning: Foundations and Concepts (Springer International, 2024).","DOI":"10.1007\/978-3-031-45468-4"},{"key":"1007_CR90","unstructured":"Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning (MIT Press, 2016)."},{"key":"1007_CR91","unstructured":"MIT Deep Learning 6.S191. http:\/\/introtodeeplearning.com (accessed 11 July 2024)."},{"key":"1007_CR92","doi-asserted-by":"publisher","first-page":"e0130140","DOI":"10.1371\/journal.pone.0130140","volume":"10","author":"S Bach","year":"2015","unstructured":"Bach, S. et al. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS One 10, e0130140 (2015).","journal-title":"PLoS One"},{"key":"1007_CR93","doi-asserted-by":"publisher","unstructured":"Karollus, A., Mauermeier, T. & Gagneur, J. Current sequence-based models capture gene expression determinants in promoters but mostly ignore distal enhancers. Preprint at bioRxiv https:\/\/doi.org\/10.1101\/2022.09.15.508087 (2022).","DOI":"10.1101\/2022.09.15.508087"},{"key":"1007_CR94","doi-asserted-by":"publisher","first-page":"e1008050","DOI":"10.1371\/journal.pcbi.1008050","volume":"16","author":"DR Kelley","year":"2020","unstructured":"Kelley, D. R. Cross-species regulatory sequence activity prediction. PLoS Comput. Biol. 16, e1008050 (2020).","journal-title":"PLoS Comput. Biol."},{"key":"1007_CR95","doi-asserted-by":"publisher","unstructured":"Linder, J., Srivastava, D., Yuan, H., Agarwal, V. & Kelley, D. R. Predicting RNA-seq coverage from DNA sequence as a unifying model of gene regulation. Nat. Genet. https:\/\/doi.org\/10.1038\/s41588-024-02053-6 (2025).","DOI":"10.1038\/s41588-024-02053-6"},{"key":"1007_CR96","doi-asserted-by":"crossref","unstructured":"Fishman, V. et al. GENA-LM: a family of open-source foundational DNA language models for long sequences, Nucleic Acids Res. 53, gkae1310 (2025).","DOI":"10.1093\/nar\/gkae1310"},{"key":"1007_CR97","doi-asserted-by":"publisher","unstructured":"Dao, T., Fu, D. Y., Ermon, S., Rudra, A. & R\u00e9, C. FlashAttention: fast and memory-efficient exact attention with IO-awareness. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2205.14135 (2022).","DOI":"10.48550\/arXiv.2205.14135"},{"key":"1007_CR98","doi-asserted-by":"publisher","unstructured":"Press, O., Smith, N. A. & Lewis, M. Train short, test long: attention with linear biases enables input length extrapolation. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2108.12409 (2022).","DOI":"10.48550\/arXiv.2108.12409"},{"key":"1007_CR99","doi-asserted-by":"publisher","unstructured":"Hu, E. J. et al. LoRA: low-rank adaptation of large language models. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2106.09685 (2021).","DOI":"10.48550\/arXiv.2106.09685"},{"key":"1007_CR100","doi-asserted-by":"publisher","unstructured":"Katharopoulos, A., Vyas, A., Pappas, N. & Fleuret, F. Transformers are RNNs: fast autoregressive transformers with linear attention. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2006.16236 (2020).","DOI":"10.48550\/arXiv.2006.16236"},{"key":"1007_CR101","doi-asserted-by":"publisher","unstructured":"Sun, Y. et al. Retentive Network: a successor to transformer for large language models. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2307.08621 (2023).","DOI":"10.48550\/arXiv.2307.08621"},{"key":"1007_CR102","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1186\/s12863-023-01123-8","volume":"24","author":"K Gresova","year":"2023","unstructured":"Gresova, K., Martinek, V., Cechak, D., Simecek, P. & Alexiou, P. Genomic benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic Data 24, 25 (2023).","journal-title":"BMC Genomic Data"},{"key":"1007_CR103","doi-asserted-by":"publisher","unstructured":"Kaplan, J. et al. Scaling laws for neural language models. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2001.08361 (2020).","DOI":"10.48550\/arXiv.2001.08361"},{"key":"1007_CR104","doi-asserted-by":"publisher","unstructured":"Serrano, Y., Ciudad, A. & Molina, A. Are protein language models compute optimal? Preprint at https:\/\/doi.org\/10.48550\/arXiv.2406.07249 (2024).","DOI":"10.48550\/arXiv.2406.07249"},{"key":"1007_CR105","doi-asserted-by":"publisher","unstructured":"Li, F.-Z., Amini, A. P., Yue, Y., Yang, K. K. & Lu, A. X. Feature reuse and scaling: understanding transfer learning with protein language models. Preprint at bioRxiv https:\/\/doi.org\/10.1101\/2024.02.05.578959 (2024).","DOI":"10.1101\/2024.02.05.578959"},{"key":"1007_CR106","doi-asserted-by":"publisher","first-page":"335","DOI":"10.1002\/qub2.68","volume":"12","author":"CV Theodoris","year":"2024","unstructured":"Theodoris, C. V. Perspectives on benchmarking foundation models for network biology. Quant. Biol. 12, 335\u2013338 (2024).","journal-title":"Quant. Biol."},{"key":"1007_CR107","doi-asserted-by":"publisher","first-page":"75","DOI":"10.1038\/nature11232","volume":"489","author":"RE Thurman","year":"2012","unstructured":"Thurman, R. E. et al. The accessible chromatin landscape of the human genome. Nature 489, 75\u201382 (2012).","journal-title":"Nature"},{"key":"1007_CR108","doi-asserted-by":"publisher","first-page":"1369","DOI":"10.1016\/j.cell.2016.09.037","volume":"167","author":"BM Javierre","year":"2016","unstructured":"Javierre, B. M. et al. Lineage-specific genome architecture links enhancers and non-coding disease variants to target gene promoters. Cell 167, 1369\u20131384 (2016).","journal-title":"Cell"},{"key":"1007_CR109","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-021-21583-9","volume":"12","author":"R Fang","year":"2021","unstructured":"Fang, R. et al. Comprehensive analysis of single cell ATAC-seq data with SnapATAC. Nat. Commun. 12, 1337 (2021).","journal-title":"Nat. Commun."},{"key":"1007_CR110","doi-asserted-by":"publisher","first-page":"1081842","DOI":"10.3389\/fgene.2022.1081842","volume":"13","author":"Y Chen","year":"2022","unstructured":"Chen, Y., Xie, M. & Wen, J. Predicting gene expression from histone modifications with self-attention based neural networks and transfer learning. Front. Genet. 13, 1081842 (2022).","journal-title":"Front. Genet."},{"key":"1007_CR111","doi-asserted-by":"publisher","first-page":"317","DOI":"10.1038\/nature14248","volume":"518","author":"A Kundaje","year":"2015","unstructured":"Kundaje, A. et al. Integrative analysis of 111 reference human epigenomes. Nature 518, 317\u2013330 (2015).","journal-title":"Nature"},{"key":"1007_CR112","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1561\/0400000042","volume":"9","author":"C Dwork","year":"2014","unstructured":"Dwork, C. & Roth, A. The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci. 9, 211\u2013407 (2014).","journal-title":"Found. Trends Theor. Comput. Sci."},{"key":"1007_CR113","doi-asserted-by":"publisher","unstructured":"McMahan, H. B., Moore, E., Ramage, D., Hampson, S. & Arcas, B. A. Y. Communication-efficient learning of deep networks from decentralized data. Preprint at https:\/\/doi.org\/10.48550\/arXiv.1602.05629 (2016).","DOI":"10.48550\/arXiv.1602.05629"},{"key":"1007_CR114","doi-asserted-by":"publisher","first-page":"bbab060","DOI":"10.1093\/bib\/bbab060","volume":"22","author":"J Clauwaert","year":"2021","unstructured":"Clauwaert, J., Menschaert, G. & Waegeman, W. Explainability in transformer models for functional genomics. Brief. Bioinform. 22, bbab060 (2021).","journal-title":"Brief. Bioinform."},{"key":"1007_CR115","doi-asserted-by":"publisher","unstructured":"Serrano, S. & Smith, N. A. Is attention interpretable? Preprint at https:\/\/doi.org\/10.48550\/arXiv.1906.03731 (2019).","DOI":"10.48550\/arXiv.1906.03731"},{"key":"1007_CR116","doi-asserted-by":"publisher","unstructured":"Chefer, H., Gur, S. & Wolf, L. Transformer interpretability beyond attention visualization. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2012.09838 (2020).","DOI":"10.48550\/arXiv.2012.09838"},{"key":"1007_CR117","doi-asserted-by":"publisher","unstructured":"Voita, E., Talbot, D., Moiseev, F., Sennrich, R. & Titov, I. Analyzing multi-head self-attention: specialized heads do the heavy lifting, the rest can be pruned. Preprint at https:\/\/doi.org\/10.48550\/arXiv.1905.09418 (2019).","DOI":"10.48550\/arXiv.1905.09418"},{"key":"1007_CR118","doi-asserted-by":"publisher","unstructured":"Abnar, S. & Zuidema, W. Quantifying attention flow in transformers. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2005.00928 (2020).","DOI":"10.48550\/arXiv.2005.00928"},{"key":"1007_CR119","doi-asserted-by":"crossref","unstructured":"Binder, A., Montavon, G., Lapuschkin, S., M\u00fcller, K.-R. & Samek, W. Layer-wise relevance propagation for neural networks with local renormalization layers. In Artificial Neural Networks and Machine Learning\u2013ICANN 2016: 25th International Conference on Artificial Neural Networks 63\u201371 (Springer, 2016).","DOI":"10.1007\/978-3-319-44781-0_8"},{"key":"1007_CR120","doi-asserted-by":"crossref","unstructured":"Selvaraju, R. R. et al. Grad-CAM: visual explanations from deep networks via gradient-based localization. In Proc. IEEE International Conference on Computer Vision 618\u2013626 (IEEE, 2017).","DOI":"10.1109\/ICCV.2017.74"},{"key":"1007_CR121","doi-asserted-by":"publisher","unstructured":"Lundberg, S. & Lee, S.-I. A unified approach to interpreting model predictions. Preprint at https:\/\/doi.org\/10.48550\/arXiv.1705.07874 (2017).","DOI":"10.48550\/arXiv.1705.07874"},{"key":"1007_CR122","doi-asserted-by":"publisher","unstructured":"Kwon, Y. & Zou, J. WeightedSHAP: analyzing and improving Shapley based feature attributions. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2209.13429 (2022).","DOI":"10.48550\/arXiv.2209.13429"},{"key":"1007_CR123","doi-asserted-by":"publisher","first-page":"e77","DOI":"10.1093\/nar\/gkab349","volume":"49","author":"F Ullah","year":"2021","unstructured":"Ullah, F. & Ben-Hur, A. A self-attention model for inferring cooperativity between regulatory features. Nucleic Acids Res. 49, e77 (2021).","journal-title":"Nucleic Acids Res."},{"key":"1007_CR124","doi-asserted-by":"publisher","first-page":"2517","DOI":"10.1038\/s41588-024-01923-3","volume":"56","author":"S Toneyan","year":"2024","unstructured":"Toneyan, S. & Koo, P. K. Interpreting cis-regulatory interactions from large-scale deep neural networks. Nat. Genet. 56, 2517\u20132527 (2024).","journal-title":"Nat. Genet."},{"key":"1007_CR125","doi-asserted-by":"publisher","first-page":"e2406285121","DOI":"10.1073\/pnas.2406285121","volume":"121","author":"Z Zhang","year":"2024","unstructured":"Zhang, Z. et al. Protein language models learn evolutionary statistics of interacting sequence motifs. Proc. Natl Acad. Sci. USA 121, e2406285121 (2024).","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"1007_CR126","doi-asserted-by":"publisher","unstructured":"Vig, J. et al. BERTology meets biology: interpreting attention in protein language models. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2006.15222 (2021).","DOI":"10.48550\/arXiv.2006.15222"},{"key":"1007_CR127","doi-asserted-by":"publisher","unstructured":"Bommasani, R. et al. On the opportunities and risks of foundation models. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2108.07258 (2022).","DOI":"10.48550\/arXiv.2108.07258"},{"key":"1007_CR128","doi-asserted-by":"publisher","unstructured":"Kedzierska, K. Z., Crawford, L., Amini, A. P. & Lu, A. X. Assessing the limits of zero-shot foundation models in single-cell biology. Preprint at bioRxiv https:\/\/doi.org\/10.1101\/2023.10.16.561085 (2023).","DOI":"10.1101\/2023.10.16.561085"},{"key":"1007_CR129","doi-asserted-by":"publisher","unstructured":"Lu, A. X., Lu, A. X. & Moses, A. Evolution is all you need: phylogenetic augmentation for contrastive learning. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2012.13475 (2020).","DOI":"10.48550\/arXiv.2012.13475"},{"key":"1007_CR130","doi-asserted-by":"publisher","unstructured":"Benegas, G., Albors, C., Aw, A. J., Ye, C. & Song, Y. S. A DNA language model based on multispecies alignment predicts the effects of genome-wide variants. Nat. Biotechnol. https:\/\/doi.org\/10.1038\/s41587-024-02511-w (2025).","DOI":"10.1038\/s41587-024-02511-w"},{"key":"1007_CR131","doi-asserted-by":"publisher","first-page":"97","DOI":"10.1186\/gm97","volume":"1","author":"VP Belancio","year":"2009","unstructured":"Belancio, V. P., Deininger, P. L. & Roy-Engel, A. M. LINE dancing in the human genome: transposable elements and disease. Genome Med. 1, 97 (2009).","journal-title":"Genome Med."},{"key":"1007_CR132","doi-asserted-by":"publisher","first-page":"852","DOI":"10.1038\/s42256-022-00534-z","volume":"4","author":"F Yang","year":"2022","unstructured":"Yang, F. et al. scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA-seq data. Nat. Mach. Intell. 4, 852\u2013866 (2022).","journal-title":"Nat. Mach. Intell."},{"key":"1007_CR133","doi-asserted-by":"publisher","unstructured":"Levine, D. et al. Cell2Sentence: teaching large language models the language of biology. Preprint at bioRxiv https:\/\/doi.org\/10.1101\/2023.09.11.557287 (2023).","DOI":"10.1101\/2023.09.11.557287"},{"key":"1007_CR134","doi-asserted-by":"publisher","first-page":"1481","DOI":"10.1038\/s41592-024-02305-7","volume":"21","author":"M Hao","year":"2024","unstructured":"Hao, M. et al. Large scale foundation model on single-cell transcriptomics. Nat. Methods 21, 1481\u20131491 (2024).","journal-title":"Nat. Methods"},{"key":"1007_CR135","doi-asserted-by":"publisher","first-page":"1430","DOI":"10.1038\/s41592-024-02353-z","volume":"21","author":"A Sza\u0142ata","year":"2024","unstructured":"Sza\u0142ata, A. et al. Transformers in single-cell omics: a review and new perspectives. Nat. Methods 21, 1430\u20131443 (2024).","journal-title":"Nat. Methods"},{"key":"1007_CR136","doi-asserted-by":"publisher","first-page":"433","DOI":"10.1002\/qub2.65","volume":"12","author":"M Hao","year":"2024","unstructured":"Hao, M. et al. Current opinions on large cellular models. Quant. Biol. 12, 433\u2013443 (2024).","journal-title":"Quant. Biol."},{"key":"1007_CR137","doi-asserted-by":"publisher","unstructured":"Hassani, A. & Shi, H. Dilated neighborhood attention transformer. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2209.15001 (2022).","DOI":"10.48550\/arXiv.2209.15001"},{"key":"1007_CR138","doi-asserted-by":"publisher","unstructured":"Bolya, D. et al. Token Merging: your ViT but faster. Preprint at https:\/\/doi.org\/10.48550\/arXiv.2210.09461 (2022).","DOI":"10.48550\/arXiv.2210.09461"},{"key":"1007_CR139","doi-asserted-by":"publisher","unstructured":"Alamdari, S. et al. Protein generation with evolutionary diffusion: sequence is all you need. Preprint at bioRxiv https:\/\/doi.org\/10.1101\/2023.09.11.556673 (2023).","DOI":"10.1101\/2023.09.11.556673"},{"key":"1007_CR140","first-page":"6840","volume":"33","author":"J Ho","year":"2020","unstructured":"Ho, J., Jain, A. & Abbeel, P. Denoising diffusion probabilistic models. Adv. Neural Inf. Process. Syst. 33, 6840\u20136851 (2020).","journal-title":"Adv. Neural Inf. Process. Syst."},{"key":"1007_CR141","doi-asserted-by":"publisher","first-page":"144","DOI":"10.1073\/pnas.41.3.144","volume":"41","author":"M Kimura","year":"1955","unstructured":"Kimura, M. Solution of a process of random genetic drift with a continuous model. Proc. Natl Acad. Sci. USA 41, 144\u2013150 (1955).","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"1007_CR142","doi-asserted-by":"publisher","first-page":"33","DOI":"10.1101\/SQB.1955.020.01.006","volume":"20","author":"M Kimura","year":"1955","unstructured":"Kimura, M. Stochastic processes and distribution of gene frequencies under natural selection. Cold Spring Harb. Symp. Quant. Biol. 20, 33\u201353 (1955).","journal-title":"Cold Spring Harb. Symp. Quant. Biol."},{"key":"1007_CR143","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1093\/genetics\/169.1.1","volume":"169","author":"J Wakeley","year":"2005","unstructured":"Wakeley, J. The limits of theoretical population. Genetics 169, 1\u20137 (2005).","journal-title":"Genetics"},{"key":"1007_CR144","doi-asserted-by":"publisher","unstructured":"DaSilva, L. F. et al. DNA-Diffusion: leveraging generative models for controlling chromatin accessibility and gene expression via synthetic regulatory elements. Preprint at bioRxiv https:\/\/doi.org\/10.1101\/2024.02.01.578352 (2024).","DOI":"10.1101\/2024.02.01.578352"}],"container-title":["Nature Machine Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s42256-025-01007-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-025-01007-9","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-025-01007-9.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,3,26]],"date-time":"2025-03-26T23:29:14Z","timestamp":1743031754000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s42256-025-01007-9"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2025,3,13]]},"references-count":144,"journal-issue":{"issue":"3","published-online":{"date-parts":[[2025,3]]}},"alternative-id":["1007"],"URL":"https:\/\/doi.org\/10.1038\/s42256-025-01007-9","relation":{},"ISSN":["2522-5839"],"issn-type":[{"value":"2522-5839","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,3,13]]},"assertion":[{"value":"29 January 2024","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"31 January 2025","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 March 2025","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}