{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,24]],"date-time":"2026-07-24T04:14:52Z","timestamp":1784866492419,"version":"3.55.0"},"reference-count":136,"publisher":"Oxford University Press (OUP)","issue":"6","license":[{"start":{"date-parts":[[2024,10,27]],"date-time":"2024-10-27T00:00:00Z","timestamp":1729987200000},"content-version":"vor","delay-in-days":34,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Shenzhen-Hong Kong Joint Funding Project","award":["SGDX20230116092056010"],"award-info":[{"award-number":["SGDX20230116092056010"]}]},{"name":"Research Grants Council of the Hong Kong SAR","award":["RGC GRF 2151185"],"award-info":[{"award-number":["RGC GRF 2151185"]}]},{"name":"Research Grants Council of the Hong Kong SAR","award":["CUHK 14222922"],"award-info":[{"award-number":["CUHK 14222922"]}]},{"name":"Research Grants Council of the Hong Kong SAR","award":["CUHK 24204023"],"award-info":[{"award-number":["CUHK 24204023"]}]},{"DOI":"10.13039\/501100004853","name":"Chinese University of Hong Kong","doi-asserted-by":"publisher","award":["8601663"],"award-info":[{"award-number":["8601663"]}],"id":[{"id":"10.13039\/501100004853","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004853","name":"Chinese University of Hong Kong","doi-asserted-by":"publisher","award":["8601603"],"award-info":[{"award-number":["8601603"]}],"id":[{"id":"10.13039\/501100004853","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004853","name":"Chinese University of Hong Kong","doi-asserted-by":"publisher","award":["5501329"],"award-info":[{"award-number":["5501329"]}],"id":[{"id":"10.13039\/501100004853","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004853","name":"Chinese University of Hong Kong","doi-asserted-by":"publisher","award":["5501517"],"award-info":[{"award-number":["5501517"]}],"id":[{"id":"10.13039\/501100004853","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004853","name":"Chinese University of Hong Kong","doi-asserted-by":"publisher","award":["4937026"],"award-info":[{"award-number":["4937026"]}],"id":[{"id":"10.13039\/501100004853","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100004853","name":"Chinese University of Hong Kong","doi-asserted-by":"publisher","award":["4937025"],"award-info":[{"award-number":["4937025"]}],"id":[{"id":"10.13039\/501100004853","id-type":"DOI","asserted-by":"publisher"}]},{"name":"Innovation and Technology Commission of the Hong Kong Special Administrative Region, China","award":["GHP\/065\/21SZ"],"award-info":[{"award-number":["GHP\/065\/21SZ"]}]},{"name":"Innovation and Technology Commission of the Hong Kong Special Administrative Region, China","award":["GHP\/065\/21SZ"],"award-info":[{"award-number":["GHP\/065\/21SZ"]}]},{"name":"Research Grants Council of the Hong Kong Special Administrative Region, China","award":["CUHK 24204023"],"award-info":[{"award-number":["CUHK 24204023"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,9,23]]},"abstract":"<jats:title>Abstract<\/jats:title>\n               <jats:p>Bioinformatics has undergone a paradigm shift in artificial intelligence (AI), particularly through foundation models (FMs), which address longstanding challenges in bioinformatics such as limited annotated data and data noise. These AI techniques have demonstrated remarkable efficacy across various downstream validation tasks, effectively representing diverse biological entities and heralding a new era in computational biology. The primary goal of this survey is to conduct a general investigation and summary of FMs in bioinformatics, tracing their evolutionary trajectory, current research landscape, and methodological frameworks. Our primary focus is on elucidating the application of FMs to specific biological problems, offering insights to guide the research community in choosing appropriate FMs for tasks like sequence analysis, structure prediction, and function annotation. Each section delves into the intricacies of the targeted challenges, contrasting the architectures and advancements of FMs with conventional methods and showcasing their utility across different biological domains. Further, this review scrutinizes the hurdles and constraints encountered by FMs in biology, including issues of data noise, model interpretability, and potential biases. This analysis provides a theoretical groundwork for understanding the circumstances under which certain FMs may exhibit suboptimal performance. Lastly, we outline prospective pathways and methodologies for the future development of FMs in biological research, facilitating ongoing innovation in the field. This comprehensive examination not only serves as an academic reference but also as a roadmap for forthcoming explorations and applications of FMs in biology.<\/jats:p>","DOI":"10.1093\/bib\/bbae548","type":"journal-article","created":{"date-parts":[[2024,10,27]],"date-time":"2024-10-27T02:17:26Z","timestamp":1729995446000},"source":"Crossref","is-referenced-by-count":57,"title":["Progress and opportunities of foundation models in bioinformatics"],"prefix":"10.1093","volume":"25","author":[{"ORCID":"https:\/\/orcid.org\/0009-0000-5628-678X","authenticated-orcid":false,"given":"Qing","family":"Li","sequence":"first","affiliation":[{"name":"Department of Computer Science and Engineering , The , Shatin, New Territories, Hong Kong SAR, 999077 ,","place":["China"]},{"name":"Chinese University of Hong Kong , The , Shatin, New Territories, Hong Kong SAR, 999077 ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Zhihang","family":"Hu","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering , The , Shatin, New Territories, Hong Kong SAR, 999077 ,","place":["China"]},{"name":"Chinese University of Hong Kong , The , Shatin, New Territories, Hong Kong SAR, 999077 ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yixuan","family":"Wang","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering , The , Shatin, New Territories, Hong Kong SAR, 999077 ,","place":["China"]},{"name":"Chinese University of Hong Kong , The , Shatin, New Territories, Hong Kong SAR, 999077 ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Lei","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering , The , Shatin, New Territories, Hong Kong SAR, 999077 ,","place":["China"]},{"name":"Chinese University of Hong Kong , The , Shatin, New Territories, Hong Kong SAR, 999077 ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yimin","family":"Fan","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering , The , Shatin, New Territories, Hong Kong SAR, 999077 ,","place":["China"]},{"name":"Chinese University of Hong Kong , The , Shatin, New Territories, Hong Kong SAR, 999077 ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Irwin","family":"King","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering , The , Shatin, New Territories, Hong Kong SAR, 999077 ,","place":["China"]},{"name":"Chinese University of Hong Kong , The , Shatin, New Territories, Hong Kong SAR, 999077 ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gengjie","family":"Jia","sequence":"additional","affiliation":[{"name":"Shenzhen Branch , Guangdong Laboratory of Lingnan Modern Agriculture, Genome Analysis Laboratory of the Ministry of Agriculture and Rural Affairs, , Shenzhen, Guangdong, 518120 ,","place":["China"]},{"name":"Agricultural Genomics Institute at Shenzhen, Chinese Academy of Agricultural Sciences , Guangdong Laboratory of Lingnan Modern Agriculture, Genome Analysis Laboratory of the Ministry of Agriculture and Rural Affairs, , Shenzhen, Guangdong, 518120 ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Sheng","family":"Wang","sequence":"additional","affiliation":[{"name":"Shanghai Zelixir Biotech Company Ltd. , Shanghai, 200030 ,","place":["China"]},{"name":"Shenzhen Institute of Advanced Technology , Xueyuan Avenue, Shenzhen University Town, Nanshan District, Shenzhen, Guangdong, 518055 ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Le","family":"Song","sequence":"additional","affiliation":[{"name":"BioMap , Zhongguancun Life Science Park, Haidian District, Beijing, 100085 ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-3664-6722","authenticated-orcid":false,"given":"Yu","family":"Li","sequence":"additional","affiliation":[{"name":"Department of Computer Science and Engineering , The , Shatin, New Territories, Hong Kong SAR, 999077 ,","place":["China"]},{"name":"Chinese University of Hong Kong , The , Shatin, New Territories, Hong Kong SAR, 999077 ,","place":["China"]}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,10,26]]},"reference":[{"key":"2024102702171886500_ref1","doi-asserted-by":"publisher","first-page":"1239","DOI":"10.1111\/j.1476-5381.2010.01127.x","article-title":"Principles of early drug discovery","volume":"162","author":"Hughes","year":"2011","journal-title":"Br J Pharmacol"},{"key":"2024102702171886500_ref2","article-title":"On the opportunities and risks of foundation models.","author":"Bommasani","year":"2021"},{"key":"2024102702171886500_ref3","doi-asserted-by":"publisher","first-page":"44","DOI":"10.1038\/s41591-018-0300-7","article-title":"High-performance medicine: the convergence of human and artificial intelligence","volume":"25","author":"Topol","year":"2019","journal-title":"Nat Med"},{"key":"2024102702171886500_ref4","doi-asserted-by":"publisher","first-page":"123","DOI":"10.1016\/B978-0-444-63623-2.00007-4","volume-title":"Developments in Environmental Modeling","author":"Park","year":"2016"},{"key":"2024102702171886500_ref5","doi-asserted-by":"publisher","first-page":"e69","DOI":"10.1093\/nar\/gky215","article-title":"DeFine: deep convolutional neural networks accurately quantify intensities of transcription factor-DNA binding and facilitate evaluation of functional non-coding variants","volume":"46","author":"Wang","year":"2018","journal-title":"Nucleic Acids Res"},{"key":"2024102702171886500_ref6","doi-asserted-by":"publisher","first-page":"3125","DOI":"10.1038\/s41467-021-23420-5","article-title":"Finding gene network topologies for given biological function with recurrent neural network","volume":"12","author":"Shen","year":"2021","journal-title":"Nat Commun"},{"key":"2024102702171886500_ref7","doi-asserted-by":"publisher","first-page":"488","DOI":"10.1038\/ng.3539","article-title":"Enhancer\u2013promoter interactions are encoded by complex genomic signatures on looping chromatin","volume":"48","author":"Whalen","year":"2016","journal-title":"Nat Genet"},{"key":"2024102702171886500_ref8","doi-asserted-by":"publisher","first-page":"1250","DOI":"10.1038\/s41592-022-01616-x","article-title":"BIONIC: biological network integration using convolutions","volume":"19","author":"Forster","year":"2022","journal-title":"Nat Methods"},{"key":"2024102702171886500_ref9","doi-asserted-by":"publisher","first-page":"1739","DOI":"10.1038\/s41467-022-29439-6","article-title":"Deciphering spatial domains from spatially resolved transcriptomics with an adaptive graph attention auto-encoder","volume":"13","author":"Dong","year":"2022","journal-title":"Nat Commun"},{"key":"2024102702171886500_ref10","doi-asserted-by":"publisher","first-page":"2063","DOI":"10.1109\/TNNLS.2018.2790388","article-title":"Applications of deep learning and reinforcement learning to biological data","volume":"29","author":"Mahmud","year":"2018","journal-title":"IEEE Trans Neural Netw Learn Syst"},{"key":"2024102702171886500_ref11","doi-asserted-by":"publisher","first-page":"e220119","DOI":"10.1148\/ryai.220119","article-title":"On the opportunities and risks of foundation models for natural language processing in radiology","volume":"4","author":"Wiggins","year":"2022","journal-title":"Radiol Artif Intell"},{"key":"2024102702171886500_ref12","first-page":"24639","article-title":"Video pretraining (vpt): learning to act by watching unlabeled online videos","volume":"35","author":"Baker","year":"2022","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024102702171886500_ref13","article-title":"The AI teacher test: measuring the pedagogical ability of blender and GPT-3 in educational dialogues","author":"Tack","year":"2022"},{"key":"2024102702171886500_ref14","doi-asserted-by":"publisher","first-page":"259","DOI":"10.1038\/s41586-023-05881-4","article-title":"Foundation models for generalist medical artificial intelligence","volume":"616","author":"Moor","year":"2023","journal-title":"Nature"},{"key":"2024102702171886500_ref15","first-page":"8844","article-title":"MSA transformer","volume-title":"International Conference on Machine Learning","author":"Rao"},{"key":"2024102702171886500_ref16","doi-asserted-by":"publisher","first-page":"1728","DOI":"10.1038\/s41467-022-29268-7","article-title":"Current progress and open challenges for applying deep learning across the biosciences","volume":"13","author":"Sapoval","year":"2022","journal-title":"Nat Commun"},{"key":"2024102702171886500_ref17","doi-asserted-by":"publisher","first-page":"616","DOI":"10.1038\/s41586-023-06139-9","article-title":"Transfer learning enables predictions in network biology","volume":"618","author":"Theodoris","year":"2023","journal-title":"Nature"},{"key":"2024102702171886500_ref18","doi-asserted-by":"publisher","first-page":"12","DOI":"10.1038\/s41588-018-0295-5","article-title":"A primer on deep learning in genomics","volume":"51","author":"Zou","year":"2019","journal-title":"Nat Genet"},{"key":"2024102702171886500_ref19","doi-asserted-by":"publisher","first-page":"73","DOI":"10.1109\/MSP.2021.3123589","article-title":"A practical guide to supervised deep learning for bioimage analysis: challenges and good practices","volume":"39","author":"Uhlmann","year":"2022","journal-title":"IEEE Signal Process Mag"},{"key":"2024102702171886500_ref20","doi-asserted-by":"publisher","first-page":"276","DOI":"10.1038\/nrg1315","article-title":"Applied bioinformatics for the identification of regulatory elements","volume":"5","author":"Wasserman","year":"2004","journal-title":"Nat Rev Genet"},{"key":"2024102702171886500_ref21","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/P18-1031","article-title":"Universal language model fine-tuning for text classification","author":"Howard","year":"2018"},{"key":"2024102702171886500_ref22","article-title":"Florence: a new foundation model for computer","author":"Yuan","year":"2021"},{"key":"2024102702171886500_ref24","article-title":"Bert: pre-training of deep bidirectional transformers for language","author":"Devlin","year":"2018"},{"key":"2024102702171886500_ref25","doi-asserted-by":"publisher","first-page":"1234","DOI":"10.1093\/bioinformatics\/btz682","article-title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","volume":"36","author":"Lee","year":"2020","journal-title":"Bioinformatics"},{"key":"2024102702171886500_ref26","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3458754","article-title":"Domain-specific language model pretraining for biomedical natural language processing","volume":"3","author":"Gu","year":"2021","journal-title":"ACM Trans Comput Healthc"},{"key":"2024102702171886500_ref27","doi-asserted-by":"publisher","first-page":"2112","DOI":"10.1093\/bioinformatics\/btab083","article-title":"DNABERT: pre-trained bidirectional encoder representations from transformers model for DNA-language in genome","volume":"37","author":"Ji","year":"2021","journal-title":"Bioinformatics"},{"key":"2024102702171886500_ref28","doi-asserted-by":"publisher","first-page":"2102","DOI":"10.1093\/bioinformatics\/btac020","article-title":"Proteinbert: a universal deep-learning model of protein sequence and function","volume":"38","author":"Brandes","year":"2022","journal-title":"Bioinformatics"},{"key":"2024102702171886500_ref29","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"2024102702171886500_ref30","article-title":"An early evaluation of gpt-4v(ision).","author":"Wu"},{"key":"2024102702171886500_ref31","first-page":"500902, 2022","article-title":"Language models of protein sequences at the scale of evolution enable accurate structure prediction","author":"Lin","journal-title":"bioRxiv"},{"key":"2024102702171886500_ref32","article-title":"Simulating 500 million years of evolution with a language model","author":"Hayes","year":"2024","journal-title":"bioRxiv"},{"key":"2024102702171886500_ref33","first-page":"1","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","volume":"21","author":"Raffel","year":"2020","journal-title":"J Mach Learn Res"},{"key":"2024102702171886500_ref34","first-page":"16857","article-title":"Mpnet: masked and permuted pre-training for language understanding","volume":"33","author":"Song","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024102702171886500_ref35","doi-asserted-by":"publisher","first-page":"1196","DOI":"10.1038\/s41592-021-01252-x","article-title":"Effective gene expression prediction from sequence by integrating long-range interactions","volume":"18","author":"Avsec","year":"2021","journal-title":"Nat Methods"},{"key":"2024102702171886500_ref36","article-title":"Protst: multi-modality learning of protein sequences and biomedical","author":"Xu","year":"2023"},{"key":"2024102702171886500_ref37","doi-asserted-by":"publisher","first-page":"4348","DOI":"10.1038\/s41467-022-32007-7","article-title":"ProtGPT2 is a deep unsupervised language model for protein design","volume":"13","author":"Ferruz","year":"2022","journal-title":"Nat Commun"},{"key":"2024102702171886500_ref38","doi-asserted-by":"crossref","article-title":"xTrimoPGLM: unified 100B-scale pre-trained transformer for deciphering the language of protein","author":"Chen","DOI":"10.1101\/2023.07.05.547496"},{"key":"2024102702171886500_ref38a","first-page":"15912","volume":"2306","author":"Liu","year":"2023","journal-title":"arXiv preprint arXiv"},{"key":"2024102702171886500_ref39","doi-asserted-by":"publisher","first-page":"23","DOI":"10.1186\/s13040-023-00338-w","article-title":"Assessment of emerging pretraining strategies in interpretable multimodal deep learning for cancer prognostication","volume":"16","author":"Azher","year":"2023","journal-title":"BioData Min"},{"key":"2024102702171886500_ref40","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbad488","article-title":"Protein-DNA binding sites prediction based on pre-trained protein language model and contrastive learning","volume":"25.1","author":"Liu","journal-title":"Briefings in Bioinformatics"},{"key":"2024102702171886500_ref41","article-title":"Hyenadna: long-range genomic sequence modeling at single nucleotide resolution","volume-title":"Advances in Neural Information Processing Systems","author":"Nguyen"},{"key":"2024102702171886500_ref42","article-title":"scGPT: towards building a foundation model for single-cell multi-omics using generative AI","author":"Cui","journal-title":"Nature Methods"},{"key":"2024102702171886500_ref57","doi-asserted-by":"publisher","first-page":"1099","DOI":"10.1038\/s41587-022-01618-2","article-title":"Large language models generate functional protein sequences across diverse families","volume":"41","author":"Madani","year":"2023","journal-title":"Nat Biotechnol"},{"key":"2024102702171886500_ref43","doi-asserted-by":"publisher","first-page":"706","DOI":"10.1038\/s41586-019-1923-7","article-title":"Improved protein structure prediction using potentials from deep learning","volume":"577","author":"Senior","year":"2020","journal-title":"Nature"},{"key":"2024102702171886500_ref44","volume-title":"Proceedings of the 29th ACM International Conference on Information & Knowledge Management","author":"Walsh"},{"key":"2024102702171886500_ref45","doi-asserted-by":"publisher","first-page":"95","DOI":"10.1016\/j.cels.2020.05.010","article-title":"Solo: doublet identification in single-cell RNA-seq via semi-supervised deep learning","volume":"11","author":"Bernstein","year":"2020","journal-title":"Cell Syst"},{"key":"2024102702171886500_ref46","doi-asserted-by":"publisher","first-page":"814","DOI":"10.1016\/j.gpb.2022.11.011","article-title":"Application of deep learning on single-cell RNA sequencing data analysis: a review","volume":"20","author":"Brendel","year":"2022","journal-title":"Genomics Proteomics Bioinformatics"},{"key":"2024102702171886500_ref47","doi-asserted-by":"publisher","first-page":"211","DOI":"10.1186\/s13059-019-1837-6","article-title":"DeepImpute: an accurate, fast, and scalable deep neural network method to impute single-cell RNA-seq data","volume":"20","author":"Arisdakessian","year":"2019","journal-title":"Genome Biol"},{"key":"2024102702171886500_ref48","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13059-019-1850-9","article-title":"A benchmark of batch-effect correction methods for single-cell RNA sequencing data","volume":"21","author":"Tran","year":"2020","journal-title":"Genome Biol"},{"key":"2024102702171886500_ref49","article-title":"Statistical methods for quantitative MS-based proteomics: part I. Preprocessing","author":"Clement"},{"key":"2024102702171886500_ref50","doi-asserted-by":"publisher","first-page":"534","DOI":"10.1186\/s12859-022-05095-x","article-title":"Pro-MAP: a robust pipeline for the pre-processing of single channel protein microarray data","volume":"23","author":"Mowoe","year":"2022","journal-title":"BMC Bioinformatics"},{"key":"2024102702171886500_ref51","article-title":"Accelerating multiple sequence alignment with dense retrieval on protein language","author":"Hong","journal-title":"bioRxiv"},{"key":"2024102702171886500_ref52","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12859-019-3019-7","article-title":"HH-suite3 for fast remote homology detection and deep protein annotation","volume":"20","author":"Steinegger","year":"2019","journal-title":"BMC Bioinformatics"},{"key":"2024102702171886500_ref53","doi-asserted-by":"publisher","first-page":"1237","DOI":"10.1093\/molbev\/msz312","article-title":"Molecular evolutionary genetics analysis (MEGA) for macOS","volume":"37","author":"Stecher","year":"2020","journal-title":"Mol Biol Evol"},{"key":"2024102702171886500_ref54","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbab577","article-title":"Capturing large genomic contexts for accurately predicting enhancer-promoter interactions","volume":"23","author":"Chen","year":"2022","journal-title":"Brief Bioinform"},{"key":"2024102702171886500_ref55","doi-asserted-by":"publisher","first-page":"125","DOI":"10.1038\/s41576-022-00532-2","article-title":"Obtaining genetics insights from deep learning via explainable artificial intelligence","volume":"24","author":"Novakovsky","year":"2023","journal-title":"Nat Rev Genet"},{"key":"2024102702171886500_ref56","doi-asserted-by":"crossref","DOI":"10.1101\/2023.01.11.523679","article-title":"The nucleotide transformer: building and evaluating robust foundation models for human genomics","volume-title":"bioRxiv","author":"Dalla-Torre"},{"key":"2024102702171886500_ref58","doi-asserted-by":"crossref","article-title":"Interpretable RNA foundation model from unannotated data for highly accurate RNA structure and function predictions","author":"Chen","DOI":"10.1101\/2022.08.06.503062"},{"key":"2024102702171886500_ref59","doi-asserted-by":"publisher","first-page":"831","DOI":"10.1038\/nbt.3300","article-title":"Predicting the sequence specificities of DNA-and RNA-binding proteins by deep learning","volume":"33","author":"Alipanahi","year":"2015","journal-title":"Nat Biotechnol"},{"key":"2024102702171886500_ref60","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3560815","article-title":"Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing","volume":"55","author":"Liu","year":"2023","journal-title":"ACM Comput Surv"},{"key":"2024102702171886500_ref61","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s13073-021-00835-9","article-title":"CADD-splice\u2014improving genome-wide variant effect prediction using deep learning-derived splice scores","volume":"13","author":"Rentzsch","year":"2021","journal-title":"Genome Med"},{"key":"2024102702171886500_ref62","doi-asserted-by":"publisher","first-page":"703","DOI":"10.1038\/s41596-019-0128-8","article-title":"Protocol update for large-scale genome and gene function analysis with the PANTHER classification system (v. 14.0)","volume":"14","author":"Mi","year":"2019","journal-title":"Nat Protoc"},{"key":"2024102702171886500_ref63","doi-asserted-by":"publisher","first-page":"43","DOI":"10.1038\/nature09906","article-title":"Mapping and analysis of chromatin state dynamics in nine human cell types","volume":"473","author":"Ernst","year":"2011","journal-title":"Nature"},{"key":"2024102702171886500_ref64","doi-asserted-by":"publisher","first-page":"W98","DOI":"10.1093\/nar\/gkx247","article-title":"GEPIA: a web server for cancer and normal gene expression profiling and interactive analyses","volume":"45","author":"Tang","year":"2017","journal-title":"Nucleic Acids Res"},{"key":"2024102702171886500_ref65","doi-asserted-by":"publisher","first-page":"547","DOI":"10.1038\/s41587-019-0071-9","article-title":"A comparison of single-cell trajectory inference methods","volume":"37","author":"Saelens","year":"2019","journal-title":"Nat Biotechnol"},{"key":"2024102702171886500_ref66","article-title":"Challenges and applications of large language","author":"Kaddour","year":"2023"},{"key":"2024102702171886500_ref67","doi-asserted-by":"publisher","first-page":"2901","DOI":"10.1609\/aaai.v34i03.5681","article-title":"K-bert: enabling language representation with knowledge graph","volume-title":"Proceedings of the AAAI Conference on Artificial Intelligence","author":"Liu","year":"2020"},{"key":"2024102702171886500_ref68","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"Brown","year":"2020","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024102702171886500_ref69","first-page":"37309","article-title":"Deep bidirectional language-knowledge graph pretraining","volume":"35","author":"Yasunaga","year":"2022","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024102702171886500_ref70","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1145\/2629489","article-title":"Wikidata: a free collaborative knowledgebase","volume":"57","author":"Denny","year":"2014","journal-title":"Communications of the ACM"},{"key":"2024102702171886500_ref71","doi-asserted-by":"crossref","article-title":"Aligning books and movies: towards story-like visual explanations by watching movies and reading books","author":"Zhu","DOI":"10.1109\/ICCV.2015.11"},{"key":"2024102702171886500_ref72","doi-asserted-by":"publisher","DOI":"10.1609\/aaai.v31i1.11164","article-title":"Conceptnet 5.5: an open multilingual graph of general knowledge","volume":"31","author":"Speer","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2024102702171886500_ref73","doi-asserted-by":"publisher","first-page":"403","DOI":"10.1038\/s43588-023-00453-y","article-title":"The high-dimensional space of human diseases built from diagnosis records and mapped to genetic loci","volume":"3","author":"Jia","year":"2023","journal-title":"Nat Comput Sci"},{"key":"2024102702171886500_ref74","doi-asserted-by":"publisher","first-page":"5508","DOI":"10.1038\/s41467-019-13455-0","article-title":"Estimating heritability and genetic correlations from large health datasets in the absence of genetic data","volume":"10","author":"Jia","year":"2019","journal-title":"Nat Commun"},{"key":"2024102702171886500_ref75","doi-asserted-by":"publisher","first-page":"172","DOI":"10.1038\/s41586-023-06291-2","article-title":"Large language models encode clinical knowledge","volume":"620","author":"Singhal","year":"2023","journal-title":"Nature"},{"key":"2024102702171886500_ref76","volume-title":"Proceedings of the 20th Workshop on Biomedical Language Processing","author":"Kanakarajan","year":"2021"},{"key":"2024102702171886500_ref77","volume-title":"Proceedings of the 14th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics","author":"Babjac","year":"2023"},{"key":"2024102702171886500_ref78","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2022.bionlp-1.9","article-title":"BioBART: Pretraining and evaluation of a biomedical generative language model","author":"Yuan","year":"2022"},{"key":"2024102702171886500_ref79","doi-asserted-by":"crossref","article-title":"Squad: 100,000+ questions for machine comprehension of text","author":"Rajpurkar","DOI":"10.18653\/v1\/D16-1264"},{"key":"2024102702171886500_ref80","doi-asserted-by":"publisher","first-page":"937","DOI":"10.1038\/nbt.4267","article-title":"How user intelligence is improving PubMed","volume":"36","author":"Fiorini","year":"2018","journal-title":"Nat Biotechnol"},{"key":"2024102702171886500_ref81","article-title":"Medical sam adapter: adapting segment anything model for medical image segmentation. arXiv preprint arXiv:2304.12620, 2023.","author":"Wu"},{"key":"2024102702171886500_ref23","doi-asserted-by":"publisher","first-page":"87","DOI":"10.1016\/j.irbm.2020.05.003","article-title":"Deep transfer learning based classification model for COVID-19 disease","volume":"43","author":"Pathak","year":"2022","journal-title":"Ing Rech Biomed"},{"key":"2024102702171886500_ref82","article-title":"Stanford crfm introduces pubmedgpt 2.7 b","author":"Bolton","year":"2022"},{"key":"2024102702171886500_ref83","article-title":"DNABERT-2: efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2306.15006, 2023.","author":"Zhou"},{"key":"2024102702171886500_ref84","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1186\/s12859-019-3306-3","article-title":"SpliceFinder: ab initio prediction of splice sites using convolutional neural network","volume":"20","author":"Wang","year":"2019","journal-title":"BMC Bioinformatics"},{"key":"2024102702171886500_ref85","doi-asserted-by":"publisher","first-page":"324","DOI":"10.1038\/s42256-021-00310-5","article-title":"Expanding functional protein sequence spaces using generative adversarial networks","volume":"3","author":"Repecka","year":"2021","journal-title":"Nat Mach Intell"},{"key":"2024102702171886500_ref86","doi-asserted-by":"crossref","DOI":"10.1101\/2022.06.08.495248","article-title":"Genomic benchmarks: a collection of datasets for genomic sequence classification","volume-title":"BMC Genomic Data","author":"Gresova"},{"key":"2024102702171886500_ref87","article-title":"High-resolution de novo structure prediction from primary sequence","author":"Wu"},{"key":"2024102702171886500_ref88","doi-asserted-by":"publisher","first-page":"1123","DOI":"10.1126\/science.ade2574","article-title":"Evolutionary-scale prediction of atomic-level protein structure with a language model","volume":"379","author":"Lin","year":"2023","journal-title":"Science"},{"key":"2024102702171886500_ref89","doi-asserted-by":"publisher","first-page":"2389","DOI":"10.1038\/s41467-023-38063-x","article-title":"Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies","volume":"14","author":"Ruffolo","year":"2023","journal-title":"Nat Commun"},{"key":"2024102702171886500_ref90","article-title":"xtrimoabfold: de novo antibody structure prediction without msa","author":"Wang"},{"key":"2024102702171886500_ref91","doi-asserted-by":"publisher","first-page":"6058","DOI":"10.1038\/s41467-020-19986-1","article-title":"Comprehensive prediction of secondary metabolite structure and biological activity from microbial genome sequences","volume":"11","author":"Skinnider","year":"2020","journal-title":"Nat Commun"},{"key":"2024102702171886500_ref92","doi-asserted-by":"publisher","first-page":"583","DOI":"10.1038\/s41586-021-03819-2","article-title":"Highly accurate protein structure prediction with AlphaFold","volume":"596","author":"Jumper","year":"2021","journal-title":"Nature"},{"key":"2024102702171886500_ref93","doi-asserted-by":"publisher","first-page":"e2016239118","DOI":"10.1073\/pnas.2016239118","article-title":"Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences","volume":"118","author":"Rives","year":"2021","journal-title":"Proc Natl Acad Sci"},{"key":"2024102702171886500_ref94","doi-asserted-by":"publisher","first-page":"520","DOI":"10.1002\/prot.25674","article-title":"NetSurfP-2.0: improved prediction of protein structural features by integrated deep learning","volume":"87","author":"Klausen","year":"2019","journal-title":"Proteins"},{"key":"2024102702171886500_ref95","doi-asserted-by":"publisher","first-page":"7112","DOI":"10.1109\/TPAMI.2021.3095381","article-title":"Prottrans: toward understanding the language of life through self-supervised learning","volume":"44","author":"Elnaggar","year":"2021","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"2024102702171886500_ref96","author":"Zhou","year":"2023"},{"key":"2024102702171886500_ref97","doi-asserted-by":"publisher","DOI":"10.7551\/mitpress\/11068.001.0001","volume-title":"The Character of Physical Law, with New Foreword","author":"Feynman","year":"2017"},{"key":"2024102702171886500_ref98","doi-asserted-by":"publisher","first-page":"1617","DOI":"10.1038\/s41587-022-01432-w","article-title":"Single-sequence protein structure prediction using a language model and deep learning","volume":"40","author":"Chowdhury","year":"2022","journal-title":"Nat Biotechnol"},{"key":"2024102702171886500_ref99","doi-asserted-by":"publisher","first-page":"6801","DOI":"10.1609\/aaai.v36i6.20636","article-title":"Self-supervised pre-training for protein embeddings using tertiary structures","volume":"36","author":"Guo","year":"2022","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2024102702171886500_ref100","doi-asserted-by":"publisher","first-page":"612","DOI":"10.1038\/s42256-023-00647-z","article-title":"Structure-inducing pre-training","volume":"5","author":"McDermott","year":"2023","journal-title":"Nat Mach Intell"},{"key":"2024102702171886500_ref101","doi-asserted-by":"publisher","first-page":"5407","DOI":"10.1038\/s41467-019-13395-9","article-title":"RNA secondary structure prediction using an ensemble of two-dimensional deep neural networks and transfer learning","volume":"10","author":"Singh","year":"2019","journal-title":"Nat Commun"},{"key":"2024102702171886500_ref102","doi-asserted-by":"publisher","first-page":"e14","DOI":"10.1093\/nar\/gkab1074","article-title":"UFold: fast and accurate RNA secondary structure prediction with deep learning","volume":"50","author":"Fu","year":"2022","journal-title":"Nucleic Acids Res"},{"key":"2024102702171886500_ref103","doi-asserted-by":"publisher","first-page":"3057","DOI":"10.1021\/acs.jcim.8b00749","article-title":"DNAPred: accurate identification of DNA-binding sites from protein sequence by ensembled hyperplane-distance-based support vector machines","volume":"59","author":"Zhu","year":"2019","journal-title":"J Chem Inf Model"},{"key":"2024102702171886500_ref104","doi-asserted-by":"publisher","first-page":"bbaa397","DOI":"10.1093\/bib\/bbaa397","article-title":"NCBRPred: predicting nucleic acid binding residues in proteins based on multilabel learning","volume":"22","author":"Zhang","year":"2021","journal-title":"Brief Bioinform"},{"key":"2024102702171886500_ref105","doi-asserted-by":"publisher","first-page":"930","DOI":"10.1093\/bioinformatics\/bty756","article-title":"Improving the prediction of protein-nucleic acids binding residues via multiple sequence profiles and the consensus of complementary methods","volume":"35","author":"Su","year":"2019","journal-title":"Bioinformatics"},{"key":"2024102702171886500_ref106","doi-asserted-by":"publisher","first-page":"25","DOI":"10.1038\/75556","article-title":"Gene ontology: tool for the unification of biology","volume":"25","author":"Ashburner","year":"2000","journal-title":"Nat Genet"},{"key":"2024102702171886500_ref107","doi-asserted-by":"publisher","first-page":"3168","DOI":"10.1038\/s41467-021-23303-9","article-title":"Structure-based protein function prediction using graph convolutional networks","volume":"12","author":"Gligorijevi\u0107","year":"2021","journal-title":"Nat Commun"},{"key":"2024102702171886500_ref108","doi-asserted-by":"publisher","first-page":"i238","DOI":"10.1093\/bioinformatics\/btac256","article-title":"DeepGOZero: improving protein function prediction from sequence and zero-shot learning based on ontology axioms","volume":"38","author":"Kulmanov","year":"2022","journal-title":"Bioinformatics"},{"key":"2024102702171886500_ref109","doi-asserted-by":"publisher","first-page":"852","DOI":"10.1038\/s42256-022-00534-z","article-title":"scBERT as a large-scale pre-trained deep language model for cell type annotation of single-cell RNA-seq data","volume":"4","author":"Yang","year":"2022","journal-title":"Nat Mach Intell"},{"key":"2024102702171886500_ref110","author":"Choromanski"},{"key":"2024102702171886500_ref111","doi-asserted-by":"publisher","first-page":"4276","DOI":"10.1609\/aaai.v35i5.16552","article-title":"Learning to pre-train graph neural networks","volume":"35","author":"Lu","year":"2021","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2024102702171886500_ref112","doi-asserted-by":"publisher","first-page":"1818","DOI":"10.1038\/s41467-020-15523-2","article-title":"SciBet as a portable and fast single cell type identifier","volume":"11","author":"Li","year":"2020","journal-title":"Nat Commun"},{"key":"2024102702171886500_ref113","doi-asserted-by":"publisher","first-page":"359","DOI":"10.1038\/nmeth.4644","article-title":"Scmap: projection of single-cell RNA-seq data across data sets","volume":"15","author":"Kiselev","year":"2018","journal-title":"Nat Methods"},{"key":"2024102702171886500_ref114","doi-asserted-by":"publisher","first-page":"198","DOI":"10.1186\/s13059-024-03338-z","article-title":"scCross: a deep generative model for unifying single-cell multi-omics with seamless integration, cross-modal generation, and in silico exploration","volume":"25","author":"Yang","year":"2024","journal-title":"Genome Biol"},{"key":"2024102702171886500_ref115","article-title":"Large-scale foundation model on single-cell transcriptomics","volume":"21","author":"Hao","journal-title":"Nat Methods"},{"key":"2024102702171886500_ref116","first-page":"36479","article-title":"Photorealistic text-to-image diffusion models with deep language understanding","volume":"35","author":"Saharia","year":"2022","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024102702171886500_ref117","doi-asserted-by":"publisher","first-page":"1458","DOI":"10.1038\/s41587-022-01284-4","article-title":"Multi-omics single-cell data integration and regulatory inference with graph-linked embedding","volume":"40","author":"Cao","year":"2022","journal-title":"Nat Biotechnol"},{"key":"2024102702171886500_ref118","doi-asserted-by":"publisher","first-page":"6474","DOI":"10.1038\/s41467-022-34213-9","article-title":"Automated identification of sequence-tailored Cas9 proteins using massive metagenomic data","volume":"13","author":"Ciciani","year":"2022","journal-title":"Nat Commun"},{"key":"2024102702171886500_ref119","doi-asserted-by":"publisher","first-page":"1796","DOI":"10.1038\/s41467-021-21770-8","article-title":"Identification of disease treatment mechanisms through the multiscale interactome","volume":"12","author":"Ruiz","year":"2021","journal-title":"Nat Commun"},{"key":"2024102702171886500_ref120","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1038\/s41576-019-0122-6","article-title":"Deep learning: new computational modeling techniques for genomics","volume":"20","author":"Eraslan","year":"2019","journal-title":"Nat Rev Genet"},{"key":"2024102702171886500_ref121","article-title":"Hyena hierarchy: towards larger convolutional language models","volume-title":"International Conference on Machine Learning","author":"Poli"},{"key":"2024102702171886500_ref122","doi-asserted-by":"crossref","DOI":"10.1101\/2023.05.23.541774","article-title":"Esmfold hallucinates native-like protein sequences","volume-title":"bioRxiv","author":"Jeliazkov"},{"key":"2024102702171886500_ref123","volume-title":"Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition","author":"Wang"},{"key":"2024102702171886500_ref124","article-title":"Evaluating self-supervised learning for molecular graph embeddings.","volume-title":"Advances in Neural Information Processing Systems","author":"Wang"},{"key":"2024102702171886500_ref125","doi-asserted-by":"publisher","first-page":"11106","DOI":"10.1609\/aaai.v35i12.17325","article-title":"Informer: beyond efficient transformer for long sequence time-series forecasting","volume":"35","author":"Zhou","year":"2021","journal-title":"Proceedings of the AAAI Conference on Artificial Intelligence"},{"key":"2024102702171886500_ref126","article-title":"Shortformer: better language modeling using shorter","author":"Press","year":"2020"},{"key":"2024102702171886500_ref127","first-page":"26736","article-title":"The stability-efficiency dilemma: investigating sequence length warmup for training GPT models","volume":"35","author":"Li","year":"2022","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024102702171886500_ref128","first-page":"16344","article-title":"Flashattention: fast and memory-efficient exact attention with io-awareness","volume":"35","author":"Dao","year":"2022","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024102702171886500_ref129","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/2023.emnlp-main.298","article-title":"GQA: training generalized multi-query transformer models from multi-head","author":"Ainslie","year":"2023"},{"key":"2024102702171886500_ref130","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3570638","article-title":"Optimization techniques for GPU programming","volume":"55","author":"Hijma","year":"2023","journal-title":"ACM Comput Surv"},{"key":"2024102702171886500_ref131","doi-asserted-by":"publisher","first-page":"110","DOI":"10.1038\/s42256-022-00445-z","article-title":"Stable learning establishes some common ground between causal inference and machine learning","volume":"4","author":"Cui","year":"2022","journal-title":"Nat Mach Intell"},{"key":"2024102702171886500_ref132","doi-asserted-by":"publisher","first-page":"1","DOI":"10.1145\/3490238","article-title":"Biomedical question answering: a survey of approaches and challenges","volume":"55","author":"Jin","year":"2022","journal-title":"ACM Comput Surv"},{"key":"2024102702171886500_ref133","doi-asserted-by":"publisher","first-page":"5381","DOI":"10.1093\/nar\/gky285","article-title":"bpRNA: large-scale automated annotation and analysis of RNA secondary structure","volume":"46","author":"Danaee","year":"2018","journal-title":"Nucleic Acids Res"},{"key":"2024102702171886500_ref134","doi-asserted-by":"publisher","first-page":"2057","DOI":"10.1038\/s41591-023-02482-6","article-title":"Machine learning for genetics-based classification and treatment response prediction in cancer of unknown primary","volume":"29","author":"Moon","year":"2023","journal-title":"Nat Med"},{"key":"2024102702171886500_ref135","doi-asserted-by":"publisher","first-page":"135","DOI":"10.1038\/s41746-023-00879-8","article-title":"The shaky foundations of large language models and foundation models for electronic health records","volume":"6","author":"Wornow","year":"2023","journal-title":"npj Digit Med"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/25\/6\/bbae548\/60105244\/bbae548.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/25\/6\/bbae548\/60105244\/bbae548.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,10,27]],"date-time":"2024-10-27T02:17:45Z","timestamp":1729995465000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbae548\/7842778"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,23]]},"references-count":136,"journal-issue":{"issue":"6","published-print":{"date-parts":[[2024,9,23]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbae548","relation":{},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,11]]},"published":{"date-parts":[[2024,9,23]]},"article-number":"bbae548"}}