{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,7]],"date-time":"2026-06-07T13:01:37Z","timestamp":1780837297463,"version":"3.54.1"},"reference-count":104,"publisher":"Oxford University Press (OUP)","issue":"3","license":[{"start":{"date-parts":[[2022,4,21]],"date-time":"2022-04-21T00:00:00Z","timestamp":1650499200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/academic.oup.com\/journals\/pages\/open_access\/funder_policies\/chorus\/standard_publication_model"}],"funder":[{"DOI":"10.13039\/501100004837","name":"Ministerio de Ciencia e Innovaci\u00f3n","doi-asserted-by":"publisher","award":["PID2019-104206GB-I00"],"award-info":[{"award-number":["PID2019-104206GB-I00"]}],"id":[{"id":"10.13039\/501100004837","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100011033","name":"Agencia Estatal de Investigaci\u00f3n","doi-asserted-by":"publisher","award":["10.13039\/ 501100011033"],"award-info":[{"award-number":["10.13039\/ 501100011033"]}],"id":[{"id":"10.13039\/501100011033","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/100013248","name":"FPI","doi-asserted-by":"publisher","award":["BES-2017-079792"],"award-info":[{"award-number":["BES-2017-079792"]}],"id":[{"id":"10.13039\/100013248","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,5,13]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>The identification of the protein fold class is a challenging problem in structural biology. Recent computational methods for fold prediction leverage deep learning techniques to extract protein fold-representative embeddings mainly using evolutionary information in the form of multiple sequence alignment (MSA) as input source. In contrast, protein language models (LM) have reshaped the field thanks to their ability to learn efficient protein representations (protein-LM embeddings) from purely sequential information in a self-supervised manner. In this paper, we analyze a framework for protein fold prediction using pre-trained protein-LM embeddings as input to several fine-tuning neural network models, which are supervisedly trained with fold labels. In particular, we compare the performance of six protein-LM embeddings: the long short-term memory-based UniRep and SeqVec, and the transformer-based ESM-1b, ESM-MSA, ProtBERT and ProtT5; as well as three neural networks: Multi-Layer Perceptron, ResCNN-BGRU (RBG) and Light-Attention (LAT). We separately evaluated the pairwise fold recognition (PFR) and direct fold classification (DFC) tasks on well-known benchmark datasets. The results indicate that the combination of transformer-based embeddings, particularly those obtained at amino acid level, with the RBG and LAT fine-tuning models performs remarkably well in both tasks. To further increase prediction accuracy, we propose several ensemble strategies for PFR and DFC, which provide a significant performance boost over the current state-of-the-art results. All this suggests that moving from traditional protein representations to protein-LM embeddings is a very promising approach to protein fold-related tasks.<\/jats:p>","DOI":"10.1093\/bib\/bbac142","type":"journal-article","created":{"date-parts":[[2022,3,29]],"date-time":"2022-03-29T07:15:05Z","timestamp":1648538105000},"source":"Crossref","is-referenced-by-count":49,"title":["An analysis of protein language model embeddings for fold prediction"],"prefix":"10.1093","volume":"23","author":[{"given":"Amelia","family":"Villegas-Morcillo","sequence":"first","affiliation":[{"name":"Department of Signal Theory, Telematics and Communications, University of Granada, Granada, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Angel M","family":"Gomez","sequence":"additional","affiliation":[{"name":"Department of Signal Theory, Telematics and Communications, University of Granada, Granada, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Victoria","family":"Sanchez","sequence":"additional","affiliation":[{"name":"Department of Signal Theory, Telematics and Communications, University of Granada, Granada, Spain"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,4,21]]},"reference":[{"issue":"7792","key":"2022051813180146200_ref1","doi-asserted-by":"crossref","first-page":"706","DOI":"10.1038\/s41586-019-1923-7","article-title":"Improved protein structure prediction using potentials from deep learning","volume":"577","author":"Senior","year":"2020","journal-title":"Nature"},{"issue":"7873","key":"2022051813180146200_ref2","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1038\/s41586-021-03819-2","article-title":"Highly accurate protein structure prediction with AlphaFold","volume":"596","author":"Jumper","year":"2021","journal-title":"Nature"},{"issue":"6557","key":"2022051813180146200_ref3","doi-asserted-by":"crossref","first-page":"871","DOI":"10.1126\/science.abj8754","article-title":"Accurate prediction of protein structures and interactions using a three-track neural network","volume":"373","author":"Baek","year":"2021","journal-title":"Science"},{"key":"2022051813180146200_ref4","volume-title":"Proteins: Structure and Function","author":"Whitford","year":"2013"},{"issue":"D1","key":"2022051813180146200_ref5","doi-asserted-by":"crossref","first-page":"D439","DOI":"10.1093\/nar\/gkab1061","article-title":"AlphaFold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models","volume":"50","author":"Varadi","year":"2022","journal-title":"Nucleic Acids Res"},{"issue":"4","key":"2022051813180146200_ref6","doi-asserted-by":"crossref","first-page":"536","DOI":"10.1016\/S0022-2836(05)80134-2","article-title":"SCOP: a structural classification of proteins database for the investigation of sequences and structures","volume":"247","author":"Murzin","year":"1995","journal-title":"J Mol Biol"},{"issue":"D1","key":"2022051813180146200_ref7","doi-asserted-by":"crossref","first-page":"D553","DOI":"10.1093\/nar\/gkab1054","article-title":"SCOPe: improvements to the structural classification of proteins\u2013extended database to facilitate variant interpretation and machine learning","volume":"50","author":"Chandonia","year":"2022","journal-title":"Nucleic Acids Res"},{"issue":"8","key":"2022051813180146200_ref8","doi-asserted-by":"crossref","first-page":"1093","DOI":"10.1016\/S0969-2126(97)00260-8","article-title":"CATH \u2013 a hierarchic classification of protein domain structures","volume":"5","author":"Orengo","year":"1997","journal-title":"Structure"},{"issue":"D1","key":"2022051813180146200_ref9","doi-asserted-by":"crossref","first-page":"D266","DOI":"10.1093\/nar\/gkaa1079","article-title":"CATH: increased structural coverage of functional space","volume":"49","author":"Sillitoe","year":"2021","journal-title":"Nucleic Acids Res"},{"issue":"1","key":"2022051813180146200_ref10","doi-asserted-by":"crossref","first-page":"235","DOI":"10.1093\/nar\/28.1.235","article-title":"The protein data bank","volume":"28","author":"Berman","year":"2000","journal-title":"Nucleic Acids Res"},{"issue":"D1","key":"2022051813180146200_ref11","doi-asserted-by":"crossref","first-page":"D437","DOI":"10.1093\/nar\/gkaa1038","article-title":"RCSB protein data bank: powerful new tools for exploring 3d structures of biological macromolecules for basic and applied research and education in fundamental biology, biomedicine, biotechnology, bioengineering and energy sciences","volume":"49","author":"Burley","year":"2021","journal-title":"Nucleic Acids Res"},{"issue":"1","key":"2022051813180146200_ref12","doi-asserted-by":"crossref","first-page":"1007","DOI":"10.1146\/annurev.bi.59.070190.005043","article-title":"The classification and origins of protein folding patterns","volume":"59","author":"Chothia","year":"1990","journal-title":"Annu Rev Biochem"},{"issue":"6381","key":"2022051813180146200_ref13","doi-asserted-by":"crossref","first-page":"86","DOI":"10.1038\/358086a0","article-title":"A new approach to protein fold recognition","volume":"358","author":"Jones","year":"1992","journal-title":"Nature"},{"issue":"3","key":"2022051813180146200_ref14","doi-asserted-by":"crossref","first-page":"613","DOI":"10.1006\/jmbi.1999.3377","article-title":"Identification of related proteins on family, superfamily and fold level","volume":"295","author":"Lindahl","year":"2000","journal-title":"J Mol Biol"},{"issue":"1\u20132","key":"2022051813180146200_ref15","first-page":"11","article-title":"Protein folds and protein folding","volume":"24","author":"Schaeffer","year":"2010","journal-title":"Protein Eng Des Sel"},{"issue":"1","key":"2022051813180146200_ref16","doi-asserted-by":"crossref","first-page":"559","DOI":"10.1146\/annurev-biophys-083012-130432","article-title":"On the universe of protein folds","volume":"42","author":"Kolodny","year":"2013","journal-title":"Annu Rev Biophys"},{"issue":"9","key":"2022051813180146200_ref17","doi-asserted-by":"crossref","first-page":"768","DOI":"10.3844\/jcssp.2008.768.776","article-title":"A survey of protein fold recognition algorithms","volume":"4","author":"Abual-Rub","year":"2008","journal-title":"J Comput Sci"},{"issue":"2","key":"2022051813180146200_ref18","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1093\/bib\/bbw108","article-title":"A comprehensive review and comparison of different computational methods for protein remote homology detection","volume":"19","author":"Chen","year":"2018","journal-title":"Brief Bioinform"},{"key":"2022051813180146200_ref19","doi-asserted-by":"crossref","first-page":"101","DOI":"10.1007\/978-3-319-94030-4_5","volume-title":"Machine Learning Paradigms","author":"Stapor","year":"2019"},{"issue":"7","key":"2022051813180146200_ref20","doi-asserted-by":"crossref","first-page":"951","DOI":"10.1093\/bioinformatics\/bti125","article-title":"Protein homology detection by HMM\u2013HMM comparison","volume":"21","author":"S\u00f6ding","year":"2005","journal-title":"Bioinformatics"},{"issue":"3","key":"2022051813180146200_ref21","doi-asserted-by":"crossref","first-page":"e1003500","DOI":"10.1371\/journal.pcbi.1003500","article-title":"MRFalign: protein homology detection through alignment of Markov random fields","volume":"10","author":"Ma","year":"2014","journal-title":"PLoS Comput Biol"},{"issue":"1","key":"2022051813180146200_ref22","doi-asserted-by":"crossref","first-page":"95","DOI":"10.1142\/S0219720003000186","article-title":"RAPTOR: optimal protein threading by linear programming","volume":"1","author":"Xu","year":"2003","journal-title":"J Bioinform Comput Biol"},{"key":"2022051813180146200_ref23","first-page":"31","volume-title":"Annual International Conference on Research in Computational Molecular Biology","author":"Peng","year":"2009"},{"issue":"15","key":"2022051813180146200_ref24","doi-asserted-by":"crossref","first-page":"2076","DOI":"10.1093\/bioinformatics\/btr350","article-title":"Improving protein fold recognition and template-based modeling by employing probabilistic-based matching between predicted one-dimensional structural properties of query and corresponding native properties of templates","volume":"27","author":"Yang","year":"2011","journal-title":"Bioinformatics"},{"issue":"12","key":"2022051813180146200_ref25","doi-asserted-by":"crossref","first-page":"i59","DOI":"10.1093\/bioinformatics\/bts213","article-title":"A conditional neural fields model for protein threading","volume":"28","author":"Ma","year":"2012","journal-title":"Bioinformatics"},{"issue":"6","key":"2022051813180146200_ref26","doi-asserted-by":"crossref","first-page":"e0197912","DOI":"10.1371\/journal.pone.0197912","article-title":"Protein alignment based on higher order conditional random fields for template-based modeling","volume":"13","author":"Morales-Cordovilla","year":"2018","journal-title":"PLoS One"},{"issue":"17","key":"2022051813180146200_ref27","doi-asserted-by":"crossref","first-page":"2684","DOI":"10.1093\/bioinformatics\/btx217","article-title":"EigenTHREADER: analogous protein fold recognition by efficient contact map threading","volume":"33","author":"Buchan","year":"2017","journal-title":"Bioinformatics"},{"issue":"10","key":"2022051813180146200_ref28","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1371\/journal.pcbi.1007411","article-title":"Detecting distant-homology protein structures by aligning deep neural-network based contact maps","volume":"15","author":"Zheng","year":"2019","journal-title":"PLoS Comput Biol"},{"issue":"12","key":"2022051813180146200_ref29","doi-asserted-by":"crossref","first-page":"1456","DOI":"10.1093\/bioinformatics\/btl102","article-title":"A machine learning information retrieval approach to protein fold recognition","volume":"22","author":"Cheng","year":"2006","journal-title":"Bioinformatics"},{"issue":"11","key":"2022051813180146200_ref30","doi-asserted-by":"crossref","first-page":"S14","DOI":"10.1186\/1471-2105-15-S11-S14","article-title":"Improving protein fold recognition by random forest","volume":"15","author":"Jo","year":"2014","journal-title":"BMC Bioinformatics"},{"key":"2022051813180146200_ref31","doi-asserted-by":"crossref","first-page":"17573","DOI":"10.1038\/srep17573","article-title":"Improving protein fold recognition by deep learning networks","volume":"5","author":"Jo","year":"2015","journal-title":"Sci Rep"},{"issue":"17","key":"2022051813180146200_ref32","doi-asserted-by":"crossref","first-page":"2982","DOI":"10.1093\/bioinformatics\/btz040","article-title":"Protein fold recognition based on multi-view modeling","volume":"35","author":"Yan","year":"2019","journal-title":"Bioinformatics"},{"issue":"6","key":"2022051813180146200_ref33","doi-asserted-by":"crossref","first-page":"2682","DOI":"10.1109\/TCBB.2020.2991268","article-title":"Protein fold recognition based on auto-weighted multi-view graph embedding learning model","volume":"18","author":"Yan","year":"2021","journal-title":"IEEE\/ACM Trans Comput Biol Bioinform"},{"key":"2022051813180146200_ref34","doi-asserted-by":"crossref","first-page":"127","DOI":"10.1016\/j.neucom.2020.09.028","article-title":"MLDH-Fold: protein fold recognition based on multi-view low-rank modeling","volume":"421","author":"Yan","year":"2021","journal-title":"Neurocomputing"},{"issue":"6","key":"2022051813180146200_ref35","doi-asserted-by":"crossref","first-page":"2185","DOI":"10.1093\/bib\/bbz139","article-title":"Fold-LTR-TCP: protein fold recognition based on triadic closure principle","volume":"21","author":"Liu","year":"2020","journal-title":"Brief Bioinform"},{"issue":"3","key":"2022051813180146200_ref36","doi-asserted-by":"crossref","DOI":"10.1093\/bib\/bbaa144","article-title":"FoldRec-C2C: protein fold recognition by combining cluster-to-cluster model and protein similarity network","volume":"22","author":"Shao","year":"2021","journal-title":"Brief Bioinform"},{"issue":"3","key":"2022051813180146200_ref37","doi-asserted-by":"crossref","DOI":"10.1093\/bib\/bbaa192","article-title":"ProtFold-DFG: protein fold recognition by combining Directed Fusion Graph and PageRank algorithm","volume":"22","author":"Shao","year":"2021","journal-title":"Brief Bioinform"},{"issue":"23","key":"2022051813180146200_ref38","doi-asserted-by":"crossref","first-page":"3749","DOI":"10.1093\/bioinformatics\/btx514","article-title":"Improving protein fold recognition by extracting fold-specific features from predicted residue\u2013residue contacts","volume":"33","author":"Zhu","year":"2017","journal-title":"Bioinformatics"},{"issue":"5","key":"2022051813180146200_ref39","doi-asserted-by":"crossref","first-page":"1733","DOI":"10.1093\/bib\/bbz098","article-title":"DeepSVM-fold: protein fold recognition by combining support vector machines and pairwise sequence similarity scores generated by deep learning networks","volume":"21","author":"Liu","year":"2020","journal-title":"Brief Bioinform"},{"key":"2022051813180146200_ref40","article-title":"MotifCNN-fold: protein fold recognition based on fold-specific features extracted by motif-based convolutional neural networks","volume":"21","author":"Li","year":"2019","journal-title":"Brief Bioinform"},{"key":"2022051813180146200_ref41","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TCBB.2020.3031888","article-title":"SelfAT-Fold: protein fold recognition based on residue-based and motif-based self-attention networks","volume":"PP","author":"Pang","year":"2020","journal-title":"IEEE\/ACM Trans Comput Biol Bioinform"},{"issue":"5","key":"2022051813180146200_ref42","doi-asserted-by":"crossref","DOI":"10.1093\/bib\/bbab001","article-title":"Why can deep convolutional neural networks improve protein fold recognition? a visual explanation by interpretation","volume":"22","author":"Liu","year":"2021","journal-title":"Brief Bioinform"},{"issue":"6","key":"2022051813180146200_ref43","doi-asserted-by":"crossref","first-page":"bbab248","DOI":"10.1093\/bib\/bbab248","article-title":"Improving protein fold recognition using triplet network and ensemble deep learning","volume":"22","author":"Liu","year":"2021","journal-title":"Brief Bioinform"},{"issue":"6","key":"2022051813180146200_ref44","doi-asserted-by":"crossref","first-page":"2848","DOI":"10.1109\/TCBB.2020.3012732","article-title":"Protein fold recognition from sequences using convolutional and recurrent neural networks","volume":"18","author":"Villegas-Morcillo","year":"2021","journal-title":"IEEE\/ACM Trans Comput Biol Bioinform"},{"issue":"1","key":"2022051813180146200_ref45","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12859-021-04419-7","article-title":"FoldHSphere: deep hyperspherical embeddings for protein fold recognition","volume":"22","author":"Villegas-Morcillo","year":"2021","journal-title":"BMC Bioinformatics"},{"issue":"12","key":"2022051813180146200_ref46","doi-asserted-by":"crossref","first-page":"2118","DOI":"10.3390\/ijms17122118","article-title":"Recent progress in machine learning-based methods for protein fold recognition","volume":"17","author":"Wei","year":"2016","journal-title":"Int J Mol Sci"},{"issue":"4","key":"2022051813180146200_ref47","doi-asserted-by":"crossref","first-page":"349","DOI":"10.1093\/bioinformatics\/17.4.349","article-title":"Multi-class protein fold recognition using support vector machines and neural networks","volume":"17","author":"Ding","year":"2001","journal-title":"Bioinformatics"},{"issue":"14","key":"2022051813180146200_ref48","doi-asserted-by":"crossref","first-page":"1717","DOI":"10.1093\/bioinformatics\/btl170","article-title":"Ensemble classifier for protein fold pattern recognition","volume":"22","author":"Shen","year":"2006","journal-title":"Bioinformatics"},{"issue":"20","key":"2022051813180146200_ref49","doi-asserted-by":"crossref","first-page":"2655","DOI":"10.1093\/bioinformatics\/btp500","article-title":"A new taxonomy-based protein fold recognition approach based on autocross-covariance transformation","volume":"25","author":"Dong","year":"2009","journal-title":"Bioinformatics"},{"issue":"7","key":"2022051813180146200_ref50","doi-asserted-by":"crossref","first-page":"2053","DOI":"10.1002\/prot.23025","article-title":"Improving taxonomy-based protein fold recognition by using global and local features","volume":"79","author":"Yang","year":"2011","journal-title":"Proteins: Structure, Function, and Bioinformatics"},{"issue":"7","key":"2022051813180146200_ref51","doi-asserted-by":"crossref","first-page":"761","DOI":"10.1109\/TNB.2015.2457906","article-title":"Advancing the accuracy of protein fold recognition by utilizing profiles from hidden Markov models","volume":"14","author":"Lyons","year":"2015","journal-title":"IEEE Trans Nanobiosci"},{"key":"2022051813180146200_ref52","first-page":"1","article-title":"ProFold: protein fold classification with additional structural features and a novel ensemble classifier","volume":"2016","author":"Chen","year":"2016","journal-title":"Biomed Res Int"},{"issue":"6","key":"2022051813180146200_ref53","doi-asserted-by":"crossref","first-page":"863","DOI":"10.1093\/bioinformatics\/btw768","article-title":"An ensemble approach to protein fold classification by integration of template-based assignment and support vector machine classifier","volume":"33","author":"Xia","year":"2016","journal-title":"Bioinformatics"},{"issue":"8","key":"2022051813180146200_ref54","doi-asserted-by":"crossref","first-page":"4201","DOI":"10.1007\/s00521-018-3346-z","article-title":"Protein fold recognition using deep kernelized extreme learning machine and linear discriminant analysis","volume":"31","author":"Ibrahim","year":"2019","journal-title":"Neural Comput Applic"},{"issue":"1","key":"2022051813180146200_ref55","doi-asserted-by":"crossref","first-page":"42","DOI":"10.1109\/TNB.2020.3022456","article-title":"An enhanced protein fold recognition for low similarity datasets using convolutional and skip-gram features with deep neural network","volume":"20","author":"Bankapur","year":"2020","journal-title":"IEEE Trans Nanobiosci"},{"issue":"6","key":"2022051813180146200_ref56","first-page":"1","article-title":"DeepFrag-k: a fragment-based deep learning approach for protein fold recognition","volume":"21","author":"Elhefnawy","year":"2020","journal-title":"BMC Bioinformatics"},{"issue":"8","key":"2022051813180146200_ref57","doi-asserted-by":"crossref","first-page":"1295","DOI":"10.1093\/bioinformatics\/btx780","article-title":"DeepSF: deep convolutional neural network for mapping protein sequences to folds","volume":"34","author":"Hou","year":"2018","journal-title":"Bioinformatics"},{"issue":"6","key":"2022051813180146200_ref58","doi-asserted-by":"crossref","first-page":"1918","DOI":"10.1109\/TCBB.2019.2911677","article-title":"Amino acid encoding methods for protein sequences: a comprehensive review and assessment","volume":"17","author":"Jing","year":"2019","journal-title":"IEEE\/ACM Trans Comput Biol Bioinform"},{"issue":"3","key":"2022051813180146200_ref59","doi-asserted-by":"crossref","first-page":"275","DOI":"10.1016\/j.eng.2019.12.014","article-title":"Progress in neural NLP: modeling, learning, and reasoning","volume":"6","author":"Zhou","year":"2020","journal-title":"Engineering"},{"key":"2022051813180146200_ref60","doi-asserted-by":"crossref","first-page":"1750","DOI":"10.1016\/j.csbj.2021.03.022","article-title":"The language of proteins: NLP, machine learning & protein sequences","volume":"19","author":"Ofer","year":"2021","journal-title":"Comput Struct Biotechnol J"},{"issue":"1","key":"2022051813180146200_ref61","doi-asserted-by":"crossref","first-page":"365","DOI":"10.1093\/nar\/gkg095","article-title":"The SWISS-PROT protein knowledgebase and its supplement TrEMBL in 2003","volume":"31","author":"Boeckmann","year":"2003","journal-title":"Nucleic Acids Res"},{"issue":"D1","key":"2022051813180146200_ref62","doi-asserted-by":"crossref","first-page":"D412","DOI":"10.1093\/nar\/gkaa913","article-title":"Pfam: the protein families database in 2021","volume":"49","author":"Mistry","year":"2021","journal-title":"Nucleic Acids Res"},{"issue":"6","key":"2022051813180146200_ref63","doi-asserted-by":"crossref","first-page":"926","DOI":"10.1093\/bioinformatics\/btu739","article-title":"UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches","volume":"31","author":"Suzek","year":"2015","journal-title":"Bioinformatics"},{"issue":"D1","key":"2022051813180146200_ref64","doi-asserted-by":"crossref","first-page":"D480","DOI":"10.1093\/nar\/gkaa1100","article-title":"UniProt: the universal protein knowledgebase in 2021","volume":"49","author":"UniProt Consortium","year":"2021","journal-title":"Nucleic Acids Res"},{"issue":"1","key":"2022051813180146200_ref65","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41467-018-04964-5","article-title":"Clustering huge protein sequence sets in linear time","volume":"9","author":"Steinegger","year":"2018","journal-title":"Nat Commun"},{"issue":"7","key":"2022051813180146200_ref66","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1038\/s41592-019-0437-4","article-title":"Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold","volume":"16","author":"Steinegger","year":"2019","journal-title":"Nat Methods"},{"issue":"11","key":"2022051813180146200_ref67","doi-asserted-by":"crossref","first-page":"e0141287","DOI":"10.1371\/journal.pone.0141287","article-title":"Continuous distributed representation of biological sequences for deep proteomics and genomics","volume":"10","author":"Asgari","year":"2015","journal-title":"PLoS One"},{"key":"2022051813180146200_ref68","volume-title":"International Conference on Learning Representations","author":"Mikolov","year":"2013"},{"key":"2022051813180146200_ref69","first-page":"3111","volume-title":"Advances in Neural Information Processing Systems","author":"Mikolov","year":"2013"},{"issue":"8","key":"2022051813180146200_ref70","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput"},{"issue":"8","key":"2022051813180146200_ref71","doi-asserted-by":"crossref","first-page":"2401","DOI":"10.1093\/bioinformatics\/btaa003","article-title":"UDSMProt: universal deep sequence models for protein classification","volume":"36","author":"Strodthoff","year":"2020","journal-title":"Bioinformatics"},{"issue":"12","key":"2022051813180146200_ref72","doi-asserted-by":"crossref","first-page":"1315","DOI":"10.1038\/s41592-019-0598-1","article-title":"Unified rational protein engineering with sequence-based deep representation learning","volume":"16","author":"Alley","year":"2019","journal-title":"Nat Methods"},{"key":"2022051813180146200_ref73","doi-asserted-by":"crossref","DOI":"10.18653\/v1\/N18-1202","article-title":"Deep contextualized word representations","author":"Peters","year":"2018"},{"issue":"1","key":"2022051813180146200_ref74","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12859-019-3220-8","article-title":"Modeling aspects of the language of life through transfer-learning protein sequences","volume":"20","author":"Heinzinger","year":"2019","journal-title":"BMC Bioinformatics"},{"key":"2022051813180146200_ref75","volume-title":"International Conference on Learning Representations","author":"Bepler","year":"2019"},{"key":"2022051813180146200_ref76","first-page":"5998","volume-title":"Advances in Neural Information Processing Systems","author":"Vaswani","year":"2017"},{"key":"2022051813180146200_ref77","volume-title":"Advances in neural information processing systems","author":"Rao","year":"2019"},{"issue":"15","key":"2022051813180146200_ref78","doi-asserted-by":"crossref","DOI":"10.1073\/pnas.2016239118","article-title":"Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences","volume":"118","author":"Rives","year":"2021","journal-title":"Proc Natl Acad Sci"},{"key":"2022051813180146200_ref79","first-page":"8844","volume-title":"Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research","author":"Rao","year":"2021"},{"key":"2022051813180146200_ref80","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TPAMI.2021.3095381","article-title":"ProtTrans: towards cracking the language of lifes code through self-supervised deep learning and high performance computing","author":"Elnaggar","year":"2021","journal-title":"IEEE Trans Pattern Anal Mach Intell"},{"key":"2022051813180146200_ref81","article-title":"Bert: pre-training of deep bidirectional transformers for language understanding","author":"Devlin","year":"2018"},{"key":"2022051813180146200_ref82","doi-asserted-by":"crossref","first-page":"2978","DOI":"10.18653\/v1\/P19-1285","volume-title":"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics","author":"Dai","year":"2019"},{"key":"2022051813180146200_ref83","volume-title":"Advances in Neural Information Processing Systems","author":"Yang","year":"2019"},{"key":"2022051813180146200_ref84","article-title":"Exploring the limits of transfer learning with a unified text-to-text transformer","author":"Raffel","year":"2019"},{"key":"2022051813180146200_ref85","article-title":"ALBERT: a lite BERT for self-supervised learning of language representations","volume-title":"International Conference on Learning Representations","author":"Lan","year":"2020"},{"key":"2022051813180146200_ref86","volume-title":"International Conference on Learning Representations","author":"Clark","year":"2020"},{"issue":"1","key":"2022051813180146200_ref87","doi-asserted-by":"crossref","first-page":"vbab035","DOI":"10.1093\/bioadv\/vbab035","article-title":"Light attention predicts protein location from the language of life","volume":"1","author":"St\u00e4rk","year":"2021","journal-title":"Bioinform Adv"},{"issue":"2","key":"2022051813180146200_ref88","doi-asserted-by":"crossref","first-page":"162","DOI":"10.1093\/bioinformatics\/btaa701","article-title":"Unsupervised protein embeddings outperform hand-crafted sequence and structure features at predicting molecular function","volume":"37","author":"Villegas-Morcillo","year":"2021","journal-title":"Bioinformatics"},{"issue":"1","key":"2022051813180146200_ref89","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1038\/s41598-020-80786-0","article-title":"Embeddings from deep learning transfer GO annotations beyond homology","volume":"11","author":"Littmann","year":"2021","journal-title":"Sci Rep"},{"key":"2022051813180146200_ref90","first-page":"1","article-title":"The power of universal contextualized protein embeddings in cross-species protein function prediction","volume":"17","author":"Bent","year":"2021","journal-title":"Evol Bioinform"},{"key":"2022051813180146200_ref91","volume-title":"Advances on Neural Information Processing Systems","author":"Meier","year":"2021"},{"key":"2022051813180146200_ref92","doi-asserted-by":"crossref","DOI":"10.1007\/s00439-021-02411-y","article-title":"Embeddings from protein language models predict conservation and variant effects","author":"Marquet","year":"2021","journal-title":"Human genetics"},{"key":"2022051813180146200_ref93","volume-title":"International Conference on Learning Representations","author":"Vig","year":"2021"},{"issue":"1","key":"2022051813180146200_ref94","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12859-019-3019-7","article-title":"HH-suite3 for fast remote homology detection and deep protein annotation","volume":"20","author":"Steinegger","year":"2019","journal-title":"BMC Bioinformatics"},{"issue":"D1","key":"2022051813180146200_ref95","doi-asserted-by":"crossref","first-page":"D170","DOI":"10.1093\/nar\/gkw1081","article-title":"Uniclust databases of clustered and deeply annotated protein sequences and alignments","volume":"45","author":"Mirdita","year":"2017","journal-title":"Nucleic Acids Res"},{"key":"2022051813180146200_ref96","article-title":"Multiplicative LSTM for sequence modelling","author":"Krause","year":"2016"},{"issue":"11","key":"2022051813180146200_ref97","doi-asserted-by":"crossref","first-page":"2673","DOI":"10.1109\/78.650093","article-title":"Bidirectional recurrent neural networks","volume":"45","author":"Schuster","year":"1997","journal-title":"IEEE Trans Signal Process"},{"key":"2022051813180146200_ref98","article-title":"Axial attention in multidimensional transformers","author":"Ho","year":"2019"},{"key":"2022051813180146200_ref99","first-page":"448","volume-title":"International Conference on Machine Learning","author":"Ioffe","year":"2015"},{"issue":"1","key":"2022051813180146200_ref100","first-page":"1929","article-title":"Dropout: a simple way to prevent neural networks from overfitting","volume":"15","author":"Srivastava","year":"2014","journal-title":"J Mach Learn Res"},{"key":"2022051813180146200_ref101","article-title":"Empirical evaluation of gated recurrent neural networks on sequence modeling","author":"Chung","year":"2014"},{"key":"2022051813180146200_ref102","first-page":"5265","volume-title":"Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)","author":"Wang","year":"2018"},{"key":"2022051813180146200_ref103","article-title":"Adam: a method for stochastic optimization","author":"Kingma","year":"2014"},{"key":"2022051813180146200_ref104","article-title":"Automatic differentiation in pytorch","volume-title":"Proceedings of Advances in Neural Information Processing Systems,","author":"Paszke","year":"2017"}],"container-title":["Briefings in Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/23\/3\/bbac142\/43745363\/bbac142.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bib\/article-pdf\/23\/3\/bbac142\/43745363\/bbac142.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,11,19]],"date-time":"2023-11-19T05:53:04Z","timestamp":1700373184000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bib\/article\/doi\/10.1093\/bib\/bbac142\/6571527"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,21]]},"references-count":104,"journal-issue":{"issue":"3","published-print":{"date-parts":[[2022,5,13]]}},"URL":"https:\/\/doi.org\/10.1093\/bib\/bbac142","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2022.02.07.479394","asserted-by":"object"}]},"ISSN":["1467-5463","1477-4054"],"issn-type":[{"value":"1467-5463","type":"print"},{"value":"1477-4054","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,5]]},"published":{"date-parts":[[2022,4,21]]},"article-number":"bbac142"}}