{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,23]],"date-time":"2026-10-23T20:03:08Z","timestamp":1792785788726,"version":"4.5.0"},"reference-count":50,"publisher":"Oxford University Press (OUP)","issue":"8","license":[{"start":{"date-parts":[[2020,1,8]],"date-time":"2020-01-08T00:00:00Z","timestamp":1578441600000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Berlin Big Data Center","award":["01IS14013A"],"award-info":[{"award-number":["01IS14013A"]}]},{"DOI":"10.13039\/501100018809","name":"Berlin Center for Machine Learning","doi-asserted-by":"crossref","award":["01IS18037I"],"award-info":[{"award-number":["01IS18037I"]}],"id":[{"id":"10.13039\/501100018809","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2020,4,15]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Motivation<\/jats:title>\n                    <jats:p>Inferring the properties of a protein from its amino acid sequence is one of the key problems in bioinformatics. Most state-of-the-art approaches for protein classification are tailored to single classification tasks and rely on handcrafted features, such as position-specific-scoring matrices from expensive database searches. We argue that this level of performance can be reached or even be surpassed by learning a task-agnostic representation once, using self-supervised language modeling, and transferring it to specific tasks by a simple fine-tuning step.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We put forward a universal deep sequence model that is pre-trained on unlabeled protein sequences from Swiss-Prot and fine-tuned on protein classification tasks. We apply it to three prototypical tasks, namely enzyme class prediction, gene ontology prediction and remote homology and fold detection. The proposed method performs on par with state-of-the-art algorithms that were tailored to these specific tasks or, for two out of three tasks, even outperforms them. These results stress the possibility of inferring protein properties from the sequence alone and, on more general grounds, the prospects of modern natural language processing methods in omics. Moreover, we illustrate the prospects for explainable machine learning methods in this field by selected case studies.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Availability and implementation<\/jats:title>\n                    <jats:p>Source code is available under https:\/\/github.com\/nstrodt\/UDSMProt.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Supplementary information<\/jats:title>\n                    <jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btaa003","type":"journal-article","created":{"date-parts":[[2020,1,2]],"date-time":"2020-01-02T15:09:17Z","timestamp":1577977757000},"page":"2401-2409","source":"Crossref","is-referenced-by-count":186,"title":["UDSMProt: universal deep sequence models for protein classification"],"prefix":"10.1093","volume":"36","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4447-0162","authenticated-orcid":false,"given":"Nils","family":"Strodthoff","sequence":"first","affiliation":[{"name":"Department of Video Coding & Analytics, Fraunhofer Heinrich Hertz Institute , Berlin 10587, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Patrick","family":"Wagner","sequence":"additional","affiliation":[{"name":"Department of Video Coding & Analytics, Fraunhofer Heinrich Hertz Institute , Berlin 10587, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Markus","family":"Wenzel","sequence":"additional","affiliation":[{"name":"Department of Video Coding & Analytics, Fraunhofer Heinrich Hertz Institute , Berlin 10587, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Wojciech","family":"Samek","sequence":"additional","affiliation":[{"name":"Department of Video Coding & Analytics, Fraunhofer Heinrich Hertz Institute , Berlin 10587, Germany"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2020,1,8]]},"reference":[{"key":"2023013110260063300_btaa003-B1","doi-asserted-by":"crossref","first-page":"4862","DOI":"10.1093\/bioinformatics\/btz422","article-title":"AlphaFold at CASP13","volume":"35","author":"AlQuraishi","year":"2019","journal-title":"Bioinformatics"},{"key":"2023013110260063300_btaa003-B2","doi-asserted-by":"crossref","first-page":"e0141287","DOI":"10.1371\/journal.pone.0141287","article-title":"Continuous distributed representation of biological sequences for deep proteomics and genomics","volume":"10","author":"Asgari","year":"2015","journal-title":"PLoS One"},{"key":"2023013110260063300_btaa003-B3","author":"Bileschi","year":"2019"},{"key":"2023013110260063300_btaa003-B4","doi-asserted-by":"crossref","first-page":"59","DOI":"10.1038\/nmeth.3176","article-title":"Fast and sensitive protein alignment using DIAMOND","volume":"12","author":"Buchfink","year":"2015","journal-title":"Nat. Methods"},{"key":"2023013110260063300_btaa003-B5","doi-asserted-by":"crossref","first-page":"231","DOI":"10.1093\/bib\/bbw108","article-title":"A comprehensive review and comparison of different computational methods for protein remote homology detection","volume":"19","author":"Chen","year":"2018","journal-title":"Brief. Bioinform"},{"key":"2023013110260063300_btaa003-B6","doi-asserted-by":"crossref","first-page":"i53","DOI":"10.1093\/bioinformatics\/btt228","article-title":"Information-theoretic evaluation of predicted ontological annotations","volume":"29","author":"Clark","year":"2013","journal-title":"Bioinformatics"},{"key":"2023013110260063300_btaa003-B7","doi-asserted-by":"crossref","first-page":"31865","DOI":"10.1038\/srep31865","article-title":"FFPred 3: feature-based function prediction for all gene ontology domains","volume":"6","author":"Cozzetto","year":"2016","journal-title":"Sci. Rep"},{"key":"2023013110260063300_btaa003-B8","doi-asserted-by":"crossref","first-page":"334","DOI":"10.1186\/s12859-018-2368-y","article-title":"ECPred: a tool for the prediction of the enzymatic functions of protein sequences based on the EC nomenclature","volume":"19","author":"Dalkiran","year":"2018","journal-title":"BMC Bioinformatics"},{"key":"2023013110260063300_btaa003-B9","author":"Devlin","year":"2019"},{"key":"2023013110260063300_btaa003-B10","doi-asserted-by":"crossref","first-page":"D427","DOI":"10.1093\/nar\/gky995","article-title":"The Pfam protein families database in 2019","volume":"47","author":"El-Gebali","year":"2019","journal-title":"Nucleic Acids Res"},{"key":"2023013110260063300_btaa003-B11","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/j.ymeth.2015.08.009","article-title":"GoFDR: a sequence alignment based method for predicting protein functions","volume":"93","author":"Gong","year":"2016","journal-title":"Methods"},{"key":"2023013110260063300_btaa003-B12","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1186\/1471-2105-8-23","article-title":"Motif kernel generated by genetic programming improves remote homology and fold detection","volume":"8","author":"H\u00e5ndstad","year":"2007","journal-title":"BMC Bioinformatics"},{"key":"2023013110260063300_btaa003-B13","doi-asserted-by":"crossref","first-page":"1728","DOI":"10.1093\/bioinformatics\/btm247","article-title":"Fast model-based protein homology detection without alignment","volume":"23","author":"Hochreiter","year":"2007","journal-title":"Bioinformatics"},{"key":"2023013110260063300_btaa003-B14","first-page":"328","author":"Howard","year":"2018"},{"key":"2023013110260063300_btaa003-B15","author":"Kingma","year":"2015"},{"key":"2023013110260063300_btaa003-B16","article-title":"DeepGOPlus: improved protein function prediction from sequence","author":"Kulmanov","year":"2019","journal-title":"Bioinformatics"},{"key":"2023013110260063300_btaa003-B17","doi-asserted-by":"crossref","first-page":"660","DOI":"10.1093\/bioinformatics\/btx624","article-title":"DeepGO: predicting protein functions from sequence and interactions using a deep ontology-aware classifier","volume":"34","author":"Kulmanov","year":"2018","journal-title":"Bioinformatics"},{"key":"2023013110260063300_btaa003-B18","author":"Li","year":"2016"},{"key":"2023013110260063300_btaa003-B19","doi-asserted-by":"crossref","first-page":"443","DOI":"10.1186\/s12859-017-1842-2","article-title":"Protein remote homology detection based on bidirectional long short-term memory","volume":"18","author":"Li","year":"2017","journal-title":"BMC Bioinformatics"},{"key":"2023013110260063300_btaa003-B20","doi-asserted-by":"crossref","first-page":"760","DOI":"10.1093\/bioinformatics\/btx680","article-title":"DEEPre: sequence-based enzyme EC number prediction by deep learning","volume":"34","author":"Li","year":"2018","journal-title":"Bioinformatics"},{"key":"2023013110260063300_btaa003-B21","author":"Liu","year":"2019"},{"key":"2023013110260063300_btaa003-B22","author":"Loshchilov","year":"2019"},{"key":"2023013110260063300_btaa003-B23","volume-title":"The NCBI Handbook","author":"Madden","year":"2013","edition":"2nd edn"},{"key":"2023013110260063300_btaa003-B24","author":"Merity","year":"2018"},{"key":"2023013110260063300_btaa003-B25","doi-asserted-by":"crossref","first-page":"536","DOI":"10.1016\/S0022-2836(05)80134-2","article-title":"SCOP: a structural classification of proteins database for the investigation of sequences and structures","volume":"247","author":"Murzin","year":"1995","journal-title":"J. Mol. Biol"},{"key":"2023013110260063300_btaa003-B26","doi-asserted-by":"crossref","first-page":"485","DOI":"10.1186\/s12859-019-3060-6","article-title":"Machine learning for discovering missing or wrong protein function annotations","volume":"20","author":"Nakano","year":"2019","journal-title":"BMC Bioinformatics"},{"key":"2023013110260063300_btaa003-B27","first-page":"2227","article-title":"Deep contextualized word representations","author":"Peters","year":"2018"},{"key":"2023013110260063300_btaa003-B28","doi-asserted-by":"crossref","first-page":"503","DOI":"10.1038\/s41586-018-0124-0","article-title":"Mutant phenotypes for thousands of bacterial genes of unknown function","volume":"557","author":"Price","year":"2018","journal-title":"Nature"},{"key":"2023013110260063300_btaa003-B29","author":"Radford","year":"2018"},{"key":"2023013110260063300_btaa003-B30","author":"Radford","year":"2019"},{"key":"2023013110260063300_btaa003-B31","first-page":"9686","author":"Rao","year":"2019"},{"key":"2023013110260063300_btaa003-B32","doi-asserted-by":"crossref","first-page":"276","DOI":"10.1016\/S0168-9525(00)02024-2","article-title":"EMBOSS: the European molecular biology open software suite","volume":"16","author":"Rice","year":"2000","journal-title":"Trends Genet"},{"key":"2023013110260063300_btaa003-B33","author":"Rives","year":"2019"},{"key":"2023013110260063300_btaa003-B34","doi-asserted-by":"crossref","first-page":"122","DOI":"10.1016\/j.compbiolchem.2007.11.004","article-title":"Subsequence-based feature map for protein function classification","volume":"32","author":"Sarac","year":"2008","journal-title":"Comput. Biol. Chem"},{"key":"2023013110260063300_btaa003-B35","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1016\/j.bbrc.2007.09.098","article-title":"EzyPred: a top\u2013down approach for predicting enzyme functional classes and subclasses","volume":"364","author":"Shen","year":"2007","journal-title":"Biochem. Biophys. Res. Commun"},{"key":"2023013110260063300_btaa003-B36","doi-asserted-by":"crossref","first-page":"31","DOI":"10.1007\/s10618-010-0175-9","article-title":"A survey of hierarchical classification across different application domains","volume":"22","author":"Silla","year":"2011","journal-title":"Data Min. Knowl. Disc"},{"key":"2023013110260063300_btaa003-B37","author":"Smith","year":"2018"},{"key":"2023013110260063300_btaa003-B38","first-page":"5926","author":"Song","year":"2019"},{"key":"2023013110260063300_btaa003-B39","first-page":"3319","author":"Sundararajan","year":"2017"},{"key":"2023013110260063300_btaa003-B40","doi-asserted-by":"crossref","first-page":"205","DOI":"10.1016\/S0022-5193(86)80075-3","article-title":"The classification of amino acid conservation","volume":"119","author":"Taylor","year":"1986","journal-title":"J. Theor. Biol"},{"key":"2023013110260063300_btaa003-B41","first-page":"D506","article-title":"UniProt: a worldwide hub of protein knowledge","volume":"47","year":"2018","journal-title":"Nucleic Acids Res"},{"key":"2023013110260063300_btaa003-B42","doi-asserted-by":"crossref","first-page":"185","DOI":"10.1007\/s10994-008-5077-3","article-title":"Decision trees for hierarchical multi-label classification","volume":"73","author":"Vens","year":"2008","journal-title":"Mach. Learn"},{"key":"2023013110260063300_btaa003-B43","first-page":"5075","volume-title":"Proceedings of the 35th International Conference on Machine Learning, Volume 80 of Proceedings of Machine Learning Research","author":"Wehrmann","year":"2018"},{"key":"2023013110260063300_btaa003-B44","author":"Qizhe","year":"2019"},{"key":"2023013110260063300_btaa003-B45","first-page":"5754","author":"Yang","year":"2019"},{"key":"2023013110260063300_btaa003-B46","doi-asserted-by":"crossref","first-page":"82","DOI":"10.1016\/j.ymeth.2018.05.026","article-title":"DeepText2go: improving large-scale protein function prediction with deep semantic text representation","volume":"145","author":"You","year":"2018","journal-title":"Methods"},{"key":"2023013110260063300_btaa003-B47","doi-asserted-by":"crossref","first-page":"2465","DOI":"10.1093\/bioinformatics\/bty130","article-title":"GOLabeler: improving sequence-based large-scale protein function prediction by learning to rank","volume":"34","author":"You","year":"2018","journal-title":"Bioinformatics"},{"key":"2023013110260063300_btaa003-B48","doi-asserted-by":"crossref","first-page":"244","DOI":"10.1186\/s13059-019-1835-8","article-title":"The CAFA challenge reports improved protein function prediction and new functional annotations for hundreds of genes through experimental screens","volume":"20","author":"Zhou","year":"2019","journal-title":"Genome Biol"},{"key":"2023013110260063300_btaa003-B49","doi-asserted-by":"crossref","first-page":"714","DOI":"10.3389\/fgene.2018.00714","article-title":"mlDEEPre: multi-functional enzyme function prediction with hierarchical multi-label deep learning","volume":"9","author":"Zou","year":"2019","journal-title":"Front. Genet"},{"key":"2023013110260063300_btaa003-B50","doi-asserted-by":"crossref","first-page":"225","DOI":"10.1038\/s42256-019-0049-9","article-title":"Leveraging implicit knowledge in neural networks for functional dissection and engineering of proteins","volume":"1","author":"Upmeier zu Belzen","year":"2019","journal-title":"Nat. Mach. Intell"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btaa003\/31955438\/btaa003.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/36\/8\/2401\/48984849\/bioinformatics_36_8_2401.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/36\/8\/2401\/48984849\/bioinformatics_36_8_2401.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,1,31]],"date-time":"2023-01-31T15:27:06Z","timestamp":1675178826000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/36\/8\/2401\/5698270"}},"subtitle":[],"editor":[{"given":"Yann","family":"Ponty","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2020,1,8]]},"references-count":50,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2020,4,15]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btaa003","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/704874","asserted-by":"object"}]},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2020,4,15]]},"published":{"date-parts":[[2020,1,8]]}}}