{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,24]],"date-time":"2026-10-24T14:54:32Z","timestamp":1792853672459,"version":"4.5.0"},"reference-count":53,"publisher":"Oxford University Press (OUP)","issue":"8","license":[{"start":{"date-parts":[[2022,1,10]],"date-time":"2022-01-10T00:00:00Z","timestamp":1641772800000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100003977","name":"Israel Science Foundation","doi-asserted-by":"crossref","award":["2753\/20"],"award-info":[{"award-number":["2753\/20"]}],"id":[{"id":"10.13039\/501100003977","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2022,4,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Summary<\/jats:title>\n                    <jats:p>Self-supervised deep language modeling has shown unprecedented success across natural language tasks, and has recently been repurposed to biological sequences. However, existing models and pretraining methods are designed and optimized for text analysis. We introduce ProteinBERT, a deep language model specifically designed for proteins. Our pretraining scheme combines language modeling with a novel task of Gene Ontology (GO) annotation prediction. We introduce novel architectural elements that make the model highly efficient and flexible to long sequences. The architecture of ProteinBERT consists of both local and global representations, allowing end-to-end processing of these types of inputs and outputs. ProteinBERT obtains near state-of-the-art performance, and sometimes exceeds it, on multiple benchmarks covering diverse protein properties (including protein structure, post-translational modifications and biophysical attributes), despite using a far smaller and faster model than competing deep-learning methods. Overall, ProteinBERT provides an efficient framework for rapidly training protein predictors, even with limited labeled data.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Availability and implementation<\/jats:title>\n                    <jats:p>Code and pretrained model weights are available at https:\/\/github.com\/nadavbra\/protein_bert.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Supplementary information<\/jats:title>\n                    <jats:p>Supplementary data are available at Bioinformatics online.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btac020","type":"journal-article","created":{"date-parts":[[2022,1,8]],"date-time":"2022-01-08T07:08:37Z","timestamp":1641625717000},"page":"2102-2110","source":"Crossref","is-referenced-by-count":1082,"title":["ProteinBERT: a universal deep-learning model of protein sequence and function"],"prefix":"10.1093","volume":"38","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-0510-2546","authenticated-orcid":false,"given":"Nadav","family":"Brandes","sequence":"first","affiliation":[{"name":"School of Computer Science and Engineering, The Hebrew University of Jerusalem , Jerusalem 9190401, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Dan","family":"Ofer","sequence":"additional","affiliation":[{"name":"Department of Biological Chemistry, The Alexander Silberman Institute of Life Sciences, The Hebrew University of Jerusalem, Jerusalem 9190401, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yam","family":"Peleg","sequence":"additional","affiliation":[{"name":"Deep Trading Ltd. , Haifa 3508401, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7218-2558","authenticated-orcid":false,"given":"Nadav","family":"Rappoport","sequence":"additional","affiliation":[{"name":"Department of Software and Information Systems Engineering, Faculty of Engineering Sciences, Ben-Gurion University of the Negev , Beer Sheva 8410501, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9357-4526","authenticated-orcid":false,"given":"Michal","family":"Linial","sequence":"additional","affiliation":[{"name":"Department of Biological Chemistry, The Alexander Silberman Institute of Life Sciences, The Hebrew University of Jerusalem, Jerusalem 9190401, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2022,2,10]]},"reference":[{"key":"2023020109022215400_btac020-B1","first-page":"265","volume-title":"12th USENIX Symposium on Operating Systems Design and Implementation (OSDI16)","author":"Abadi","year":"2016"},{"key":"2023020109022215400_btac020-B2","doi-asserted-by":"crossref","first-page":"1315","DOI":"10.1038\/s41592-019-0598-1","article-title":"Unified rational protein engineering with sequence-based deep representation learning","volume":"16","author":"Alley","year":"2019","journal-title":"Nat. Methods"},{"key":"2023020109022215400_btac020-B3","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1016\/S0022-2836(05)80360-2","article-title":"Basic local alignment search tool","volume":"215","author":"Altschul","year":"1990","journal-title":"J. Mol. Biol"},{"key":"2023020109022215400_btac020-B4","doi-asserted-by":"crossref","first-page":"3389","DOI":"10.1093\/nar\/25.17.3389","article-title":"Gapped BLAST and PSI-BLAST\u202f: a new generation of protein database search programs","volume":"25","author":"Altschul","year":"1997","journal-title":"Nucleic Acids Res"},{"key":"2023020109022215400_btac020-B5","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1038\/75556","article-title":"Gene ontology: tool for the unification of biology","volume":"25","author":"Ashburner","year":"2000","journal-title":"Nat. Genet"},{"key":"2023020109022215400_btac020-B45","doi-asserted-by":"crossref","first-page":"D310","DOI":"10.1093\/nar\/gkt1242","article-title":"SCOP2 prototype: a new approach to protein structure mining","volume":"42","author":"Andreeva","year":"2014","journal-title":"NucleicAcids Res"},{"key":"2023020109022215400_btac020-B47"},{"key":"2023020109022215400_btac020-B46","doi-asserted-by":"crossref","first-page":"D376","DOI":"10.1093\/nar\/gkz1064","article-title":"The SCOP database in 2020: expanded classification of representativefamily and superfamily domains of known protein structures","volume":"48","author":"Andreeva","year":"2020","journal-title":"Nucleic Acids Res"},{"key":"2023020109022215400_btac020-B6","author":"Bepler","year":"2019"},{"key":"2023020109022215400_btac020-B7","doi-asserted-by":"crossref","first-page":"23","DOI":"10.1007\/978-1-4939-3167-5_2","volume-title":"Plant Bioinformatics","author":"Boutet","year":"2016"},{"key":"2023020109022215400_btac020-B50","doi-asserted-by":"crossref","DOI":"10.1093\/database\/baw133","article-title":"ASAP: Amachine learning framework for local protein properties","volume":"2016","author":"Brandes","year":"2016","journal-title":"Database"},{"key":"2023020109022215400_btac020-B8","author":"Brown","year":"2020"},{"key":"2023020109022215400_btac020-B9","author":"Chen","year":"2020"},{"key":"2023020109022215400_btac020-B10","author":"Chollet","year":"2015"},{"key":"2023020109022215400_btac020-B11","author":"Choromanski","year":"2020"},{"key":"2023020109022215400_btac020-B12","author":"Clark","year":"2020"},{"key":"2023020109022215400_btac020-B13","author":"Devlin","year":"2018"},{"key":"2023020109022215400_btac020-B14","first-page":"299","article-title":"Transfer learning for text classification","volume":"18","author":"Do","year":"2005","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"2023020109022215400_btac020-B15","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1109\/TPAMI.2021.3095381","article-title":"ProtTrans: towards cracking the language of lifes code through self-supervised deep learning and high performance computing","volume":"1","author":"Elnaggar","year":"2021","journal-title":"IEEE Trans. Pattern Anal. Mach. Intell"},{"key":"2023020109022215400_btac020-B16","doi-asserted-by":"crossref","first-page":"D222","DOI":"10.1093\/nar\/gkt1223","article-title":"Pfam: the protein families database","volume":"42","author":"Finn","year":"2014","journal-title":"Nucleic Acids Res"},{"key":"2023020109022215400_btac020-B17","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s12859-019-3220-8","article-title":"Modeling aspects of the language of life through transfer-learning protein sequences","volume":"20","author":"Heinzinger","year":"2019","journal-title":"BMC Bioinformatics"},{"key":"2023020109022215400_btac020-B18","author":"Hendrycks","year":"2016"},{"key":"2023020109022215400_btac020-B19","first-page":"237","author":"Hochreiter","year":"2001"},{"key":"2023020109022215400_btac020-B20","author":"Howard","year":"2018"},{"key":"2023020109022215400_btac020-B48","doi-asserted-by":"crossref","first-page":"D512","DOI":"10.1093\/nar\/gku1267","article-title":"PhosphoSitePlus, 2014: mutations, PTMs and recalibrations","volume":"43","author":"Hornbeck","year":"2015","journal-title":"Nucleic AcidsRes"},{"key":"2023020109022215400_btac020-B21","author":"Keskar","year":"2019"},{"key":"2023020109022215400_btac020-B22","author":"Lan","year":"2019"},{"key":"2023020109022215400_btac020-B23","author":"Madani","year":"2020"},{"key":"2023020109022215400_btac020-B44","doi-asserted-by":"crossref","first-page":"7","DOI":"10.1002\/prot.25415","article-title":"Critical assessment of methods of protein structure prediction (CASP)\u2014Round XII","volume":"86","author":"Moult","year":"2018","journal-title":"Proteins Struct Funct Bioinforma"},{"key":"2023020109022215400_btac020-B24","first-page":"1","volume-title":"Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics","author":"Nambiar","year":"2020"},{"key":"2023020109022215400_btac020-B25","doi-asserted-by":"crossref","first-page":"328","DOI":"10.18653\/v1\/K19-1031","volume-title":"Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)","author":"Neishi","year":"2019"},{"key":"2023020109022215400_btac020-B51","doi-asserted-by":"crossref","first-page":"931","DOI":"10.1093\/bioinformatics\/btt725","article-title":"NeuroPID: a predictorfor identifying neuropeptide precursors from metazoan proteomes","volume":"30","author":"Ofer","year":"2014","journal-title":"Bioinformatics"},{"key":"2023020109022215400_btac020-B49","doi-asserted-by":"crossref","first-page":"3429","DOI":"10.1093\/bioinformatics\/btv345","article-title":"ProFET: Featureengineering captures high-level protein functions","volume":"31","author":"Ofer","year":"2015","journal-title":"Bioinformatics"},{"key":"2023020109022215400_btac020-B26","doi-asserted-by":"crossref","first-page":"1750","DOI":"10.1016\/j.csbj.2021.03.022","article-title":"The language of proteins: NLP, machine learning & protein sequences","volume":"19","author":"Ofer","year":"2021","journal-title":"Comput. Struct. Biotechnol. J"},{"key":"2023020109022215400_btac020-B27","doi-asserted-by":"crossref","first-page":"1345","DOI":"10.1109\/TKDE.2009.191","article-title":"A survey on transfer learning","volume":"22","author":"Pan","year":"2010","journal-title":"IEEE Trans. Knowl. Data Eng"},{"key":"2023020109022215400_btac020-B28","author":"Radford","year":"2018"},{"key":"2023020109022215400_btac020-B29","first-page":"9","article-title":"Language models are unsupervised multitask learners","volume":"1","author":"Radford","year":"2019","journal-title":"OpenAI Blog"},{"key":"2023020109022215400_btac020-B53"},{"key":"2023020109022215400_btac020-B30","author":"Raffel","year":"2019"},{"key":"2023020109022215400_btac020-B31","first-page":"9689","article-title":"Evaluating protein transfer learning with tape","volume":"32","author":"Rao","year":"2019","journal-title":"Adv. Neural Inf. Process. Syst"},{"key":"2023020109022215400_btac020-B32","author":"Rao","year":"2021"},{"key":"2023020109022215400_btac020-B33","doi-asserted-by":"crossref","first-page":"e2016239118","DOI":"10.1073\/pnas.2016239118","article-title":"Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences","volume":"118","author":"Rives","year":"2021","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023020109022215400_btac020-B52","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1038\/nature17995","article-title":"Local fitness landscape of the green fluorescent protein","volume":"533","author":"Sarkisyan","year":"2016","journal-title":"Nature"},{"key":"2023020109022215400_btac020-B34","doi-asserted-by":"crossref","first-page":"148","DOI":"10.1016\/S0006-3495(96)79210-X","article-title":"The Shannon information entropy of protein sequences","volume":"71","author":"Strait","year":"1996","journal-title":"Biophys. J"},{"key":"2023020109022215400_btac020-B35","doi-asserted-by":"crossref","first-page":"2401","DOI":"10.1093\/bioinformatics\/btaa003","article-title":"UDSMProt: universal deep sequence models for protein classification","volume":"36","author":"Strodthoff","year":"2020","journal-title":"Bioinformatics"},{"key":"2023020109022215400_btac020-B36","author":"Sturmfels","year":"2020"},{"key":"2023020109022215400_btac020-B37","doi-asserted-by":"crossref","first-page":"1282","DOI":"10.1093\/bioinformatics\/btm098","article-title":"UniRef: comprehensive and non-redundant UniProt reference clusters","volume":"23","author":"Suzek","year":"2007","journal-title":"Bioinformatics"},{"key":"2023020109022215400_btac020-B38","first-page":"640","volume-title":"Advances in Neural Information Processing Systems","author":"Thrun","year":"1996"},{"key":"2023020109022215400_btac020-B39","author":"Vaswani","year":"2017"},{"key":"2023020109022215400_btac020-B40","author":"Wang","year":"2019"},{"key":"2023020109022215400_btac020-B41","author":"Yang","year":"2019"},{"key":"2023020109022215400_btac020-B42","doi-asserted-by":"crossref","first-page":"1496","DOI":"10.1073\/pnas.1914677117","article-title":"Improved protein structure prediction using predicted interresidue orientations","volume":"117","author":"Yang","year":"2020","journal-title":"Proc. Natl. Acad. Sci. USA"},{"key":"2023020109022215400_btac020-B43","author":"Zaheer","year":"2020"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btac020\/42490343\/btac020.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/38\/8\/2102\/49009610\/btac020.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/38\/8\/2102\/49009610\/btac020.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,2,1]],"date-time":"2023-02-01T15:50:39Z","timestamp":1675266639000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/38\/8\/2102\/6502274"}},"subtitle":[],"editor":[{"given":"Pier Luigi","family":"Martelli","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2022,2,10]]},"references-count":53,"journal-issue":{"issue":"8","published-print":{"date-parts":[[2022,4,12]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btac020","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.05.24.445464","asserted-by":"object"}]},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2022,4,15]]},"published":{"date-parts":[[2022,2,10]]}}}