{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,1,27]],"date-time":"2026-01-27T09:49:08Z","timestamp":1769507348014,"version":"3.49.0"},"reference-count":42,"publisher":"Oxford University Press (OUP)","issue":"11","license":[{"start":{"date-parts":[[2025,10,29]],"date-time":"2025-10-29T00:00:00Z","timestamp":1761696000000},"content-version":"vor","delay-in-days":5,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2025,11,1]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Motivation<\/jats:title>\n                    <jats:p>Protein language models (PLMs) have transformed protein research by learning rich representations from sequence data alone, yet they largely ignore the wealth of structural information now available through advances in structure prediction. Current methods that incorporate structural data often require substantial computational resources and complex architectures, limiting their practical adoption. We present a novel joint sequence and structure embedding method that achieves computational and parameter efficiency while maintaining high performance. Our approach introduces a lightweight integration framework that combines pretrained sequence transformers\u2019 self-attention with specialized structural adapters, enabling seamless incorporation of structural knowledge into existing PLMs through these enhanced self-attention mechanisms.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>The method demonstrates remarkable efficiency, requiring only modest pretraining on 542K protein structures, three orders of magnitude less than the data used to train PLMs, using standard masked language modeling objectives. Despite this lightweight approach, our joint embeddings consistently outperform sequence-only models like ESM-2 while achieving comparable results to more complex structure-based methods that use significantly more parameters and computational resources. This work establishes a new paradigm for protein representation learning that balances performance with practical constraints. By providing computationally efficient joint sequence-structure embeddings, we offer the scientific community an accessible tool that captures both sequential and structural protein information without the computational overhead typically associated with structure-aware models.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Availability and implementation<\/jats:title>\n                    <jats:p>code and links to checkpoints are available at https:\/\/github.com\/BorgwardtLab\/PST.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btaf582","type":"journal-article","created":{"date-parts":[[2025,10,21]],"date-time":"2025-10-21T11:38:33Z","timestamp":1761046713000},"source":"Crossref","is-referenced-by-count":3,"title":["Endowing protein language models with structural knowledge"],"prefix":"10.1093","volume":"41","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-1067-7651","authenticated-orcid":false,"given":"Philip","family":"Hartout","sequence":"first","affiliation":[{"name":"Department of Machine Learning and Systems Biology, Max Planck Institute of Biochemistry , Bavaria, Martinsried 82152,","place":["Germany"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Dexiong","family":"Chen","sequence":"additional","affiliation":[{"name":"Department of Machine Learning and Systems Biology, Max Planck Institute of Biochemistry , Bavaria, Martinsried 82152,","place":["Germany"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Paolo","family":"Pellizzoni","sequence":"additional","affiliation":[{"name":"Department of Machine Learning and Systems Biology, Max Planck Institute of Biochemistry , Bavaria, Martinsried 82152,","place":["Germany"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Carlos","family":"Oliver","sequence":"additional","affiliation":[{"name":"Department of Machine Learning and Systems Biology, Max Planck Institute of Biochemistry , Bavaria, Martinsried 82152,","place":["Germany"]}],"role":[{"role":"author","vocabulary":"crossref"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7221-2393","authenticated-orcid":false,"given":"Karsten","family":"Borgwardt","sequence":"additional","affiliation":[{"name":"Department of Machine Learning and Systems Biology, Max Planck Institute of Biochemistry , Bavaria, Martinsried 82152,","place":["Germany"]}],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"286","published-online":{"date-parts":[[2025,10,24]]},"reference":[{"key":"2025111103174039500_btaf582-B1","doi-asserted-by":"crossref","first-page":"3520","DOI":"10.1016\/j.cell.2022.07.019","article-title":"Accurate de novo design of membrane-traversing macrocycles","volume":"185","author":"Bhardwaj","year":"2022","journal-title":"Cell"},{"key":"2025111103174039500_btaf582-B2","doi-asserted-by":"crossref","first-page":"5061","DOI":"10.1021\/jm100112j","article-title":"A medicinal chemist\u2019s guide to molecular interactions","volume":"53","author":"Bissantz","year":"2010","journal-title":"J Med Chem"},{"key":"2025111103174039500_btaf582-B3","doi-asserted-by":"crossref","first-page":"D553","DOI":"10.1093\/nar\/gkab1054","article-title":"SCOPe: improvements to the structural classification of proteins\u2013extended database to facilitate variant interpretation and machine learning","volume":"50","author":"Chandonia","year":"2022","journal-title":"Nucleic Acids Res"},{"key":"2025111103174039500_btaf582-B4","author":"Chen D, O\u2019Bray L, Borgwardt K."},{"key":"2025111103174039500_btaf582-B5","doi-asserted-by":"crossref","first-page":"49","DOI":"10.1126\/science.add2187","article-title":"Robust deep learning\u2013based protein sequence design using ProteinMPNN","volume":"378","author":"Dauparas","year":"2022","journal-title":"Science"},{"key":"2025111103174039500_btaf582-B6","first-page":"4171","author":"Devlin J, Chang M-W, Lee K","year":"2019"},{"key":"2025111103174039500_btaf582-B7","author":"Elnaggar A, Heinzinger M, Dallago C","year":"2021"},{"key":"2025111103174039500_btaf582-B8","author":"Elnaggar A., Essam H., Salah-Eldin W.","year":"2023"},{"key":"2025111103174039500_btaf582-B9","doi-asserted-by":"crossref","first-page":"154","DOI":"10.1002\/prot.24495","article-title":"Assessment of ligand binding site predictions in CASP10","volume":"82","author":"Gallo Cassarino","year":"2014","journal-title":"Proteins"},{"key":"2025111103174039500_btaf582-B10","author":"Gao Z, Tan C, Li S Z","year":"2023"},{"key":"2025111103174039500_btaf582-B11","doi-asserted-by":"crossref","first-page":"3168","DOI":"10.1038\/s41467-021-23303-9","article-title":"Structure-based protein function prediction using graph convolutional networks","volume":"12","author":"Gligorijevi\u0107","year":"2021","journal-title":"Nat Commun"},{"key":"2025111103174039500_btaf582-B12","doi-asserted-by":"crossref","first-page":"eads0018","DOI":"10.1126\/science.ads0018","article-title":"Simulating 500 million years of evolution with a language model","volume":"387","author":"Hayes","year":"2025","journal-title":"Science"},{"key":"2025111103174039500_btaf582-B13","doi-asserted-by":"crossref","first-page":"1295","DOI":"10.1093\/bioinformatics\/btx780","article-title":"DeepSF: deep convolutional neural network for mapping protein sequences to folds","volume":"34","author":"Hou","year":"2018","journal-title":"Bioinformatics"},{"key":"2025111103174039500_btaf582-B14","author":"Hu M, Yuan F, Yang K K"},{"key":"2025111103174039500_btaf582-B15","doi-asserted-by":"crossref","first-page":"499","DOI":"10.1002\/prot.22458","article-title":"Structure is three to ten times more conserved than sequence\u2014a study of structural response in protein cores","volume":"77","author":"Illerg\u00e5rd","year":"2009","journal-title":"Proteins: Structure, Function, and Bioinformatics"},{"key":"2025111103174039500_btaf582-B16","author":"Ingraham J, Garg V K, Barzilay R","year":"2019"},{"key":"2025111103174039500_btaf582-B17","author":"Jing B, Eismann S, Suriana P","year":"2021"},{"key":"2025111103174039500_btaf582-B18","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1038\/s41586-021-03819-2","article-title":"Highly accurate protein structure prediction with AlphaFold","volume":"596","author":"Jumper","year":"2021","journal-title":"Nature"},{"key":"2025111103174039500_btaf582-B19","doi-asserted-by":"crossref","first-page":"4139","DOI":"10.1038\/s41467-023-39840-4","article-title":"Turnover number predictions for kinetically uncharacterized enzymes using machine and deep learning","volume":"14","author":"Kroll","year":"2023","journal-title":"Nat Commun"},{"key":"2025111103174039500_btaf582-B20","author":"Kucera T, Oliver C, Chen D"},{"key":"2025111103174039500_btaf582-B21","doi-asserted-by":"crossref","first-page":"1123","DOI":"10.1126\/science.ade2574","article-title":"Evolutionary-scale prediction of atomic-level protein structure with a language model","volume":"379","author":"Lin","year":"2023","journal-title":"Science"},{"key":"2025111103174039500_btaf582-B22","author":"Meier J, Rao R, Verkuil R","year":"2021"},{"key":"2025111103174039500_btaf582-B23","doi-asserted-by":"crossref","first-page":"D412","DOI":"10.1093\/nar\/gkaa913","article-title":"Pfam: the protein families database in 2021","volume":"49","author":"Mistry","year":"2021","journal-title":"Nucleic Acids Res"},{"key":"2025111103174039500_btaf582-B24","first-page":"16990","author":"Notin P, Dias M, Frazer J"},{"key":"2025111103174039500_btaf582-B25","first-page":"331","author":"Notin P, Kollasch A, Ritter D","year":"2023"},{"key":"2025111103174039500_btaf582-B26","author":"Rao R, Bhattacharya N, Thomas N","year":"2019"},{"key":"2025111103174039500_btaf582-B27","doi-asserted-by":"crossref","first-page":"1443","DOI":"10.1039\/D2SC05709C","article-title":"AlphaFold accelerates artificial intelligence powered drug discovery: efficient discovery of a novel CDK20 small molecule inhibitor","volume":"14","author":"Ren","year":"2023","journal-title":"Chem Sci"},{"key":"2025111103174039500_btaf582-B28","doi-asserted-by":"crossref","first-page":"816","DOI":"10.1038\/s41592-018-0138-4","article-title":"Deep generative models of genetic variation capture the effects of mutations","volume":"15","author":"Riesselman","year":"2018","journal-title":"Nat Methods"},{"key":"2025111103174039500_btaf582-B29","volume":"118","author":"Rives A, Meier J, Sercu T","year":"2021","journal-title":"Proc Natl Acad Sci U S A"},{"key":"2025111103174039500_btaf582-B30","author":"Shanehsazzadeh A, Belanger D, Dohan D","year":"2020"},{"key":"2025111103174039500_btaf582-B31","doi-asserted-by":"crossref","first-page":"127063","DOI":"10.1016\/j.neucom.2023.127063","article-title":"Roformer: enhanced transformer with rotary position embedding","volume":"568","author":"Su","year":"2024","journal-title":"Neurocomputing (Amst)"},{"key":"2025111103174039500_btaf582-B32","doi-asserted-by":"crossref","first-page":"6338","DOI":"10.1021\/acs.jcim.4c00689","article-title":"Simple, efficient, and scalable structure-aware adapter boosts protein language models","volume":"64","author":"Tan","year":"2024","journal-title":"J Chem Inf Model"},{"key":"2025111103174039500_btaf582-B33","author":"Townshend RJL, V\u00f6gele M, Suriana P","year":"2020"},{"key":"2025111103174039500_btaf582-B34","doi-asserted-by":"crossref","first-page":"D439","DOI":"10.1093\/nar\/gkab1061","article-title":"AlphaFold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models","volume":"50","author":"Varadi","year":"2022","journal-title":"Nucleic Acids Res"},{"key":"2025111103174039500_btaf582-B35","first-page":"5998","author":"Vaswani A, Shazeer N, Parmar N","year":"2017"},{"key":"2025111103174039500_btaf582-B36","doi-asserted-by":"crossref","first-page":"70","DOI":"10.1038\/s41467-024-54816-8","article-title":"DPFunc: accurately predicting protein function via deep learning with domain-guided structure information","volume":"16","author":"Wang","year":"2025","journal-title":"Nat Commun"},{"key":"2025111103174039500_btaf582-B37","doi-asserted-by":"crossref","first-page":"6832","DOI":"10.1038\/s41598-022-10775-y","article-title":"Lm-gvp: an extensible sequence and structure informed deep learning framework for protein property prediction","volume":"12","author":"Wang","year":"2022","journal-title":"Sci Rep"},{"key":"2025111103174039500_btaf582-B38","author":"Xu K, Hu W, Leskovec J","year":"2019"},{"key":"2025111103174039500_btaf582-B39","doi-asserted-by":"crossref","first-page":"W248","DOI":"10.1093\/nar\/gkae381","article-title":"GPSFun: geometry-aware protein sequence function predictions with language models","volume":"52","author":"Yuan","year":"2024","journal-title":"Nucleic Acids Res"},{"key":"2025111103174039500_btaf582-B40","author":"Zhang Z, Xu M, Jamasb A"},{"key":"2025111103174039500_btaf582-B41","author":"Zhang Z, Wang C, Xu M"},{"key":"2025111103174039500_btaf582-B42","author":"Zhang Z, Xu M, Lozano A","year":"2023"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btaf582\/65019961\/btaf582.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/41\/11\/btaf582\/65019961\/btaf582.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/41\/11\/btaf582\/65019961\/btaf582.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,11,11]],"date-time":"2025-11-11T08:17:50Z","timestamp":1762849070000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/doi\/10.1093\/bioinformatics\/btaf582\/8305579"}},"subtitle":[],"editor":[{"given":"Jianlin","family":"Cheng","sequence":"additional","affiliation":[],"role":[{"role":"editor","vocabulary":"crossref"}]}],"short-title":[],"issued":{"date-parts":[[2025,10,24]]},"references-count":42,"journal-issue":{"issue":"11","published-print":{"date-parts":[[2025,11,1]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btaf582","relation":{},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2025,11]]},"published":{"date-parts":[[2025,10,24]]},"article-number":"btaf582"}}