{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"institution":[{"name":"bioRxiv"}],"indexed":{"date-parts":[[2026,10,16]],"date-time":"2026-10-16T09:44:08Z","timestamp":1792143848514,"version":"4.4.0"},"posted":{"date-parts":[[2026,6,7]]},"group-title":"Bioengineering","reference-count":92,"publisher":"openRxiv","license":[{"start":{"date-parts":[[2026,6,7]],"date-time":"2026-06-07T00:00:00Z","timestamp":1780790400000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by-nc\/4.0\/"}],"funder":[{"award":["HDTRA1201001"],"award-info":[{"award-number":["HDTRA1201001"]}],"id":[{"id":"https:\/\/ror.org\/04tz64554","id-type":"ROR","asserted-by":"publisher"}]},{"award":["1R01GM146093-01A1"],"award-info":[{"award-number":["1R01GM146093-01A1"]}],"id":[{"id":"https:\/\/ror.org\/01cwqze88","id-type":"ROR","asserted-by":"publisher"}]},{"award":["F-1654"],"award-info":[{"award-number":["F-1654"]}],"id":[{"id":"https:\/\/ror.org\/00np6vq88","id-type":"ROR","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"accepted":{"date-parts":[[2026,10,12]]},"abstract":"<jats:title>SUMMARY<\/jats:title>\n                <jats:p>Predicting mutation effects is critical for protein engineering. A commonly pursued strategy is zero-shot prediction with deep-learning models, whose performance is typically evaluated using aggregate benchmark scores that obscure multidimensional biological tradeoffs. Here, we evaluate sequence, structure, and hybrid foundation models across curated datasets to expose the practical limitations of zero-shot predictions. We demonstrate that model performance is highly variable and heavily influenced by experimental noise and competing phenotypic goals. Notably, predictions from competing models often correlate more strongly with one another than with experimental outcomes. While current models effectively filter deleterious mutations and enrich high-scoring variants for natural functions, they struggle to identify precise optimal mutations and consistently fail to predict mutations that drive new-to-nature functions. These models learn from evolutionary constraints that tend to oppose the extreme biophysical optimizations required in targeted engineering. Consequently, zero-shot scoring serves as a valuable initial filter but cannot substitute for task-specific prediction.<\/jats:p>\n                <jats:sec>\n                  <jats:title>THE BIGGER PICTURE<\/jats:title>\n                  <jats:p>Deep learning provides a powerful toolkit for predicting mutational effects in protein engineering. However, realizing the full potential of deep-learning models requires a shift in how we evaluate them. Current benchmarks report aggregate scores across many proteins, and this practice masks high variability in performance across different proteins. Furthermore, because models are trained on natural sequences, their predictions reflect evolutionary compromises rather than targeted goals. Consequently, many models struggle to predict mutations that push proteins beyond what nature requires or that introduce entirely new-to-nature functions. The next breakthrough in computational design will rely on objective-specific evaluations rather than simply scaling model architectures. Refining our metrics for success will directly enable the next generation of targeted biotechnologies.<\/jats:p>\n                <\/jats:sec>\n                <jats:sec>\n                  <jats:title>HIGHLIGHTS<\/jats:title>\n                  <jats:list list-type=\"bullet\">\n                    <jats:list-item>\n                      <jats:p>Aggregate benchmark scores mask high protein-to-protein variability.<\/jats:p>\n                    <\/jats:list-item>\n                    <jats:list-item>\n                      <jats:p>A single prediction score cannot decouple competing stability and activity traits.<\/jats:p>\n                    <\/jats:list-item>\n                    <jats:list-item>\n                      <jats:p>Model rankings enrich functional variants but miss exact optimal mutations.<\/jats:p>\n                    <\/jats:list-item>\n                    <jats:list-item>\n                      <jats:p>Models consistently fail to predict mutations for new-to-nature functions.<\/jats:p>\n                    <\/jats:list-item>\n                  <\/jats:list>\n                <\/jats:sec>","DOI":"10.64898\/2026.06.04.730121","type":"posted-content","created":{"date-parts":[[2026,6,8]],"date-time":"2026-06-08T05:30:12Z","timestamp":1780896612000},"source":"Crossref","is-referenced-by-count":3,"title":["Inherent limitations of zero-shot mutation prediction in protein engineering"],"prefix":"10.64898","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4473-9424","authenticated-orcid":false,"given":"Phillip R.","family":"Woolley","sequence":"first","affiliation":[{"name":"Department of Molecular Biosciences, The University of Texas at Austin"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-4476-1026","authenticated-orcid":false,"given":"Aaron L.","family":"Feller","sequence":"additional","affiliation":[{"name":"Department of Molecular Biosciences, The University of Texas at Austin"},{"name":"Department of Integrative Biology, The University of Texas at Austin"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-6246-5338","authenticated-orcid":false,"given":"Andrew D.","family":"Ellington","sequence":"additional","affiliation":[{"name":"Department of Molecular Biosciences, The University of Texas at Austin"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-7470-9261","authenticated-orcid":false,"given":"Claus O.","family":"Wilke","sequence":"additional","affiliation":[{"name":"Department of Integrative Biology, The University of Texas at Austin"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"54368","reference":[{"key":"2026101601450519000_2026.06.04.730121v3.1","doi-asserted-by":"crossref","first-page":"221","DOI":"10.1038\/nmeth.2340","article-title":"A large-scale evaluation of computational protein function prediction","volume":"10","year":"2013","journal-title":"Nature Methods"},{"key":"2026101601450519000_2026.06.04.730121v3.2","doi-asserted-by":"publisher","DOI":"10.1038\/s41587-024-02123-4"},{"key":"2026101601450519000_2026.06.04.730121v3.3","doi-asserted-by":"crossref","first-page":"13","DOI":"10.1038\/s43586-025-00383-1","article-title":"Computational protein design","volume":"5","year":"2025","journal-title":"Nature Reviews Methods Primers"},{"key":"2026101601450519000_2026.06.04.730121v3.4","doi-asserted-by":"publisher","DOI":"10.1126\/science.adg7492"},{"key":"2026101601450519000_2026.06.04.730121v3.5","doi-asserted-by":"crossref","first-page":"630","DOI":"10.1038\/s41422-024-00989-2","article-title":"Zero-shot prediction of mutation effects with multimodal deep representation learning guides protein engineering","volume":"34","year":"2024","journal-title":"Cell Research"},{"key":"2026101601450519000_2026.06.04.730121v3.6","doi-asserted-by":"publisher","DOI":"10.1002\/pro.4780"},{"key":"2026101601450519000_2026.06.04.730121v3.7","unstructured":"Wang, L. , Li, X. , Zhang, H. , Wang, J. , Jiang, D. , Xue, Z. , and Wang, Y. (2025). A comprehensive review of protein language models. arXiv:2502.06881.."},{"key":"2026101601450519000_2026.06.04.730121v3.8","doi-asserted-by":"publisher","DOI":"10.1126\/science.ade2574"},{"key":"2026101601450519000_2026.06.04.730121v3.9","doi-asserted-by":"publisher","DOI":"10.1126\/science.add2187"},{"key":"2026101601450519000_2026.06.04.730121v3.10","first-page":"8844","article-title":"MSA Transformer","volume":"139","year":"2021","journal-title":"PMLR"},{"key":"2026101601450519000_2026.06.04.730121v3.11","first-page":"29287","article-title":"Language models enable zero-shot prediction of the effects of mutations on protein function","volume":"34","year":"2021","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2026101601450519000_2026.06.04.730121v3.12","first-page":"64331","article-title":"ProteinGym: Large-scale benchmarks for protein fitness prediction and design","volume":"36","year":"2023","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2026101601450519000_2026.06.04.730121v3.13","doi-asserted-by":"publisher","DOI":"10.64898\/2026.02.23.707496"},{"key":"2026101601450519000_2026.06.04.730121v3.14","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gkaa981"},{"key":"2026101601450519000_2026.06.04.730121v3.15","first-page":"04234","article-title":"Statistical uncertainty quantification for aggregate performance metrics in machine learning benchmarks","volume":"2501","year":"2025","journal-title":"arXiv"},{"key":"2026101601450519000_2026.06.04.730121v3.16","doi-asserted-by":"publisher","DOI":"10.1186\/s13059-017-1272-5"},{"key":"2026101601450519000_2026.06.04.730121v3.17","doi-asserted-by":"publisher","DOI":"10.1016\/S0006-3495(04)74329-5"},{"key":"2026101601450519000_2026.06.04.730121v3.18","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.0510098103"},{"key":"2026101601450519000_2026.06.04.730121v3.19","doi-asserted-by":"publisher","DOI":"10.1126\/science.1217405"},{"key":"2026101601450519000_2026.06.04.730121v3.20","doi-asserted-by":"crossref","first-page":"687","DOI":"10.1038\/s41592-019-0496-6","article-title":"Machine-learning-guided directed evolution for protein engineering","volume":"16","year":"2019","journal-title":"Nature Methods"},{"key":"2026101601450519000_2026.06.04.730121v3.21","doi-asserted-by":"publisher","DOI":"10.1038\/nrg2808"},{"key":"2026101601450519000_2026.06.04.730121v3.22","doi-asserted-by":"publisher","DOI":"10.1002\/anie.201708408"},{"key":"2026101601450519000_2026.06.04.730121v3.23","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-45630-3"},{"key":"2026101601450519000_2026.06.04.730121v3.24","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.76903"},{"key":"2026101601450519000_2026.06.04.730121v3.25","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1915562116"},{"key":"2026101601450519000_2026.06.04.730121v3.26","doi-asserted-by":"publisher","DOI":"10.1016\/j.ajhg.2021.07.001"},{"key":"2026101601450519000_2026.06.04.730121v3.27","doi-asserted-by":"publisher","DOI":"10.1016\/j.molcel.2024.04.008"},{"key":"2026101601450519000_2026.06.04.730121v3.28","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2020.08.012"},{"key":"2026101601450519000_2026.06.04.730121v3.29","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.58026"},{"key":"2026101601450519000_2026.06.04.730121v3.30","doi-asserted-by":"publisher","DOI":"10.1016\/j.jmb.2012.09.014"},{"key":"2026101601450519000_2026.06.04.730121v3.31","doi-asserted-by":"publisher","DOI":"10.1093\/molbev\/msu081"},{"key":"2026101601450519000_2026.06.04.730121v3.32","doi-asserted-by":"publisher","DOI":"10.1073\/pnas.1215206110"},{"key":"2026101601450519000_2026.06.04.730121v3.33","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2015.01.035"},{"key":"2026101601450519000_2026.06.04.730121v3.34","doi-asserted-by":"publisher","DOI":"10.1016\/j.cell.2015.09.055"},{"key":"2026101601450519000_2026.06.04.730121v3.35","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-45621-4"},{"key":"2026101601450519000_2026.06.04.730121v3.36","doi-asserted-by":"publisher","DOI":"10.1038\/s41588-018-0204-y"},{"key":"2026101601450519000_2026.06.04.730121v3.37","doi-asserted-by":"publisher","DOI":"10.1016\/j.molcel.2018.06.012"},{"key":"2026101601450519000_2026.06.04.730121v3.38","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.15802"},{"key":"2026101601450519000_2026.06.04.730121v3.39","doi-asserted-by":"publisher","DOI":"10.1016\/j.jmb.2013.01.032"},{"key":"2026101601450519000_2026.06.04.730121v3.40","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2014.09.072"},{"key":"2026101601450519000_2026.06.04.730121v3.41","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.16965"},{"key":"2026101601450519000_2026.06.04.730121v3.42","doi-asserted-by":"publisher","DOI":"10.1016\/j.molcel.2019.02.003"},{"key":"2026101601450519000_2026.06.04.730121v3.43","doi-asserted-by":"publisher","DOI":"10.1016\/j.chembiol.2023.08.005"},{"key":"2026101601450519000_2026.06.04.730121v3.44","doi-asserted-by":"publisher","DOI":"10.1093\/molbev\/msaa211"},{"key":"2026101601450519000_2026.06.04.730121v3.45","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.53810"},{"key":"2026101601450519000_2026.06.04.730121v3.46","doi-asserted-by":"publisher","DOI":"10.1016\/j.celrep.2016.03.046"},{"key":"2026101601450519000_2026.06.04.730121v3.47","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-47671-0"},{"key":"2026101601450519000_2026.06.04.730121v3.48","doi-asserted-by":"publisher","DOI":"10.7554\/eLife.53476"},{"key":"2026101601450519000_2026.06.04.730121v3.49","doi-asserted-by":"publisher","DOI":"10.1126\/science.1231434"},{"key":"2026101601450519000_2026.06.04.730121v3.50","doi-asserted-by":"publisher","DOI":"10.1002\/anie.201402809"},{"key":"2026101601450519000_2026.06.04.730121v3.51","doi-asserted-by":"publisher","DOI":"10.1074\/jbc.M112.346288"},{"key":"2026101601450519000_2026.06.04.730121v3.52","doi-asserted-by":"publisher","DOI":"10.1046\/j.1432-1033.2002.03143.x"},{"key":"2026101601450519000_2026.06.04.730121v3.53","doi-asserted-by":"publisher","DOI":"10.1016\/j.cub.2006.10.063"},{"key":"2026101601450519000_2026.06.04.730121v3.54","doi-asserted-by":"publisher","DOI":"10.1038\/ncomms2246"},{"key":"2026101601450519000_2026.06.04.730121v3.55","doi-asserted-by":"crossref","first-page":"e1000002","DOI":"10.1371\/journal.pcbi.1000002","article-title":"How protein stability and new functions trade off","volume":"4","year":"2008","journal-title":"PLoS computational biology"},{"key":"2026101601450519000_2026.06.04.730121v3.56","first-page":"84641","article-title":"Zero-shot protein stability prediction by inverse folding models: a free energy interpretation","volume":"38","year":"2026","journal-title":"Advances in Neural Information Processing Systems"},{"key":"2026101601450519000_2026.06.04.730121v3.57","doi-asserted-by":"publisher","DOI":"10.1101\/2024.09.23.614603"},{"key":"2026101601450519000_2026.06.04.730121v3.58","doi-asserted-by":"publisher","DOI":"10.64898\/2026.06.03.729735"},{"key":"2026101601450519000_2026.06.04.730121v3.59","doi-asserted-by":"publisher","DOI":"10.1126\/science.ads0018"},{"key":"2026101601450519000_2026.06.04.730121v3.60","doi-asserted-by":"crossref","unstructured":"Ouyang-Zhang, J. , Gong, C. , Zhao, Y. , Kr\u00e4henb\u00fchl, P. , Klivans, A.R. , and Diaz, D.J. (2025). Distilling structural representations into protein sequence models. The Thirteenth International Conference on Learning Representations, https:\/\/openreview.net\/forum?id=KXrgDM3mVD..","DOI":"10.1101\/2024.11.08.622579"},{"key":"2026101601450519000_2026.06.04.730121v3.61","doi-asserted-by":"crossref","unstructured":"Su, J. , Han, C. , Zhou, Y. , Shan, J. , Zhou, X. , and Yuan, F. (2024). SaProt: Protein language modeling with structure-aware vocabulary. The Twelfth International Conference on Learning Representations, https:\/\/openreview.net\/forum?id=6MRm3G4NiU.","DOI":"10.1101\/2023.10.01.560349"},{"key":"2026101601450519000_2026.06.04.730121v3.62","doi-asserted-by":"publisher","DOI":"10.1093\/nargab\/lqae150"},{"key":"2026101601450519000_2026.06.04.730121v3.63","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-024-07601-y"},{"key":"2026101601450519000_2026.06.04.730121v3.64","doi-asserted-by":"publisher","DOI":"10.1186\/s13073-024-01340-5"},{"key":"2026101601450519000_2026.06.04.730121v3.65","doi-asserted-by":"publisher","DOI":"10.7717\/peerj.19919"},{"key":"2026101601450519000_2026.06.04.730121v3.66","doi-asserted-by":"publisher","DOI":"10.64898\/2026.03.08.710389"},{"key":"2026101601450519000_2026.06.04.730121v3.67","doi-asserted-by":"publisher","DOI":"10.15252\/msb.202211474"},{"key":"2026101601450519000_2026.06.04.730121v3.68","doi-asserted-by":"publisher","DOI":"10.1002\/pro.4688"},{"key":"2026101601450519000_2026.06.04.730121v3.69","doi-asserted-by":"crossref","first-page":"801","DOI":"10.1038\/nmeth.3027","article-title":"Deep mutational scanning: a new style of protein science","volume":"11","year":"2014","journal-title":"Nature Methods"},{"key":"2026101601450519000_2026.06.04.730121v3.70","doi-asserted-by":"crossref","first-page":"e70526","DOI":"10.1002\/pro.70526","article-title":"Supervised learning of protein variant effects across large-scale mutagenesis datasets","volume":"35","year":"2026","journal-title":"Protein Science"},{"key":"2026101601450519000_2026.06.04.730121v3.71","doi-asserted-by":"crossref","first-page":"13280","DOI":"10.1038\/s41598-023-40247-w","article-title":"Two sequence- and two structure-based ml models have learned different aspects of protein biochemistry","volume":"13","year":"2023","journal-title":"Scientific Reports"},{"key":"2026101601450519000_2026.06.04.730121v3.72","doi-asserted-by":"crossref","unstructured":"Berry, S.P. , Gaudet, R. , and Marks, D.S . (2026). Differences between protein fitness models can be used to design variants of altered specificity. bioRxiv pp. 2026\u201306.","DOI":"10.64898\/2026.06.10.731299"},{"key":"2026101601450519000_2026.06.04.730121v3.73","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-022-04599-z"},{"key":"2026101601450519000_2026.06.04.730121v3.74","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-025-56751-8"},{"key":"2026101601450519000_2026.06.04.730121v3.75","doi-asserted-by":"publisher","DOI":"10.1021\/jacs.3c10941"},{"key":"2026101601450519000_2026.06.04.730121v3.76","doi-asserted-by":"crossref","first-page":"eadr6006","DOI":"10.1126\/science.adr6006","article-title":"Rapid in silico directed evolution by a protein language model with EVOLVEpro","volume":"387","year":"2025","journal-title":"Science"},{"key":"2026101601450519000_2026.06.04.730121v3.77","doi-asserted-by":"publisher","DOI":"10.1038\/s41467-024-51844-2"},{"key":"2026101601450519000_2026.06.04.730121v3.78","doi-asserted-by":"crossref","first-page":"eaea1820","DOI":"10.1126\/science.aea1820","article-title":"Rapid directed evolution guided by protein language models and epistatic interactions","volume":"392","year":"2026","journal-title":"Science"},{"key":"2026101601450519000_2026.06.04.730121v3.79","doi-asserted-by":"crossref","first-page":"btaf064","DOI":"10.1093\/bioinformatics\/btaf064","article-title":"Foldx force field revisited, an improved version","volume":"41","year":"2025","journal-title":"Bioinformatics"},{"key":"2026101601450519000_2026.06.04.730121v3.80","doi-asserted-by":"crossref","first-page":"3769","DOI":"10.1021\/ct400230y","article-title":"Primo: a transferable coarse-grained force field for proteins","volume":"9","year":"2013","journal-title":"Journal of chemical theory and computation"},{"key":"2026101601450519000_2026.06.04.730121v3.81","doi-asserted-by":"publisher","DOI":"10.1038\/s41592-020-0848-2"},{"key":"2026101601450519000_2026.06.04.730121v3.82","doi-asserted-by":"publisher","DOI":"10.1016\/j.molcel.2016.06.012"},{"key":"2026101601450519000_2026.06.04.730121v3.83","doi-asserted-by":"crossref","first-page":"1868","DOI":"10.1038\/s41592-025-02776-2","article-title":"Biophysics-based protein language models for protein engineering","volume":"22","year":"2025","journal-title":"Nature Methods"},{"key":"2026101601450519000_2026.06.04.730121v3.84","doi-asserted-by":"publisher","DOI":"10.1101\/2024.03.07.584001"},{"key":"2026101601450519000_2026.06.04.730121v3.85","doi-asserted-by":"publisher","DOI":"10.1145\/3786319"},{"key":"2026101601450519000_2026.06.04.730121v3.86","doi-asserted-by":"publisher","DOI":"10.1038\/s41586-021-03819-2"},{"key":"2026101601450519000_2026.06.04.730121v3.87","doi-asserted-by":"publisher","DOI":"10.3390\/v8060155"},{"key":"2026101601450519000_2026.06.04.730121v3.88","doi-asserted-by":"publisher","DOI":"10.1038\/srep04942"},{"key":"2026101601450519000_2026.06.04.730121v3.89","doi-asserted-by":"publisher","DOI":"10.1093\/nar\/gky1049"},{"key":"2026101601450519000_2026.06.04.730121v3.90","doi-asserted-by":"publisher","DOI":"10.1093\/NAR\/GKAB1061"},{"key":"2026101601450519000_2026.06.04.730121v3.91","doi-asserted-by":"publisher","DOI":"10.1101\/2025.06.14.659707"},{"key":"2026101601450519000_2026.06.04.730121v3.92","doi-asserted-by":"publisher","DOI":"10.1038\/s41587-023-01773-0"}],"container-title":[],"original-title":[],"link":[{"URL":"https:\/\/syndication.highwire.org\/content\/doi\/10.64898\/2026.06.04.730121","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,10,16]],"date-time":"2026-10-16T08:45:16Z","timestamp":1792140316000},"score":1,"resource":{"primary":{"URL":"http:\/\/biorxiv.org\/lookup\/doi\/10.64898\/2026.06.04.730121"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,7]]},"references-count":92,"URL":"https:\/\/doi.org\/10.64898\/2026.06.04.730121","relation":{},"subject":[],"published":{"date-parts":[[2026,6,7]]},"subtype":"preprint"}}