{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,10,24]],"date-time":"2026-10-24T00:30:38Z","timestamp":1792801838636,"version":"4.5.0"},"reference-count":62,"publisher":"Springer Science and Business Media LLC","issue":"2","license":[{"start":{"date-parts":[[2024,2,23]],"date-time":"2024-02-23T00:00:00Z","timestamp":1708646400000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,2,23]],"date-time":"2024-02-23T00:00:00Z","timestamp":1708646400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100000266","name":"RCUK | Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/T517811\/1"],"award-info":[{"award-number":["EP\/T517811\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"RCUK | Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/W522582\/1"],"award-info":[{"award-number":["EP\/W522582\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["Nat Mach Intell"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Protein representations from deep language models have yielded state-of-the-art performance across many tasks in computational protein engineering. In recent years, progress has primarily focused on parameter count, with recent models\u2019 capacities surpassing the size of the very datasets they were trained on. Here we propose an alternative direction. We show that large language models trained on codons, instead of amino acid sequences, provide high-quality representations that outperform comparable state-of-the-art models across a variety of tasks. In some tasks, such as species recognition, prediction of protein and transcript abundance or melting point estimation, we show that a language model trained on codons outperforms every other published protein language model, including some that contain over 50 times more parameters. These results indicate that, in addition to commonly studied scale and model complexity, the information content of biological data provides an orthogonal direction to improve the power of machine learning in biology.<\/jats:p>","DOI":"10.1038\/s42256-024-00791-0","type":"journal-article","created":{"date-parts":[[2024,2,23]],"date-time":"2024-02-23T12:02:30Z","timestamp":1708689750000},"page":"170-179","update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":94,"title":["Codon language embeddings provide strong signals for use in protein engineering"],"prefix":"10.1038","volume":"6","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-1408-5554","authenticated-orcid":false,"given":"Carlos","family":"Outeiral","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-1388-2252","authenticated-orcid":false,"given":"Charlotte M.","family":"Deane","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,2,23]]},"reference":[{"key":"791_CR1","doi-asserted-by":"publisher","first-page":"521","DOI":"10.1038\/s42256-022-00499-z","volume":"4","author":"N Ferruz","year":"2022","unstructured":"Ferruz, N. & H\u00f6cker, B. Controllable protein design with language models. Nat. Mach. Intell. 4, 521\u2013532 (2022).","journal-title":"Nat. Mach. Intell."},{"key":"791_CR2","doi-asserted-by":"publisher","first-page":"1282","DOI":"10.1093\/bioinformatics\/btm098","volume":"23","author":"BE Suzek","year":"2007","unstructured":"Suzek, B. E., Huang, H., McGarvey, P., Mazumder, R. & Wu, C. H. UniRef: comprehensive and non-redundant UniProt reference clusters. Bioinformatics 23, 1282\u20131288 (2007).","journal-title":"Bioinformatics"},{"key":"791_CR3","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1186\/s12859-020-3439-4","volume":"21","author":"J Reeb","year":"2020","unstructured":"Reeb, J., Wirth, T. & Rost, B. Variant effect predictions capture some aspects of deep mutational scanning experiments. BMC Bioinformatics 21, 107 (2020).","journal-title":"BMC Bioinformatics"},{"key":"791_CR4","first-page":"29287","volume":"34","author":"J Meier","year":"2021","unstructured":"Meier, J. et al. Language models enable zero-shot prediction of the effects of mutations on protein function. Adv. Neural Inf. Proc. Syst. 34, 29287\u201329303 (2021).","journal-title":"Adv. Neural Inf. Proc. Syst."},{"key":"791_CR5","doi-asserted-by":"crossref","unstructured":"Marquet, C. et al. Embeddings from protein language models predict conservation and variant effects. Hum. Genet. 141, 1629\u20131647 (2021).","DOI":"10.1007\/s00439-021-02411-y"},{"key":"791_CR6","unstructured":"Notin, P. et al. Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval. In Proc. International Conference on Machine Learning, 16990\u201317017 (PMLR, 2022)."},{"key":"791_CR7","doi-asserted-by":"publisher","first-page":"1160","DOI":"10.1038\/s41598-020-80786-0","volume":"11","author":"M Littmann","year":"2021","unstructured":"Littmann, M., Heinzinger, M., Dallago, C., Olenyi, T. & Rost, B. Embeddings from deep learning transfer GO annotations beyond homology. Sci. Rep. 11, 1160 (2021).","journal-title":"Sci. Rep."},{"key":"791_CR8","doi-asserted-by":"publisher","first-page":"227","DOI":"10.1038\/s42256-022-00457-9","volume":"4","author":"S Unsal","year":"2022","unstructured":"Unsal, S. et al. Learning functional properties of proteins with language models. Nat. Mach. Intell. 4, 227\u2013245 (2022).","journal-title":"Nat. Mach. Intell."},{"key":"791_CR9","doi-asserted-by":"publisher","first-page":"W228","DOI":"10.1093\/nar\/gkac278","volume":"50","author":"V Thumuluri","year":"2022","unstructured":"Thumuluri, V., Almagro Armenteros, J. J., Johansen, A. R., Nielsen, H. & Winther, O. Deeploc 2.0: multi-label subcellular localization prediction using protein language models. Nucleic Acids Res. 50, W228\u2013W234 (2022).","journal-title":"Nucleic Acids Res."},{"key":"791_CR10","doi-asserted-by":"publisher","first-page":"941","DOI":"10.1093\/bioinformatics\/btab801","volume":"38","author":"V Thumuluri","year":"2022","unstructured":"Thumuluri, V. et al. NetSolP: predicting protein solubility in Escherichia coli using language models. Bioinformatics 38, 941\u2013946 (2022).","journal-title":"Bioinformatics"},{"key":"791_CR11","doi-asserted-by":"publisher","first-page":"23916","DOI":"10.1038\/s41598-021-03431-4","volume":"11","author":"M Littmann","year":"2021","unstructured":"Littmann, M., Heinzinger, M., Dallago, C., Weissenow, K. & Rost, B. Protein embeddings and deep learning predict binding residues for various ligand classes. Sci. Rep. 11, 23916 (2021).","journal-title":"Sci. Rep."},{"key":"791_CR12","doi-asserted-by":"crossref","unstructured":"Teufel, F. et al. Signalp 6.0 predicts all five types of signal peptides using protein language models. Nat. Biotechnol. 40, 1023\u20131025 (2022).","DOI":"10.1038\/s41587-021-01156-3"},{"key":"791_CR13","doi-asserted-by":"crossref","unstructured":"Indriani, F., Mahmudah, K. R., Purnama, B. & Satou, K. ProtTrans-glutar: incorporating features from pre-trained transformer-based models for predicting glutarylation sites. Front. Genet. 13, 885929 (2022).","DOI":"10.3389\/fgene.2022.885929"},{"key":"791_CR14","doi-asserted-by":"crossref","unstructured":"Ilzhoefer, D., Heinzinger, M. & Rost, B. Seth predicts nuances of residue disorder from protein embeddings. Frontiers in Bioinformatics 2: 1019597 (2022)","DOI":"10.3389\/fbinf.2022.1019597"},{"key":"791_CR15","doi-asserted-by":"publisher","unstructured":"H\u00f8ie, M. H. et al. Netsurfp-3.0: accurate and fast prediction of protein structural features by protein language models and deep learning. Nucleic Acids Res. 50, W510\u2013W515, (2022). https:\/\/doi.org\/10.1093\/nar\/gkac439","DOI":"10.1093\/nar\/gkac439"},{"key":"791_CR16","doi-asserted-by":"publisher","first-page":"326","DOI":"10.1186\/s12859-022-04873-x","volume":"23","author":"M Bernhofer","year":"2022","unstructured":"Bernhofer, M. & Rost, B. TMbed: transmembrane proteins predicted through language model embeddings. BMC Bioinformatics 23, 326 (2022).","journal-title":"BMC Bioinformatics"},{"key":"791_CR17","doi-asserted-by":"crossref","unstructured":"Chowdhury, R., Bouatta, N., Biswas, S. et al. Single-sequence protein structure prediction using a language model and deep learning. Nat Biotechnol 40, 1617\u20131623 (2022).","DOI":"10.1038\/s41587-022-01432-w"},{"key":"791_CR18","doi-asserted-by":"publisher","unstructured":"Wu, R. et al. High-resolution de novo structure prediction from primary sequence. Preprint at bioRxiv (2022). https:\/\/doi.org\/10.1101\/2022.07.21.500999","DOI":"10.1101\/2022.07.21.500999"},{"key":"791_CR19","doi-asserted-by":"crossref","unstructured":"Lin, Zeming, et al. \"Evolutionary-scale prediction of atomic-level protein structure with a language model.\" Science 379.6637: 1123-1130. (2023)","DOI":"10.1126\/science.ade2574"},{"key":"791_CR20","doi-asserted-by":"publisher","first-page":"155a","DOI":"10.1016\/j.bpj.2021.11.1942","volume":"121","author":"JA Ruffolo","year":"2022","unstructured":"Ruffolo, J. A. & Gray, J. J. Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies. Biophysical J. 121, 155a\u2013156a (2022).","journal-title":"Biophysical J."},{"key":"791_CR21","doi-asserted-by":"publisher","unstructured":"Wei\u00dfenow, K., Heinzinger, M. & Rost, B. Protein language-model embeddings for fast, accurate, and alignment-free protein structure prediction. Structure 30(8), P1169-1177 (2022). https:\/\/doi.org\/10.1016\/j.str.2022.05.001","DOI":"10.1016\/j.str.2022.05.001"},{"key":"791_CR22","unstructured":"Kaplan, J. et al. Scaling laws for neural language models. Preprint at arXiv arXiv:2001.08361 (2020). https:\/\/arxiv.org\/abs\/2001.08361"},{"key":"791_CR23","doi-asserted-by":"crossref","unstructured":"Rao, R. et al. Evaluating protein transfer learning with TAPE. Adv. Neural Inf. Processing Syst. 32, 9689\u20139701 (2019).","DOI":"10.1101\/676825"},{"key":"791_CR24","doi-asserted-by":"publisher","first-page":"e2016239118","DOI":"10.1073\/pnas.2016239118","volume":"118","author":"A Rives","year":"2021","unstructured":"Rives, A. et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc. Natl Acad. Sci. USA 118, e2016239118 (2021).","journal-title":"Proc. Natl Acad. Sci. USA"},{"key":"791_CR25","unstructured":"Nijkamp, E., Ruffolo, J., Weinstein, E. N., Naik, N. & Madani, A. Progen2: exploring the boundaries of protein language models. Preprint at arXiv arXiv:2206.13517 (2022). https:\/\/arxiv.org\/abs\/2206.13517"},{"key":"791_CR26","doi-asserted-by":"crossref","unstructured":"Elnaggar, A. et al. ProtTrans: towards cracking the language of lifes code through self-supervised deep learning and high performance computing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10), 7112-7127 (2021)","DOI":"10.1109\/TPAMI.2021.3095381"},{"key":"791_CR27","doi-asserted-by":"publisher","first-page":"6719","DOI":"10.1093\/nar\/gkq495","volume":"38","author":"R Saunders","year":"2010","unstructured":"Saunders, R. & Deane, C. M. Synonymous codon usage influences the local protein structure observed. Nucleic Acids Res. 38, 6719\u20136728 (2010).","journal-title":"Nucleic Acids Res."},{"key":"791_CR28","doi-asserted-by":"publisher","first-page":"2815","DOI":"10.1038\/s41467-022-30390-9","volume":"13","author":"AA Rosenberg","year":"2022","unstructured":"Rosenberg, A. A., Marx, A. & Bronstein, A. M. Codon-specific Ramachandran plots show amino acid backbone conformation depends on identity of the translated codon. Nat. Commun. 13, 2815 (2022).","journal-title":"Nat. Commun."},{"key":"791_CR29","doi-asserted-by":"publisher","unstructured":"Lin, B. C., Kaissarian, N. M. & Kimchi-Sarfaty, C. Implementing computational methods in tandem with synonymous gene recoding for therapeutic development. Trends Pharmacol. Sci. 44(2), P73-84 (2022). https:\/\/doi.org\/10.1016\/j.tips.2022.09.008","DOI":"10.1016\/j.tips.2022.09.008"},{"key":"791_CR30","doi-asserted-by":"crossref","unstructured":"Shen, X., Song, S., Li, C. & Zhang, J. Synonymous mutations in representative yeast genes are mostly strongly non-neutral. Nature 606, 725\u2013731 (2022).","DOI":"10.1038\/s41586-022-04823-w"},{"key":"791_CR31","doi-asserted-by":"publisher","unstructured":"Kruglyak, L. et al. No evidence that synonymous mutations in yeast genes are mostly deleterious. Preprint at bioRxiv (2022). https:\/\/doi.org\/10.1101\/2022.07.14.500130","DOI":"10.1101\/2022.07.14.500130"},{"key":"791_CR32","doi-asserted-by":"crossref","unstructured":"Dhindsa, Ryan S., et al. \"A minimal role for synonymous variation in human disease.\" The American Journal of Human Genetics 109.12: 2105-2109 (2022).","DOI":"10.1016\/j.ajhg.2022.10.016"},{"key":"791_CR33","doi-asserted-by":"publisher","first-page":"17892","DOI":"10.1021\/ja510082j","volume":"136","author":"DA Nissley","year":"2014","unstructured":"Nissley, D. A. & O\u2019Brien, E. P. Timing is everything: unifying codon translation rates and nascent proteome behavior. J. Am. Chem. Soc. 136, 17892\u201317898 (2014).","journal-title":"J. Am. Chem. Soc."},{"key":"791_CR34","doi-asserted-by":"publisher","first-page":"858","DOI":"10.1021\/ja411302m","volume":"136","author":"IM Sander","year":"2014","unstructured":"Sander, I. M., Chaney, J. L. & Clark, P. L. Expanding Anfinsen\u2019s principle: contributions of synonymous codon selection to rational protein design. J. Am. Chem. Soc. 136, 858\u2013861 (2014).","journal-title":"J. Am. Chem. Soc."},{"key":"791_CR35","doi-asserted-by":"publisher","first-page":"143","DOI":"10.1146\/annurev-biophys-060414-034333","volume":"44","author":"JL Chaney","year":"2015","unstructured":"Chaney, J. L. & Clark, P. L. Roles for synonymous codon usage in protein biogenesis. Ann. Rev. Biophys. 44, 143\u2013166 (2015).","journal-title":"Ann. Rev. Biophys."},{"key":"791_CR36","doi-asserted-by":"publisher","first-page":"375","DOI":"10.1146\/annurev-biochem-071320-112701","volume":"90","author":"Y Liu","year":"2021","unstructured":"Liu, Y., Yang, Q. & Zhao, F. Synonymous but not silent: the codon usage code for gene expression and protein folding. Ann. Rev. Biochem. 90, 375 (2021).","journal-title":"Ann. Rev. Biochem."},{"key":"791_CR37","doi-asserted-by":"crossref","unstructured":"Jiang, Yang, et al. \"How synonymous mutations alter enzyme structure and function over long timescales.\" Nature Chemistry 15.3: 308-318 (2023).","DOI":"10.1038\/s41557-022-01091-z"},{"key":"791_CR38","doi-asserted-by":"publisher","first-page":"3081","DOI":"10.1038\/s41467-022-30548-5","volume":"13","author":"DA Nissley","year":"2022","unstructured":"Nissley, D. A. et al. Universal protein misfolding intermediates can bypass the proteostasis network and remain soluble and less functional. Nat. Commun. 13, 3081 (2022).","journal-title":"Nat. Commun."},{"key":"791_CR39","doi-asserted-by":"publisher","first-page":"D106","DOI":"10.1093\/nar\/gkab1051","volume":"50","author":"C Cummins","year":"2022","unstructured":"Cummins, C. et al. The European Nucleotide Archive in 2021. Nucleic Acids Res. 50, D106\u2013D110 (2022).","journal-title":"Nucleic Acids Res."},{"key":"791_CR40","doi-asserted-by":"publisher","first-page":"1181","DOI":"10.1093\/oxfordjournals.molbev.a004176","volume":"19","author":"JA Birdsell","year":"2002","unstructured":"Birdsell, J. A. Integrating genomics, bioinformatics, and classical genetics to study the effects of recombination on genome evolution. Mol. Biol. Evol. 19, 1181\u20131197 (2002).","journal-title":"Mol. Biol. Evol."},{"key":"791_CR41","doi-asserted-by":"publisher","first-page":"292","DOI":"10.1093\/nar\/28.1.292","volume":"28","author":"Y Nakamura","year":"2000","unstructured":"Nakamura, Y., Gojobori, T. & Ikemura, T. Codon usage tabulated from international DNA sequence databases: status for the year 2000. Nucleic Acids Res. 28, 292\u2013292 (2000).","journal-title":"Nucleic Acids Res."},{"key":"791_CR42","doi-asserted-by":"publisher","first-page":"msac157","DOI":"10.1093\/molbev\/msac157","volume":"39","author":"K Subramanian","year":"2022","unstructured":"Subramanian, K., Payne, B., Feyertag, F. & Alvarez-Ponce, D. The codon statistics database: a database of codon usage bias. Mol. Biology Evol. 39, msac157 (2022).","journal-title":"Mol. Biology Evol."},{"key":"791_CR43","doi-asserted-by":"publisher","unstructured":"Dallago, C. et al. FLIP: Benchmark tasks in fitness landscape inference for proteins. Preprint at bioRxiv (2021). https:\/\/doi.org\/10.1101\/2021.11.09.467890","DOI":"10.1101\/2021.11.09.467890"},{"key":"791_CR44","unstructured":"Nelson, D. L., Lehninger, A. L. & Cox, M. M. Lehninger Principles of Biochemistry (Macmillan, 2008)."},{"key":"791_CR45","doi-asserted-by":"publisher","first-page":"1281","DOI":"10.1093\/nar\/15.3.1281","volume":"15","author":"PM Sharp","year":"1987","unstructured":"Sharp, P. M. & Li, W.-H. The codon adaptation index\u2014a measure of directional synonymous codon usage bias, and its potential applications. Nucleic Acids Res. 15, 1281\u20131295 (1987).","journal-title":"Nucleic Acids Res."},{"key":"791_CR46","doi-asserted-by":"publisher","first-page":"492","DOI":"10.1074\/mcp.O111.014704","volume":"11","author":"M Wang","year":"2012","unstructured":"Wang, M. et al. PAXdb, a database of protein abundance averages across all three domains of life. Mol. Cell. Proteom. 11, 492\u2013500 (2012).","journal-title":"Mol. Cell. Proteom."},{"key":"791_CR47","doi-asserted-by":"publisher","first-page":"1881","DOI":"10.1093\/bioinformatics\/btab881","volume":"38","author":"C Outeiral","year":"2022","unstructured":"Outeiral, C., Nissley, D. A. & Deane, C. M. Current structure predictors are not learning the physics of protein folding. Bioinformatics 38, 1881\u20131887 (2022).","journal-title":"Bioinformatics"},{"key":"791_CR48","doi-asserted-by":"crossref","unstructured":"Sun, C., Shrivastava, A., Singh, S. & Gupta, A. Revisiting unreasonable effectiveness of data in deep learning era. In Proc. IEEE International Conference on Computer Vision, 843\u2013852 (IEEE, 2017).","DOI":"10.1109\/ICCV.2017.97"},{"key":"791_CR49","unstructured":"Hoffmann, J. et al. Training compute-optimal large language models. Preprint arXiv arXiv:2203.15556 (2022). https:\/\/arxiv.org\/abs\/2203.15556"},{"key":"791_CR50","doi-asserted-by":"publisher","first-page":"570","DOI":"10.1162\/tacl_a_00385","volume":"9","author":"LA Hendricks","year":"2021","unstructured":"Hendricks, L. A., Mellor, J., Schneider, R., Alayrac, J.-B. & Nematzadeh, A. Decoupling the role of data, attention, and losses in multimodal transformers. Trans. Assoc. Comput. Linguist. 9, 570\u2013585 (2021).","journal-title":"Trans. Assoc. Comput. Linguist."},{"key":"791_CR51","unstructured":"Klarner, L., Reutlinger, M., Schindler, T., Deane, C. & Morris, G. Bias in the benchmark: systematic experimental errors in bioactivity databases confound multi-task and meta-learning algorithms. In Proc. ICML 2022 2nd AI for Science Workshop (2022). https:\/\/openreview.net\/forum?id=Gc5oq8sr6A3"},{"key":"791_CR52","doi-asserted-by":"publisher","first-page":"3150","DOI":"10.1093\/bioinformatics\/bts565","volume":"28","author":"L Fu","year":"2012","unstructured":"Fu, L., Niu, B., Zhu, Z., Wu, S. & Li, W. CD-HIT: accelerated for clustering the next-generation sequencing data. Bioinformatics 28, 3150\u20133152 (2012).","journal-title":"Bioinformatics"},{"key":"791_CR53","doi-asserted-by":"publisher","first-page":"1063","DOI":"10.1093\/bib\/bbx117","volume":"20","author":"MY Galperin","year":"2019","unstructured":"Galperin, M. Y., Kristensen, D. M., Makarova, K. S., Wolf, Y. I. & Koonin, E. V. Microbial genome analysis: the COG approach. Brief. Bioinformatics 20, 1063\u20131070 (2019).","journal-title":"Brief. Bioinformatics"},{"key":"791_CR54","doi-asserted-by":"publisher","unstructured":"Breuza, L. et al. The UniProtkb guide to the human proteome. Database, bav120 (2016). https:\/\/doi.org\/10.1093\/database\/bav120","DOI":"10.1093\/database\/bav120"},{"key":"791_CR55","doi-asserted-by":"publisher","first-page":"495","DOI":"10.1038\/s41592-020-0801-4","volume":"17","author":"A Jarzab","year":"2020","unstructured":"Jarzab, A. et al. Meltome atlas\u2014thermal proteome stability across the tree of life. Nat. Methods 17, 495\u2013503 (2020).","journal-title":"Nat. Methods"},{"key":"791_CR56","doi-asserted-by":"publisher","first-page":"1155","DOI":"10.1038\/s41467-019-09107-y","volume":"10","author":"S Sridharan","year":"2019","unstructured":"Sridharan, S. et al. Proteome-wide solubility and thermal stability profiling reveals distinct regulatory roles for ATP. Nat. Commun. 10, 1155 (2019).","journal-title":"Nat. Commun."},{"key":"791_CR57","unstructured":"Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. Preprint arXiv arXiv:1810.04805 (2018). https:\/\/arxiv.org\/abs\/1810.04805"},{"key":"791_CR58","unstructured":"Su, J., Lu, Y., Pan, S., Wen, B. & Liu, Y. Roformer: enhanced transformer with rotary position embedding. Preprint at arXiv arXiv:2104.09864 (2021). https:\/\/arxiv.org\/abs\/2104.09864"},{"key":"791_CR59","unstructured":"Liu, Y. et al. Roberta: a robustly optimized BERT pretraining approach. Preprint at arXiv arXiv:1907.11692 (2019). https:\/\/arxiv.org\/abs\/1907.11692"},{"key":"791_CR60","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825\u20132830 (2011).","journal-title":"J. Mach. Learn. Res."},{"key":"791_CR61","unstructured":"Bishop, C. M. Pattern Recognition and Machine Learning (Springer, 2006)."},{"key":"791_CR62","unstructured":"Outeiral, C. Codon adaptation language model (CaLM) (CodeOcean, 2023)."}],"container-title":["Nature Machine Intelligence"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.nature.com\/articles\/s42256-024-00791-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-024-00791-0","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.nature.com\/articles\/s42256-024-00791-0.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,10]],"date-time":"2024-04-10T18:01:46Z","timestamp":1712772106000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.nature.com\/articles\/s42256-024-00791-0"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,2,23]]},"references-count":62,"journal-issue":{"issue":"2","published-online":{"date-parts":[[2024,2]]}},"alternative-id":["791"],"URL":"https:\/\/doi.org\/10.1038\/s42256-024-00791-0","relation":{},"ISSN":["2522-5839"],"issn-type":[{"value":"2522-5839","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,2,23]]},"assertion":[{"value":"15 December 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"9 January 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"23 February 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"The authors declare no competing interests.","order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}]}}