{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,7,9]],"date-time":"2026-07-09T23:43:34Z","timestamp":1783640614622,"version":"3.55.0"},"reference-count":55,"publisher":"Oxford University Press (OUP)","issue":"4","license":[{"start":{"date-parts":[[2024,4,12]],"date-time":"2024-04-12T00:00:00Z","timestamp":1712880000000},"content-version":"vor","delay-in-days":14,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"name":"Azrieli Foundation Early Career Faculty Fellowship"},{"DOI":"10.13039\/501100004375","name":"Tel Aviv University","doi-asserted-by":"publisher","id":[{"id":"10.13039\/501100004375","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100003977","name":"Israel Science Foundation","doi-asserted-by":"publisher","award":["2818\/21"],"award-info":[{"award-number":["2818\/21"]}],"id":[{"id":"10.13039\/501100003977","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2024,3,29]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Motivation<\/jats:title>\n                    <jats:p>Deep-learning models are transforming biological research, including many bioinformatics and comparative genomics algorithms, such as sequence alignments, phylogenetic tree inference, and automatic classification of protein functions. Among these deep-learning algorithms, models for processing natural languages, developed in the natural language processing (NLP) community, were recently applied to biological sequences. However, biological sequences are different from natural languages, such as English, and French, in which segmentation of the text to separate words is relatively straightforward. Moreover, biological sequences are characterized by extremely long sentences, which hamper their processing by current machine-learning models, notably the transformer architecture. In NLP, one of the first processing steps is to transform the raw text to a list of tokens. Deep-learning applications to biological sequence data mostly segment proteins and DNA to single characters. In this work, we study the effect of alternative tokenization algorithms on eight different tasks in biology, from predicting the function of proteins and their stability, through nucleotide sequence alignment, to classifying proteins to specific families.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>We demonstrate that applying alternative tokenization algorithms can increase accuracy and at the same time, substantially reduce the input length compared to the trivial tokenizer in which each character is a token. Furthermore, applying these tokenization algorithms allows interpreting trained models, taking into account dependencies among positions. Finally, we trained these tokenizers on a large dataset of protein sequences containing more than 400 billion amino acids, which resulted in over a 3-fold decrease in the number of tokens. We then tested these tokenizers trained on large-scale data on the above specific tasks and showed that for some tasks it is highly beneficial to train database-specific tokenizers. Our study suggests that tokenizers are likely to be a critical component in future deep-network analysis of biological sequence data.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Availability and implementation<\/jats:title>\n                    <jats:p>Code, data, and trained tokenizers are available on https:\/\/github.com\/technion-cs-nlp\/BiologicalTokenizers.<\/jats:p>\n                  <\/jats:sec>","DOI":"10.1093\/bioinformatics\/btae196","type":"journal-article","created":{"date-parts":[[2024,4,11]],"date-time":"2024-04-11T11:44:44Z","timestamp":1712835884000},"source":"Crossref","is-referenced-by-count":44,"title":["Effect of tokenization on transformers for biological sequences"],"prefix":"10.1093","volume":"40","author":[{"given":"Edo","family":"Dotan","sequence":"first","affiliation":[{"name":"The Henry and Marilyn Taub Faculty of Computer Science, Technion \u2013 Israel Institute of Technology , Haifa 3200003, Israel"},{"name":"The Shmunis School of Biomedicine and Cancer Research, George S. Wise Faculty of Life Sciences, Tel Aviv University , Tel Aviv 69978, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Gal","family":"Jaschek","sequence":"additional","affiliation":[{"name":"Department of Genetics, Yale University School of Medicine , New Haven, CT 06510, United States"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9463-2575","authenticated-orcid":false,"given":"Tal","family":"Pupko","sequence":"additional","affiliation":[{"name":"The Shmunis School of Biomedicine and Cancer Research, George S. Wise Faculty of Life Sciences, Tel Aviv University , Tel Aviv 69978, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Yonatan","family":"Belinkov","sequence":"additional","affiliation":[{"name":"The Henry and Marilyn Taub Faculty of Computer Science, Technion \u2013 Israel Institute of Technology , Haifa 3200003, Israel"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"286","published-online":{"date-parts":[[2024,4,12]]},"reference":[{"key":"2024042711205920200_btae196-B1","doi-asserted-by":"crossref","first-page":"e0239381","DOI":"10.1371\/journal.pone.0239381","article-title":"Short K-Mer abundance profiles yield robust machine learning features and accurate classifiers for RNA viruses","volume":"15","author":"Alam","year":"2020","journal-title":"PLoS One"},{"key":"2024042711205920200_btae196-B2","doi-asserted-by":"crossref","first-page":"26","DOI":"10.1186\/s40246-022-00396-x","article-title":"A review of deep learning applications in human genomics using next-generation sequencing data","volume":"16","author":"Alharbi","year":"2022","journal-title":"Hum Genomics"},{"key":"2024042711205920200_btae196-B3","doi-asserted-by":"crossref","first-page":"403","DOI":"10.1016\/S0022-2836(05)80360-2","article-title":"Basic local alignment search tool","volume":"215","author":"Altschul","year":"1990","journal-title":"J Mol Biol"},{"key":"2024042711205920200_btae196-B4","doi-asserted-by":"crossref","first-page":"D376","DOI":"10.1093\/nar\/gkz1064","article-title":"The SCOP database in 2020: expanded classification of representative family and superfamily domains of known protein structures","volume":"48","author":"Andreeva","year":"2020","journal-title":"Nucleic Acids Res"},{"key":"2024042711205920200_btae196-B5","doi-asserted-by":"crossref","first-page":"2102","DOI":"10.1093\/bioinformatics\/btac020","article-title":"ProteinBERT: a universal deep-learning model of protein sequence and function.","volume":"38","author":"Brandes","year":"2022","journal-title":"Bioinformatics"},{"key":"2024042711205920200_btae196-B6","doi-asserted-by":"crossref","first-page":"27","DOI":"10.1016\/j.ejphar.2009.10.015","article-title":"Neuropeptides from concept to online database www.neuropeptides.nl","volume":"626","author":"Burbach","year":"2010","journal-title":"Eur J Pharmacol"},{"key":"2024042711205920200_btae196-B7","first-page":"4171","volume-title":"Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)","author":"Devlin","year":"2019"},{"key":"2024042711205920200_btae196-B8","author":"Dotan","year":"2023"},{"key":"2024042711205920200_btae196-B9","doi-asserted-by":"crossref","first-page":"733","DOI":"10.1162\/coli_a_00445","article-title":"Position information in transformers: an overview","volume":"48","author":"Dufter","year":"2022","journal-title":"Comput Linguist"},{"key":"2024042711205920200_btae196-B10","doi-asserted-by":"crossref","first-page":"389","DOI":"10.1038\/s41576-019-0122-6","article-title":"Deep learning: new computational modelling techniques for genomics","volume":"20","author":"Eraslan","year":"2019","journal-title":"Nat Rev Genet"},{"key":"2024042711205920200_btae196-B11","first-page":"23","volume-title":"C Users J","author":"Gage","year":"1994"},{"key":"2024042711205920200_btae196-B12","doi-asserted-by":"crossref","first-page":"1295","DOI":"10.1093\/bioinformatics\/btx780","article-title":"DeepSF: deep convolutional neural network for mapping protein sequences to folds","volume":"34","author":"Hou","year":"2018","journal-title":"Bioinformatics"},{"key":"2024042711205920200_btae196-B13","doi-asserted-by":"crossref","first-page":"2112","DOI":"10.1093\/bioinformatics\/btab083","article-title":"DNABERT: pre-trained bidirectional encoder representations from transformers model for DNA-language in genome","volume":"37","author":"Ji","year":"2021","journal-title":"Bioinformatics"},{"key":"2024042711205920200_btae196-B14","doi-asserted-by":"crossref","first-page":"583","DOI":"10.1038\/s41586-021-03819-2","article-title":"Highly accurate protein structure prediction with AlphaFold","volume":"596","author":"Jumper","year":"2021","journal-title":"Nature"},{"key":"2024042711205920200_btae196-B15","author":"Kokhlikyan","year":"2020"},{"key":"2024042711205920200_btae196-B16","doi-asserted-by":"crossref","first-page":"1466","DOI":"10.1016\/j.csbj.2020.06.017","article-title":"Deep learning models in genomics; are we there yet?","volume":"18","author":"Koumakis","year":"2020","journal-title":"Comput Struct Biotechnol J"},{"key":"2024042711205920200_btae196-B17","author":"Kudo","year":"2018"},{"key":"2024042711205920200_btae196-B18","doi-asserted-by":"crossref","first-page":"660","DOI":"10.1093\/bioinformatics\/btx624","article-title":"DeepGO: predicting protein functions from sequence and interactions using a deep ontology-aware classifier","volume":"34","author":"Kulmanov","year":"2018","journal-title":"Bioinformatics"},{"key":"2024042711205920200_btae196-B19","doi-asserted-by":"crossref","first-page":"436","DOI":"10.1038\/nature14539","article-title":"Deep learning","volume":"521","author":"LeCun","year":"2015","journal-title":"Nature"},{"key":"2024042711205920200_btae196-B20","first-page":"111","volume-title":"AI Open","author":"Lin","year":"2022"},{"key":"2024042711205920200_btae196-B21","doi-asserted-by":"crossref","first-page":"39","DOI":"10.1186\/s13062-016-0145-2","article-title":"Unity and disunity in evolutionary sciences: process-based analogies open common research avenues for biology and linguistics","volume":"11","author":"List","year":"2016","journal-title":"Biol Direct"},{"key":"2024042711205920200_btae196-B22","doi-asserted-by":"crossref","first-page":"5769","DOI":"10.1093\/molbev\/msab266","article-title":"A probabilistic model for indel evolution: differentiating insertions from deletions","volume":"38","author":"Loewenthal","year":"2021","journal-title":"Mol Biol Evol"},{"key":"2024042711205920200_btae196-B23","doi-asserted-by":"crossref","first-page":"D115","DOI":"10.1093\/nar\/gkr1044","article-title":"IMG: the integrated microbial genomes database and comparative analysis system","volume":"40","author":"Markowitz","year":"2012","journal-title":"Nucleic Acids Res"},{"key":"2024042711205920200_btae196-B24","doi-asserted-by":"crossref","first-page":"442","DOI":"10.1016\/0005-2795(75)90109-9","article-title":"Comparison of the predicted and observed secondary structure of T4 phage lysozyme","volume":"405","author":"Matthews","year":"1975","journal-title":"Biochim Biophys Acta"},{"key":"2024042711205920200_btae196-B25","doi-asserted-by":"crossref","first-page":"5731","DOI":"10.1038\/s41467-022-33397-4","article-title":"Deciphering microbial gene function using natural language processing","volume":"13","author":"Miller","year":"2022","journal-title":"Nat Commun"},{"key":"2024042711205920200_btae196-B26","doi-asserted-by":"crossref","first-page":"D412","DOI":"10.1093\/nar\/gkaa913","article-title":"Pfam: the protein families database in 2021","volume":"49","author":"Mistry","year":"2021","journal-title":"Nucleic Acids Res"},{"key":"2024042711205920200_btae196-B27","doi-asserted-by":"crossref","DOI":"10.1128\/microbiolspec.VMBF-0004-2015","article-title":"The structure and function of type III secretion systems","volume":"4","author":"Notti","year":"2016","journal-title":"Microbiol Spectr"},{"key":"2024042711205920200_btae196-B28","doi-asserted-by":"crossref","first-page":"44","DOI":"10.1126\/science.abj6987","article-title":"The complete sequence of a human genome","volume":"376","author":"Nurk","year":"2022","journal-title":"Science"},{"key":"2024042711205920200_btae196-B2900","doi-asserted-by":"crossref","first-page":"931","DOI":"10.1093\/bioinformatics\/btt725","article-title":"NeuroP ID: a predictor for identifying neuropeptide precursors from metazoan proteomes","volume":"30","author":"Ofer","year":"2014","journal-title":"Bioinformatics"},{"key":"2024042711205920200_btae196-B29","doi-asserted-by":"crossref","first-page":"e11456","DOI":"10.7717\/peerj.11456","article-title":"K-Mer-based machine learning method to classify LTR-retrotransposons in plant genomes","volume":"9","author":"Orozco-Arias","year":"2021","journal-title":"PeerJ"},{"key":"2024042711205920200_btae196-B30","first-page":"48","author":"Ott","year":"2019"},{"key":"2024042711205920200_btae196-B31","doi-asserted-by":"crossref","first-page":"1759","DOI":"10.1093\/molbev\/msq066","article-title":"An alignment confidence score capturing robustness to guide tree uncertainty","volume":"27","author":"Penn","year":"2010","journal-title":"Mol Biol Evol"},{"key":"2024042711205920200_btae196-B32","author":"Radford","year":"2018"},{"key":"2024042711205920200_btae196-B33","first-page":"9689","article-title":"Evaluating protein transfer learning with TAPE","volume":"32","author":"Rao","year":"2019","journal-title":"Adv Neural Inf Process Syst"},{"key":"2024042711205920200_btae196-B34","first-page":"8844","author":"Rao","year":"2021"},{"key":"2024042711205920200_btae196-B35","doi-asserted-by":"crossref","first-page":"686","DOI":"10.1128\/MMBR.00011-08","article-title":"Comparative genomics and molecular dynamics of DNA repeats in eukaryotes","volume":"72","author":"Richard","year":"2008","journal-title":"Microbiol Mol Biol Rev"},{"key":"2024042711205920200_btae196-B36","doi-asserted-by":"crossref","first-page":"168","DOI":"10.1126\/science.aan0693","article-title":"Global analysis of protein folding using massively parallel design, synthesis, and testing","volume":"357","author":"Rocklin","year":"2017","journal-title":"Science"},{"key":"2024042711205920200_btae196-B37","doi-asserted-by":"crossref","first-page":"e0000106","DOI":"10.1371\/journal.pdig.0000106","article-title":"Automated large-scale prediction of exudative AMD progression using Machine-Read OCT biomarkers","volume":"2","author":"Rudas","year":"2023","journal-title":"PLOS Digit Health"},{"key":"2024042711205920200_btae196-B38","doi-asserted-by":"crossref","first-page":"397","DOI":"10.1038\/nature17995","article-title":"Local fitness landscape of the green fluorescent protein","volume":"533","author":"Sarkisyan","year":"2016","journal-title":"Nature"},{"key":"2024042711205920200_btae196-B39","first-page":"5149","author":"Schuster","year":"2012"},{"key":"2024042711205920200_btae196-B40","first-page":"1715","author":"Sennrich","year":"2016"},{"key":"2024042711205920200_btae196-B41","doi-asserted-by":"crossref","first-page":"1026","DOI":"10.1038\/nbt.3988","article-title":"MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets","volume":"35","author":"Steinegger","year":"2017","journal-title":"Nat Biotechnol"},{"key":"2024042711205920200_btae196-B42","doi-asserted-by":"crossref","first-page":"2542","DOI":"10.1038\/s41467-018-04964-5","article-title":"Clustering huge protein sequence sets in linear time","volume":"9","author":"Steinegger","year":"2018","journal-title":"Nat Commun"},{"key":"2024042711205920200_btae196-B43","first-page":"3319","author":"Sundararajan","year":"2017"},{"key":"2024042711205920200_btae196-B44","doi-asserted-by":"crossref","first-page":"bbaa177","DOI":"10.1093\/bib\/bbaa177","article-title":"Interpretation of deep learning in genomics and epigenomics","volume":"22","author":"Talukder","year":"2021","journal-title":"Brief Bioinform"},{"key":"2024042711205920200_btae196-B45","first-page":"365","article-title":"Amino acid K-Mer feature extraction for quantitative antimicrobial resistance (AMR) prediction by machine learning and model interpretation for biological insights","volume":"9","author":"ValizadehAslani","year":"2020","journal-title":"Biology (Basel)"},{"key":"2024042711205920200_btae196-B46","doi-asserted-by":"crossref","first-page":"550","DOI":"10.1038\/514550a","article-title":"The top 100 papers","volume":"514","author":"Van Noorden","year":"2014","journal-title":"Nature"},{"key":"2024042711205920200_btae196-B47","author":"Vaswani","year":"2017"},{"key":"2024042711205920200_btae196-B48","doi-asserted-by":"crossref","first-page":"7068349","DOI":"10.1155\/2018\/7068349","article-title":"Deep learning for computer vision: a brief review","volume":"2018","author":"Voulodimos","year":"2018","journal-title":"Comput Intell Neurosci"},{"key":"2024042711205920200_btae196-B49","doi-asserted-by":"crossref","first-page":"1024405","DOI":"10.3389\/fpls.2022.1024405","article-title":"Natural language processing approach to model the secretion signal of type III effectors","volume":"13","author":"Wagner","year":"2022","journal-title":"Front Plant Sci"},{"key":"2024042711205920200_btae196-B50","doi-asserted-by":"crossref","first-page":"80","DOI":"10.2307\/3001968","article-title":"Individual comparisons by ranking methods","volume":"1","author":"Wilcoxon","year":"1945","journal-title":"Biometrics Bull"},{"key":"2024042711205920200_btae196-B51","first-page":"38","author":"Wolf","year":"2020"},{"key":"2024042711205920200_btae196-B52","doi-asserted-by":"crossref","first-page":"257","DOI":"10.1186\/s13059-019-1891-0","article-title":"Improved metagenomic analysis with kraken 2","volume":"20","author":"Wood","year":"2019","journal-title":"Genome Biol"},{"key":"2024042711205920200_btae196-B53","doi-asserted-by":"crossref","first-page":"3636","DOI":"10.1073\/pnas.1814684116","article-title":"Grammar of protein domain architectures","volume":"116","author":"Yu","year":"2019","journal-title":"Proc Natl Acad Sci USA"},{"key":"2024042711205920200_btae196-B54","author":"Zhou","year":"2023"}],"container-title":["Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/academic.oup.com\/bioinformatics\/advance-article-pdf\/doi\/10.1093\/bioinformatics\/btae196\/57226869\/btae196.pdf","content-type":"application\/pdf","content-version":"am","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/40\/4\/btae196\/57345801\/btae196.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"syndication"},{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article-pdf\/40\/4\/btae196\/57345801\/btae196.pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,4,27]],"date-time":"2024-04-27T07:21:43Z","timestamp":1714202503000},"score":1,"resource":{"primary":{"URL":"https:\/\/academic.oup.com\/bioinformatics\/article\/doi\/10.1093\/bioinformatics\/btae196\/7645044"}},"subtitle":[],"editor":[{"given":"Alfonso","family":"Valencia","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2024,3,29]]},"references-count":55,"journal-issue":{"issue":"4","published-print":{"date-parts":[[2024,3,29]]}},"URL":"https:\/\/doi.org\/10.1093\/bioinformatics\/btae196","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2023.08.15.553415","asserted-by":"object"}]},"ISSN":["1367-4803","1367-4811"],"issn-type":[{"value":"1367-4803","type":"print"},{"value":"1367-4811","type":"electronic"}],"subject":[],"published-other":{"date-parts":[[2024,4,1]]},"published":{"date-parts":[[2024,3,29]]},"article-number":"btae196"}}