{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,5,30]],"date-time":"2026-05-30T00:57:07Z","timestamp":1780102627596,"version":"3.54.0"},"reference-count":14,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2024,9,13]],"date-time":"2024-09-13T00:00:00Z","timestamp":1726185600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2024,9,13]],"date-time":"2024-09-13T00:00:00Z","timestamp":1726185600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"funder":[{"DOI":"10.13039\/501100002957","name":"Technische Universit\u00e4t Dresden","doi-asserted-by":"crossref","id":[{"id":"10.13039\/501100002957","id-type":"DOI","asserted-by":"crossref"}]}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Bioinformatics"],"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:p>Transformer-based large language models (LLMs) are very suited for biological sequence data, because of analogies to natural language. Complex relationships can be learned, because a concept of \"words\" can be generated through tokenization. Training the models with masked token prediction, they learn both token sequence identity and larger sequence context. We developed methodology to interrogate model learning, which is both relevant for the interpretability of the model and to evaluate its potential for specific tasks. We used DNABERT, a DNA language model trained on the human genome with overlapping k-mers as tokens. To gain insight into the model\u2032s learning, we interrogated how the model performs predictions, extracted token embeddings, and defined a fine-tuning benchmarking task to predict the next tokens of different sizes without overlaps. This task evaluates foundation models without interrogating specific genome biology, it does not depend on tokenization strategies, vocabulary size, the dictionary, or the number of training parameters. Lastly, there is no leakage of information from token identity into the prediction task, which makes it particularly useful to evaluate the learning of sequence context. We discovered that the model with overlapping k-mers struggles to learn larger sequence context. Instead, the learned embeddings largely represent token sequence. Still, good performance is achieved for genome-biology-inspired fine-tuning tasks. Models with overlapping tokens may be used for tasks where a larger sequence context is of less relevance, but the token sequence directly represents the desired learning features. This emphasizes the need to interrogate knowledge representation in biological LLMs.<\/jats:p>","DOI":"10.1186\/s12859-024-05869-5","type":"journal-article","created":{"date-parts":[[2024,9,13]],"date-time":"2024-09-13T07:03:40Z","timestamp":1726211020000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":13,"title":["Distinguishing word identity and sequence context in DNA language models"],"prefix":"10.1186","volume":"25","author":[{"given":"Melissa","family":"Sanabria","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Jonas","family":"Hirsch","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"given":"Anna R.","family":"Poetsch","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"297","published-online":{"date-parts":[[2024,9,13]]},"reference":[{"key":"5869_CR1","unstructured":"Vaswani, A. et al. Attention is all you need. Advances in neural information processing systems 30, (2017). https:\/\/proceedings.neurips.cc\/paper\/7181-attention-is-all"},{"key":"5869_CR2","doi-asserted-by":"publisher","first-page":"389","DOI":"10.1038\/s41576-019-0122-6","volume":"20","author":"G Eraslan","year":"2019","unstructured":"Eraslan G, Avsec Z, Gagneur J, Theis FJ. Deep learning: new computational modelling techniques for genomtics. Nat Rev Genet. 2019;20:389\u2013403.","journal-title":"Nat Rev Genet"},{"key":"5869_CR3","doi-asserted-by":"publisher","first-page":"1196","DOI":"10.1038\/s41592-021-01252-x","volume":"18","author":"Z Avsec","year":"2021","unstructured":"Avsec Z, et al. Effective gene expression prediction from sequence by integrating long-range interactions. Nat Methods. 2021;18:1196\u2013203.","journal-title":"Nat Methods"},{"key":"5869_CR4","first-page":"1877","volume":"33","author":"T Brown","year":"2020","unstructured":"Brown T, et al. Language models are few-shot learners. Adv Neural Inf Process Syst. 2020;33:1877\u2013901.","journal-title":"Adv Neural Inf Process Syst"},{"key":"5869_CR5","doi-asserted-by":"publisher","DOI":"10.1101\/2023.01.11.523679.abstract","author":"H Dalla-Torre","year":"2023","unstructured":"Dalla-Torre H, et al. The nucleotide transformer: building and evaluating robust foundation models for human genomics. bioRxiv. 2023. https:\/\/doi.org\/10.1101\/2023.01.11.523679.abstract.","journal-title":"bioRxiv"},{"key":"5869_CR6","doi-asserted-by":"publisher","first-page":"2112","DOI":"10.1093\/bioinformatics\/btab083","volume":"37","author":"Y Ji","year":"2021","unstructured":"Ji Y, Zhou Z, Liu H, Davuluri RV. DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome. Bioinformatics. 2021;37:2112\u201320.","journal-title":"Bioinformaics"},{"key":"5869_CR7","doi-asserted-by":"publisher","unstructured":"Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018). https:\/\/doi.org\/10.48550\/arXiv.1810.04805","DOI":"10.48550\/arXiv.1810.04805"},{"key":"5869_CR8","doi-asserted-by":"publisher","first-page":"1248","DOI":"10.1038\/nature08473","volume":"461","author":"R Rohs","year":"2009","unstructured":"Rohs R, et al. The role of DNA shape in protein-DNA recognition. Nature. 2009;461:1248\u201353.","journal-title":"Nature"},{"key":"5869_CR9","doi-asserted-by":"crossref","unstructured":"Ethayarajh, K. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. arXiv preprint arXiv:1909.00512 (2019)","DOI":"10.18653\/v1\/D19-1006"},{"key":"5869_CR10","unstructured":"Mikolov, T., Chen, K., Corrado, G. & Dean, J. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)"},{"key":"5869_CR11","doi-asserted-by":"crossref","unstructured":"McInnes, L., Healy, J. & Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018). https:\/\/arxiv.org\/pdf\/1802.03426.pdf?source=post_page","DOI":"10.21105\/joss.00861"},{"key":"5869_CR12","doi-asserted-by":"publisher","DOI":"10.1038\/s42256-024-00872-0","author":"M Sanabria","year":"2024","unstructured":"Sanabria M, Hirsch J, Joubert PM, Poetsch AR. DNA language model GROVER learns sequence context in the human genome. Nature Machine Intelligence. 2024. https:\/\/doi.org\/10.1038\/s42256-024-00872-0.","journal-title":"Nature Machine Intelligence"},{"key":"5869_CR13","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.8407874","author":"S Melissa","year":"2023","unstructured":"Melissa S, Jonas H, Poetsch AR. Distinguishing word identity and sequence context in DNA language models - the code to the paper. 2023. Zenodo. https:\/\/doi.org\/10.5281\/zenodo.8407874."},{"key":"5869_CR14","doi-asserted-by":"publisher","DOI":"10.5281\/zenodo.8407817","author":"S Melissa","year":"2023","unstructured":"Melissa S, Jonas H, Poetsch AR. Next-kmer-prediction fine-tuning to compare DNA language models, a tutorial. 2023. Zenodo. https:\/\/doi.org\/10.5281\/zenodo.8407817."}],"container-title":["BMC Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-024-05869-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12859-024-05869-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-024-05869-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2024,9,13]],"date-time":"2024-09-13T07:04:46Z","timestamp":1726211086000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/s12859-024-05869-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,13]]},"references-count":14,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2024,12]]}},"alternative-id":["5869"],"URL":"https:\/\/doi.org\/10.1186\/s12859-024-05869-5","relation":{"references":[{"id-type":"uri","id":"","asserted-by":"subject"},{"id-type":"uri","id":"","asserted-by":"subject"}],"has-preprint":[{"id-type":"doi","id":"10.1101\/2023.07.11.548593","asserted-by":"object"}]},"ISSN":["1471-2105"],"issn-type":[{"value":"1471-2105","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,13]]},"assertion":[{"value":"16 September 2023","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"12 July 2024","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"13 September 2024","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"Not applicable.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Ethics approval and consent to participate"}},{"value":"Not applicable.","order":3,"name":"Ethics","group":{"name":"EthicsHeading","label":"Consent for publication"}},{"value":"The authors declare no competing interests.","order":4,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"301"}}