{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,3]],"date-time":"2026-06-03T12:24:57Z","timestamp":1780489497964,"version":"3.54.1"},"update-to":[{"DOI":"10.1371\/journal.pcbi.1012755","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2025,1,23]],"date-time":"2025-01-23T00:00:00Z","timestamp":1737590400000}}],"reference-count":50,"publisher":"Public Library of Science (PLoS)","issue":"1","license":[{"start":{"date-parts":[[2025,1,10]],"date-time":"2025-01-10T00:00:00Z","timestamp":1736467200000},"content-version":"vor","delay-in-days":0,"URL":"http:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/100006132","name":"Office of Science","doi-asserted-by":"publisher","award":["DE-SC-0024459"],"award-info":[{"award-number":["DE-SC-0024459"]}],"id":[{"id":"10.13039\/100006132","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":["www.ploscompbiol.org"],"crossmark-restriction":false},"short-container-title":["PLoS Comput Biol"],"abstract":"<jats:p>Complex deep learning models trained on very large datasets have become key enabling tools for current research in natural language processing and computer vision. By providing pre-trained models that can be fine-tuned for specific applications, they enable researchers to create accurate models with minimal effort and computational resources. Large scale genomics deep learning models come in two flavors: the first are large language models of DNA sequences trained in a self-supervised fashion, similar to the corresponding natural language models; the second are supervised learning models that leverage large scale genomics datasets from ENCODE and other sources. We argue that these models are the equivalent of foundation models in natural language processing in their utility, as they encode within them chromatin state in its different aspects, providing useful representations that allow quick deployment of accurate models of gene regulation. We demonstrate this premise by leveraging the recently created Sei model to develop simple, interpretable models of intron retention, and demonstrate their advantage over models based on the DNA language model DNABERT-2. Our work also demonstrates the impact of chromatin state on the regulation of intron retention. Using representations learned by Sei, our model is able to discover the involvement of transcription factors and chromatin marks in regulating intron retention, providing better accuracy than a recently published custom model developed for this purpose.<\/jats:p>","DOI":"10.1371\/journal.pcbi.1012755","type":"journal-article","created":{"date-parts":[[2025,1,10]],"date-time":"2025-01-10T15:20:36Z","timestamp":1736522436000},"page":"e1012755","update-policy":"https:\/\/doi.org\/10.1371\/journal.pcbi.corrections_policy","source":"Crossref","is-referenced-by-count":8,"title":["The role of chromatin state in intron retention: A case study in leveraging large scale deep learning models"],"prefix":"10.1371","volume":"21","author":[{"given":"Ahmed","family":"Daoud","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8269-6942","authenticated-orcid":true,"given":"Asa","family":"Ben-Hur","sequence":"additional","affiliation":[],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"340","published-online":{"date-parts":[[2025,1,10]]},"reference":[{"issue":"7","key":"pcbi.1012755.ref001","doi-asserted-by":"crossref","first-page":"389","DOI":"10.1038\/s41576-019-0122-6","article-title":"Deep learning: new computational modelling techniques for genomics","volume":"20","author":"G Eraslan","year":"2019","journal-title":"Nature Reviews Genetics"},{"key":"pcbi.1012755.ref002","first-page":"2","article-title":"BERT: Pre-training of deep bidirectional transformers for language understanding","volume":"1","author":"J Devlin","year":"2019","journal-title":"Proceedings of naacL-HLT"},{"key":"pcbi.1012755.ref003","first-page":"1877","article-title":"Language models are few-shot learners","volume":"33","author":"T Brown","year":"2020","journal-title":"Advances in neural information processing systems"},{"key":"pcbi.1012755.ref004","unstructured":"Bommasani R, Hudson DA, Adeli E, Altman R, Arora S, von Arx S, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:210807258. 2021;."},{"issue":"10","key":"pcbi.1012755.ref005","doi-asserted-by":"crossref","first-page":"7112","DOI":"10.1109\/TPAMI.2021.3095381","article-title":"ProtTrans: Toward understanding the language of life through self-supervised learning","volume":"44","author":"A Elnaggar","year":"2021","journal-title":"IEEE transactions on pattern analysis and machine intelligence"},{"issue":"15","key":"pcbi.1012755.ref006","doi-asserted-by":"crossref","first-page":"e2016239118","DOI":"10.1073\/pnas.2016239118","article-title":"Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences","volume":"118","author":"A Rives","year":"2021","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"15","key":"pcbi.1012755.ref007","doi-asserted-by":"crossref","first-page":"2112","DOI":"10.1093\/bioinformatics\/btab083","article-title":"DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome","volume":"37","author":"Y Ji","year":"2021","journal-title":"Bioinformatics"},{"issue":"10","key":"pcbi.1012755.ref008","doi-asserted-by":"crossref","first-page":"931","DOI":"10.1038\/nmeth.3547","article-title":"Predicting effects of noncoding variants with deep learning\u2013based sequence model","volume":"12","author":"J Zhou","year":"2015","journal-title":"Nature methods"},{"issue":"7","key":"pcbi.1012755.ref009","doi-asserted-by":"crossref","first-page":"990","DOI":"10.1101\/gr.200535.115","article-title":"Basset: learning the regulatory code of the accessible genome with deep convolutional neural networks","volume":"26","author":"DR Kelley","year":"2016","journal-title":"Genome research"},{"issue":"5","key":"pcbi.1012755.ref010","doi-asserted-by":"crossref","first-page":"739","DOI":"10.1101\/gr.227819.117","article-title":"Sequential regulatory activity prediction across chromosomes with convolutional neural networks","volume":"28","author":"DR Kelley","year":"2018","journal-title":"Genome Research"},{"issue":"10","key":"pcbi.1012755.ref011","doi-asserted-by":"crossref","first-page":"1196","DOI":"10.1038\/s41592-021-01252-x","article-title":"Effective gene expression prediction from sequence by integrating long-range interactions","volume":"18","author":"\u017d Avsec","year":"2021","journal-title":"Nature methods"},{"issue":"8","key":"pcbi.1012755.ref012","doi-asserted-by":"crossref","first-page":"1171","DOI":"10.1038\/s41588-018-0160-6","article-title":"Deep learning sequence-based ab initio prediction of variant effects on expression and disease risk","volume":"50","author":"J Zhou","year":"2018","journal-title":"Nature genetics"},{"issue":"7","key":"pcbi.1012755.ref013","doi-asserted-by":"crossref","first-page":"940","DOI":"10.1038\/s41588-022-01102-2","article-title":"A sequence-based global map of regulatory activity for deciphering human genetics","volume":"54","author":"KM Chen","year":"2022","journal-title":"Nature genetics"},{"issue":"9","key":"pcbi.1012755.ref014","doi-asserted-by":"crossref","first-page":"1616","DOI":"10.1101\/gr.134445.111","article-title":"Deep sequencing of subcellular RNA fractions shows splicing to be predominantly co-transcriptional in the human genome but inefficient for lncRNAs","volume":"22","author":"H Tilgner","year":"2012","journal-title":"Genome research"},{"key":"pcbi.1012755.ref015","doi-asserted-by":"crossref","first-page":"165","DOI":"10.1146\/annurev-biochem-060614-034242","article-title":"Regulation of alternative splicing through coupling with transcription and chromatin structure","volume":"84","author":"S Naftelberg","year":"2015","journal-title":"Annual Review of Biochemistry"},{"key":"pcbi.1012755.ref016","doi-asserted-by":"crossref","first-page":"168614","DOI":"10.1016\/j.jmb.2024.168614","article-title":"The Functional Relationship between RNA Splicing and the Chromatin Landscape","author":"JC Yustis","year":"2024","journal-title":"Journal of Molecular Biology"},{"issue":"1","key":"pcbi.1012755.ref017","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13059-023-02885-1","article-title":"Evidence for the role of transcription factors in the co-transcriptional regulation of intron retention","volume":"24","author":"F Ullah","year":"2023","journal-title":"Genome Biology"},{"key":"pcbi.1012755.ref018","article-title":"Evaluating the representational power of pre-trained DNA language models for regulatory genomics","author":"Z Tang","year":"2024","journal-title":"bioRxiv"},{"issue":"7294","key":"pcbi.1012755.ref019","doi-asserted-by":"crossref","first-page":"53","DOI":"10.1038\/nature09000","article-title":"Deciphering the splicing code","volume":"465","author":"Y Barash","year":"2010","journal-title":"Nature"},{"issue":"11","key":"pcbi.1012755.ref020","doi-asserted-by":"crossref","first-page":"1774","DOI":"10.1101\/gr.177790.114","article-title":"Widespread intron retention in mammals functionally tunes transcription","volume":"24","author":"B U","year":"2014","journal-title":"Genome Res"},{"issue":"14","key":"pcbi.1012755.ref021","doi-asserted-by":"crossref","first-page":"i274","DOI":"10.1093\/bioinformatics\/btx268","article-title":"Integrative deep models for alternative splicing","volume":"33","author":"A Jha","year":"2017","journal-title":"Bioinformatics"},{"issue":"6","key":"pcbi.1012755.ref022","doi-asserted-by":"crossref","first-page":"e1008006","DOI":"10.1371\/journal.pcbi.1008006","article-title":"Epigenome-based splicing prediction using a recurrent neural network","volume":"16","author":"D Lee","year":"2020","journal-title":"PLoS computational biology"},{"issue":"3","key":"pcbi.1012755.ref023","doi-asserted-by":"crossref","first-page":"535","DOI":"10.1016\/j.cell.2018.12.015","article-title":"Predicting splicing from primary sequence with deep learning","volume":"176","author":"K Jaganathan","year":"2019","journal-title":"Cell"},{"issue":"1","key":"pcbi.1012755.ref024","doi-asserted-by":"crossref","first-page":"103","DOI":"10.1186\/s13059-022-02664-4","article-title":"Predicting RNA splicing from DNA sequence using Pangolin","volume":"23","author":"T Zeng","year":"2022","journal-title":"Genome Biology"},{"key":"pcbi.1012755.ref025","unstructured":"Zhou Z, Ji Y, Li W, Dutta P, Davuluri R, Liu H. DNABERT-2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:230615006. 2023;."},{"issue":"2","key":"pcbi.1012755.ref026","doi-asserted-by":"crossref","first-page":"553","DOI":"10.1093\/nar\/gkr752","article-title":"Modeling the relative relationship of transcription factor binding and histone modifications to gene expression levels in mouse embryonic stem cells","volume":"40","author":"C Cheng","year":"2012","journal-title":"Nucleic acids research"},{"issue":"37","key":"pcbi.1012755.ref027","doi-asserted-by":"crossref","first-page":"13367","DOI":"10.1073\/pnas.1412081111","article-title":"Transcription factor binding predicts histone modifications in human cell lines","volume":"111","author":"D Benveniste","year":"2014","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"1","key":"pcbi.1012755.ref028","doi-asserted-by":"crossref","first-page":"25","DOI":"10.1186\/s13059-023-02867-3","article-title":"H3.3 contributes to chromatin accessibility and transcription factor binding at promoter-proximal regulatory elements in embryonic stem cells","volume":"24","author":"A Tafessu","year":"2023","journal-title":"Genome biology"},{"key":"pcbi.1012755.ref029","doi-asserted-by":"crossref","first-page":"19","DOI":"10.3389\/fcvm.2017.00019","article-title":"Coordinating regulation of gene expression in cardiovascular disease: interactions between chromatin modifiers and transcription factors","volume":"4","author":"AJ Bauer","year":"2017","journal-title":"Frontiers in cardiovascular medicine"},{"issue":"9","key":"pcbi.1012755.ref030","doi-asserted-by":"crossref","first-page":"563","DOI":"10.1038\/s41576-022-00468-7","article-title":"Histone post-translational modifications\u2014cause and consequence of genome function","volume":"23","author":"G Mill\u00e1n-Zambrano","year":"2022","journal-title":"Nature Reviews Genetics"},{"issue":"10","key":"pcbi.1012755.ref031","doi-asserted-by":"crossref","first-page":"792","DOI":"10.1016\/j.tcb.2021.04.001","article-title":"The transcription factor code: a beacon for histone methyltransferase docking","volume":"31","author":"GT Garcia","year":"2021","journal-title":"Trends in Cell Biology"},{"issue":"9","key":"pcbi.1012755.ref032","doi-asserted-by":"crossref","first-page":"497","DOI":"10.1016\/j.tibs.2010.03.010","article-title":"The carboxy terminal domain of RNA polymerase II and alternative splicing","volume":"35","author":"MJ Mu\u00f1oz","year":"2010","journal-title":"Trends in biochemical sciences"},{"issue":"6","key":"pcbi.1012755.ref033","doi-asserted-by":"crossref","first-page":"2547","DOI":"10.1093\/brain\/awad046","article-title":"RNA polymerase II-associated proteins reveal pathways affected in VCP-related amyotrophic lateral sclerosis","volume":"146","author":"MR Rafiee","year":"2023","journal-title":"Brain"},{"issue":"10","key":"pcbi.1012755.ref034","doi-asserted-by":"crossref","first-page":"5269","DOI":"10.1073\/pnas.1915975117","article-title":"Pathway-guided analysis identifies Myc-dependent alternative pre-mRNA splicing in aggressive prostate cancers","volume":"117","author":"JW Phillips","year":"2020","journal-title":"Proceedings of the National Academy of Sciences"},{"issue":"7558","key":"pcbi.1012755.ref035","doi-asserted-by":"crossref","first-page":"96","DOI":"10.1038\/nature14351","article-title":"MYC regulates the core pre-mRNA splicing machinery as an essential step in lymphomagenesis","volume":"523","author":"CM Koh","year":"2015","journal-title":"Nature"},{"issue":"8","key":"pcbi.1012755.ref036","doi-asserted-by":"crossref","DOI":"10.1016\/j.celrep.2022.111704","article-title":"MYC regulates a pan-cancer network of co-expressed oncogenic splicing factors","volume":"41","author":"L Urbanski","year":"2022","journal-title":"Cell reports"},{"key":"pcbi.1012755.ref037","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/s13073-015-0168-9","article-title":"Widespread intron retention diversifies most cancer transcriptomes","volume":"7","author":"H Dvinge","year":"2015","journal-title":"Genome medicine"},{"issue":"14","key":"pcbi.1012755.ref038","doi-asserted-by":"crossref","first-page":"7825","DOI":"10.1093\/nar\/gkab520","article-title":"CTCF as a regulator of alternative splicing: new tricks for an old player","volume":"49","author":"AB Alharbi","year":"2021","journal-title":"Nucleic acids research"},{"issue":"8","key":"pcbi.1012755.ref039","doi-asserted-by":"crossref","first-page":"2450","DOI":"10.1016\/j.celrep.2019.10.066","article-title":"The bromodomain protein 4 contributes to the regulation of alternative splicing","volume":"29","author":"S Uppal","year":"2019","journal-title":"Cell reports"},{"issue":"1","key":"pcbi.1012755.ref040","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1186\/gb-2012-13-1-r4","article-title":"SpliceGrapher: detecting patterns of alternative splicing from RNA-Seq data in the context of gene models and EST data","volume":"13","author":"MF Rogers","year":"2012","journal-title":"Genome biology"},{"issue":"13","key":"pcbi.1012755.ref041","doi-asserted-by":"crossref","first-page":"1658","DOI":"10.1093\/bioinformatics\/btl158","article-title":"CD-HIT: a fast program for clustering and comparing large sets of protein or nucleotide sequences","volume":"22","author":"W Li","year":"2006","journal-title":"Bioinformatics"},{"issue":"1","key":"pcbi.1012755.ref042","doi-asserted-by":"crossref","first-page":"15","DOI":"10.1093\/bioinformatics\/bts635","article-title":"STAR: ultrafast universal RNA-seq aligner","volume":"29","author":"A Dobin","year":"2013","journal-title":"Bioinformatics"},{"key":"pcbi.1012755.ref043","unstructured":"Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning. pmlr; 2015. p. 448\u2013456."},{"issue":"7","key":"pcbi.1012755.ref044","doi-asserted-by":"crossref","first-page":"e1008050","DOI":"10.1371\/journal.pcbi.1008050","article-title":"Cross-species regulatory sequence activity prediction","volume":"16","author":"DR Kelley","year":"2020","journal-title":"PLoS computational biology"},{"key":"pcbi.1012755.ref045","doi-asserted-by":"crossref","unstructured":"Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, et al. Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations; 2020. p. 38\u201345.","DOI":"10.18653\/v1\/2020.emnlp-demos.6"},{"key":"pcbi.1012755.ref046","article-title":"Lightgbm: A highly efficient gradient boosting decision tree","volume":"30","author":"G Ke","year":"2017","journal-title":"Advances in neural information processing systems"},{"key":"pcbi.1012755.ref047","unstructured":"Loshchilov I, Hutter F. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:160803983. 2016;."},{"key":"pcbi.1012755.ref048","unstructured":"Sundararajan M, Taly A, Yan Q. Axiomatic attribution for deep networks. In: International conference on machine learning. PMLR; 2017. p. 3319\u20133328."},{"key":"pcbi.1012755.ref049","unstructured":"Kokhlikyan N, Miglani V, Martin M, Wang E, Alsallakh B, Reynolds J, et al. Captum: A unified and generic model interpretability library for pytorch. arXiv preprint arXiv:200907896. 2020;."},{"issue":"D1","key":"pcbi.1012755.ref050","doi-asserted-by":"crossref","first-page":"D165","DOI":"10.1093\/nar\/gkab1113","article-title":"JASPAR 2022: the 9th release of the open-access database of transcription factor binding profiles","volume":"50","author":"JA Castro-Mondragon","year":"2022","journal-title":"Nucleic acids research"}],"updated-by":[{"DOI":"10.1371\/journal.pcbi.1012755","type":"new_version","label":"New version","source":"publisher","updated":{"date-parts":[[2025,1,23]],"date-time":"2025-01-23T00:00:00Z","timestamp":1737590400000}}],"container-title":["PLOS Computational Biology"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1012755","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,1,23]],"date-time":"2025-01-23T13:37:43Z","timestamp":1737639463000},"score":1,"resource":{"primary":{"URL":"https:\/\/dx.plos.org\/10.1371\/journal.pcbi.1012755"}},"subtitle":[],"editor":[{"given":"Joel S.","family":"Bader","sequence":"first","affiliation":[],"role":[{"vocabulary":"crossref","role":"editor"}]}],"short-title":[],"issued":{"date-parts":[[2025,1,10]]},"references-count":50,"journal-issue":{"issue":"1","published-online":{"date-parts":[[2025,1,10]]}},"URL":"https:\/\/doi.org\/10.1371\/journal.pcbi.1012755","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2024.01.26.577402","asserted-by":"object"}]},"ISSN":["1553-7358"],"issn-type":[{"value":"1553-7358","type":"electronic"}],"subject":[],"published":{"date-parts":[[2025,1,10]]}}}