{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,4,12]],"date-time":"2026-04-12T09:10:14Z","timestamp":1775985014370,"version":"3.50.1"},"reference-count":40,"publisher":"Springer Science and Business Media LLC","issue":"1","license":[{"start":{"date-parts":[[2022,4,2]],"date-time":"2022-04-02T00:00:00Z","timestamp":1648857600000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"},{"start":{"date-parts":[[2022,4,2]],"date-time":"2022-04-02T00:00:00Z","timestamp":1648857600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0"}],"content-domain":{"domain":["link.springer.com"],"crossmark-restriction":false},"short-container-title":["BMC Bioinformatics"],"published-print":{"date-parts":[[2022,12]]},"abstract":"<jats:title>Abstract<\/jats:title>\n                  <jats:sec>\n                    <jats:title>Motivation<\/jats:title>\n                    <jats:p>Deep learning has become a prevalent method in identifying genomic regulatory sequences such as promoters. In a number of recent papers, the performance of deep learning models has continually been reported as an improvement over alternatives for sequence-based promoter recognition. However, the performance improvements in these models do not account for the different datasets that models are evaluated on. The lack of a consensus dataset and procedure for benchmarking purposes has made the comparison of each model\u2019s true performance difficult to assess.<\/jats:p>\n                  <\/jats:sec>\n                  <jats:sec>\n                    <jats:title>Results<\/jats:title>\n                    <jats:p>\n                      We present a framework called Supervised Promoter Recognition Framework (\u2018SUPR REF\u2019) capable of streamlining the complete process of training, validating, testing, and comparing promoter recognition models in a systematic manner.\n                      <jats:italic>SUPR REF<\/jats:italic>\n                      includes the creation of biologically relevant benchmark datasets to be used in the evaluation process of deep learning promoter recognition models. We showcase this framework by comparing the models\u2019 performances on alternative datasets, and properly evaluate previously published models on new benchmark datasets. Our results show that the reliability of deep learning ab initio promoter recognition models on eukaryotic genomic sequences is still not at a sufficient level, as overall performance is still low. These results originate from a subset of promoters, the well-known RNA Polymerase II core promoters. Furthermore, given the observational nature of these data, cross-validation results from small promoter datasets need to be interpreted with caution.\n                    <\/jats:p>\n                  <\/jats:sec>","DOI":"10.1186\/s12859-022-04647-5","type":"journal-article","created":{"date-parts":[[2022,4,2]],"date-time":"2022-04-02T12:09:07Z","timestamp":1648901347000},"update-policy":"https:\/\/doi.org\/10.1007\/springer_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["Supervised promoter recognition: a benchmark framework"],"prefix":"10.1186","volume":"23","author":[{"given":"Raul I.","family":"Perez Martell","sequence":"first","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Alison","family":"Ziesel","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Hosna","family":"Jabbari","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]},{"given":"Ulrike","family":"Stege","sequence":"additional","affiliation":[],"role":[{"role":"author","vocabulary":"crossref"}]}],"member":"297","published-online":{"date-parts":[[2022,4,2]]},"reference":[{"issue":"4","key":"4647_CR1","doi-asserted-by":"publisher","first-page":"650","DOI":"10.1016\/j.cell.2018.01.029","volume":"172","author":"S Lambert","year":"2018","unstructured":"Lambert S, et al. The human transcription factors. Cell. 2018;172(4):650\u201365. https:\/\/doi.org\/10.1016\/j.cell.2018.01.029.","journal-title":"Cell"},{"issue":"D1","key":"4647_CR2","doi-asserted-by":"publisher","first-page":"7","DOI":"10.1093\/nar\/gkz1001","volume":"48","author":"O Fornes","year":"2019","unstructured":"Fornes O, et al. JASPAR 2020: update of the open-access database of transcription factor binding profiles. Nucleic Acids Res. 2019;48(D1):7\u201392. https:\/\/doi.org\/10.1093\/nar\/gkz1001.","journal-title":"Nucleic Acids Res"},{"key":"4647_CR3","doi-asserted-by":"publisher","DOI":"10.1038\/s41598-019-39878-9","author":"P Krzyzanowski","year":"2019","unstructured":"Krzyzanowski P, et al. Regional perturbation of gene transcription is associated with intrachromosomal rearrangements and gene fusion transcripts in high grade ovarian cancer. Sci Rep. 2019. https:\/\/doi.org\/10.1038\/s41598-019-39878-9.","journal-title":"Sci Rep"},{"issue":"1","key":"4647_CR4","doi-asserted-by":"publisher","first-page":"89","DOI":"10.1021\/acssynbio.5b00116","volume":"5","author":"T Rudge","year":"2016","unstructured":"Rudge T, et al. Characterization of intrinsic properties of promoters. ACS Synth Biol. 2016;5(1):89\u201398. https:\/\/doi.org\/10.1021\/acssynbio.5b00116.","journal-title":"ACS Synth Biol"},{"key":"4647_CR5","doi-asserted-by":"publisher","first-page":"6","DOI":"10.1016\/j.biosystems.2015.10.002","volume":"138","author":"Y Li","year":"2015","unstructured":"Li Y, et al. The identification of cis-regulatory elements: a review from a machine learning perspective. BioSystems. 2015;138:6\u201317. https:\/\/doi.org\/10.1016\/j.biosystems.2015.10.002.","journal-title":"BioSystems"},{"issue":"1\u20132","key":"4647_CR6","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1016\/j.artmed.2005.02.005","volume":"35","author":"V Narang","year":"2005","unstructured":"Narang V, et al. Computational modeling of oligonucleotide positional densities for human promoter prediction. Artif Intell Med. 2005;35(1\u20132):107\u201319. https:\/\/doi.org\/10.1016\/j.artmed.2005.02.005.","journal-title":"Artif Intell Med"},{"issue":"1","key":"4647_CR7","doi-asserted-by":"publisher","first-page":"3","DOI":"10.1186\/gb-2006-7-s1-s3","volume":"7","author":"V Bajic","year":"2006","unstructured":"Bajic V, et al. Performance assessment of promoter predictions on ENCODE regions in the EGASP experiment. Genome Biol. 2006;7(1):3\u2013113. https:\/\/doi.org\/10.1186\/gb-2006-7-s1-s3.","journal-title":"Genome Biol"},{"key":"4647_CR8","doi-asserted-by":"publisher","DOI":"10.1093\/bib\/bbab198","author":"OM de Medeiros","year":"2021","unstructured":"de Medeiros OM, et al. TSSFinder\u2013fast and accurate ab initio prediction of the core promoter in eukaryotic genomes. Brief Bioinform. 2021. https:\/\/doi.org\/10.1093\/bib\/bbab198.","journal-title":"Brief Bioinform"},{"issue":"10","key":"4647_CR9","doi-asserted-by":"publisher","first-page":"931","DOI":"10.1038\/nmeth.3547","volume":"12","author":"J Zhou","year":"2015","unstructured":"Zhou J, Troyanskaya OG. Predicting effects of noncoding variants with deep learning-based sequence model. Nat Methods. 2015;12(10):931\u20134. https:\/\/doi.org\/10.1038\/nmeth.3547.","journal-title":"Nat Methods"},{"issue":"8","key":"4647_CR10","doi-asserted-by":"publisher","first-page":"831","DOI":"10.1038\/nbt.3300","volume":"33","author":"B Alipanahi","year":"2015","unstructured":"Alipanahi B, et al. Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning. Nat Biotechnol. 2015;33(8):831\u20138. https:\/\/doi.org\/10.1038\/nbt.3300.","journal-title":"Nat Biotechnol"},{"issue":"11","key":"4647_CR11","doi-asserted-by":"publisher","first-page":"107","DOI":"10.1093\/nar\/gkw226","volume":"44","author":"D Quang","year":"2016","unstructured":"Quang D, Xie X. DanQ: a hybrid convolutional and recurrent deep neural network for quantifying the function of DNA sequences. Nucleic Acids Res. 2016;44(11):107. https:\/\/doi.org\/10.1093\/nar\/gkw226.","journal-title":"Nucleic Acids Res"},{"issue":"3","key":"4647_CR12","doi-asserted-by":"publisher","first-page":"159","DOI":"10.1093\/bib\/bbaa159","volume":"22","author":"J Li","year":"2021","unstructured":"Li J, et al. DeepATT: a hybrid category attention neural network for identifying functional effects of DNA sequences. Brief Bioinform. 2021;22(3):159. https:\/\/doi.org\/10.1093\/bib\/bbaa159.","journal-title":"Brief Bioinform"},{"key":"4647_CR13","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-642-23166-7_12","author":"T Raeder","year":"2012","unstructured":"Raeder T, et al. Learning from imbalanced data: evaluation matters. Intell Syst Ref Libr. 2012. https:\/\/doi.org\/10.1007\/978-3-642-23166-7_12.","journal-title":"Intell Syst Ref Libr"},{"issue":"2","key":"4647_CR14","doi-asserted-by":"publisher","first-page":"0171410","DOI":"10.1371\/journal.pone.0171410","volume":"12","author":"R Umarov","year":"2017","unstructured":"Umarov R, Solovyev V. Recognition of prokaryotic and eukaryotic promoters using convolutional deep learning neural networks. PLoS ONE. 2017;12(2):0171410. https:\/\/doi.org\/10.1371\/journal.pone.0171410.","journal-title":"PLoS ONE"},{"key":"4647_CR15","doi-asserted-by":"publisher","unstructured":"Qian Y et al. An improved promoter recognition model using convolutional neural network. In: 2018 IEEE 42nd annual COMPSAC, 2018. IEEE, Tokyo, Japan. p. 471\u2013476. https:\/\/doi.org\/10.1109\/COMPSAC.2018.00072.","DOI":"10.1109\/COMPSAC.2018.00072"},{"key":"4647_CR16","doi-asserted-by":"publisher","first-page":"286","DOI":"10.3389\/fgene.2019.00286","volume":"10","author":"M Oubounyt","year":"2019","unstructured":"Oubounyt M, et al. DeePromoter: robust promoter predictor using deep learning. Front Genet. 2019;10:286\u2013286. https:\/\/doi.org\/10.3389\/fgene.2019.00286.","journal-title":"Front Genet"},{"issue":"15","key":"4647_CR17","doi-asserted-by":"publisher","first-page":"2112","DOI":"10.1093\/bioinformatics\/btab083","volume":"37","author":"Y Ji","year":"2021","unstructured":"Ji Y, et al. DNABERT: pre-trained bidirectional encoder representations from transformers model for DNA-language in genome. Bioinformatics. 2021;37(15):2112\u201320. https:\/\/doi.org\/10.1093\/bioinformatics\/btab083.","journal-title":"Bioinformatics"},{"issue":"4","key":"4647_CR18","doi-asserted-by":"publisher","first-page":"315","DOI":"10.1038\/s41592-019-0360-8","volume":"16","author":"K Chen","year":"2019","unstructured":"Chen K, et al. Selene: a PyTorch-based deep learning library for sequence data. Nat Methods. 2019;16(4):315\u20138. https:\/\/doi.org\/10.1038\/s41592-019-0360-8.","journal-title":"Nat Methods"},{"issue":"1","key":"4647_CR19","doi-asserted-by":"publisher","first-page":"3488","DOI":"10.1038\/s41467-020-17155-y","volume":"11","author":"W Kopp","year":"2020","unstructured":"Kopp W, et al. Deep learning for genomics using Janggu. Nat Commun. 2020;11(1):3488. https:\/\/doi.org\/10.1038\/s41467-020-17155-y.","journal-title":"Nat Commun"},{"issue":"17","key":"4647_CR20","doi-asserted-by":"publisher","first-page":"3035","DOI":"10.1093\/bioinformatics\/bty222","volume":"34","author":"S Budach","year":"2018","unstructured":"Budach S, Marsico A. pysster: classification of biological sequences by learning sequence and structure motifs with convolutional neural networks. Bioinformatics. 2018;34(17):3035\u20137. https:\/\/doi.org\/10.1093\/bioinformatics\/bty222.","journal-title":"Bioinformatics"},{"issue":"6","key":"4647_CR21","doi-asserted-by":"publisher","first-page":"592","DOI":"10.1038\/s41587-019-0140-0","volume":"37","author":"Z Avsec","year":"2019","unstructured":"Avsec Z, et al. The Kipoi repository accelerates community exchange and reuse of predictive models for genomics. Nat Biotechnol. 2019;37(6):592\u2013600. https:\/\/doi.org\/10.1038\/s41587-019-0140-0.","journal-title":"Nat Biotechnol"},{"key":"4647_CR22","doi-asserted-by":"publisher","unstructured":"Shirley MD et al. Efficient \u201cpythonic\u201d access to FASTA files using pyfaidx. Technical Report e1196, PeerJ Inc. 2015. https:\/\/doi.org\/10.7287\/peerj.preprints.970v1.","DOI":"10.7287\/peerj.preprints.970v1"},{"issue":"D1","key":"4647_CR23","doi-asserted-by":"publisher","first-page":"51","DOI":"10.1093\/nar\/gkw1069","volume":"45","author":"R Dr\u00e9os","year":"2017","unstructured":"Dr\u00e9os R, et al. The eukaryotic promoter database in its 30th year: focus on non-vertebrate organisms. Nucleic Acids Res. 2017;45(D1):51\u20135. https:\/\/doi.org\/10.1093\/nar\/gkw1069.","journal-title":"Nucleic Acids Res"},{"key":"4647_CR24","doi-asserted-by":"publisher","first-page":"853","DOI":"10.1093\/nar\/gky1095","volume":"47","author":"M Haeussler","year":"2019","unstructured":"Haeussler M, et al. The UCSC genome browser database: 2019 update. Nucleic Acids Res. 2019;47:853\u20138. https:\/\/doi.org\/10.1093\/nar\/gky1095.","journal-title":"Nucleic Acids Res"},{"issue":"7493","key":"4647_CR25","doi-asserted-by":"publisher","first-page":"462","DOI":"10.1038\/nature13182","volume":"507","author":"The FANTOM Consortium et al.","year":"2014","unstructured":"The FANTOM Consortium et al. A promoter-level mammalian expression atlas. Nature. 2014;507(7493):462\u201370. https:\/\/doi.org\/10.1038\/nature13182.","journal-title":"Nature"},{"key":"4647_CR26","unstructured":"Perez\u00a0Martell R. Deep learning for promoter recognition: a robust testing methodology. M.Sc. Thesis, University of Victoria; 2020."},{"issue":"4","key":"4647_CR27","doi-asserted-by":"publisher","first-page":"529","DOI":"10.1101\/gr.10.4.529","volume":"10","author":"M Reese","year":"2000","unstructured":"Reese M, et al. Genie\u2013gene finding in Drosophila melanogaster. Genome Res. 2000;10(4):529\u201338. https:\/\/doi.org\/10.1101\/gr.10.4.529.","journal-title":"Genome Res"},{"issue":"1","key":"4647_CR28","doi-asserted-by":"publisher","first-page":"85","DOI":"10.1093\/nar\/27.1.85","volume":"27","author":"The FlyBase Consortium","year":"1999","unstructured":"The FlyBase Consortium. The FlyBase database of the Drosophila genome projects and community literature. Nucleic Acids Res. 1999;27(1):85\u20138. https:\/\/doi.org\/10.1093\/nar\/27.1.85.","journal-title":"Nucleic Acids Res"},{"issue":"16","key":"4647_CR29","doi-asserted-by":"publisher","first-page":"2730","DOI":"10.1093\/bioinformatics\/bty1068","volume":"35","author":"R Umarov","year":"2019","unstructured":"Umarov R, et al. Promoter analysis and prediction in the human genome using sequence-based deep learning models. Bioinformatics. 2019;35(16):2730\u20137. https:\/\/doi.org\/10.1093\/bioinformatics\/bty1068.","journal-title":"Bioinformatics"},{"key":"4647_CR30","unstructured":"Devlin J, et al. BERT: pre-training of deep bidirectional transformers for language understanding; 2019. arXiv:1810.04805 [cs]."},{"issue":"D1","key":"4647_CR31","doi-asserted-by":"publisher","first-page":"175","DOI":"10.1093\/nar\/gkx995","volume":"46","author":"R Dr\u00e9os","year":"2018","unstructured":"Dr\u00e9os R, et al. MGA repository: a curated data resource for ChIP-seq and other genome annotated data. Nucleic Acids Res. 2018;46(D1):175\u201380. https:\/\/doi.org\/10.1093\/nar\/gkx995.","journal-title":"Nucleic Acids Res"},{"key":"4647_CR32","doi-asserted-by":"publisher","DOI":"10.3389\/fgene.2019.01184","author":"Y Ju","year":"2019","unstructured":"Ju Y, et al. CircSLNN: identifying RBP-binding sites on circRNAs via sequence labeling neural networks. Front Genet. 2019. https:\/\/doi.org\/10.3389\/fgene.2019.01184.","journal-title":"Front Genet"},{"key":"4647_CR33","unstructured":"Tietz M et al. Skorch: a scikit-learn compatible neural network library that wraps PyTorch. Online: skorch.readthedocs.io\/en\/stable\/; 2017. https:\/\/skorch.readthedocs.io\/en\/stable\/."},{"key":"4647_CR34","first-page":"2825","volume":"12","author":"F Pedregosa","year":"2011","unstructured":"Pedregosa F, et al. Scikit-learn: machine learning in python. J Mach Learn Res. 2011;12:2825\u201330.","journal-title":"J Mach Learn Res"},{"key":"4647_CR35","unstructured":"Buitinck L et al. API design for machine learning software: experiences from the scikit-learn project. In: ECML PKDD: languages for data mining and machine learning; 2013. p. 108\u2013122."},{"key":"4647_CR36","unstructured":"Fidler F, Wilcox J. Reproducibility of scientific results; 2021."},{"issue":"6377","key":"4647_CR37","doi-asserted-by":"publisher","first-page":"725","DOI":"10.1126\/science.359.6377.725","volume":"359","author":"M Hutson","year":"2018","unstructured":"Hutson M. Artificial intelligence faces reproducibility crisis. Science. 2018;359(6377):725\u20136. https:\/\/doi.org\/10.1126\/science.359.6377.725.","journal-title":"Science"},{"key":"4647_CR38","doi-asserted-by":"publisher","DOI":"10.1093\/gigascience\/giy077","author":"Y-M Kim","year":"2018","unstructured":"Kim Y-M, et al. Experimenting with reproducibility: a case study of robustness in bioinformatics. GigaScience. 2018. https:\/\/doi.org\/10.1093\/gigascience\/giy077.","journal-title":"GigaScience"},{"key":"4647_CR39","unstructured":"Bates S et al. Cross-validation: what does it estimate and how well does it do it?. 2021. arXiv:2104.00673."},{"key":"4647_CR40","doi-asserted-by":"publisher","unstructured":"Perez I. ivanpmartell\/suprref: SUPRREF. Zenodo. 2022. https:\/\/doi.org\/10.5281\/zenodo.5823112; https:\/\/zenodo.org\/record\/5823112.","DOI":"10.5281\/zenodo.5823112"}],"container-title":["BMC Bioinformatics"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-022-04647-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/article\/10.1186\/s12859-022-04647-5\/fulltext.html","content-type":"text\/html","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/link.springer.com\/content\/pdf\/10.1186\/s12859-022-04647-5.pdf","content-type":"application\/pdf","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2022,4,2]],"date-time":"2022-04-02T12:10:12Z","timestamp":1648901412000},"score":1,"resource":{"primary":{"URL":"https:\/\/bmcbioinformatics.biomedcentral.com\/articles\/10.1186\/s12859-022-04647-5"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2022,4,2]]},"references-count":40,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2022,12]]}},"alternative-id":["4647"],"URL":"https:\/\/doi.org\/10.1186\/s12859-022-04647-5","relation":{"has-preprint":[{"id-type":"doi","id":"10.1101\/2021.09.07.459345","asserted-by":"object"}]},"ISSN":["1471-2105"],"issn-type":[{"value":"1471-2105","type":"electronic"}],"subject":[],"published":{"date-parts":[[2022,4,2]]},"assertion":[{"value":"10 January 2022","order":1,"name":"received","label":"Received","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"16 March 2022","order":2,"name":"accepted","label":"Accepted","group":{"name":"ArticleHistory","label":"Article History"}},{"value":"2 April 2022","order":3,"name":"first_online","label":"First Online","group":{"name":"ArticleHistory","label":"Article History"}},{"order":1,"name":"Ethics","group":{"name":"EthicsHeading","label":"Declarations"}},{"value":"The authors declare that they have no competing interests.","order":2,"name":"Ethics","group":{"name":"EthicsHeading","label":"Competing interests"}}],"article-number":"118"}}